Imaging device, information processing method, and information processing system

Abstract

An imaging device according to the present disclosure comprises: an image processing unit that performs image processing, according to a parameter, on a captured image of a subject, which has been captured and output by an imaging unit, and outputs the processed captured image as image data; an inference unit that uses a machine learning model to perform an inference process on the basis of the image data; and a degradation processing unit that uses the image data and a result of the inference by the inference process to detect degradation of the captured image.

Description

Imaging device, information processing method, and information processing system

[0001] The present disclosure relates to an imaging device, an information processing method, and an information processing system.

[0002] Conventionally, there has been known an imaging device that irradiates an imaging unit with light from a subject via an optical system, and performs predetermined image processing such as development processing on pixel signals output from the imaging unit in response to the irradiated light, and outputs the result as image data. Also known is an imaging device that has a configuration that performs recognition processing based on the captured image and outputs the recognition result.

[0003] International Publication No. 2020 / 116046

[0004] In an imaging device, camera parameters (internal parameters) and parameters related to development processing are set to fixed values ​​during operation of the imaging device. Therefore, if the optical system, including the lens, deteriorates over time, is damaged, or the environment changes, the fixed parameters may result in unintended images being developed. For example, the distortion coefficient, which is one of the camera's internal parameters, is generally calculated before the camera is shipped. However, due to factors such as lens deterioration over time, the distortion coefficient may not be fully corrected using the initial parameters, resulting in distorted images being output near the edges of the screen.

[0005] In addition, there are also known imaging devices that are configured to perform recognition processing based on captured images and output the recognition results. However, such imaging devices may not output the captured images, in which case it is difficult to notice deterioration in the captured images. If the recognition processing is performed without noticing the deterioration in the captured images, there is a risk that the recognition accuracy will decrease.

[0006] Therefore, an object of the present disclosure is to provide an imaging device, an information processing method, and an information processing system that can suppress deterioration in image quality of captured images due to aging.

[0007] The imaging device according to the present disclosure includes an image processing unit that performs image processing on an image captured by an imaging unit, capturing an image of a subject and outputting the captured image in accordance with parameters, and outputs the image data; an inference unit that performs inference processing based on the image data using a machine learning model; and a degradation processing unit that detects degradation of the captured image using the image data and the inference results from the inference processing.

[0008] 1 is a block diagram schematically illustrating an example configuration of a camera according to an existing technology. FIG. 1 is a schematic diagram illustrating an information processing system according to an embodiment. FIG. 1 is a schematic diagram for explaining image deterioration detection processing by an information processing system according to an existing technology. FIG. 1 is a schematic diagram for explaining image deterioration detection processing by an information processing system according to an embodiment. FIG. 2 is a block diagram illustrating an example configuration of an information processing system applicable to an embodiment. FIG. 3 is a functional block diagram for explaining functions of a user device according to an embodiment. FIG. 4 is a functional block diagram for explaining functions of a server according to an embodiment. FIG. 5 is a block diagram illustrating a hardware configuration of an example user device according to an embodiment. FIG. 6 is a block diagram illustrating a configuration of an example imaging device applicable to a second embodiment. FIG. 7 is a perspective view schematically illustrating a structure of an example imaging device applicable to an embodiment. FIG. 8 is a block diagram illustrating a hardware configuration of an example server according to an embodiment. FIG. 9 is a block diagram illustrating an example basic configuration of a user device according to an embodiment. FIG. 10 is a schematic diagram for explaining an example utilization of an inference result by an inference unit according to an embodiment. FIG. 11 is a schematic diagram for explaining an example utilization of an inference result by an inference unit according to an embodiment. FIG. 12 is a schematic diagram for explaining a process for detecting a misalignment in the angle of view of a captured image according to an embodiment. FIG. 13 is a schematic diagram for explaining a process for correcting a misalignment in the angle of view of a captured image according to an embodiment. FIG. 1 is a schematic diagram for explaining a process for detecting and correcting a color misalignment according to an embodiment. FIG. 2 is a flowchart of an example of a deterioration detection process according to an embodiment. FIG. 3 is a flowchart of an example of a deployment process of a correction model according to an embodiment. FIG. 4 is a time chart of an example of a deterioration detection process according to an embodiment. FIG. 5 is a time chart of an example of a parameter update process according to an embodiment. FIG. 6 is a schematic diagram for explaining variations in output data according to an embodiment.

[0009] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. In the following embodiments, the same components are denoted by the same reference numerals, and redundant description will be omitted.

[0010] Hereinafter, embodiments according to the present disclosure will be described in the following order: 1. Existing technology 2. Overview of embodiments according to the present disclosure 3. Configuration examples applicable to embodiments according to the present disclosure 4. More specific description of embodiments according to the present disclosure 4-1. Basic configuration 4-2. Example of use of inference results by inference unit 4-3. Captured image degradation detection and correction parameter calculation process 4-4. Overall flow of processing according to embodiment 4-5. Variations of output data

[0011] (1. Regarding Existing Technology) Prior to describing the embodiments of the present disclosure, existing technology will be described to facilitate understanding.

[0012] Fig. 1 is a block diagram showing a schematic configuration of an example of a camera according to existing technology. The camera shown in Fig. 1 performs recognition processing based on a captured image and outputs a recognition result, and is, for example, a surveillance camera that is fixedly installed and used. Section (a) of Fig. 1 shows an example in which the camera outputs a recognition result in which recognition was performed correctly. Section (b) of Fig. 1 shows an example in which the camera outputs a recognition result in which recognition performance has deteriorated.

[0013] In section (a) of Fig. 1 , light from a subject is irradiated onto the light-receiving surface of an image sensor 510 via an optical system 500 including a lens, an autofocus mechanism, an auto-diaphragm mechanism, etc. The image sensor 510 outputs a captured image corresponding to the light irradiated onto the light-receiving surface. This captured image is a RAW image that has not been subjected to development processing, etc. The image sensor 510 outputs the captured image to an image processing unit 511.

[0014] The image processing unit 511 performs predetermined image processing on the captured image output from the image sensor 510 in accordance with the camera parameters 512. For example, the image processing unit 511 performs AD (Analog to Digital) conversion processing on the captured image, which is an analog signal, to convert it into a digital signal, and then performs development processing on the captured image converted into a digital signal to generate a color image (RGB image) using the colors R (red), G (green), and B (blue). For example, the image processing unit 511 may also perform gain adjustment using AGC (Auto Gain Control) or white balance processing on the captured image. Furthermore, for example, the image processing unit 511 may perform distortion correction. Without being limited to these, the image processing unit 511 may also perform noise removal processing, defective pixel correction processing, and the like on the captured image.

[0015] The image processing unit 511 outputs image data 513, which is the captured image that has been subjected to image processing, to the recognition processing unit 514. The image based on the image data 513 is an image in which image distortion caused by the optical system 500, for example, has been corrected by the image processing unit 511.

[0016] The recognition processing unit 514 performs recognition processing based on the image data 513 output from the image processing unit 511. The recognition processing performed by the recognition processing unit 514 is not particularly limited, and may be processing to recognize a face included in an image based on the image data 513, or processing to recognize a license plate of a vehicle included in the image. The recognition processing unit 514 performs recognition processing based on an image in which image distortion has been corrected by the image processing unit 511, and therefore can output a correctly recognized recognition result.

[0017] Meanwhile, the camera parameters 512, including the development parameters used in the development process in the image processing unit 511, are fixed values ​​when the camera is in operation. Therefore, if the lens in the optical system 500 deteriorates over time, is damaged, or the environment changes, for example, there is a possibility that an unintended image will be developed if the parameters are fixed values.

[0018] Furthermore, for example, a distortion coefficient for performing distortion correction, which is one of the camera parameters 512, is generally calculated when the camera is shipped. However, due to factors such as aging of the lenses in the optical system 500, the distortion may not be fully corrected using the initial distortion coefficient. As a result, the image based on the image data 513 output from the image processing unit 511 will be distorted near the periphery of the screen.

[0019] 1, the degree of deterioration of the optical system 500 and the image sensor 510 is large, and the image based on the image data 513' output from the image processing unit 511 is not sufficiently corrected by the correction process performed by the image processing unit 511 in accordance with the camera parameters 512, and distortion remains. Therefore, when the recognition processing unit 514 performs recognition processing based on this image data 513', the recognition performance deteriorates, and there is a risk that a correct recognition result will not be obtained.

[0020] In particular, it is difficult to notice changes in the image quality of a captured image with a camera that outputs only the recognition results and does not output the image itself, as shown in Figure 1. This results in a decrease in the recognition performance of the recognition processing unit 514, making it difficult to provide the value of the solution.

[0021] In an embodiment of the present disclosure, a camera and an information processing system are proposed that are capable of suppressing degradation in image quality of captured images due to aging and continuously outputting highly accurate recognition results.

[0022] (2. Overview of the Embodiments According to the Present Disclosure) Next, an overview of the embodiments according to the present disclosure will be described.

[0023] 2 is a schematic diagram illustrating an information processing system according to an embodiment. In FIG. 2, a user device 10, such as a camera, is connected to a server 30 configured in a cloud network 3 via a communication network (not shown). The three user devices 10 shown in the figure represent the same user device 10 in chronological order. The user device 10 executes recognition processing based on image data obtained by developing captured images and outputs the recognition results.

[0024] The user device 10 automatically detects deterioration of the captured image using its own pre-existing algorithms and / or machine learning models in accordance with its own pre-existing camera parameters, based on the image data developed from the captured image and the recognition results obtained by performing recognition processing on the image data (step S1).

[0025] Note that the term "algorithm" used here refers to a specific procedure for executing some kind of processing, and machine learning models can also be considered a type of algorithm. Hereinafter, unless otherwise specified, algorithms and machine learning models will be collectively referred to as "algorithms." In other words, when the term "algorithm" is simply used, the term may also include machine learning models.

[0026] When the user device 10 detects degradation of the captured image, it transmits a degradation notification including information indicating the nature of the degradation to the server 30 (step S2). The server 30 stores multiple algorithms for performing calibration according to the image degradation of the user device 10. Each of these multiple algorithms performs processing to estimate camera parameters that correct the image degradation, for example.

[0027] Based on the degradation notification sent from the user device 10, the server 30 selects an algorithm for making corrections according to the degradation content, sends the selected algorithm to the user device 10, and deploys the selected algorithm to the user device 10 (step S3). At the same time, the server 30 acquires from the user device 10 and stores the algorithm that the user device 10 has in advance.

[0028] The user device 10 performs calibration using the most recently captured image in accordance with the deployed algorithm transmitted from the server 30, and updates the camera parameters to correct image degradation (step S4). Upon completing the calibration, the user device 10 transmits a calibration completion notification to the server 30 (step S5).

[0029] In response to the calibration completion notification sent from the user device 10, the server 30 sends the algorithm previously stored in the user device 10, acquired from the user device 10 in step S3, to the user device 10 and deploys it (step S6). The user device 10 is driven according to the latest camera parameters updated in step S4 (step S7). That is, the user device 10 is returned to the state it was in when degradation was detected in step S1, except for the camera parameters.

[0030] In the embodiment, with the above-described configuration, calibration of the user device 10 can be automatically performed in cooperation with the server 30. Furthermore, the user device 10 automatically detects image degradation using its own pre-defined algorithm, making it possible to detect image degradation early. Furthermore, the above-described information processing system can calibrate the user device 10 without outputting images captured by the user device 10 to an external device, thereby realizing a secure system.

[0031] (Comparison with Existing Technology) Image deterioration detection according to the embodiment will be described in comparison with image deterioration detection according to existing technology using FIGS. 3 and 4. FIG.

[0032] 3 is a schematic diagram illustrating an image degradation detection process performed by an information processing system according to existing technology. In a user device 10', such as a camera, an image processing unit 11 develops a captured image supplied from an imaging device (not shown) and outputs image data 12 in the form of an RGB image. The user device 10' transmits the image data 12 to a server 30'.

[0033] The server 30' executes processing (e.g., recognition processing) for a specific task using a task-specific deep neural network (DNN) 31 on the image data 12 transmitted from the user device 10', and passes the processing result to a post-processing unit 32. The post-processing unit 32 performs predetermined processing, such as statistical processing, on the processing result passed from the task-specific DNN 31 and outputs the result. The output of the post-processing unit 32 is used as the value of a solution related to the information processing system.

[0034] The image data 12 is transmitted to the server 30' and also supplied to the image deterioration detection unit 34. The image deterioration detection unit 34 detects whether or not image deterioration has occurred based on the image data 12. The image deterioration detection unit 34 may detect image deterioration, for example, by comparing the most recently acquired image data 12 with image data 12 acquired in the past. The image deterioration detection unit 34 outputs the detection result as an image deterioration detection notification.

[0035] 4 is a schematic diagram illustrating an image degradation detection process performed by the information processing system according to the embodiment. In a user device 10, such as a camera, an image processing unit 11 performs development processing on a captured image supplied from an imaging device (not shown) and outputs image data 12 in the form of an RGB image.

[0036] The user device 10 according to the embodiment includes a task-specific DNN 13 and image degradation detection units 15a and 15b (also shown as image degradation detection units #1 and #2 in the figure, respectively).

[0037] The image data 12 is supplied to the specific task DNN 31 and the image deterioration detection unit 15b. The specific task DNN 13 executes processing (e.g., recognition processing) according to a specific task on the image data 12, and transmits the processing result 14 to the server 30. The server 30 passes the processing result 14 to the post-processing unit 32. The post-processing unit 32 performs predetermined processing, such as statistical processing, on the processing result 14 passed from the specific task DNN 13 and outputs the result. The output of the post-processing unit 32 is used as the value of a solution related to the information processing system.

[0038] The image deterioration detection unit 15a performs processing related to image deterioration detection based on the processing result 14 and passes the processing result to the image deterioration detection unit 15b. The image deterioration detection unit 15b detects the presence or absence of image deterioration based on the processing result passed from the image deterioration detection unit 15a and the image data 12. Details of the processing by the image deterioration detection units 15a and 15b will be described later. The image deterioration detection unit 15b outputs the detection result as an image deterioration detection notification and transmits it to the server 30.

[0039] In this way, in the information processing system according to the embodiment, image degradation detection is performed inside the user device 10, so that a secure system can be realized.

[0040] (3. Configuration Examples Applicable to Embodiments of the Present Disclosure) Next, configuration examples applicable to embodiments of the present disclosure will be described. Fig. 5 is a block diagram showing the configuration of an example of an information processing system applicable to the embodiments.

[0041] In the following, neural networks such as DNNs will be simply referred to as "networks" to distinguish them from communication networks such as the Internet and LANs (Local Area Networks).

[0042] 5, an information processing system 1 according to the embodiment includes a server 30 and one or more user devices 10, 10, ... connected via a communication network 2. The communication network 2 may be the Internet, or a LAN (Local Area Network) established in a closed environment such as an in-house network.

[0043] 5, the server 30 is shown as being configured in a cloud network 3 connected to a communication network 2. However, the server 30 is not limited to this and may be configured as a single computer or may be configured as a distributed server across multiple computers.

[0044] The user devices 10, 10, ... may be cameras, such as fixedly installed surveillance cameras. However, the user devices 10, 10, ... may also be in-vehicle cameras, cameras built into smartphones or tablet devices, or compact digital cameras. The user devices 10, 10, ... may be configured to perform recognition processing based on captured images and output the recognition results. Each of the user devices 10, 10, ... has unique identification information.

[0045] 6 is a functional block diagram illustrating the functions of the user device 10 according to the embodiment. In FIG. 6, the user device 10 includes an overall control unit 100, a communication unit 110, an image acquisition unit 120, an image processing unit 130, an inference unit 140, and a calibration processing unit 150.

[0046] The overall control unit 100, communication unit 110, image acquisition unit 120, image processing unit 130, inference unit 140, and calibration processing unit 150 may be realized, for example, by running an imaging control program according to the embodiment on a CPU (Central Processing Unit) of the user device 10. However, without being limited to this, some or all of the overall control unit 100, communication unit 110, image acquisition unit 120, image processing unit 130, inference unit 140, and calibration processing unit 150 may be configured by hardware circuits that operate in cooperation with each other.

[0047] The overall control unit 100 controls the overall operation of the user device 10. The communication unit 110 controls communication over the communication network 2, and transmits and receives data to and from the server 30, for example. The image acquisition unit 120 acquires captured images (RAW images) output from an imaging device included in the user device 10.

[0048] The image processing unit 130 performs predetermined image processing, such as development processing, on the captured image acquired by the image acquisition unit 120 in accordance with camera parameters to generate image data, for example, an RGB image. The inference unit 140 performs inference processing using a machine learning model that has been trained in advance, based on the image data generated by the image processing unit 130. For example, the inference unit 140 performs recognition processing on the image data using the machine learning model.

[0049] The calibration processing unit 150 performs calibration processing related to image degradation of the captured image. More specifically, the calibration processing unit 150 detects image degradation of the captured image using the image data generated by the image processing unit 130 and the inference result by the inference unit 140, in accordance with an algorithm based on the degradation detection model. Furthermore, the calibration processing unit 150 may infer camera parameters to be used by the image processing unit 130 for image processing using a correction model that corrects the image degradation in accordance with the detection result of the image degradation, and update the original camera parameters with the inferred camera parameters.

[0050] 7 is a functional block diagram illustrating an example of the functions of the server 30 according to the embodiment. The server 30 includes an overall control unit 300, a communication unit 310, an algorithm management unit 320, and an algorithm storage unit 330.

[0051] The overall control unit 300, the communication unit 310, the algorithm management unit 320, and the algorithm storage unit 330 may be realized by running an information processing program according to the embodiment on a CPU included in the server 30. However, some or all of the overall control unit 300, the communication unit 310, the algorithm management unit 320, and the algorithm storage unit 330 may be configured by hardware circuits that operate in cooperation with each other.

[0052] The overall control unit 300 controls the overall operation of the server 30. The communication unit 310 controls communication over the communication network 2, and transmits and receives data to and from the user device 10, for example.

[0053] The algorithm storage unit 330 stores multiple algorithms including machine learning models in, for example, a non-volatile storage medium. More specifically, the algorithm storage unit 330 may store algorithms for performing calibration for image degradation occurring in the user device 10 in the storage medium for each type of image degradation. The algorithm storage unit 330 may also temporarily store algorithms acquired from the user device 10 in the storage medium.

[0054] In the following, unless otherwise specified, "the algorithm storage unit 330 stores data (such as an algorithm) in a storage medium" will be expressed as "the algorithm storage unit 330 stores data," etc.

[0055] The algorithm management unit 320 selects an algorithm corresponding to the image degradation notification from the user device 10 from the algorithms stored in the algorithm storage unit 330. The algorithm management unit 320 also retrieves from the algorithm storage unit 330 the algorithm acquired from the user device 10 and temporarily stored in the algorithm storage unit 330 in order to transmit it to the user device 10.

[0056] FIG. 8 is a block diagram showing an example of a hardware configuration of the user device 10 according to the embodiment.

[0057] 8 , the user device 10 includes an image capture device 1000, a degradation processing CPU 1010, a memory 1011, a non-volatile memory 1012, and a communication I / F (interface) 1020. The image capture device 1000, the degradation processing CPU 1010, and the communication I / F 1020 are connected to each other via a bus 1030 so as to be able to communicate with each other.

[0058] The imaging device 1000 includes an optical system, an imaging unit, and a signal processing unit, and generates a captured image in response to light irradiated onto the imaging unit in the imaging unit via the optical system. The imaging device 1000 also generates image data by performing predetermined signal processing, such as development, on the generated captured image. The imaging device 1000 then performs inference processing, such as recognition processing, based on the generated image data and outputs the inference results. The imaging device 1000 also includes a CPU, which may control the overall operation of the user device 10. That is, the imaging device 1000 implements the functions of the overall control unit 100, image acquisition unit 120, image processing unit 130, and inference unit 140 shown in FIG. 6 . A more detailed configuration of the imaging device 1000 will be described later.

[0059] The degradation processing CPU 1010 is connected to a memory 1011 and a non-volatile memory 1012. The memory 1011 includes a read-only memory (ROM) and a random access memory (RAM). The non-volatile memory 1012 stores algorithms for executing processing related to image degradation in a non-volatile manner. The degradation processing CPU 1010 uses the RAM of the memory 1011 as a work memory and executes processing according to the algorithms stored in the non-volatile memory 1012 in accordance with the programs stored in the ROM of the memory 1011. In other words, the degradation processing CPU 1010 realizes the function of the calibration processing unit 150 shown in FIG. 6 .

[0060] The communication I / F 1020 controls communication with the communication network 2 in accordance with instructions from the CPU included in the imaging device 1000 or the degradation processing CPU 1010. In other words, the communication I / F 1020 realizes the function of the communication unit 110 shown in FIG.

[0061] Fig. 9 is a block diagram showing an example of the configuration of an image pickup apparatus 1000 applicable to the second embodiment. In Fig. 9, the image pickup apparatus 1000 is configured as a CIS (CMOS (Complementary Metal Oxide Semiconductor) Image Sensor) in which an image pickup element including a pixel array and a signal processing unit that performs predetermined signal processing such as image processing on an image captured by the image pickup element are configured on a single chip, and includes an image pickup unit 1100 and a signal processing unit 1120. The image pickup unit 1100 and the signal processing unit 1120 are electrically connected by connection lines CL1, CL2, and CL3, which are internal buses, respectively.

[0062] The imaging unit 1100 has an imaging section 1111, an imaging processing section 1112, an output control section 1113, an output I / F 1114, and an imaging control section 1110, and captures an image of a subject to obtain a captured image.

[0063] The image capturing unit 1111 includes a pixel array in which a plurality of pixels, which are light receiving elements that output signals according to received light through photoelectric conversion, are arranged in a matrix. The image capturing unit 1111 is driven by the image capturing processing unit 1112 to capture an image of a subject.

[0064] That is, light from an optical system (not shown) is incident on the imaging unit 1111. The imaging unit 1111 receives the incident light from the optical system at each pixel included in the pixel array, performs photoelectric conversion, and outputs an analog image signal corresponding to the incident light.

[0065] The size of the image generated by the image signal output by the imaging unit 1111 can be selected from a plurality of sizes, such as width x height, 3968 pixels x 2976 pixels, 1920 pixels x 1080 pixels, and 640 pixels x 480 pixels. The image sizes that can be output by the imaging unit 1111 are not limited to these examples. Furthermore, the image output by the imaging unit 1111 can be selected to be, for example, a color image of RGB (red, green, blue) or a black-and-white image of luminance only. These selections for the imaging unit 1111 can be made as a type of shooting mode setting.

[0066] Note that information based on the output of each pixel arranged in a matrix in the pixel array is called a frame. The imaging device 1000 repeatedly acquires information on the matrix of pixels in chronological order at a predetermined rate (frame rate) in the imaging unit 1111. The imaging device 1000 outputs each piece of acquired information collectively for each frame.

[0067] The imaging processing unit 1112 is, for example, an ISP (Image Signal Processor), and performs imaging processing related to capturing images in the imaging unit 1111, such as driving the imaging unit 1111, AD (Analog to Digital) conversion of the analog image signal output by the imaging unit 1111, and imaging signal processing, under the control of the imaging control unit 1110.

[0068] The imaging signal processing performed by the imaging processing unit 1112 includes, for example, a process of calculating brightness for each small region of the image output by the imaging unit 1111 by calculating the average value of pixel values ​​for each predetermined small region, a high dynamic range (HDR) conversion process of converting the image output by the imaging unit 1111 into an HDR image, defect correction, development, etc. The imaging processing unit 1112 performs such imaging signal processing in accordance with camera parameters stored in a memory 1123 or a register group 1117, which will be described later.

[0069] The imaging processing unit 1112 outputs a digital image signal obtained by AD conversion of the analog image signal output by the imaging unit 1111 as image data of the captured image. The imaging processing unit 1112 can also output a RAW image that has not been subjected to processing such as development as image data. An image in which each pixel has information on each of the RGB colors, obtained by processing such as development of a RAW image, is called an RGB image.

[0070] The image data output by the imaging processing unit 1112 is supplied to the output control unit 1113 and also to the image compression unit 1125 of the signal processing unit 1120 via a connection line CL2.

[0071] The output control section 1113 is supplied with image data from the imaging processing section 1112, and also with the results of signal processing using image data and the like from the signal processing unit 1120 via a connection line CL3.

[0072] The output control unit 1113 performs output control to selectively output image data from the imaging processing unit 1112 and the signal processing result from the signal processing unit 1120 from (one) output I / F 1114 to the data processing unit 2200. In other words, the output control unit 1113 selects the image data from the imaging processing unit 1112 or the signal processing result from the signal processing unit 1120 and supplies it to the output I / F 1114.

[0073] The output I / F 1114 is an I / F that outputs image data and signal processing results supplied from the output control unit 1113 to the outside. As the output I / F 1114, for example, a relatively high-speed parallel I / F such as MIPI (Mobile Industry Processor Interface) can be adopted.

[0074] The output I / F 1114 outputs to the outside the image data from the imaging processing unit 1112 or the signal processing result from the signal processing unit 1120 in accordance with the output control of the output control unit 1113. Therefore, for example, when only the signal processing result from the signal processing unit 1120 is needed outside and the image data itself is not needed, it is possible to output only the signal processing result, and it is possible to reduce the amount of data output from the output I / F 1114 to the outside.

[0075] Furthermore, the signal processing unit 1120 performs signal processing that produces the signal processing results required externally, and outputs the signal processing results from the output I / F 1114, thereby eliminating the need to perform signal processing externally and reducing the load on external blocks.

[0076] The imaging control unit 1110 includes a communication I / F 1116 and a register group 1117 .

[0077] The communication I / F 1116 is, for example, 2 The first communication I / F is a serial communication I / F such as an Inter-Integrated Circuit (C), and exchanges necessary information such as information to be read and written to the register group 1117 with the outside (for example, the degradation processing CPU 1010).

[0078] The register group 1117 has a plurality of registers and stores imaging information related to the imaging of an image by the imaging unit 1111 and various other information. The register group 1117 may store camera parameters. For example, the register group 1117 stores imaging information received from the outside via the communication I / F 1116 and the results of imaging signal processing by the imaging processing unit 1112 (e.g., brightness of each small region of the captured image, etc.).

[0079] The imaging information stored in the register group 1117 includes, for example, ISO sensitivity (analog gain during AD conversion in the imaging processing unit 1112), exposure time (shutter speed), frame rate, focus, shooting mode, cropping range, etc. (information representing the same).

[0080] The shooting modes include a manual mode in which the exposure time, frame rate, etc. are manually set, and an automatic mode in which the settings are automatically set according to the scene. The automatic mode includes modes according to various shooting scenes, such as night scenes and human faces.

[0081] The cut-out range refers to the range cut out from the image output by the imaging unit 1111 when the imaging processing unit 1112 cuts out a part of the image output by the imaging unit 1111 and outputs it as image data. By specifying the cut-out range, it becomes possible to cut out, for example, only the range in which a person appears from the image output by the imaging unit 1111. In addition to a method of cutting out an image from the image output by the imaging unit 1111, there is also a method of reading out only the image (signal) of the cut-out range from the imaging unit 1111.

[0082] The imaging control unit 1110 controls the imaging processing unit 1112 in accordance with the imaging information stored in the register group 1117 , thereby controlling the imaging of images by the imaging unit 1111 .

[0083] The register group 1117 can store imaging information, the results of imaging signal processing in the imaging processing unit 1112, and output control information related to output control in the output control unit 1113. The output control unit 1113 can perform output control to selectively output image data and signal processing results in accordance with the output control information stored in the register group 1117.

[0084] Furthermore, in the imaging device 1000, the imaging control unit 1110 and the CPU 1121 of the signal processing unit 1120 are connected via a connection line CL1, and the CPU 1121 can read and write information from and to the register group 1117 via the connection line CL1. That is, in the imaging device 1000, reading and writing of information from and to the register group 1117 can be performed not only from the communication I / F 1116 but also from the CPU 1121.

[0085] The signal processing unit 1120 has a CPU 1121, a DSP (Digital Signal Processor) 1122, a memory 1123, a communication I / F 1124, an image compression unit 1125, and an input I / F 1126, and performs predetermined signal processing using image data obtained by the imaging unit 1100. Note that the CPU 1121 is not limited to this, and may also be an MPU (Micro Processor Unit) or an MCU (Micro Controller Unit).

[0086] The CPU 1121, DSP 1122, memory 1123, communication I / F 1124, and input I / F 1126 that make up the signal processing unit 1120 are interconnected via a bus, and can exchange information as necessary.

[0087] The CPU 1121 executes programs stored in the memory 1123 to control the signal processing unit 1120, read and write information to the register group 1117 of the imaging control unit 1110 via the connection line CL1, and perform various other processes.

[0088] For example, by executing a program, the CPU 1121 functions as an imaging information calculation unit that calculates imaging information using the signal processing results obtained by signal processing in the DSP 1122, and feeds back and stores the new imaging information calculated using the signal processing results to the register group 1117 of the imaging control unit 1110 via the connection line CL1.

[0089] Therefore, the CPU 1121 can control the image capturing in the image capturing unit 1111 and the image capturing signal processing in the image capturing processing unit 1112 according to the signal processing result of the captured image. The CPU 1121 may also control the overall operation of the user device 10 in which the image capturing apparatus 1000 is mounted.

[0090] Furthermore, the imaging information stored in the register group 1117 by the CPU 1121 can be provided (output) to the outside from the communication I / F 1116. For example, focus information among the imaging information stored in the register group 1117 can be provided from the communication I / F 1116 to a focus driver (not shown) that controls focus.

[0091] The DSP 1122 executes a program stored in the memory 1123, thereby functioning as a signal processing unit that performs signal processing using image data supplied from the imaging processing unit 1112 to the signal processing unit 1120 via the connection line CL2 and information received from the outside by the input I / F 1126. The DSP 1122 may also perform inference processing using a machine learning model stored in the memory 1123.

[0092] The memory 1123 is configured by an SRAM (Static Random Access Memory), a DRAM (Dynamic RAM), or the like, and stores data necessary for processing by the signal processing unit 1120, and the like.

[0093] For example, the memory 1123 stores programs received from outside via the communication I / F 1124, captured images compressed by the image compression unit 1125 and used for signal processing in the DSP 1122, the signal processing results of the signal processing performed by the DSP 1122, information received by the input I / F 1126, etc.

[0094] The memory 1123 may include a volatile area and a nonvolatile area. The volatile area may be used as a work memory by the CPU 1121, for example. The nonvolatile area is a rewritable storage area and may store programs for the operation of the CPU 1121. Furthermore, the nonvolatile area may store the machine learning model used by the DSP 1122 described above.

[0095] The communication I / F 1124 is a second communication I / F, such as a serial communication I / F such as an SPI (Serial Peripheral Interface), and exchanges necessary information such as programs executed by the CPU 1121 and DSP 1122 with the outside (for example, a memory or control unit not shown).

[0096] For example, the communication I / F 1124 downloads programs to be executed by the CPU 1121 and the DSP 1122 from an external device and supplies and stores the programs in the memory 1123. Therefore, the programs downloaded by the communication I / F 1124 enable the CPU 1121 and the DSP 1122 to execute various processes.

[0097] The communication I / F 1124 can exchange programs and other arbitrary data with the outside. For example, the communication I / F 1124 can output to the outside the signal processing results obtained by the signal processing in the DSP 1122. The communication I / F 1124 can also output information according to instructions from the CPU 1121 to an external device, thereby controlling the external device according to instructions from the CPU 1121.

[0098] The signal processing results (such as inference results) obtained by signal processing in the DSP 1122 can be output to the outside from the communication I / F 1124, and can also be written to the register group 1117 of the imaging control unit 1110 by the CPU 1121. The signal processing results written to the register group 1117 can be output to the outside from the communication I / F 1116. The same applies to the processing results performed by the CPU 1121.

[0099] The captured image is supplied from the imaging processing unit 1112 to the image compression unit 1125 via a connection line CL2. The image compression unit 1125 performs a compression process to compress the image data, and generates compressed image data with a smaller amount of data than the original image data. The compressed image data generated by the image compression unit 1125 is supplied to the memory 1123 via a bus and stored therein.

[0100] Here, the signal processing in the DSP 1122 can be performed using the image data itself, or can be performed using compressed image data generated from the image data by the image compression unit 1125. Since the compressed image data has a smaller data volume than the original image data, it is possible to reduce the load of signal processing in the DSP 1122 and save the storage capacity of the memory 1123 that stores the compressed image data.

[0101] The compression process in the image compression unit 1125 can be, for example, scaling down the image data of a captured image of 3968 pixels x 2976 pixels to image data of 640 pixels x 480 pixels. Furthermore, if the signal processing in the DSP 1122 is performed on luminance and the image data is an RGB image, the compression process can be YUV conversion, which converts the RGB image into a YUV image, for example.

[0102] The image compression unit 1125 can be realized by software or by dedicated hardware provided by an ISP or the like.

[0103] The input I / F 1126 is an I / F that receives information from the outside. For example, the input I / F 1126 receives the output of an external sensor (external sensor output) from the external sensor and supplies it to the memory 1123 via the bus for storage. As with the output I / F 1114, for example, a parallel I / F such as MIPI can be used as the input I / F 1126.

[0104] Furthermore, as the external sensor, for example, a distance sensor that senses information regarding distance can be adopted.Furthermore, as the external sensor, for example, an image sensor that senses light and outputs an image corresponding to that light, that is, an image sensor separate from the imaging device 1000, can be adopted.

[0105] In addition to using image data from captured images or compressed image data generated from image data, the DSP 1122 can perform signal processing using external sensor output received by the input I / F 1126 from an external sensor such as those described above and stored in memory 1123.

[0106] In the imaging device 1000 configured as described above, signal processing including network processing is performed by the DSP 1122 using image data obtained by imaging in the imaging unit 1111 or compressed image data generated from the image data, and the signal processing results of the signal processing and the captured image are selectively output from the output I / F 1114. Therefore, it is possible to configure a compact imaging device that outputs information required by the user.

[0107] FIG. 10 is a perspective view schematically illustrating the structure of an example of the image capturing apparatus 1000 that can be applied to the embodiment described with reference to FIG.

[0108] The imaging device 1000 can be configured as a one-chip semiconductor device having a stacked structure in which multiple dies are stacked, for example, as shown in Fig. 10. In the example of Fig. 10, the imaging device 1000 is configured as a one-chip semiconductor device in which two dies, dies 1130 and 1131, are stacked.

[0109] A die refers to a small thin piece of silicon in which an electronic circuit is fabricated, and an individual in which one or more dies are sealed is called a chip.

[0110] 10 , an imaging unit 1111 is mounted on an upper die 1130. An imaging processing unit 1112, an output control unit 1113, an output I / F 1114, and an imaging control unit 1115 are mounted on a lower die 1131. As such, in the example of FIG. 10 , the imaging unit 1111 of the imaging unit 1100 is mounted on the die 1130, and the parts other than the imaging unit 1111 are mounted on the die 1131. A signal processing unit 1120 including a CPU 1121, a DSP 1122, a memory 1123, a communication I / F 1124, an image compression unit 1125, and an input I / F 1126 is further mounted on the die 1131.

[0111] The upper die 1130 and the lower die 1131 are electrically connected, for example, by forming a through-hole that penetrates the die 1130 and reaches the die 1131. Without being limited to this, the dies 1130 and 1131 may be electrically connected by metal-metal wiring such as Cu-Cu bonding that directly connects metal wiring such as Cu exposed on the lower surface side of the die 1130 with metal wiring such as Cu exposed on the upper surface side of the die 1131.

[0112] Here, the imaging processing unit 1112 may use, for example, a column-parallel AD method or an area AD method to perform AD conversion on the image signal output from the imaging unit 1111 .

[0113] In the column-parallel AD method, for example, an ADC (AD Converter) is provided for each column of pixels that constitutes the imaging unit 1111, and the ADC for each column is responsible for AD conversion of the pixel signals of the pixels in that column, thereby performing AD conversion of the image signals of the pixels in each column of one row in parallel. When the column-parallel AD method is adopted, part of the imaging processing unit 1112 that performs AD conversion in the column-parallel AD method may be mounted on the upper die 1130.

[0114] In the area AD system, the pixels constituting the imaging unit 1111 are divided into multiple blocks, and an ADC is provided for each block. The ADC in each block is responsible for AD conversion of the pixel signals of the pixels in that block, thereby performing AD conversion of the image signals of the pixels in multiple blocks in parallel. In the area AD system, the block is the smallest unit, and AD conversion (reading and AD conversion) of the image signals can be performed only on necessary pixels among the pixels constituting the imaging unit 1111.

[0115] If the area of ​​the imaging device 1000 is allowed to be large, the imaging device 1000 can be configured with a single die.

[0116] 10, two dies 1130 and 1131 are stacked to form the one-chip imaging device 1000, but three or more dies can be stacked to form the one-chip imaging device 1000. For example, when three dies are stacked to form the one-chip imaging device 1000, the memory 1123 mounted on the die 1131 in FIG. 10 can be mounted on a die different from the dies 1130 and 1031.

[0117] Here, in an imaging device in which the sensor chip, memory chip, and DSP chip are connected in parallel with multiple bumps (hereinafter also referred to as a bump-connected imaging device), the thickness increases significantly and the device becomes larger compared to the single-chip imaging device 1000 configured in a stacked structure.

[0118] Furthermore, in a bump-connected imaging device, due to signal degradation at the bump connection points, it may be difficult to ensure a sufficient rate for outputting image data from the imaging processing unit 1112 to the output control unit 1113.

[0119] The imaging device 1000 having a stacked structure can prevent the device from becoming larger as described above, and can prevent a situation in which a sufficient rate cannot be ensured between the imaging processing unit 1112 and the output control unit 1113. Therefore, the imaging device 1000 having a stacked structure can make the imaging device that outputs information required for processing in a subsequent stage of the imaging device 1000 compact.

[0120] When the information required at a later stage is image data of a captured image, the imaging device 1000 can output image data (such as a RAW image or an RGB image). When the information required at a later stage is obtained by signal processing using image data (such as processing using an AI (Artificial Intelligence) model), the imaging device 1000 can obtain and output the signal processing results as information required by the user by performing the signal processing in the DSP 1122.

[0121] The signal processing performed by the image capturing apparatus 1000, that is, the signal processing by the DSP 1122, may be, for example, recognition processing for recognizing a predetermined recognition target from image data.

[0122] The imaging device 1000 is not limited to the example configured as the CIS described above, but may also have a general camera configuration, such as a configuration in which an image is captured by an imaging element and image data is output that has undergone predetermined signal processing such as noise removal and level adjustment.

[0123] In the user device 10, for example, the CPU 1121 executes the imaging control program according to the embodiment, thereby configuring the above-mentioned overall control unit 100, communication unit 110, image acquisition unit 120, image processing unit 130 and inference unit 140, for example, as modules in the main memory area of ​​the memory 1123.

[0124] The imaging control program can be obtained from the outside via the communication network 2, for example, by communication via the communication I / F 1020, or can be obtained from a storage medium connected to a data I / F not shown, and installed on the user device 10.

[0125] Similarly, in the user device 10, for example, the degradation processing CPU 1010 configures the above-mentioned calibration processing unit, for example as a module, in the main memory area of ​​the memory 1011 by executing the degradation processing program of the embodiment.

[0126] The degradation processing program can be obtained from the outside via the communication network 2, for example, by communication via the communication I / F 1020, or from a storage medium connected to a data I / F not shown, and installed on the user device 10.

[0127] Fig. 11 is a block diagram showing an example of a hardware configuration of the server 30 according to the embodiment. For the sake of explanation, Fig. 11 shows the server 30 as being configured by a single computer.

[0128] In FIG. 11, the server 30 includes a CPU 3000, a ROM 3001, a RAM 3002, a storage device 3003, a data I / F 3004, and a communication I / F 3005, and these components are connected to each other via a bus 3010 so that they can communicate with each other.

[0129] The storage device 3003 is a non-volatile storage medium such as a hard disk drive, flash memory, etc. The CPU 3000 operates in accordance with programs stored in the storage device 3003 or the ROM 3001, using the RAM 3002 as a work memory, and controls the overall operation of the server 30.

[0130] The storage device 3003 or the RAM 3002 may include an algorithm storage area in its storage area for storing algorithms acquired from the user device 10 .

[0131] The data I / F 3004 transmits and receives data to and from external devices. The communication I / F 3005 controls communication with the communication network 2.

[0132] In the server 30, the CPU 3000 executes the information processing program according to the embodiment, thereby configuring the above-mentioned overall control unit 300, communication unit 310, algorithm management unit 320 and algorithm storage unit 330, for example, as modules in the main memory area of ​​the RAM 3002.

[0133] The information processing program can be obtained from outside via the communication network 2, for example, by communication via the communication I / F 3005, or can be obtained from a storage medium connected to the data I / F 3004 and installed on the server 30.

[0134] (4. More Specific Description of Embodiments According to the Present Disclosure) Next, embodiments according to the present disclosure will be described more specifically.

[0135] 12 is a block diagram showing an example of the basic configuration of the user device 10 according to the embodiment. In FIG. 12, the user device 10 according to the embodiment includes an imaging unit 1111, an image processing unit 130, a memory 132, an inference unit 140, and a calibration processing unit 150.

[0136] The captured image output from the imaging unit 1111 is supplied to the image processing unit 130. The image processing unit 130 may correspond to the imaging processing unit 1112 in Fig. 9, and converts the imaging signal supplied from the imaging device 1000 into image data by AD conversion processing, and performs predetermined image processing such as development processing on the image data in accordance with camera parameters 131. The image data that has been subjected to image processing by the image processing unit 130 is stored in a memory 132, which is, for example, a frame memory, on a frame-by-frame basis. The inference unit 140 performs inference processing such as recognition processing on the image data stored in the memory 132, and outputs metadata obtained as a result of the inference processing.

[0137] The calibration processing unit 150 includes a degradation processing unit 160 , a storage unit 170 , and a communication unit 180 .

[0138] The storage unit 170 stores data in, for example, the memory 1011 and the non-volatile memory 1012, and reads data from the memory 1011 and the non-volatile memory 1012. Hereinafter, storing data in the memory 1011 by the storage unit 170 will be described as "storing" by the storage unit 170, and reading data from the memory 1011 by the storage unit 170 will be described as "reading from the storage unit 170," etc.

[0139] The degradation processing unit 160 acquires the image data stored in the memory 132 and the metadata output from the inference unit 140. The degradation processing unit 160 stores the image data acquired from the memory 132 in the storage unit 170.

[0140] The degradation processing unit 160 detects degradation of the captured image output from the imaging unit 1111, for example, based on image data based on the captured image captured at a first time point acquired from the memory 132, image data based on the captured image captured at a second time point earlier than the first time point and stored in the storage unit 170, and metadata acquired from the inference unit 140. The communication unit 180 transmits the detection result of the degradation detection by the degradation processing unit 160 to the server 30 together with the metadata output from the inference unit 140.

[0141] The captured image captured at the first time is the most recent captured image. However, the captured image captured at the first time may be an image captured at a time immediately prior to the most recent captured image (for example, one to several frames prior). The second time may be a time one frame prior to the first time, or may be a time multiple frames prior to the first time.

[0142] Furthermore, the communication unit 180 receives an algorithm selected and transmitted from the server 30 according to the detection result, for example, and passes the received algorithm to the degradation processing unit 160. The degradation processing unit 160 uses the algorithm passed from the communication unit 180 to infer camera parameters that correct the detected degradation of the captured image, and updates the original camera parameters 131 with the inferred camera parameters.

[0143] With this configuration, the information processing system 1 according to the embodiment can detect deterioration of the captured image output from the imaging unit 1111 and automatically execute a process to correct the detected deterioration, thereby suppressing deterioration in the image quality of the captured image due to aging. Furthermore, the information processing system 1 according to the embodiment does not transmit the captured image or image data outside the user device 10 when detecting deterioration of the captured image, thereby realizing a more secure system.

[0144] (4-2. Examples of utilization of inference results by inference unit) Next, examples of utilization of inference results by the inference unit 140 according to the embodiment will be described with reference to Fig. 13A to Fig. 13C. Fig. 13A to Fig. 13C are schematic diagrams for explaining examples of utilization of inference results by the inference unit 140 according to the embodiment.

[0145] The inference unit 140 executes inference processing using a machine learning model based on the image data stored in the memory 132, and outputs metadata. The inference processing executed by the inference unit 140 may be a recognition processing for recognizing an object based on the image data.

[0146] Note that a known method for detecting degradation of a captured image is to use a degradation detection algorithm that calculates a difference image between a plurality of captured images captured at different times with the same angle of view, and performs detection based on this difference image.

[0147] (First Example of Use of Inference Results) FIG. 13A shows a process of identifying a background portion in an image made of image data based on the inference results, as a first example of use of the inference results.

[0148] The inference unit 140a performs object detection by performing recognition processing on the image data supplied from the image processing unit 130. The degradation processing unit 160a performs background determination 1601 on one frame of image data output from the image processing unit 130 based on the object detection result of the recognition processing by the inference unit 140a, and extracts the background part of the image based on the image data.

[0149] The degradation processing unit 160a determines the presence or absence of a foreground object in an image based on the image data, for example, by using a machine learning model. The degradation processing unit 160a stores image data in which no foreground object is detected as image data of a background image in the storage unit 170.

[0150] The degradation processing unit 160a performs index calculation 1602 to calculate an index for determining degradation of the captured image based on a background image based on the latest image data stored in the storage unit 170 and a background image based on image data acquired in the past (e.g., one frame before) with respect to the image data. The index calculation 1602 may be, for example, a process of calculating a difference image between the background image based on the latest image data and the background image based on image data acquired in the past. The degradation processing unit 160a performs degradation determination 1603 based on the index calculated by the index calculation 1602 to determine degradation of the captured image. For example, the degradation processing unit 160a may determine that the captured image has deteriorated when the difference indicated in the difference image is equal to or greater than a predetermined value.

[0151] In this deterioration determination 1603, for example, it is possible to determine and detect deviations in the angle of view, deviations in color, abnormal luminance values, pixel values ​​due to defective pixels, and shading in the captured image.

[0152] The degradation processing unit 160a transmits the result of the detection of degradation of the captured image by the degradation determination 1603 to the server 30 via the communication unit 180 (not shown) together with the metadata that is the inference result of the inference unit 140a.

[0153] That is, if degradation detection for captured images is constantly performed, the degradation detection algorithm may not operate correctly due to foreground objects. For example, if a captured image contains a foreground object, a difference image cannot be obtained properly, making it difficult to detect deviations in the angle of view, etc. In a first application example of the inference results according to this embodiment, a captured image (reference image) that does not contain a foreground object is extracted using the inference results of the inference unit 140a and used for degradation detection. Therefore, it is easy to obtain a difference image, and degradation of the captured image can be detected with high accuracy.

[0154] In the above description, the degradation processing unit 160a has been described as constantly executing the degradation determination 1603. However, this is not limited to this example. For example, the degradation processing unit 160a may execute the degradation determination 1603 only when the recognition score of the object detection in the inference result supplied from the inference unit 140a becomes equal to or less than a predetermined value.

[0155] (Second Example of Use of Inference Results) FIG. 13B shows a second example of use of the inference results, in which a region of interest (ROI) is identified and extracted from an image of image data based on the inference results.

[0156] The inference unit 140a performs recognition processing on the image data supplied from the image processing unit 130 to perform object detection. The degradation processing unit 160b performs ROI extraction 1611 on one frame of image data output from the image processing unit 130 based on the object detection result of the recognition processing by the inference unit 140a, and extracts an ROI in the image based on the image data. In the example of FIG. 13B , the degradation processing unit 160b extracts, as the ROI, a region 401 in which multiple people are detected in the image 400 based on the image data, as a result of object detection by the inference unit 140a, using the ROI extraction 1611. The degradation processing unit 160b cuts out the ROI region 401 from the image 400 and stores it in the storage unit 170 as an ROI image 402.

[0157] The degradation processing unit 160b performs index calculation 1612 to obtain an index for determining degradation of the captured image, similar to index calculation 1602 in Figure 13A, based on an ROI image 402 based on the latest image data stored in the storage unit 170 and an ROI image 402 based on image data acquired in the past (e.g., one frame before) with respect to the image data.

[0158] The index calculation 1612 may be, for example, a process of calculating a difference image between the ROI image 402 based on the latest image data and the ROI image 402 based on image data acquired in the past. The degradation processing unit 160b performs degradation determination 1613 based on the index calculated by the index calculation 1612, and determines the degradation of the captured image in the same manner as described above.

[0159] In this deterioration determination 1613, it is possible to determine and detect, for example, color shifts in the captured image, abnormal luminance values, and pixel values ​​due to defective pixels.

[0160] The degradation processing unit 160b transmits the result of the detection of degradation of the captured image by the degradation determination 1613 to the server 30 via the communication unit 180 (not shown) together with the metadata that is the inference result of the inference unit 140a.

[0161] That is, the ROI within the angle of view differs for each scene and each camera. Meanwhile, depending on the degradation detection algorithm used to detect degradation in the captured image, it may react to minute changes in the image outside the ROI in the captured image. In a second application example of the inference results according to this embodiment, degradation detection is performed using only the ROI image 402 extracted from an image based on image data of the captured image. That is, in this second application example, degradation detection for the captured image is performed by focusing on an area where objects are detected frequently. In this way, by performing processing using the degradation detection algorithm by focusing on a more meaningful area, it is possible to detect degradation in the captured image with higher accuracy.

[0162] (Third Example of Use of Inference Results) FIG. 13C shows a third example of use of the inference results, in which deterioration detection is performed for each object of a target category included in an image of image data based on the inference results.

[0163] The inference unit 140a performs recognition processing on the image data supplied from the image processing unit 130 to detect objects. The degradation processing unit 160c performs object classification 1621 for each detection target category based on the object detection result of the recognition processing by the inference unit 140a. In the example of Fig. 13C, the degradation processing unit 160c classifies the images of detected objects into the categories of "bicycle," "car," and "person" by the object classification 1621 and stores them in the storage unit 170.

[0164] The degradation processing unit 160c performs index calculation 1622 for each category based on the image of the object for each category based on the latest image data and the image of the object for each category based on image data acquired in the past (for example, one frame before) with respect to the image data, both of which are stored in the storage unit 170. The index calculation 1622 may be, for example, a process of calculating, for each category, a difference image between the image of each category based on the latest image data and the image of each category acquired in the past.

[0165] The degradation processing unit 160c performs degradation determination 1623 based on the index (difference image) for each category calculated in the index calculation. For example, the degradation processing unit 160c performs a category-by-category determination on the index calculated for each category in the degradation determination 1623 and takes the logical sum of the determinations. The degradation processing unit 160c may determine that the captured image is degraded if the degradation determination 1623 results in a failure in any of the categories. The degradation processing unit 160c may perform degradation determination 1623 by setting determination conditions for the index for each category.

[0166] In this deterioration determination 1623, it is possible to determine and detect, for example, color shifts and abnormal brightness values ​​in the captured image.

[0167] The degradation processing unit 160c transmits the result of detection of degradation of the captured image by the degradation determination 1623 to the server 30 via the communication unit 180 (not shown) together with the metadata that is the inference result of the inference unit 140a.

[0168] That is, appropriate image setting values ​​(brightness value, noise resistance, etc.) may differ for each object, i.e., category. Therefore, if each category is judged using a uniform image quality setting, the performance of the degradation judgment 1623 may not be fully utilized. In this third application example, degradation detection is performed on captured images for each category. Therefore, it is possible to detect degradation of captured images with higher accuracy.

[0169] The first to third examples of utilization of the inference results described above can be implemented in combination with each other.

[0170] (4-3. Detection of deterioration of captured image and calculation of correction parameters) Next, a first example and a second example of the deterioration detection of the captured image according to the embodiment and the calculation of correction parameters according to the deterioration detection results will be explained in more detail using Figures 14A and 14B, as well as Figure 15.

[0171] (First Example of Deterioration Detection and Correction Parameter Calculation Processing) First, a first example of the deterioration detection and correction parameter calculation processing will be described. Fig. 14A is a schematic diagram for describing processing for detecting a misalignment in the angle of view of a captured image according to the embodiment, according to the first example. In the example of Fig. 14A, it is assumed that a machine learning model for performing object detection used to detect a misalignment in the angle of view is deployed in advance in the inference unit 140a.

[0172] The image processing unit 130 performs AD conversion processing on the captured image output from the imaging unit 1111 (not shown), and performs predetermined image processing such as development processing on the image data obtained by AD conversion of the captured image in accordance with camera parameters 131 (not shown).

[0173] The inference unit 140a performs object detection by performing recognition processing on the image data output from the image processing unit 130. The degradation processing unit 160d performs background determination 1631 on one frame of image data output from the image processing unit 130 based on the object detection result of the recognition processing by the inference unit 140a, and extracts the background part of the image based on the image data.

[0174] The degradation processing unit 160d determines the presence or absence of a foreground object in an image based on the image data, for example, by using a machine learning model. The degradation processing unit 160d stores image data in which no foreground object is detected as image data of a background image in the storage unit 170.

[0175] The degradation processing unit 160d calculates, for example, a difference image between the background image based on the latest image data and the background image based on image data acquired one frame before the current image data, both of which are stored in the storage unit 170, and performs index calculation 1632 to calculate an index for determining degradation of the captured image based on the calculated difference image.The degradation processing unit 160d performs degradation determination 1633 based on the index calculated by index calculation 1632, and determines the angle of view deviation of the captured image.

[0176] The degradation processing unit 160d transmits, via the communication unit 180, a degradation notification indicating degradation of the angle of view deviation determined by the degradation determination 1603 to the server 30 together with the metadata that is the inference result of the inference unit 140a.

[0177] At this time, for example, the communication unit 180 may include information identifying the detection performed by the degradation processing unit 160d (in this example, the detection of an angle of view shift) and information identifying the user device 10 in the degradation result and transmit this to the server 30. In addition, the communication unit 180 transmits to the server 30 the algorithm used by the degradation processing unit 160d for the degradation detection and the machine learning model used by the inference unit 140a for object detection.

[0178] That is, if the user device 10 is a fixed camera that is installed and used in a fixed position, the installation position or lens position of the camera may change due to external factors. In the embodiment, based on the object detection results by the inference unit 140a, only a background image (reference image) without a foreground object is stored in the storage unit 170, and degradation of the captured image due to a deviation in the angle of view is detected by calculating the difference between the stored background image and a background image based on the most recent captured image. This makes it possible to improve the robustness and detection accuracy of the degradation detection algorithm.

[0179] FIG. 14B is a schematic diagram for explaining the process of correcting the angle of view deviation of the captured image according to the embodiment, according to the first example.

[0180] 14A from the user device 10. The server 30 also receives the algorithm used by the degradation processing unit 160d for degradation detection and the machine learning model used by the inference unit 140a for object detection, both of which are transmitted from the user device 10. The algorithm management unit 320 temporarily stores the received algorithm and machine learning model in the algorithm storage unit 330.

[0181] The server 30, for example, acquires, via the algorithm management unit 320, an algorithm used by the degradation processing unit 160d to correct the angle of view shift and a machine learning model used by the inference unit 140a from the algorithms stored in the algorithm storage unit 330, in accordance with information identifying the detection of angle of view shift included in the received degradation notification. The server 30 transmits the acquired algorithm and machine learning model to the user device 10 identified by the identification information included in the degradation notification.

[0182] The user device 10 receives the algorithm and the machine learning model transmitted from the server 30. The user device 10 deploys the received algorithm and replaces the algorithm of the degradation processing unit 160 with the received algorithm. Similarly, the user device 10 deploys the received machine learning model and replaces the machine learning model held by the inference unit 140 with the received machine learning model.

[0183] In Figure 14B, the image processing unit 130 performs AD conversion processing on the captured image output from the imaging unit 1111 (not shown), and performs predetermined image processing such as development processing on the image data obtained by AD conversion of the captured image in accordance with camera parameters 131.

[0184] The inference unit 140b functions as a feature detector by replacing the machine learning model from an object detection model to a model that detects features based on image data. The degradation processing unit 160e replaces the algorithm from a degradation detection algorithm to a correction algorithm for correcting the camera parameters 131.

[0185] The inference unit 140b performs feature amount detection processing on the image data output from the image processing unit 130 and extracts feature points of the image based on the image data. The inference unit 140b passes the extracted feature points to the degradation processing unit 160e. The degradation processing unit 160e stores the feature points passed from the inference unit 140b in the storage unit 170. The degradation processing unit 160e performs feature point matching 1641 based on the feature points of the current frame passed from the inference unit 140b and feature points (reference feature points) extracted based on frames prior to the current frame and stored in the storage unit 170. The degradation processing unit 160e performs homography transformation 1642 based on the result of the feature point matching 1641 and calculates a transformation matrix that converts the coordinates of the current frame to the coordinates of the previous frame. The degradation processing unit 160e updates the camera parameters 131 using the calculated transformation matrix.

[0186] When the degradation processing unit 160e has completed updating the camera parameters 131, it notifies the communication unit 180 of this fact. In response to this notification, the communication unit 180 transmits to the server 30 a completion notification indicating that the updating of the camera parameters 131 has been completed.

[0187] In response to the completion notification sent from the user device 10, the server 30 causes the algorithm management unit 320 to acquire the algorithm and machine learning model temporarily stored in the algorithm storage unit 330. The algorithm management unit 320 transmits the algorithm and machine learning model acquired from the algorithm storage unit 330 to the user device 10 that sent the completion notification.

[0188] The algorithm and machine learning model transmitted from the server 30 are deployed in the user device 10. That is, in the user device 10, the algorithm of the degradation processing unit 160e is replaced with the algorithm transmitted from the server 30. Also, the machine learning model of the inference unit 140b is replaced with the machine learning model transmitted from the server 30. That is, the degradation processing unit 160e and the inference unit 140b are each returned to the state before the camera parameters 131 were updated.

[0189] (First Example of Deterioration Detection and Correction Parameter Calculation Process) Next, a second example of the deterioration detection and correction parameter calculation process will be described. Fig. 15 is a schematic diagram for explaining the process of detecting and correcting color misalignment according to the embodiment in the second example. Note that the color here may be at least one of hue and saturation.

[0190] In the example of FIG. 15, it is assumed that the inference unit 140c is previously deployed with a machine learning model for performing segmentation processing, which is used to detect color misalignment.

[0191] The image processing unit 130 performs AD conversion processing on the captured image output from the imaging unit 1111 (not shown), and performs predetermined image processing such as development processing on the image data obtained by AD conversion of the captured image in accordance with camera parameters 131.

[0192] The inference unit 140c performs segmentation processing on the image data output from the image processing unit 130. The degradation processing unit 160f extracts the foreground in an image formed by one frame of image data output from the image processing unit 130, based on each part identified by the segmentation processing of the inference unit 140a. The degradation processing unit 160f performs foreground masking 1651 on the image data output from the image processing unit 130, and masks the area of ​​the image data that corresponds to the extracted foreground.

[0193] The degradation processing unit 160f performs color measurement 1652 based on the image data in which the foreground is masked. For example, the degradation processing unit 160f acquires the value of the R component (referred to as the R value), the value of the G component (referred to as the G value), and the value of the B component (referred to as the B value) of each pixel in the image data in which the foreground is masked, using the color measurement 1652, and calculates, for example, the proportion of each component for each pixel based on the acquired R value, G value, and B value, thereby determining the color for each pixel.

[0194] The degradation processing unit 160f stores information indicating the color of each pixel in the storage unit 170 as a result of the color measurement 1652. Here, the degradation processing unit 160f stores the results of the color measurement 1652 obtained for each frame of the captured image in the storage unit 170 sequentially.

[0195] The degradation processing unit 160f performs index calculation 1653 to calculate an index of color shift based on the color measurement results sequentially stored in the storage unit 170. The degradation processing unit 160f calculates a systematic error of the color measurement results for each pixel, for example, a time-series fluctuation of the color measurement results for each pixel, based on each color measurement result stored in the storage unit 170. The degradation processing unit 160f performs index calculation 1653 to calculate an index for determining color deterioration of the captured image based on the results of this systematic error measurement. The degradation processing unit 160f performs degradation determination 1654 based on the index calculated by index calculation 1653, and determines color shift of the captured image.

[0196] The degradation processing unit 160f transmits, via the communication unit 180 (not shown), a degradation notification indicating degradation of color misalignment determined by the degradation judgment 1654 to the server 30 together with the metadata that is the inference result of the inference unit 140c.

[0197] 14B , the server 30 may select, via the algorithm management unit 320, an algorithm and a machine learning model to be stored in the algorithm storage unit 330 and used for correcting color based on the degradation notification and metadata transmitted from the user device 10, and transmit these to the user device 10. At the same time, the server 30 may acquire, via the algorithm management unit 320, the degradation detection algorithm of the degradation processing unit 160f and the machine learning model of the inference unit 140c from the user device 10, and temporarily store them in the algorithm storage unit 330.

[0198] The subsequent processing in the server 30 and the user device 10 is the same as the processing described using Figures 14A and 14B, so a description thereof will be omitted here.

[0199] (Modification of the embodiment) A modification of the embodiment will be described. In the case of color shift correction, the deterioration processing unit 160f can directly obtain a correction value for updating the camera parameters 131 based on the result of the deterioration determination 1654 by color shift detection.

[0200] For example, the degradation processor 160f can obtain changes in each RGB value as a result of measuring systematic errors based on the color measurement results for each pixel using the index calculation 653. The degradation processor 160f calculates, for each pixel, correction values ​​for each RGB value that offset these changes. The degradation processor 160f uses these correction values ​​for each pixel to correct coefficients for each pixel in the development process in the camera parameters 131.

[0201] In this way, in the modified example of the embodiment, the update process of the camera parameters 131 in response to deterioration of color in the captured image can be completed within the user device 10 without notifying the server 30.

[0202] (4-4. Overall Flow of Processing According to the Embodiment) Next, the overall flow of processing according to the embodiment will be described.

[0203] (Flow of Deterioration Detection Processing) Fig. 16 is a flowchart showing an example of deterioration detection processing according to an embodiment. In Fig. 16, steps S100 to S154 represent processing in the user device 10, and steps S30 and S31 represent processing in the server 30. Also, steps S100 to S106 represent processing by the overall control unit 100, step S130 represents processing by the image processing unit 130, step S140 represents processing by the inference unit 140, and steps S150a to S154 represent processing by the calibration processing unit 150 (deterioration processing unit 160).

[0204] In step S100, the overall control unit 100 issues an instruction to start processing the captured image of the latest frame output by the imaging unit 1111. In response to this instruction, the image processing unit 130 performs AD conversion processing (not shown) and development processing on the captured image output from the imaging unit 1111 (step S130). The inference unit 140 performs inference processing based on the image data developed by the image processing unit 130. Metadata based on the inference result by the inference unit 140 is passed to the overall control unit 100 and the calibration processing unit 150.

[0205] The calibration processing unit 150 calculates the shape of a bounding box (bbox) in the image data of the captured image based on the metadata passed from the inference unit 140 (step S150a). The calibration processing unit 150 sequentially stores information about the bounding box whose shape has been calculated for each frame in, for example, the storage unit 170. The calibration processing unit 150 calculates a systematic error of the bounding box shape based on the information about the bounding box calculated in step S150a and stored in the storage unit 170 (step S151a).

[0206] The calibration processing unit 150 also sequentially stores the image data developed by the image processing unit 130 in the storage unit 170 for each frame. The calibration processing unit 150 creates a difference image between the latest image data developed by the image processing unit 130 and past image data (e.g., one frame before) stored in the storage unit 170 (step S150b), and detects a misalignment in the angle of view of the captured image based on the created difference image (step S151b). The calibration processing unit 150 also detects the color difference between the latest image data developed by the image processing unit 130 and each of the past image data stored in the storage unit 170, calculates a systematic error in the color difference of each of the detected image data, and detects any color abnormalities (step S151c).

[0207] In this figure, the processes of steps S150a and S151a, the processes of steps S150b and S151b, and the processes of steps S150c and S151c are shown to be executed in parallel, but this is not limited to this example. For example, the processes of steps S150a and S151a, the processes of steps S150b and S151b, and the processes of steps S150c and S151c may be executed in series.

[0208] The calibration processing unit 150 determines whether or not to send a degradation notification to the server 30 based on the systematic error of the bounding box calculated in step S151a, the detection result of the angle of view shift in step S151b, and the detection result of the color abnormality in step S151c (step S152).

[0209] If the calibration processing unit 150 determines that it is not necessary to send a degradation notification to the server 30 (step S152, "Notification not required"), it instructs the overall control unit 100 to that effect. When the overall control unit 100 receives the instruction that a notification is not required from the calibration processing unit 150, it proceeds to processing of the next frame (step S102). At the same time, the overall control unit 100 outputs the metadata received from the inference unit 140 to, for example, the server 30 (step S101).

[0210] On the other hand, if the calibration processing unit 150 determines in step S152 that it is necessary to send a degradation notification to the server 30 (step S152, "Notification required"), it checks the license information in the user device 10 (step S153).

[0211] The license information is stored in advance in a nonvolatile storage medium of the user device 10 , such as the nonvolatile memory 1012 or a nonvolatile area in the memory 1123 .

[0212] The license information may be, for example, unique information for each user device 10, and may be linked to a table indicating whether calibration processing for degradation of captured images is permitted in the user device 10, or to what stage the calibration processing is permitted. It is also possible to link billing information for the user to the license information. For example, the license information can be used to select parameters to be corrected, and billing can be set according to the selected parameters. It is also possible to limit the number of algorithms that can be deployed according to the license information. Furthermore, for example, the frequency of degradation detection can be set according to the license information, and it is also possible to set it so that degradation of captured images is constantly monitored, and degradation notifications and parameter updates can be performed outside of business hours.

[0213] If the calibration processing unit 150 determines that the license information in the user device 10 is invalid due to expiration or the like (step S153, "Invalid"), it notifies the overall control unit 100 of this fact. The overall control unit 100 then notifies, for example, the user, that the license information is invalid (step S103).

[0214] On the other hand, if the calibration processing unit 150 determines in step S153 that the license information for the user device 10 is valid (step S153, "Valid"), it notifies the overall control unit 100 of this fact. In response to this notification from the calibration processing unit 150, the overall control unit 100 transmits a degradation notification to the server 30 (step S104). The overall control unit 100 may transmit the degradation notification to the server 30 including identification information for identifying the user device 10.

[0215] The server 30 receives the degradation notification transmitted from the user device 10 (overall control unit 100) (step S30). The server 30, using the algorithm management unit 320, selects an algorithm to be deployed to the user device 10 from the algorithms stored in the algorithm storage unit 330, in accordance with the information indicating the type of degradation included in the degradation notification (step S31). The server 30 further selects a machine learning model to be deployed to the user device 10 from the machine learning models stored in the algorithm storage unit 330, in accordance with the information indicating the type of degradation.

[0216] The server 30 transmits the selected algorithm and machine learning model to the user device 10. Although not shown in the figure, the server 30 also acquires from the user device 10 the algorithm held by the degradation processing unit 160 in the calibration processing unit 150 and the machine learning model held by the inference unit 140, and temporarily stores them in the storage unit 170.

[0217] In the user device 10, the overall control unit 100 receives the algorithm and machine learning model transmitted from the server 30 and performs deployment control to deploy the received algorithm and machine learning model to the calibration processing unit 150 (degradation processing unit 160) and the inference unit 140, respectively (step S105). Through this deployment control, the machine learning model is deployed to the inference unit 140 (step S141), and the algorithm is deployed to the calibration processing unit 150 (degradation processing unit 160) (step S154).

[0218] In the user device 10, the overall control unit 100 ends the processing of the current frame in response to the transition to the next frame processing in step S102 (step S106), returns the processing to step S100, and starts processing the next frame.

[0219] (Flow of Correction Model Deployment Process) Fig. 17 is a flowchart of an example of the correction model deployment process according to the embodiment. In Fig. 17, steps S200 to S254 represent processes in the user device 10, and steps S32 and S311 represent processes in the server 30. Also, steps S200 to S206 represent processes by the overall control unit 100, step S230 represents processes by the image processing unit 130, step S240 represents processes by the inference unit 140, and steps S250 to S254 represent processes by the calibration processing unit 150 (deterioration processing unit 160).

[0220] It is assumed that, prior to the processing of the flowchart in Fig. 17, the machine learning model used for correction is deployed in the inference unit 140 in step S141 of the flowchart in Fig. 16. Similarly, it is assumed that the algorithm used for correction is deployed in the calibration processing unit 150 (deterioration processing unit 160) in step S154 of the flowchart in Fig. 16.

[0221] In step S200, the overall control unit 100 issues an instruction to start processing the captured image of the latest frame output by the imaging unit 1111. In response to this instruction, the image processing unit 130 performs AD conversion processing (not shown) and development processing on the captured image output from the imaging unit 1111 (step S230). The inference unit 140 performs inference processing based on the image data developed by the image processing unit 130. Metadata based on the inference result by the inference unit 140 is passed to the overall control unit 100 and the calibration processing unit 150.

[0222] The calibration processing unit 150 calculates parameters for correcting degradation of the captured image based on the metadata passed from the inference unit 140 and the image data output from the image processing unit 130 (step S250), and updates the camera parameters 131 using the calculated parameters (step S251).

[0223] The calibration processing unit 150 determines whether the update of the camera parameters 131 has been completed in the parameter update in step S251. As an example, consider a case where the calibration processing unit 150 performs feature point matching 1641 shown in Fig. 14B and uses the similarity obtained thereby as an index. In this case, if the similarity is less than a predetermined value, it can be determined that the matching was not successful and the update of the camera parameters 131 was insufficient.

[0224] If the calibration processing unit 150 determines that the camera parameters 131 have not been updated sufficiently (step S252, "insufficient update"), it notifies the overall control unit 100 of this fact. Upon receiving notification of insufficient update from the calibration processing unit 150, the overall control unit 100 proceeds to processing of the next frame (step S202). At the same time, the overall control unit 100 outputs the metadata received from the inference unit 140 to, for example, the server 30 (step S201).

[0225] On the other hand, if the calibration processing unit 150 determines in step S252 that the update of the camera parameters 131 has been completed (step S252, "update completed"), it checks the license information in the user device 10 (step S253).

[0226] If the calibration processing unit 150 determines that the license information in the user device 10 is invalid due to expiration or the like (step S253, "Invalid"), it notifies the overall control unit 100. The overall control unit 100 notifies, for example, the user, that the license information is invalid (step S203).

[0227] On the other hand, if the calibration processing unit 150 determines in step S253 that the license information for the user device 10 is valid (step S253, "Valid"), it notifies the overall control unit 100 of this fact. In response to this notification from the calibration processing unit 150, the overall control unit 100 transmits a completion notification to the server 30 (step S204). The overall control unit 100 may transmit the completion notification to the server 30 including identification information for identifying the user device 10.

[0228] The server 30 receives the completion notification sent from the user device 10 (overall control unit 100) (step S32). The server 30, using the algorithm management unit 320, selects the algorithm and machine learning model acquired from the user device 10 in step S31 of FIG. 16 and temporarily stored in the algorithm storage unit 330 (step S33), and transmits them to the user device 10.

[0229] In the user device 10, the overall control unit 100 receives the algorithm and machine learning model transmitted from the server 30 and performs deployment control to deploy the received algorithm and machine learning model to the calibration processing unit 150 (degradation processing unit 160) and the inference unit 140, respectively (step S205). Through this deployment control, the algorithm is deployed to the calibration processing unit 150 (degradation processing unit 160) (step S254), and the machine learning model is deployed to the inference unit 140 (step S241).

[0230] In the user device 10, in response to the transition to the next frame processing in step S202, the overall control unit 100 ends the processing of the current frame (step S206), returns the processing to step S200, and starts processing of the next frame.

[0231] 18 is a time chart of an example of the deterioration detection process according to the embodiment. In FIG. 18, the passage of time is shown to the right, and from the top, the processes by the overall control unit 100, image processing unit 130, inference unit 140, and calibration processing unit 150 in the user device 10 are shown, and below that, the processes by the server 30 are shown.

[0232] For example, when processing of the first frame (also referred to as frame #1 in the figure) begins, the image processing unit 130 performs AD conversion processing and development processing on the image signal output from the imaging unit 1111 (step S3001). Upon completion of the development processing and other processes, the image processing unit 130 passes control to the overall control unit 100. After a predetermined processing time has elapsed (step S3011), the overall control unit 100 instructs the inference unit 140 to perform inference processing.

[0233] In FIG. 18 and FIG. 19 described later, the xth frame (x is an integer equal to or greater than 1) is indicated as frame #x.

[0234] In response to an instruction from the overall control unit 100, the inference unit 140 executes inference processing using a pre-stored machine learning model based on the image data output from the image processing unit 130 (step S3021). When the inference processing by the inference unit 140 is completed, the overall control unit 100 outputs metadata (also referred to as "meta output" in the figure) based on the inference result (step S3031). At the same time, the calibration processing unit 150 performs degradation detection of the captured image based on the inference result by the inference unit 140 and the image data output from the image processing unit 130 (step S3041).

[0235] When the calibration processing unit 150 finishes the degradation detection, it passes control to the overall control unit 100. After a predetermined processing time (step S3051), the overall control unit 100 transmits a degradation notification to the server 30 (step S310). The degradation notification transmitted from the user device 10 is received by the server 30 via the communication network 2. It is thought that it takes several frames for the degradation notification to be transmitted from the user device 10 and received by the server 30, although this depends on the communication environment of the communication network 2. In the example shown in the figure, the degradation notification is received at a timing corresponding to the mth frame in the user device 10, several frames, several tens of frames, or even more, after the first frame.

[0236] In response to the degradation notification received from the user device 10, the server 30 selects, from the algorithm storage unit 330, a deployment target (algorithm, machine learning model) to be deployed to the user device 10 using the algorithm management unit 320 (step S311). This deployment target selection process can be completed within one frame period of the user device 10, for example. The server 30 transmits the selected deployment target to the user device 10 (step S312).

[0237] In addition, the server 30 acquires from the user device 10 the algorithm deployed in the calibration processing unit 150 (degradation processing unit 160) and the machine learning model deployed in the inference unit 140, and temporarily stores the acquired algorithm and machine learning model in the algorithm storage unit 330.

[0238] Meanwhile, in the user device 10, in the mth frame, as in the first frame, the image processing unit 130 performs AD conversion processing and development processing on the image signal output from the image capturing unit 1111 (step S300 m ), and when development processing and the like are completed, control is passed to the overall control unit 100.

[0239] After a predetermined processing time has elapsed (step S301 m ), and instructs the inference unit 140 to perform inference processing. The inference unit 140 executes inference processing using a pre-stored machine learning model based on the image data output from the image processing unit 130 in response to the instruction from the overall control unit 100 (step S302 m When the inference process by the inference unit 140 is completed, the overall control unit 100 outputs metadata based on the inference results (step S303 m At the same time, the calibration processing unit 150 detects deterioration of the captured image based on the inference result by the inference unit 140 and the image data output from the image processing unit 130 (step S304 m ).

[0240] The calibration processing unit 150 may omit the deterioration detection process during the deterioration notification process in step S310.

[0241] In the illustrated example, the deployment target sent from the server 30 is received by the user device 10 in the middle of the nth frame after several frames or several tens of frames or more have passed through the communication network 2.

[0242] In the user device 10, the image processing unit 130 performs development processing and the like on the nth frame in the same manner as the first frame (step S300 n After the development process and the like are completed, a predetermined processing time is passed in the overall control unit 100 (step S301 n ), the inference unit 140 executes the inference process (step S302 n When the inference process by the inference unit 140 is completed, the overall control unit 100 outputs metadata based on the inference results (step S303 n ).

[0243] In the illustrated example, the overall control unit 100 executes step S303 n When the metadata output is completed, the overall control unit 100 executes deployment control to deploy the deployment target (algorithm, machine learning model) transmitted from the server 30 (step S306). The overall control unit 100 may execute deployment control of the deployment target at any timing after receiving the deployment target, not limited to after the metadata output.

[0244] In the example shown in the figure, in the next (n+1)th frame, the image processing unit 130 in the user device 10 performs AD conversion processing and development processing on the image signal output from the image capturing unit 1111 (step S300 n+1 ), and when development processing and the like are completed, control is passed to the overall control unit 100.

[0245] After a predetermined processing time has elapsed (step S301 n+1), and executes deployment of the deployment targets transmitted from the server 30 in step S312. That is, the overall control unit 100 deploys the machine learning model among the deployment targets to the inference unit 140 (step S320), and deploys the algorithm to the calibration processing unit 150 (degradation processing unit 160) (step S321). Note that in the figure, the deployment process in step S320 is shown as model deployment, and the deployment process in step S321 is shown as algorithm deployment.

[0246] The overall control unit 100 may deploy the deployment target at any timing. For example, the deployment process does not affect the processing of the image processing unit 130, so the deployment process may overlap with the development process by the image processing unit 130. Furthermore, the deployment of the deployment target does not have to be completed within one frame period in the user device 10.

[0247] In this manner, in the embodiment, the degradation detection and degradation notification processes are executed using a period of one frame, and then the deployment process for the deployment target in response to the degradation detection is executed over a period of multiple frames.

[0248] (Parameter Update Process) Fig. 19 is a time chart of an example of a parameter update process according to an embodiment. The process shown in Fig. 19 is a process performed after the deployment process in steps S320 and S321 in Fig. 18 is completed. For the sake of explanation, the first frame is shown as the first frame in the figure.

[0249] For example, when processing of the first frame starts, the image processing unit 130 performs AD conversion processing and development processing on the image signal output from the imaging unit 1111 (step S4001). When the image processing unit 130 completes the development processing and the like, it passes control to the overall control unit 100. After a predetermined processing time has elapsed (step S4011), the overall control unit 100 instructs the inference unit 140 to perform inference processing.

[0250] In response to an instruction from the overall control unit 100, the inference unit 140 executes inference processing using the machine learning model deployed in step S320 of Fig. 18 based on the image data output from the image processing unit 130 (step S4021). When the inference processing by the inference unit 140 ends, the calibration processing unit 150 calculates parameters using the algorithm deployed in step S321 of Fig. 18 based on the inference result by the inference unit 140 and the image data output from the image processing unit 130 (step S4031), and updates the camera parameters 131 with the calculated parameters (step S4041).

[0251] When the calibration processing unit 150 completes updating the camera parameters 131, it passes control to the overall control unit 100. After a predetermined processing time has elapsed (step S405), the overall control unit 100 transmits a completion notification indicating that the updating of the camera parameters 131 has been completed to the server 30 (step S410). The completion notification is received by the server 30 after a period corresponding to multiple frames in the user device 10. In the example shown in the figure, the server 30 receives the completion notification during the m-th frame period in the user device 10.

[0252] In response to the received completion notification, the server 30 selects from the algorithm storage unit 330, via the algorithm management unit 320, the algorithm and machine learning model (collectively referred to as the original model) that the user device 10 had before receiving the degradation notification, which were temporarily stored in the process of step S312 in FIG. 18 (step S411). This original model selection process can be completed within, for example, one frame period in the user device 10. The server 30 transmits the selected original model to the user device 10 via the algorithm management unit 320.

[0253] Meanwhile, in the user device 10, in the mth frame, as in the first frame, the image processing unit 130 performs AD conversion processing and development processing on the image signal output from the image capturing unit 1111 (step S400 mAfter the development process and the like are completed, the control is passed to the overall control unit 100. The overall control unit 100 returns to the initial state after a predetermined processing time (step S401 m ), and instructs the inference unit 140 to perform inference processing. The inference unit 140 executes inference processing using a pre-stored machine learning model based on the image data output from the image processing unit 130 in response to the instruction from the overall control unit 100 (step S402 m ).

[0254] Note that the parameter calculation and update process by the calibration processing unit 150 has already been completed in the first frame. Therefore, after the update of the camera parameters 131 is completed in step S404, the calibration processing unit 150 may stop its operation until algorithm deployment (step S421), which will be described later.

[0255] In the illustrated example, the original model transmitted from the server 30 is received by the user device 10 in the middle of the nth frame via the communication network 2 after several frames or several tens of frames or more.

[0256] In the user device 10, the image processing unit 130 performs development processing and the like on the nth frame in the same manner as the first frame (step S400 n After the development process and the like are completed, a predetermined processing time is passed in the overall control unit 100 (step S401 n ), the inference unit 140 executes the inference process (step S402 n In the example shown in the figure, when the inference processing by the inference unit 140 is completed, the overall control unit 100 outputs metadata based on the inference result (step S406 n ).

[0257] In the illustrated example, the overall control unit 100 executes step S406 nWhen the metadata output is completed, the overall control unit 100 executes deployment control to deploy the original model (algorithm, machine learning model) transmitted from the server 30 (step S407). The overall control unit 100 may execute deployment control of the original model at any timing after receiving the original model, not limited to after the metadata output.

[0258] In the example shown in the figure, in the next (n+1)th frame, the image processing unit 130 in the user device 10 performs AD conversion processing and development processing on the image signal output from the image capturing unit 1111 (step S400 n+1 After the development process and the like are completed, the control is passed to the overall control unit 100. The overall control unit 100 returns to the initial state after a predetermined processing time (step S401 n+1 ), and then deploys the original model transmitted from the server 30 in step S412. That is, the overall control unit 100 deploys the machine learning model of the original model to the inference unit 140 (step S420), and deploys the algorithm to the calibration processing unit 150 (deterioration processing unit 160) (step S421).

[0259] The overall control unit 100 may deploy the original model at any timing. For example, the deployment process does not affect the processing of the image processing unit 130, so the deployment process may overlap with the development process by the image processing unit 130. Furthermore, the deployment of the original model does not have to be completed within one frame period in the user device 10. In the example shown in the figure, the deployment of the original model in steps S420 and S421 is completed immediately before the (n+k)th frame, which is k-1 frames after the (n+1)th frame.

[0260] In the user device 10, from the (n+k)th frame onwards, processing is performed by the calibration processing unit 150 and the inference unit 140, to which the original models are respectively deployed. That is, in the user device 10, development processing and the like are performed in the image processing unit 130 in the (n+k)th frame (step S400 n+k After the development process and the like are completed, a predetermined processing time is passed in the overall control unit 100 (step S401n+k ), the inference unit 140 executes the inference process (step S402 n+k ).

[0261] When the inference process by the inference unit 140 is completed, the overall control unit 100 outputs the metadata based on the inference result (step S406 n+k ). At the same time, the calibration processing unit 150 detects deterioration of the captured image based on the inference result by the inference unit 140 and the image data output from the image processing unit 130 (step S407). When the calibration processing unit 150 finishes the deterioration detection, it passes control to the overall control unit 100. After a predetermined processing time has elapsed (step S408), the overall control unit 100 executes the subsequent processing.

[0262] (4-5. Variations of Output Data) Next, a description will be given of variations of output data output from the user device 10 to the server 30 according to the embodiment. Fig. 20 is a schematic diagram for explaining variations of output data according to the embodiment.

[0263] The above-described user device 10 does not output images, and therefore the only information outputted is metadata based on the inference results of the inference unit 140. Therefore, it is necessary to use this metadata to appropriately notify the user device 10 of the state inside the user device 10 and degradation information indicating the degradation of the captured image.

[0264] 20 , the output data is roughly divided into output data in a degradation notification phase in which the user device 10 notifies the server 30 of the degradation, and output data in a correction notification phase in which the user device 10 notifies the server 30 of the completion of correction. The communication unit 180 transmits the output data in the degradation notification phase and the correction notification phase as a degradation notification to the server 30. The degradation notification may be added to the metadata output by the inference unit 140 and transmitted to the server 30, for example.

[0265] In the degradation notification phase, the output data includes the following items: "Whether image quality degradation has been detected," "Details of degradation," "Frame and time (latency) at which degradation was detected," "Whether internal correction has been completed," and "Setting value at the time of degradation."

[0266] The item "Whether image quality degradation has been detected" is an item for informing a subsequent stage (e.g., the server 30) whether image degradation has occurred, and its value is either "True" or "False." If the value of the item "Whether image quality degradation has been detected" is "True," the subsequent stage begins processing to determine what type of degradation has occurred. If the value is "False," the user device 10 begins operating as usual.

[0267] The item "degradation details" is an item for informing the subsequent stage of what kind of image degradation has occurred, and the value indicates the type of degradation, for example, it includes at least the following values: - Angle of view deviation - Color deviation - Abnormal brightness value - Occurrence of distortion - Defective pixel value and its location - Shading

[0268] In a later stage, for example, the algorithm management unit 320 of the server 30 determines which correction algorithm to deploy depending on which image degradation has occurred. For example, if correction of angle of view deviation is required, the algorithm management unit 320 deploys a machine learning model that functions as a feature extractor to the user device 10 and calculates a transformation matrix for image correction.

[0269] The item "Frame and time (latency) where degradation was detected" is used in subsequent analysis to determine the point in time at which degradation occurred, and includes a "timestamp" and a "frame ID." By outputting the frame ID, metadata from the machine learning model of the target frame can be referenced in subsequent processing, making it possible to know the situation at the time degradation was detected in subsequent processing. The value of this item can also be used in subsequent processing to reevaluate and determine whether correction is necessary.

[0270] The item "Internal correction completed or not" is an item for determining whether correction is necessary at a later stage, and the value is "True" or "False." If the value is "True," deterioration correction has already been completed within the user device 10, so no special processing is required at the later stage. If the value is "False," the later stage may make a determination and deploy an appropriate correction algorithm to the user device 10.

[0271] The item "Setting value at time of deterioration" is an item for checking the current setting value in subsequent processing, and the value is various setting values ​​in the camera parameters 131. By referring to the value of this item, the current correction value applied to the user device 10 can be checked, and subsequent processing can also reconfirm whether the value is abnormal.

[0272] In the correction notification phase, the output data includes an item "correction success status." The item "correction success status" is an item for reporting the status of the correction to subsequent processing, and its value is one of "success," "timeout," and "not_accurate." Of these, the value "success" indicates that the correction was successful in the user device 10, and the server 30 may deploy the original model to the user device 10. The value "timeout" indicates that data for correction was not obtained within a specified time. For example, the server 30 may request the user device 10 to perform correction again. The value "not_accurate" indicates that the correction was performed in the user device 10 but was not accurate enough. For example, the server 30 may request the user device 10 to perform correction again.

[0273] The user device 10 may change the frequency of outputting the output data shown in FIG. 20 depending on the situation, such as "every frame," "every fixed period," or "only during business hours." For example, if the detected image degradation is of low importance, the user device 10 may output the information shown in FIG. 20 all at once at the end of the operating hours of the day. This reduces communication costs. Furthermore, by outputting the degradation notification at an appropriate frequency, the user device 10 can achieve a well-balanced configuration between communication costs and processing volume and image quality and recognition accuracy.

[0274] The effects described in this specification are merely examples and are not limiting, and other effects may also be present.

[0275] The present technology may also be configured as follows. (1) An imaging device comprising: an image processing unit that performs image processing on a captured image output by an imaging unit after capturing an image of a subject in accordance with parameters and outputs the image data; an inference unit that performs inference processing based on the image data using a machine learning model; and a degradation processing unit that detects degradation of the captured image using the image data and an inference result from the inference processing. (2) The imaging device described in (1), wherein the degradation processing unit updates the parameters based on a detection result of the degradation. (3) The imaging device described in (2), wherein the degradation processing unit calculates the parameters to be updated using an algorithm according to the detection result. (4) The imaging device described in (3), wherein the degradation processing unit transmits the detection result to a server and calculates the parameters to be updated using the algorithm transmitted from the server in accordance with the detection result. (5) The imaging device described in any one of (1) to (4), wherein the degradation processing unit detects degradation of the captured image based on a background image extracted from the image data based on a result of object detection by the inference processing of the inference unit. (6) The imaging device according to any one of (1) to (5), wherein the degradation processing unit detects degradation of the captured image based on a region of interest extracted from the image data based on a result of object detection by the inference processing of the inference unit. (7) The imaging device according to any one of (1) to (6), wherein the degradation processing unit detects objects of different categories from each other from the image data based on a result of object detection by the inference processing of the inference unit, and detects degradation of the captured image for each category. (8) The imaging device according to any one of (1) to (7), wherein the degradation processing unit detects degradation of the captured image based on first image data based on the captured image captured at a first time among the image data, and second image data based on the captured image captured at a second time earlier than the first time. (9) The imaging device according to (8), wherein the degradation processing unit detects a shift in angle of view in the captured image as degradation of the captured image based on a difference between the first image data and the second image data.(10) The imaging device according to (9), wherein, when the angle of view shift is detected, the inference unit detects feature points in the image data by the inference processing using an algorithm corresponding to the angle of view shift, and the degradation processing unit calculates the parameters based on the feature points and feature points detected by the inference unit from image data of the image data that is older than the image data used by the inference unit to detect the feature points. (11) The imaging device according to any one of (8) to (10), wherein the degradation processing unit detects color shift in the captured image as degradation of the captured image based on a difference between the first image data and the second image data. (12) The imaging device according to (11), wherein the inference unit extracts a foreground image from the image data by the inference processing, and the degradation processing unit detects the color shift in image data in which a region corresponding to the foreground image is masked from the image data. (13) An information processing method including: an image processing step executed by a processor, in which an image processing is performed on a captured image, which is output by an imaging unit after capturing an image of a subject, according to parameters, and the image processing is output as image data; an inference step, in which an inference process is performed based on the image data using a machine learning model; and a degradation processing step, in which degradation of the captured image is detected using the image data and an inference result from the inference process.(14) An information processing system including an imaging device and a server communicatively connected to the imaging device via a communication network, wherein the imaging device comprises: an imaging unit that captures an image of a subject and outputs the captured image; an image processing unit that performs image processing on the captured image in accordance with parameters and outputs the result as image data; an inference unit that performs inference processing based on the image data using a machine learning model; and a degradation processing unit that detects degradation of the captured image using the image data and an inference result of the inference processing; and the server comprises: a storage unit that stores an algorithm related to the processing by which the degradation processing unit detects degradation of the captured image; and a management unit that manages the algorithms stored in the storage unit, and the management unit selects the algorithm to be stored in the storage unit according to a detection result of detection of degradation of the captured image transmitted from the imaging device, and transmits the selected algorithm to the imaging device to deploy it. (15) The information processing system according to (14), wherein the management unit acquires from the imaging device, in accordance with the detection result, an algorithm used by the degradation processing unit to detect degradation of the captured image, and stores it, transmits the selected algorithm from the storage unit to the imaging device to deploy it, and in accordance with a notification of completion of processing by the algorithm transmitted from the imaging device, transmits the stored algorithm to the imaging device to deploy it. (16) The information processing system according to (14) or (15), wherein the storage unit further stores the machine learning model used by the inference unit to perform the inference processing, and the management unit further transmits the machine learning model stored in the storage unit to the imaging device to deploy it in accordance with the detection result. (17) The information processing system described in any one of (14) to (16), wherein the imaging device has license information unique to the imaging device regarding the use of the algorithm stored in the storage unit, and the server checks the license information of the imaging device in response to a detection result of the deterioration of the captured image being transmitted from the imaging device, and if the license information is valid, permits the imaging device to use the algorithm stored in the storage unit.

[0276] REFERENCE SIGNS LIST 1 Information processing system 2 Communication network 3 Cloud network 10, 101, 102 User device 11 Image processing unit 13 Task-specific DNN 15a, 15b Image deterioration detection unit 30 Server 100, 300 Overall control unit 110, 310 Communication unit 120 Image acquisition unit 131 Camera parameters 132, 1011, 1123 Memory 140, 140a, 140b, 140c Inference unit 150 Calibration processing unit 160a, 160b, 160c, 160d, 160e, 160f Degradation processing unit 170 Storage unit 180 Communication unit 320 Algorithm management unit 330 Algorithm storage unit 402 ROI image 1000 Imaging device 1010 Degradation processing CPU 1111 Imaging unit 1112 Image capture processing unit 1121, 3000 CPU 1122 DSP 1601, 1631 Background determination 1602, 1612, 1622, 1632, 1653 Index calculation 1603, 1613, 1623, 1633, 1654 Degradation determination 1611 ROI extraction 1621 Object classification 1641 Feature point matching 1642 Homography conversion 1651 Foreground mask 1652 Color measurement

Claims

1. An imaging device comprising: an image processing unit that performs image processing on a captured image output by an imaging unit after capturing an image of a subject in accordance with parameters and outputs the processed image as image data; an inference unit that performs inference processing based on the image data using a machine learning model; and a degradation processing unit that detects degradation of the captured image using the image data and the inference results of the inference processing.

2. The imaging device according to claim 1, wherein the degradation processing unit updates the parameters based on the detection result of the degradation.

3. The imaging device according to claim 2, wherein the degradation processing unit calculates the parameters to be updated using an algorithm according to the detection result.

4. The imaging device according to claim 3, wherein the degradation processing unit transmits the detection result to a server and calculates the parameters to be updated using the algorithm transmitted from the server in accordance with the detection result.

5. The imaging device according to claim 1, wherein the degradation processing unit detects degradation of the captured image based on a background image extracted from the image data based on the result of object detection by the inference processing of the inference unit.

6. The imaging device according to claim 1, wherein the degradation processing unit detects degradation of the captured image based on an area of ​​interest extracted from the image data based on the result of object detection by the inference processing of the inference unit.

7. The imaging device according to claim 1, wherein the degradation processing unit detects objects of different categories from the image data based on the results of object detection by the inference processing of the inference unit, and detects degradation of the captured image for each category.

8. The imaging device described in claim 1, wherein the degradation processing unit detects degradation of the captured image based on first image data based on the captured image captured at a first time among the image data, and second image data based on the captured image captured at a second time that is earlier than the first time.

9. The imaging device according to claim 8, wherein the degradation processing section detects a deviation in the angle of view in the captured image as degradation of the captured image based on a difference between the first image data and the second image data.

10. The imaging device described in claim 9, wherein, when the angle of view shift is detected, the inference unit detects feature points in the image data by the inference process using an algorithm corresponding to the angle of view shift, and the degradation processing unit calculates the parameters based on the feature points and feature points detected by the inference unit from image data of the image data that is older than the image data used by the inference unit to detect the feature points.

11. The imaging device according to claim 8, wherein the degradation processing unit detects color shift in the captured image as degradation of the captured image based on the difference between the first image data and the second image data.

12. The imaging device described in claim 11, wherein the inference unit extracts a foreground image from the image data by the inference processing, and the degradation processing unit detects the color shift in image data in which an area corresponding to the foreground image is masked from the image data.

13. An information processing method including: an image processing step executed by a processor, in which an image captured by an imaging unit captures an image of a subject and outputs the image as image data, the image processing step subjecting the captured image to image processing in accordance with parameters; an inference step performing inference processing based on the image data using a machine learning model; and a degradation processing step detecting degradation of the captured image using the image data and the inference results of the inference processing.

14. An information processing system including an imaging device and a server communicatively connected to the imaging device via a communication network, wherein the imaging device comprises: an imaging unit that captures an image of a subject and outputs the captured image; an image processing unit that performs image processing on the captured image in accordance with parameters and outputs the result as image data; an inference unit that performs inference processing based on the image data using a machine learning model; and a degradation processing unit that detects degradation of the captured image using the image data and the inference result of the inference processing; and the server comprises: a storage unit that stores an algorithm related to the processing by which the degradation processing unit detects degradation of the captured image; and a management unit that manages the algorithms stored in the storage unit, and the management unit selects the algorithm to be stored in the storage unit in accordance with the detection result of detection of degradation of the captured image transmitted from the imaging device, and transmits the selected algorithm to the imaging device to deploy it.

15. The information processing system described in claim 14, wherein the management unit acquires and stores an algorithm used by the degradation processing unit to detect degradation of the captured image from the imaging device in accordance with the detection result, transmits the selected algorithm from the storage unit to the imaging device to deploy it, and transmits the stored algorithm to the imaging device to deploy it in accordance with a notification from the imaging device that processing using the transmitted algorithm has been completed.

16. The information processing system of claim 14, wherein the storage unit further stores the machine learning model used by the inference unit to perform the inference processing, and the management unit further transmits the machine learning model stored in the storage unit to the imaging device and deploys it in accordance with the detection result.

17. The information processing system of claim 14, wherein the imaging device has license information unique to the imaging device regarding the use of the algorithm stored in the storage unit, and the server checks the license information of the imaging device in response to a detection result transmitted from the imaging device indicating that the captured image has deteriorated, and if the license information is valid, permits the imaging device to use the algorithm stored in the storage unit.

Images