Information processing device, information processing method, and information processing system

Abstract

This information processing device comprises: an inference unit that uses an AI model to perform inference processing on a captured image; a score calculation unit that calculates an inference uncertainty score on the basis of inference result information obtained by the inference processing; a determination processing unit that determines, on the basis of the uncertainty score, whether drift has occurred; and a transmission processing unit that, if the determination processing unit determines that drift has occurred, performs processing for transmitting the captured image, which is subjected to inference, to an external device.

Description

Information Processing Apparatus, Information Processing Method, and Information Processing System

[0001] The present technology relates to an information processing apparatus, an information processing method, and an information processing system, and particularly relates to the technical field of correcting drift of an AI (Artificial Intelligence) model.

[0002] An image recognition system that performs AI (Artificial Intelligence) processing related to recognition of image content such as object detection processing or object recognition processing on a captured image has been put into practical use. For an AI model that performs image recognition, as known as "drift", the inference performance of the AI model may deteriorate due to some unexpected change.

[0003] Conventionally, re-learning of an AI model for correcting drift has been performed as follows. That is, an imaging device that obtains an image to be inferred transmits the captured image to a server device that performs re-learning processing. The server device performs a drift check on the transmitted captured image by a method of change point detection or anomaly detection. When drift is detected, annotation work for generating correct data is performed on the captured images transmitted so far, and re-learning of the AI model is performed using the learning dataset obtained by this annotation work.

[0004] Note that the following Patent Document 1 can be cited as a related conventional technology. Patent Document 1 discloses a technique for collecting data used for re-learning in response to detection of drift.

[0005] U.S. Patent Application Publication No. 2023 / 0129390

[0006] However, according to the conventional re-learning method as described above, the imaging device needs to transmit all the captured images that are the inference targets to the server device, leading to an increase in data communication volume in realizing re-learning, an increase in the amount of data handled in re-learning, an increase in processing burden, and an increase in learning time.

[0007] This technology was developed in light of the above circumstances, and aims to reduce the amount of data communication required for retraining to correct drift, reduce the processing burden during retraining, and shorten the training time.

[0008] The information processing device according to this technology comprises: an inference unit that performs inference processing using an AI model on captured images; a score calculation unit that calculates an uncertainty score for the inference based on the inference result information from the inference processing; a determination processing unit that determines whether or not drift has occurred based on the uncertainty score; and a transmission processing unit that, when the determination processing unit determines that drift has occurred, performs processing to transmit the captured images that were the subject of inference to an external device. This makes it possible to transmit only the captured images that caused the drift, in other words, only the captured images necessary for retraining to correct the drift, to the external device.

[0009] This is an overview diagram of an information processing system configured with an information processing device as an embodiment. This is a block diagram showing an example configuration of an information processing device (imaging device) as the first embodiment. This is an explanatory diagram of NMS processing. This is a block diagram showing an example hardware configuration of a server device included in the information processing system as an embodiment. This is a functional block diagram for explaining the functions of the information processing device as the first embodiment. This is an explanatory diagram of an example of an uncertainty score calculation method as the first embodiment. This is a flowchart showing an example of a specific processing procedure to be executed in order to realize the drift detection method as the first embodiment. This is a flowchart of the uncertainty score calculation process in Figure 7. This is a flowchart showing the process for accumulating images for retraining. This is a flowchart showing the process for retraining the AI ​​model using the accumulated images. This is a block diagram showing an example of the internal configuration of an information processing device (imaging device) as the second embodiment. This is a functional block diagram for explaining the functions of the information processing device as the second embodiment. This is an explanatory diagram of a specific calculation example of the uncertainty score in the second embodiment. This is a flowchart showing an example of a specific processing procedure to be executed in order to realize the drift detection method as the second embodiment. This is a flowchart of the uncertainty score calculation process in Figure 14.

[0010] The embodiments of this technology will be described below in the following order, with reference to the attached drawings. <1. First Embodiment> (1-1. Overview of the Information Processing System as an Embodiment) (1-2. Example of Imaging Device Configuration) (1-3. Example of Server Device Configuration) (1-4. Drift Detection Method as the First Embodiment) (1-5. Processing Procedure) <2. Second Embodiment> <3. Modified Examples> <4. Summary of Embodiments> <5. This Technology>

[0011] <1. First Embodiment> (1-1. Overview of the Information Processing System as an Embodiment) Figure 1 is an overview diagram of an information processing system configured with an information processing device as an embodiment of the present technology. As shown in the figure, the information processing system of the embodiment comprises an imaging device 1, a server device 2, and a user terminal 3. In this information processing system, the imaging device 1 corresponds to one embodiment of the information processing device according to the present technology.

[0012] In this information processing system, the server device 2 and the user terminal 3 are configured as computer devices each equipped with a microcomputer having a CPU (Central Processing Unit), ROM (Read Only Memory), and RAM (Random Access Memory). In this example, the server device 2 is configured to enable data communication with the imaging device 1 and the user terminal 3 via a network NT, such as the Internet.

[0013] The imaging device 1 captures an image of a subject and obtains an image. Here, "imaging" in this specification broadly means obtaining image data that captures a subject. The image data referred to here is a general term for data consisting of multiple pixel data, and the pixel data is a broad concept that includes not only data indicating the amount of light received from the subject, but also data indicating, for example, the distance to the subject, the polarization information of the subject, and temperature information. In other words, the "image data" obtained by "imaging" (image data) includes data as a grayscale image that shows the amount of light received for each pixel, data as a distance image that shows the distance to the subject for each pixel, data as a polarization image that shows the polarization information of incident light for each pixel, and data as a thermal image that shows temperature information for each pixel. Furthermore, the "image data" also includes data as an event image obtained by an EVS (Event-based Vision Sensor) having an event sensor in which multiple event detection pixels that detect changes in the amount of light received as events are arranged in two dimensions. This event image data can be rephrased as image data that shows whether or not an event has occurred for each pixel, and can be expressed as image data that captures the movement of the subject.

[0014] As an example, the imaging device 1 in this example is configured to obtain the above-mentioned grayscale image as the captured image, similar to a typical digital camera. As shown in the figure, the information processing system in this example is provided with multiple imaging devices 1.

[0015] In the information processing system, server device 2 is a computer device intended to be used by the provider of services using the information processing system. User terminal 3 is a computer device intended to be used by the user who receives the service.

[0016] The information processing system in this example is configured to provide a service that performs inference processing using an AI (Artificial Intelligence) model on captured images obtained by the imaging device 1, generates analytical information showing the analysis results of the subject based on the inference results, and presents it to the user.

[0017] Here, assuming a service that presents the user with analysis information of the subject based on the inference results as described above, various applications for the imaging device 1 can be considered as surveillance cameras. For example, applications include surveillance cameras for indoors such as stores, offices, and residences; surveillance cameras for monitoring outdoors such as parking lots and streets (including traffic surveillance cameras, etc.); surveillance cameras for manufacturing lines in FA (Factory Automation) and IA (Industrial Automation); and surveillance cameras for monitoring inside and outside vehicles.

[0018] For example, in the case of surveillance cameras in a store, multiple imaging devices 1 could be placed at designated locations within the store, allowing users to observe the demographics of customers (such as gender and age group) and their behavior within the store (movement patterns). In this case, the analysis information could include data on customer demographics, movement patterns within the store, and congestion at checkout counters (e.g., waiting times at checkout counters). Alternatively, in the case of traffic surveillance cameras, each imaging device 1 could be placed at various locations near the road, allowing users to recognize information such as license plates, vehicle colors, and vehicle types of passing vehicles. In this case, the analysis information could include data on license plates, vehicle colors, and vehicle types.

[0019] Furthermore, if traffic surveillance cameras are used in a parking lot, the cameras should be positioned to monitor each parked vehicle, checking for suspicious individuals behaving inappropriately around each vehicle. If a suspicious person is detected, the system could notify users of their presence and their characteristics (gender, age group, clothing, etc.). Additionally, it is conceivable to monitor vacant spaces in the city and parking lots and notify users of available parking spaces.

[0020] In this example, the inference processing targeting the captured images is performed by the imaging device 1. Specifically, the imaging device 1 is equipped with an AI model for performing inference processing, and information indicating the results of the inference processing using the AI ​​model (hereinafter referred to as "inference result information") is transmitted to the server device 2. Based on the inference result information transmitted from the imaging device 1 in this manner, the server device 2 performs the various analysis processes described above.

[0021] By employing this method in which the imaging device 1 performs inference processing and transmits the inference result information to the server device 2, the amount of communication data required to perform inference processing can be significantly reduced compared to the case in which the server device 2 performs inference processing on captured images transmitted from the imaging device 1. Furthermore, since it is no longer necessary to transmit captured images from the imaging device 1 to the server device 2, it is possible to prevent the leakage of captured images containing personal information to external parties, thereby protecting privacy.

[0022] In this example, the AI ​​model used by the imaging device 1 is transmitted from the server device 2 to the imaging device 1. In this example, the imaging device 1 is owned by the user, and the user can transmit the AI ​​model from the server device 2 to the imaging device 1 by purchasing the right to use the AI ​​model through payment of a fee to the service provider.

[0023] It should be noted that this premise is merely an example for illustrative purposes, and the AI ​​model may be one that is pre-configured in the imaging device 1.

[0024] Furthermore, in the information processing system of this example, the server device 2 also performs retraining processing for the AI ​​model used by the imaging device 1. The retraining processing is performed using images captured by the imaging device 1, that is, actual images captured at the site where the subject to inference exists, as training input images. The retraining processing as an embodiment will be explained in more detail later.

[0025] Although Figure 1 shows an example where the information processing system is equipped with multiple imaging devices 1, the number of imaging devices 1 only needs to be at least one. Also, although Figure 1 shows that there is only one user terminal 3 in the information processing system, there may be multiple user terminals 3. In other words, it is conceivable that there may be multiple users receiving services from the inference system.

[0026] (1-2. Example of Imaging Device Configuration) Figure 2 is a block diagram showing an example of the configuration of the imaging device 1. As shown in the figure, the imaging device 1 includes an image sensor 10, an imaging optical system 11, an optical system drive unit 12, a camera control unit 13, a memory unit 14, and a communication unit 15. The image sensor 10, the camera control unit 13, the memory unit 14, and the communication unit 15 are connected via a bus 16 and are capable of communicating data with each other.

[0027] In this example, the image sensor 10 is configured as a grayscale image sensor that obtains the aforementioned grayscale image. Specifically, the image sensor 10 is configured as a solid-state image sensor such as a CCD (Charge Coupled Device) type or a CMOS (Complementary Metal Oxide Semiconductor) type.

[0028] The imaging optical system 11 includes lenses such as a cover lens, zoom lens, and focus lens, as well as an aperture (iris) mechanism. This imaging optical system 11 guides light (incident light) from the subject and focuses it onto the light-receiving surface (imaging surface) of the image sensor 10.

[0029] The optical system drive unit 12 comprehensively represents the drive units for the zoom lens, focus lens, and aperture mechanism of the imaging optical system 11. Specifically, the optical system drive unit 12 includes actuators for driving the zoom lens, focus lens, and aperture mechanism, and drive circuits for said actuators.

[0030] The camera control unit 13 is configured with, for example, a microcomputer having a CPU, ROM, and RAM, and performs overall control of the imaging device 1 by having the CPU execute various processes according to a program stored in the ROM or a program loaded into the RAM.

[0031] Furthermore, the camera control unit 13 issues drive instructions to the optical system drive unit 12 for the zoom lens, focus lens, aperture mechanism, etc. In response to these drive instructions, the optical system drive unit 12 will perform actions such as moving the focus lens and zoom lens, and opening and closing the aperture blades of the aperture mechanism.

[0032] Furthermore, the camera control unit 13 controls the writing and reading of various data to and from the memory unit 14. The memory unit 14 is a non-volatile storage device such as an HDD (Hard Disk Drive) or flash memory device, and is used to store data used by the camera control unit 13 when performing various processes. The memory unit 14 can also be used as a storage location (recording location) for image data output from the image sensor 10.

[0033] The camera control unit 13 performs various data communications with external devices via the communication unit 15. In this example, the communication unit 15 is configured to enable communication via the network NT shown in Figure 1, and to perform data communications with external devices connected to the network NT, particularly in this example, at least with the server device 2.

[0034] As shown in the figure, the image sensor 10 comprises an imaging unit 41, an image signal processing unit 42, an internal sensor control unit 43, an AI processing unit 44, a memory unit 45, a communication interface (I / F) 46, a pre-processing unit 47, and a post-processing unit 48, each connected via a bus 50 to enable data communication between them.

[0035] The imaging unit 41 includes a pixel array unit in which pixels having photoelectric conversion elements (light-receiving elements) such as photodiodes are arranged in two dimensions, and a readout circuit that reads out electrical signals (received signals) obtained by photoelectric conversion from each pixel of the pixel array unit. In this readout circuit, the electrical signals obtained by photoelectric conversion are subjected to processes such as CDS (Correlated Double Sampling) processing and AGC (Automatic Gain Control) processing, and further A / D (Analog to Digital) conversion processing.

[0036] The image signal processing unit 42 performs preprocessing, syncing, YC generation, and codec processing on the captured image signal as digital data after A / D conversion. Preprocessing includes clamping, which clamps the black levels of R (red), G (green), and B (blue) to predetermined levels, and correction processing between R, G, and B color channels. Preprocessing can also include brightness adjustment processing such as gamma correction and color adjustment processing such as white balance adjustment. In syncing processing, color separation processing is performed so that the image data for each pixel has all R, G, and B color components. For example, in the case of an image sensor using a Bayer array color filter, demosaicing is performed as the color separation processing. In YC generation processing, luminance (Y) signals and color (C) signals are generated (separated) from the R, G, and B image data. In codec processing, the image data that has undergone the above various processing is subjected to encoding processing for recording or communication, and file generation, for example. Codec processing allows for the creation of video files in formats such as MPEG-2 (MPEG: Moving Picture Experts Group) and H.264. It also allows for the creation of still image files in formats such as JPEG (Joint Photographic Experts Group), TIFF (Tagged Image File Format), and GIF (Graphics Interchange Format).

[0037] The sensor control unit 43 is configured with a microcomputer, for example, which includes a CPU, ROM, RAM, etc., and comprehensively controls the operation of the image sensor 10. For example, the sensor control unit 43 issues instructions to the imaging unit 41 to control the execution of imaging operations. It also controls the execution of processing to the image signal processing unit 42.

[0038] The AI ​​processing unit 44 is configured with a programmable computing device such as a DSP (Digital Signal Processor) or FPGA (Field Programmable Gate Array), and performs inference processing (AI processing) using an AI model on captured images.

[0039] In this example, the AI ​​model used by the AI ​​processing unit 44 is an image recognition AI model that performs image recognition processing. Image recognition processing here refers to the process of recognizing the content of an image. Examples of image recognition processing include object detection processing, which detects the region in which an object exists; object recognition processing, which recognizes what kind of object is depicted in the image; semantic segmentation processing; and regression processing, which estimates the size of a subject. Object detection processing here includes not only detecting the region in which an object exists, but also recognizing what kind of object it is, such as YOLO (You Only Look Once) or SSD (Single Shot Multibox Detector).

[0040] The memory unit 45 is used to hold (temporarily store) data necessary for AI processing by the AI ​​processing unit 44. Specifically, the memory unit 45 is used to store AI model data, which is the necessary parameter data for constructing an AI model (image recognition AI model) in the AI ​​processing unit 44. If the AI ​​model has a neural network such as a CNN (Convolutional Neural Network), this AI model data corresponds to parameters that indicate the structure of the neural network, or parameters that serve as filter coefficients used in convolution processing, etc.

[0041] Here, for convenience of explanation, such AI model data is treated as equivalent to an AI model hereinafter. As shown in the figure, the memory unit 45 stores the main AI model Mm and the detection AI model Md. The AI processing unit 44 in the present embodiment is capable of executing, as inference processing, inference processing using the main AI model Mm and inference processing using the detection AI model Md as inference processing. The main AI model Mm is an AI model for obtaining inference result information used in the above-described analysis processing. In other words, it is an AI model to which the inference result information is transmitted to the server device 2. The detection AI model Md is an AI model prepared for detecting the drift of the main AI model. Details of the detection AI model Md will be described later.

[0042] In this example, the memory unit 45 is also used to hold the captured image that is the input data to the AI processing unit 44.

[0043] The communication interface 46 is an interface that communicates with each unit connected via the bus 16, such as the camera control unit 13 and the memory unit 14 outside the image sensor 10. For example, the communication interface 46 communicates to obtain an AI model used by the AI processing unit 44 from the outside based on the control of the in-sensor control unit 43. Also, it is possible to output the result information (inference result information) of the inference processing by the AI processing unit 44 to the outside of the image sensor 10 via the communication interface 46.

[0044] The preprocessing unit 47 performs predetermined image signal processing as preprocessing on the image input to the AI model used by the AI processing unit 44. As this preprocessing, for example, image format conversion processing for adapting to the input tensor of the AI model, such as normalization processing of luminance values and resizing processing to the input tensor size of the AI model, is performed.

[0045] The post-processing unit 48 performs predetermined signal processing as post-processing on the output of the AI model in the AI processing unit 44. The content of this post-processing may vary depending on the content of the inference task of the AI model. For example, when the AI model performs object detection processing, the post-processing unit 48 performs processing to delete extra bounding boxes BB for the bounding boxes BB obtained as the output of the AI model. Specifically, processing as NMS (Non-Maximum Suppression) is performed.

[0046] As is well known, in an AI model that performs object detection processing, it does not predict (prediction) one bounding box BB for one object from the beginning. Instead, as illustrated in FIG. 3, it predicts a plurality of bounding boxes BB for one object. At this time, the AI model that performs object detection processing outputs score information for each of those plurality of bounding boxes BB. The score mentioned here is a value indicating the probability that an object exists within the bounding box BB.

[0047] In the NMS process, based on the value of this score, processing is performed to remove other bounding boxes BB so that one bounding box BB remains for one object. Note that since NMS is a known technique, detailed description of its content is omitted.

[0048] Regarding the post-processing unit 48, when the AI model performs classification processing or regression processing of the subject class, post-processing such as processing to delete extra bounding boxes BB like NMS is not performed.

[0049] (1-3. Configuration Example of Server Device) FIG. 4 is a block diagram showing a hardware configuration example of the server device 2. Note that it is conceivable that the computer device as the user terminal 3 shown in FIG. 1 also adopts a hardware configuration similar to that shown in FIG. 4.

[0050] As shown in the figure, the server device 2 is equipped with a CPU 21. The CPU 21 executes various processes according to the program stored in the ROM 22 or the program loaded from the storage unit 29 into the RAM 23. The RAM 23 also appropriately stores data necessary for the CPU 21 to execute various processes.

[0051] The CPU 21, ROM 22, and RAM 23 are interconnected via a bus 24. An input / output interface (I / F) 25 is also connected to this bus 24.

[0052] An input unit 26, consisting of operators or operating devices, is connected to the input / output interface 25. For example, the input unit 26 can be various operators or operating devices such as a keyboard, mouse, keys, dial, touch panel, touchpad, or remote controller. User operations are detected by the input unit 26, and the signals corresponding to the input operations are interpreted by the CPU 21.

[0053] Furthermore, a display unit 27, consisting of an LCD (Liquid Crystal Display) or an organic EL (Electro-Luminescence) panel, and an audio output unit 28, consisting of a speaker, are connected to the input / output interface 25, either as an integrated unit or as separate components. The display unit 27 is used for displaying various types of information and is composed of, for example, a display device provided on the casing of a computer device or a separate display device connected to a computer device.

[0054] The display unit 27 displays images for various image processing tasks and videos to be processed on the display screen based on instructions from the CPU 21. The display unit 27 also displays various operation menus, icons, messages, etc., i.e., GUI (Graphical User Interface), based on instructions from the CPU 21.

[0055] The input / output interface 25 may also be connected to a storage unit 29, which consists of an HDD or solid memory, or a communication unit 30, which consists of a modem or the like.

[0056] The communication unit 30 performs communication processing via transmission lines such as the Internet, and communicates with various devices via wired / wireless communication, bus communication, etc.

[0057] A drive 31 is also connected to the input / output interface 25 as needed, and a removable recording medium 32 such as a magnetic disk, optical disk, magneto-optical disk, or semiconductor memory is appropriately mounted there.

[0058] The drive 31 can read data files such as programs used for various processes from the removable recording medium 32. The read data files are stored in the storage unit 29, or images and sounds contained in the data files are output by the display unit 27 and the sound output unit 28. Computer programs and other data read from the removable recording medium 32 are installed in the storage unit 29 as needed.

[0059] In the server device 2 having the hardware configuration described above, for example, the software for processing in this embodiment can be installed via network communication by the communication unit 30 or via the removable recording medium 32. Alternatively, the software may be stored in advance in the ROM 22 or storage unit 29, etc. The CPU 21 performs processing operations based on various programs, thereby executing the necessary information processing and communication processing for the server device 2.

[0060] Furthermore, server device 2 is not limited to being a single computer device as shown in Figure 4, but may be configured as a system of multiple computer devices. These multiple computer devices may be systematized via a LAN (Local Area Network), or they may be located remotely via a VPN (Virtual Private Network) using the Internet, etc. The multiple computer devices may also include computer devices that function as a group of servers (cloud) available through cloud computing services.

[0061] (1-4. Drift Detection Method as a First Embodiment) As described above, in the information processing system of this embodiment, the server device 2 performs retraining on the AI ​​model used by the imaging device 1, and this retraining process is executed in response to the detection of drift in the AI ​​model. Here, drift refers to a deterioration in the inference performance of the AI ​​model due to some unexpected change.

[0062] In this embodiment, drift detection of the AI ​​model is performed on the imaging device 1 side, not the server device 2. Specifically, in this example, drift detection is performed by a sensor control unit 43 located within the image sensor 10 in the imaging device 1.

[0063] Figure 5 is a functional block diagram illustrating the functions of the sensor internal control unit 43 as an embodiment. As shown in the figure, the sensor internal control unit 43 includes a score calculation unit F1, a determination processing unit F2, and a transmission processing unit F3. The score calculation unit F1 calculates an uncertainty score for the inference based on the inference result information obtained by the inference processing of the AI ​​processing unit 44.

[0064] The determination processing unit F2 determines whether drift has occurred based on the uncertainty score calculated by the score calculation unit F1. Furthermore, if the determination processing unit F2 determines that drift has occurred, the transmission processing unit F3 transmits the captured image targeted for inference to an external device. Specifically, it transmits the captured image targeted for inference to the server device 2. In this example, data communication between the sensor internal control unit 43 and the server device 2 is performed via the communication interface 46 and the communication unit 15.

[0065] By having the above-described score calculation unit F1, judgment processing unit F2, and transmission processing unit F3 configuration, it becomes possible to transmit only the captured images that caused the drift, in other words, only the captured images necessary for relearning to correct the drift, to an external device. Therefore, it is possible to reduce the amount of data communication required for relearning to correct the drift, and to reduce the processing burden and shorten the learning time by reducing the amount of data handled in relearning.

[0066] A specific example of the uncertainty score calculated by the score calculation unit F1 will be explained. Here, the AI ​​model used by the AI ​​processing unit 44 is assumed to be an AI model that performs object detection processing. Specifically, here, the object detection processing is assumed to be the detection of the vehicle number written on the license plate of a vehicle.

[0067] As mentioned earlier, within the image sensor 10, the AI ​​processing unit 44 has two AI models available for use: a main AI model Mm and a detection AI model Md. In this example, using the same captured image, inference processing is performed using multiple AI models, namely the main AI model Mm and the detection AI model Md, and an inference uncertainty score is calculated based on the inference result information obtained from each AI model.

[0068] The detection AI model Md is prepared to perform inference processing using the same inference task as the main AI model Mm (in this example, object detection processing with the vehicle number as the target class). However, the detection AI model Md is prepared to have at least some of the parameters used for AI processing different from those of the main AI model Mm. This makes it possible to create a difference in the uncertainty scores calculated between the AI ​​models, and improve the accuracy of the uncertainty score at the captured image level, which is calculated based on the uncertainty scores calculated for each AI model.

[0069] The following explanation illustrates the case where multiple AI models are used as the detection AI model Md. When using multiple detection AI models Md, at least some parameters should be different among the detection AI models Md. Increasing the number of AI models used to calculate the uncertainty score allows for a more multifaceted evaluation of the uncertainty of the inference, thereby improving the accuracy of the uncertainty score. In the following, multiple detection AI models Md will be distinguished by adding an underscore "_" and a numerical value to the end of their names.

[0070] Figure 6 is an explanatory diagram of an example of an uncertainty score calculation method as a first embodiment. First, in the imaging device 1 as a first embodiment, the AI ​​processing unit 44 performs inference processing using the main AI model Mm and multiple detection AI models Md (Md_1 to Md_n) on the same captured image. In this case, the AI ​​processing unit 44 is equipped with multiple hardware devices (n+1 in this example), such as DSPs, for performing inference processing using the AI ​​models, and it is conceivable to perform inference processing using each AI model simultaneously. Alternatively, it is conceivable to use fewer than n+1 hardware devices and perform some of the inference processing using time-sharing.

[0071] In this case, depending on the AI ​​model, multiple bounding boxes BB may be detected, and a score (likelihood) is calculated for each bounding box BB. This information, along with the bounding boxes BB and their scores, is then input to the post-processing unit 48. Here, the scores calculated by each AI model are calculated for all classes of the target object. Specifically, in this example where vehicle numbers are the target class for object detection, each AI model calculates a score indicating the likelihood for each value from 0 to 9. Hereafter, the likelihood score calculated for each class in this manner will be referred to as "score p".

[0072] In the post-processing by the post-processing unit 48, for example, the aforementioned NMS processing selects only one bounding box BB for each target object. Here, since the vehicle number is a four-digit number, the number of bounding boxes BB after post-processing for each AI model is narrowed down to four, as illustrated in the figure.

[0073] The score calculation unit F1 in this example calculates an uncertainty score based on the inference result information of each AI model after post-processing by the post-processing unit 48. Specifically, the inference result information after post-processing for each AI model includes information on the four bounding boxes BB and information on the class-specific score p calculated for each bounding box BB.

[0074] In this example, the score calculation unit F1 calculates an uncertainty score Um based on the score p information for each bounding box BB when multiple AI models each detect multiple objects. It also groups bounding boxes BB that are estimated to have detected the same object, and calculates the variance of the uncertainty scores Um among the AI ​​models for each group to obtain a group-level uncertainty score Ug. Furthermore, the score calculation unit F1 calculates an image-level uncertainty score Uu based on the group-level uncertainty score Ug.

[0075] In this example, the judgment processing unit F2 determines whether or not drift has occurred based on such an image-level uncertainty score Uu.

[0076] Regarding the processing of the score calculation unit F1 described above, the uncertainty score Um for each bounding box BB is calculated for each AI model using the score p for the predicted class (i.e., the highest score p among the scores p calculated for each class) according to the following [Equation 1]: Um = 1 - p ... [Equation 1]

[0077] Furthermore, the grouping of bounding boxes BB that are presumed to have detected the same object will be based on each bounding box BB detected by the main AI model Mm. That is, for each bounding box BB detected by the main AI model Mm, the grouping will be performed by selecting the bounding box BB that is closest in position from among the bounding box BBs detected by the detection AI model Md. In this case, since each AI model detects four bounding boxes BB, four groups will be formed.

[0078] Here, regarding grouping, we have given an example where the bounding box BB of the detection AI model Md is narrowed down to one bounding box BB per object, as the bounding box BB after post-processing. However, it is also possible to use the bounding box BB before post-processing for the bounding box BB of the detection AI model Md used for grouping. In this case, grouping can be performed based on IoU (Intersection over Union), using each bounding box BB detected by the main AI model Mm as the basis. Specifically, for each bounding box BB detected by the main AI model Mm, the bounding box BB with the highest IoU value among the bounding box BBs detected by each detection AI model Md is selected as the bounding box BB for the same group.

[0079] Furthermore, when using bounding boxes BB before post-processing for grouping, the number of bounding boxes BB of the detection AI model Md selected for each group may be more than one. In that case, as a selection based on IoU as described above, it is conceivable to select bounding boxes BB with an IoU value of or greater than a predetermined value (for example, 0.5 or greater) as the bounding boxes BB of the same group.

[0080] The score calculation unit F1 performs the above grouping and then calculates the variance of the uncertainty score Um between AI models for each group to obtain the group-level uncertainty score Ug. Here, the group-level uncertainty score Ug is calculated for each group according to the following [Equation 2], where M is the number of AI models used in the calculation (1 + n in this example).

[0081] Furthermore, the score calculation unit F1 calculates an image-level uncertainty score Uu based on the group-level uncertainty score Ug. The image-level uncertainty score Uu only needs to be calculated as a value that reflects the uncertainty score Ug calculated for each group. For example, the image-level uncertainty score Uu could be calculated as the sum or average of all uncertainty scores Ug. Alternatively, it could be calculated as the average of the top predetermined number of uncertainty scores Ug with the largest values ​​(for example, up to the 3rd or 5th place).

[0082] Here, we have given an example of calculating the uncertainty score Um for each bounding box BB of each AI model using [Equation 1], but it is also possible to calculate the uncertainty score Um as an entropy-based score. When using an entropy-based score, the score p for each class to be predicted is used in the score calculation. Specifically, in this case, the uncertainty score Um is calculated using [Equation 3] below, where C is the total number of classes to be predicted.

[0083] The entropy of the score p for each class is a suitable indicator for estimating the degree of uncertainty in inference. Therefore, the uncertainty score can be calculated appropriately, and the presence or absence of drift can be appropriately determined.

[0084] Here, as described above, the transmission processing unit F3 transmits the captured image to the server device 2 when the determination processing unit F2 determines that drift has occurred. In this example, the transmission processing unit F3 performs a process to send notification information to the server device 2 informing it of the occurrence of drift (hereinafter referred to as "drift notification process") in response to the determination processing unit F2 determining that drift has occurred. In this example, as part of the drift notification process, the transmission processing unit F3 transmits the identification information of its own device along with the information notifying it of the occurrence of drift.

[0085] Upon receiving notification through the drift notification process described above, the server device 2 stores the captured image transmitted from the imaging device 1 as data for relearning the main AI model Mm currently being used by the transmitting imaging device 1 (i.e., the AI ​​model transmitted by the server device 2 to the imaging device 1 in this example). At this time, the server device 2 manages the correspondence between the imaging device 1 and the main AI model Mm being used by the imaging device 1 based on the identification information of the imaging device 1. Therefore, as described above, the identification information of the imaging device 1 is notified through the drift notification process, allowing the server device 2 to identify the AI ​​model to be relearned.

[0086] When the number of captured images transmitted from the imaging device 1 reaches a predetermined number, the server device 2 notifies the operator involved in the retraining process. Upon receiving this notification, the operator performs annotation work on the stored captured images and generates a training dataset for retraining.

[0087] Server device 2 uses the training dataset generated by the annotation process described above to retrain the AI ​​model. Once the retraining process is complete, server device 2 transmits the retrained AI model to the imaging device 1.

[0088] As a result, when drift is detected in the imaging device 1, the server device 2 performs a retraining process for the main AI model Mm used by the imaging device 1, and the main AI model Mm used by the imaging device 1 is replaced with the retrained AI model. Therefore, the drift condition is resolved.

[0089] (1-5. Processing Procedure) Referring to the flowcharts in Figures 7 and 8, specific examples of processing procedures that the sensor control unit 43 should perform to realize the drift detection method as the first embodiment described above will be explained. The main entity executing the processing in Figures 7 and 8 is the CPU of the sensor control unit 43, and the CPU performs the processing shown in Figures 7 and 8 based on a program stored in a predetermined storage device such as the ROM of the sensor control unit 43. In the following explanation, the main entity executing the processing in Figures 7 and 8 will be the sensor control unit 43. Here, the sensor control unit 43 repeatedly performs the processing shown in Figure 7 for each frame of the captured image.

[0090] In Figure 7, the sensor control unit 43 waits in step S101 until the inference processing of each AI model is completed. That is, it waits until all of the main AI model Mm and the detection AI model Md have completed the inference processing for the captured image for one frame that was to be processed.

[0091] When the inference process for each AI model is completed, the sensor-internal control unit 43 performs the uncertainty score calculation process in step S102.

[0092] Figure 8 is a flowchart of the uncertainty score calculation process in step S102. First, in step S111, the sensor control unit 43 calculates the uncertainty score Um for each bounding box BB of each AI model. That is, based on the information of the bounding box BB after post-processing (after NMS processing) of each AI model and the information of its score p, the uncertainty score Um for each bounding box BB is calculated for each AI model using the above [Equation 1].

[0093] In step S111, it is also possible to calculate the uncertainty score Um based on the entropy-based score explained above with reference to [Equation 3].

[0094] In step S112, following step S111, the sensor control unit 43 performs a grouping process of bounding boxes BB between AI models. That is, as described above, the bounding boxes BB detected by the detection AI model Md are grouped based on each bounding box BB detected by the main AI model Mm. Note that the details of the grouping have already been explained, so a redundant explanation will be avoided.

[0095] In step S113, following step S112, the sensor control unit 43 calculates a group-level uncertainty score Ug based on the uncertainty score Um. Specifically in this example, the group-level uncertainty score Ug is calculated using the above-mentioned [Equation 2].

[0096] In step S114, following step S113, the sensor control unit 43 calculates an image-level uncertainty score Uu based on the uncertainty score Ug. As mentioned above, the image-level uncertainty score Uu is calculated as a value that reflects the uncertainty scores Ug calculated for each group, for example, by calculating it as the sum or average of all uncertainty scores Ug.

[0097] The sensor's internal control unit 43 completes the uncertainty score calculation process in step S102, in accordance with the calculation of the image-level uncertainty score Uu in step S114.

[0098] In Figure 7, the sensor control unit 43 proceeds to step S103 in response to having performed the calculation process in step S102, and determines whether the uncertainty score is equal to or greater than a predetermined threshold THu. That is, it determines whether the image-level uncertainty score Uu calculated in step S102 is equal to or greater than the threshold THu.

[0099] In step S103, if it is determined that the uncertainty score Uu is greater than or equal to the threshold THu (i.e., if drift is determined to have occurred), the sensor's internal control unit 43 proceeds to step S104, performs drift notification processing to the server device 2, and then in step S105, transmits the captured image that was the subject of inference to the server device 2, thus completing the series of processes shown in Figure 7. As explained earlier, in the drift notification processing in this example, the device sends its own identification information to the server device 2 along with information notifying the occurrence of drift.

[0100] On the other hand, if in step S103 it is determined that the uncertainty score Uu is not equal to or greater than the threshold THu, the sensor internal control unit 43 completes the series of processes shown in Figure 7.

[0101] Figures 9 and 10 are flowcharts showing the processing on the server device 2 side. Figure 9 shows the process for accumulating images for retraining, and Figure 10 shows the process for retraining the AI ​​model using the accumulated images. Although the processing in Figures 9 and 10 is executed by the CPU 21 of the server device 2 based on a program stored in a predetermined storage device such as ROM 22 or memory unit 29, for convenience, the server device 2 is referred to as the entity executing the processing in Figures 9 and 10.

[0102] In Figure 9, the server device 2 determines in step S201 whether or not a drift notification has been received. That is, it determines whether or not the drift notification information transmitted by the drift notification process in step S104, as described earlier, has been received. If it is determined in step S201 that there was no drift notification, the server device 2 determines in step S202 whether or not the process has ended. That is, it determines whether or not a predetermined condition has been met for the process shown in Figure 9 to be terminated, such as when an operation input instructing the termination of the process in Figure 9 has been made.

[0103] If it is determined in step S202 that processing has not yet ended, the server device 2 returns to step S101. This causes the server device 2 to wait for either a drift notification or processing completion.

[0104] If it is determined in step S201 that a drift notification has been received, the server device 2 proceeds to step S203 and performs the process of storing the captured image. That is, it stores the captured image that was sent to the server device 2 in the process of step S105 along with the drift notification in a predetermined storage device such as the storage unit 29.

[0105] In step S204, following step S203, the server device 2 determines whether the number of images required for relearning has been accumulated. If it determines that the number of images required for relearning has been accumulated, it proceeds to step S205 to perform a readiness completion notification process. This readiness completion notification process in step S205 is a process that notifies the operator that the number of captured images required for relearning has been accumulated. For example, this can be done by displaying predetermined information on the display unit 27.

[0106] The server device 2 completes the series of processes shown in Figure 9 in response to executing the preparation completion notification process in step S205.

[0107] Furthermore, if the server device 2 determines that processing is complete in step S202, it also completes the series of processes shown in Figure 9.

[0108] Upon receiving the notification in step S205, the worker performs annotation work on the captured images stored in the server device 2 and generates a training dataset for retraining.

[0109] The process shown in Figure 10 is performed after the training dataset for retraining has been generated through this annotation process.

[0110] In Figure 10, the server device 2 waits for a retraining instruction operation in step S211. That is, it waits for an input operation to instruct the execution of retraining.

[0111] If a retraining instruction is received, the server device 2 proceeds to step S212 and executes the retraining process for the corresponding AI model. Specifically, it performs the retraining process using the training dataset generated by the annotation work described above, targeting the AI ​​model identified from the identification information notified by the imaging device 1.

[0112] In step S213, following step S212, the server device 2 performs a process to transmit the retrained AI model to the corresponding imaging device 1. That is, it performs a process to transmit the AI ​​model, which has been retrained by the retraining process in step S212, to the imaging device 1 identified by the above identification information.

[0113] The server device 2 completes the series of processes shown in Figure 10 in accordance with the execution of the process in step S213.

[0114] Here, we explained that the AI ​​model is retrained in response to the detection of drift based on the uncertainty score Uu. However, it is also possible to calibrate the main AI model Mm in response to the detection of drift based on the uncertainty score Uu. Specifically, this could involve adjusting the class-specific score p calculated by the main AI model Mm in a way that lowers its value, or adjusting the judgment threshold for the score p (the threshold used to determine whether or not to accept a prediction result) in a way that raises its value.

[0115] The occurrence of drift suggests that an object unknown to the AI ​​model may have been captured in the image. Since the AI ​​model is configured to derive prediction results based on a score p, there is a risk that it may incorrectly assign the unknown object to some class. By performing the calibration described above, it becomes possible to make it less likely for the score p calculated by the AI ​​model to exceed the judgment threshold when an unknown object is captured in the image, thereby preventing incorrect answers that assign the unknown object to an existing class of object.

[0116] Furthermore, while the above describes an example of calculating the uncertainty score Uu when the AI ​​model is an AI model that performs object detection processing, the uncertainty score Uu in this embodiment can also be calculated when the AI ​​model is an AI model that performs classification processing or an AI model that performs regression processing.

[0117] When the AI ​​model performs classification processing, the score calculation unit F1 calculates an image-level uncertainty score Uu based on the likelihood information for each AI model. Specifically, first, for each AI model, an uncertainty score is calculated using the score p for the predicted class. For example, the uncertainty score is calculated for each AI model using the above [Equation 1]. Alternatively, the uncertainty score for each AI model can be calculated based on entropy. In that case, the score calculation is performed using the above [Equation 2] with the scores p of all classes to be predicted. Then, the score calculation unit F1 in this case calculates an image-level uncertainty score Uu based on the uncertainty score calculated for each AI model as described above. For example, using a similar approach to the above [Equation 2], it is conceivable to use a value based on the variance of the uncertainty score calculated for each AI model as the image-level uncertainty score Uu, such as the mean square of the variances of the uncertainty scores calculated for each AI model.

[0118] Furthermore, if the AI ​​model performs regression processing, the score calculation unit F1 calculates an image-level uncertainty score Uu based on the variance of the inference values ​​for each AI model. For example, it is conceivable to calculate the image-level uncertainty score as the mean square of the variances of the inference values ​​for each AI model.

[0119] <2. Second Embodiment> The second embodiment calculates the uncertainty score using only one AI model. Figure 11 is a block diagram showing an example of the internal configuration of the imaging device 1A as the second embodiment. In the following description, parts that are the same as those already described will be denoted by the same reference numerals and their description will be omitted.

[0120] The imaging device 1A differs from the imaging device 1 in that it is equipped with an image sensor 10A instead of the image sensor 10. The image sensor 10A differs from the image sensor 10 in that only the AI ​​model Ms is stored in the memory unit 45, and that it is equipped with an in-sensor control unit 43A instead of the in-sensor control unit 43. The AI ​​model Ms corresponds to an AI model for obtaining inference result information used in analysis processing, similar to the main AI model Mm in the first embodiment.

[0121] Here, the method of calculating the uncertainty score using only one AI model is applicable when the AI ​​model performs object detection processing, but is difficult to apply when the AI ​​model performs classification processing or regression processing. For this reason, in the second embodiment, the AI ​​model Ms is assumed to perform object detection processing (for example, in this case, object detection processing with vehicle numbers as the target class).

[0122] Figure 12 is a functional block diagram illustrating the functions of the sensor internal control unit 43A in the second embodiment. The sensor internal control unit 43A differs from the sensor internal control unit 43 (see Figure 5) in that it has a score calculation unit F1A instead of the score calculation unit F1.

[0123] The score calculation unit F1A calculates an image-level uncertainty score Uu based on the information of each bounding box BB obtained as the inference result information of the AI ​​model Ms, and the score p for each bounding box BB.

[0124] Figure 13 is an explanatory diagram illustrating a specific example of uncertainty score calculation in the second embodiment. In this case as well, the inference result information after post-processing by the post-processing unit 48 is used to calculate the uncertainty score. That is, in the case of a four-digit vehicle number as in this example, only the bounding box BB (four in total) remaining for each digit and the score p calculated for each of those bounding boxes BB (in this example, only the score p for the class that was considered the prediction result) are used.

[0125] Specifically, the score calculation unit F1A calculates an uncertainty score Uc for each predicted class based on the inference result information after the post-processing described above. For example, in the case of vehicle numbers, if the four digits are all different values ​​(classes), then four values ​​will be calculated as the uncertainty score Uc for each predicted class. Specifically, the uncertainty score Uc is calculated for each predicted class using the score p of the predicted class, by the formula "Uc = 1 - p".

[0126] The score calculation unit F1A then calculates an image-level uncertainty score Uu based on the uncertainty score Uc. In this case, the image-level uncertainty score Uu only needs to be calculated as a value that reflects the uncertainty score Uc calculated for each class. For example, it could be calculated as the sum or average of all uncertainty scores Uc, or as the average of the top predetermined number of uncertainty scores Uc with the largest values ​​(e.g., up to the 3rd or 5th place).

[0127] Figure 14 is a flowchart showing a specific example of processing steps that the sensor control unit 43A should perform to realize the drift detection method as a second embodiment. The main entity executing the processing in Figure 14 is the CPU of the sensor control unit 43A, which executes the processing shown in Figure 14 based on a program stored in a predetermined storage device, such as the ROM of the sensor control unit 43A. In the following description, the main entity executing the processing in Figure 14 will be referred to as the sensor control unit 43A. Here, the sensor control unit 43A repeatedly executes the processing shown in Figure 14 for each frame of the captured image.

[0128] The process shown in Figure 14 differs from the process in the first embodiment shown in Figure 7 in that step S101A is performed instead of step S101, and step S102A is performed instead of step S102.

[0129] In step S101A, the sensor's internal control unit 43A waits until the AI ​​model's inference processing is completed. That is, it waits until the AI ​​model Ms has completed the inference processing for the captured image for one frame that was targeted for processing.

[0130] In step S102A, following step S101A, the sensor control unit 43A executes the process shown in Figure 15 as the calculation process for the uncertainty score.

[0131] In Figure 15, the sensor control unit 43A first calculates the uncertainty score Uc for each prediction class in step S301. That is, based on the score p information for each bounding box BB obtained as post-processed inference result information, specifically the score p information for the predicted class, the uncertainty score Uc is calculated for each predicted class using the formula "Uc = 1 - p".

[0132] In step S302, following step S301, the sensor control unit 43A calculates an image-level uncertainty score Uu based on the uncertainty score Uc for each class. Since an example of the method for calculating the image-level uncertainty score Uu based on the uncertainty score Uc has already been explained, a redundant explanation will be avoided.

[0133] The sensor's internal control unit 43A completes the calculation of the uncertainty score in step S102A, in accordance with having executed the process in step S302.

[0134] Here, in the second embodiment, the processing performed on the server device 2 side is the same as in the first embodiment (see Figures 9 and 10), so we will avoid repeating the explanation.

[0135] In addition, the uncertainty score in the second embodiment can also be calculated as an entropy-based value. In that case, the information of each bounding box BB remaining after post-processing and the information of the score p (C values) for each predictable class calculated for each bounding box BB are used. Specifically, in this case, the uncertainty score is calculated for each bounding box BB using the score p for each predictable class according to [Equation 3]. Hereinafter, the uncertainty score for each bounding box BB obtained in this way will be referred to as the "uncertainty score Ud". Then, the image-level uncertainty score Uu is calculated based on these uncertainty scores Ud for each bounding box BB. In this case, the image-level uncertainty score Uu only needs to be calculated as a value that reflects the uncertainty score Ud calculated for each bounding box BB. For example, it can be calculated as the sum or average of all uncertainty scores Ud, or as the average of the top predetermined number of uncertainty scores Ud with large values ​​(e.g., up to the 3rd or 5th place).

[0136] <3. Modifications> Although embodiments of the present technology have been described above, the embodiments are not limited to the specific examples described above, and various modified configurations can be adopted. For example, in the above example, when the processing of the score calculation unit F1 (or F1A), the determination processing unit F2, and the transmission processing unit F3 is performed by an edge-side device as the imaging device 1 (or 1A), an example was given in which these processes are performed by a processor in the image sensor as the sensor internal control unit 43 or 43A. However, at least a part of these processes can also be performed by a processor provided in a part of the imaging device that is outside the image sensor, such as a camera control unit 13.

[0137] Furthermore, while the above example describes a configuration in which the device performing the score calculation unit F1 (or F1A), the determination processing unit F2, and the transmission processing unit F3 is configured as an imaging device equipped with an imaging unit, it is not essential that the device performing these processing functions is equipped with an imaging unit. For example, in the case where a computer device acting as a fog server is interposed between the imaging device and the server device, it is conceivable that the processing of the score calculation unit F1 (or F1A), the determination processing unit F2, and the transmission processing unit F3 could be performed by the fog server. In this case, the fog server could perform inference processing using an AI model on the captured images transmitted by each imaging device, and then perform processing as the score calculation unit F1 (or F1A), and processing as the determination processing unit F2 and the transmission processing unit F3 based on the inference result information from the inference processing.

[0138] <4. Summary of Embodiments> As described above, the information processing device (imaging device 1 or 1A) as an embodiment comprises: an inference unit (AI processing unit 44) that performs inference processing using an AI model on captured images; a score calculation unit (F1 or F1A) that calculates an uncertainty score for the inference based on the inference result information from the inference processing; a determination processing unit (F2) that determines whether or not drift has occurred based on the uncertainty score; and a transmission processing unit (F3) that, when the determination processing unit determines that drift has occurred, performs processing to transmit the captured images that were the subject of inference to an external device. This makes it possible to transmit only the captured images that caused the drift, in other words, only the captured images necessary for retraining to correct the drift, to the external device. Therefore, it is possible to reduce the amount of data communication required for retraining to correct the drift, and to reduce the processing burden and shorten the training time by reducing the amount of data handled in retraining.

[0139] Furthermore, in the information processing device (imaging device 1) as an embodiment, the inference unit performs inference processing on the captured image using multiple AI models that perform inference processing for the same inference task, and the score calculation unit (F1) calculates an uncertainty score based on the inference result information from the multiple AI models. By using the inference result information from multiple AI models to calculate the uncertainty score, the uncertainty score can be calculated with greater accuracy than when using only the inference result information from a single AI model. Therefore, the accuracy of determining whether or not drift occurs can be improved.

[0140] Furthermore, in the information processing device as an embodiment, the multiple AI models are AI models that perform object detection processing or classification processing, and the score calculation unit calculates an uncertainty score based on the likelihood information (score p) of the inferences calculated by the multiple AI models. In the inference processing as object detection processing or classification processing, the likelihood for each class is calculated, and this likelihood information is used in the calculation of the uncertainty score. Since the likelihood information indicates the certainty of the inference result, by using this likelihood information in the calculation of the uncertainty score, the uncertainty score can be calculated with high accuracy.

[0141] Furthermore, in the information processing apparatus as an embodiment, the multiple AI models are AI models that perform object detection processing, and the score calculation unit calculates an uncertainty score based on likelihood information for each bounding box when each of the multiple AI models detects multiple objects, and groups bounding boxes that are estimated to have detected the same object, calculates the variance of uncertainty scores among the AI ​​models for each group to obtain a group-level uncertainty score, calculates an image-level uncertainty score based on the group-level uncertainty score, and the judgment processing unit determines whether or not drift has occurred based on the image-level uncertainty score. As a result, when the inference task is object detection processing, the uncertainty score can be appropriately calculated using the inference result information of the multiple AI models, and the determination of whether or not drift has occurred can be appropriately made.

[0142] Furthermore, in the information processing device as an embodiment, multiple AI models are AI models that perform classification processing, and the score calculation unit calculates an image-level uncertainty score based on the likelihood information for each AI model. This makes it possible to appropriately calculate the uncertainty score in response to the case where an AI model is an AI model that performs classification processing, and to appropriately determine whether or not drift has occurred.

[0143] Furthermore, in the information processing apparatus as an embodiment, the score calculation unit calculates an uncertainty score based on the likelihood entropy for each class. The likelihood entropy for each class is suitable as an indicator for estimating the degree of uncertainty in inference. Therefore, with the above configuration, the uncertainty score can be calculated appropriately, and the presence or absence of drift can be appropriately determined.

[0144] Furthermore, in the information processing device as an embodiment, the multiple AI models are AI models that perform regression processing, and the score calculation unit calculates an uncertainty score based on the variance of the inference values ​​for each AI model. This makes it possible to appropriately calculate the uncertainty score in response to cases where the AI ​​model is an AI model that performs regression processing, and to appropriately determine whether or not drift has occurred.

[0145] Furthermore, in the information processing device as an embodiment, the transmission processing unit performs the process of sending notification information to an external device to notify it of the occurrence of drift, in response to the determination processing unit determining the occurrence of drift. This allows the external device to manage the state of whether or not drift has occurred for the AI ​​model being used by the information processing device.

[0146] Furthermore, the information processing device as an embodiment is configured as an imaging device equipped with an imaging unit for obtaining captured images. This allows for a reduction in the amount of data communication required for retraining to correct drift, a reduction in the amount of data handled during retraining, thereby reducing the processing burden and shortening the training time, in cases where the imaging device is configured to perform inference processing using an AI model.

[0147] The information processing method as an embodiment is an information processing device that is performed by an information processing device equipped with an inference unit that performs inference processing using an AI model on captured images. The method calculates an uncertainty score for the inference based on the inference result information obtained from the inference processing, determines whether or not drift has occurred based on the uncertainty score, and, if it is determined that drift has occurred, transmits the captured image that was the target of the inference to an external device. The same functions and effects as the information processing device as an embodiment described above can be obtained by such an information processing method.

[0148] An information processing system as an embodiment comprises an information processing device that performs inference processing using an AI model on captured images, and a server device that can communicate with the information processing device. The information processing device includes a score calculation unit that calculates an uncertainty score for inference based on the inference result information obtained from the inference processing, a determination processing unit that determines whether or not drift has occurred based on the uncertainty score, and a transmission processing unit that transmits the captured images targeted for inference to the server device when the determination processing unit determines that drift has occurred. The server device includes a retraining processing unit (see Figure 10) that performs retraining processing for the AI ​​model using the captured images transmitted by the transmission processing unit as training input images. Such an information processing system can also obtain the same functions and effects as the information processing device as the embodiment described above. Furthermore, depending on the information processing system, the drift state can be resolved by retraining the AI ​​model.

[0149] Furthermore, the effects described herein are merely illustrative and not limited to those described herein, and other effects may also occur.

[0150] <5. The Technology> The technology can also be configured as follows: (1) An information processing device comprising: an inference unit that performs inference processing using an AI model on an captured image; a score calculation unit that calculates an uncertainty score for the inference based on the inference result information from the inference processing; a determination processing unit that determines whether or not drift has occurred based on the uncertainty score; and a transmission processing unit that transmits the captured image that has been the subject of inference to an external device when the determination processing unit determines that drift has occurred. (2) The information processing device according to (1), wherein the inference unit has multiple AI models that perform inference processing on the captured image for the same inference task, and the score calculation unit calculates the uncertainty score based on the inference result information from the multiple AI models. (3) The information processing device according to (2), wherein the multiple AI models are AI models that perform object detection processing or classification processing, and the score calculation unit calculates the uncertainty score based on the likelihood information of the inference calculated by the multiple AI models. (4) The plurality of AI models are AI models that perform object detection processing, and the score calculation unit calculates an uncertainty score based on the likelihood information for each bounding box when each of the plurality of AI models detects a plurality of objects, groups bounding boxes that are estimated to have detected the same object, calculates the variance of uncertainty scores among the AI ​​models for each group to obtain a group-level uncertainty score, calculates an image-level uncertainty score based on the group-level uncertainty score, and the judgment processing unit determines whether or not drift has occurred based on the image-level uncertainty score, as described in (3). (5) The plurality of AI models are AI models that perform classification processing, and the score calculation unit calculates an image-level uncertainty score based on the likelihood information for each of the AI ​​models, as described in (3). (6) The score calculation unit calculates a score based on the entropy of the likelihood for each class as the uncertainty score, as described in any of (3) to (5).(7) The information processing device according to (2), wherein the plurality of AI models are AI models that perform regression processing, and the score calculation unit calculates a value based on the variance of the inference values ​​for each AI model as the uncertainty score. (8) The information processing device according to any one of (1) to (7), wherein the transmission processing unit performs the process of transmitting notification information to the external device to notify it of the occurrence of drift in response to the determination processing unit determining the occurrence of drift. (9) The information processing device according to any one of (1) to (8), configured as an imaging device equipped with an imaging unit that obtains the captured image. (10) An information processing method executed by an information processing device equipped with an inference unit that performs inference processing using an AI model on an captured image, wherein the inference processing unit calculates an uncertainty score for the inference based on the inference result information obtained from the inference processing unit, determines whether or not drift has occurred based on the uncertainty score, and transmits the captured image that was the subject of the inference to an external device if it is determined that drift has occurred. (11) An information processing system comprising an information processing device that performs inference processing using an AI model on captured images, and a server device that can communicate with the information processing device, wherein the information processing device comprises: a score calculation unit that calculates an uncertainty score for inference based on inference result information obtained from the inference processing; a determination processing unit that determines whether or not drift has occurred based on the uncertainty score; and a transmission processing unit that, when the determination processing unit determines that drift has occurred, transmits the captured images that were the subject of inference to the server device, and the server device comprises a retraining processing unit that performs retraining processing for the AI ​​model using the captured images transmitted by the transmission processing unit as learning input images.

[0151] 1, 1A Imaging device 2 Server device 10, 10A Image sensor 11 Imaging optical system 12 Optical system drive unit 13 Camera control unit 14 Memory unit 15 Communication unit 16 Bus 41 Imaging unit 42 Image signal processing unit 43, 43A Sensor internal control unit 44 AI processing unit 45 Memory unit 46 Communication interface 47 Pre-processing unit 48 Post-processing unit 50 Bus Mm Main AI model Md Detection AI model BB Bounding box 21 CPU 27 Display unit 29 Storage unit 30 Communication unit F1, F1A Score calculation unit F2 Judgment processing unit F3 Transmission processing unit Ms AI model

Claims

1. An information processing device comprising: an inference unit that performs inference processing using an AI model on an captured image; a score calculation unit that calculates an uncertainty score for the inference based on the inference result information obtained from the inference processing; a determination processing unit that determines whether or not drift has occurred based on the uncertainty score; and a transmission processing unit that, when the determination processing unit determines that drift has occurred, performs processing to transmit the captured image that was the subject of inference to an external device.

2. The information processing apparatus according to claim 1, wherein the inference unit has multiple AI models that perform inference processing on the captured image, and the score calculation unit calculates the uncertainty score based on the inference result information from the multiple AI models.

3. The information processing apparatus according to claim 2, wherein the plurality of AI models are AI models that perform object detection processing or classification processing, and the score calculation unit calculates the uncertainty score based on the likelihood information of the inferences calculated by the plurality of AI models.

4. The information processing apparatus according to claim 3, wherein the plurality of AI models are AI models that perform object detection processing, the score calculation unit calculates an uncertainty score based on the likelihood information for each bounding box when each of the plurality of AI models detects a plurality of objects, groups bounding boxes that are estimated to have detected the same object, calculates the variance of uncertainty scores among the AI ​​models for each group to obtain a group-level uncertainty score, calculates an image-level uncertainty score based on the group-level uncertainty score, and the determination processing unit determines whether or not drift has occurred based on the image-level uncertainty score.

5. The information processing apparatus according to claim 3, wherein the plurality of AI models are AI models that perform classification processing, and the score calculation unit calculates an image-level uncertainty score based on the likelihood information for each of the AI ​​models.

6. The information processing apparatus according to claim 3, wherein the score calculation unit calculates a score based on the likelihood entropy for each class as the uncertainty score.

7. The information processing apparatus according to claim 2, wherein the plurality of AI models are AI models that perform regression processing, and the score calculation unit calculates a value based on the variance of the inference values ​​for each of the AI ​​models as the uncertainty score.

8. The information processing apparatus according to claim 1, wherein the transmission processing unit performs a process of transmitting notification information to the external device to notify it of the occurrence of drift, in response to the determination processing unit determining the occurrence of drift.

9. The information processing device according to claim 1, configured as an imaging device equipped with an imaging unit for obtaining the captured image.

10. An information processing method performed by an information processing device equipped with an inference unit that performs inference processing using an AI model on captured images, the method comprising: calculating an uncertainty score for the inference based on the inference result information obtained from the inference processing; determining whether or not drift has occurred based on the uncertainty score; and, if it is determined that drift has occurred, transmitting the captured image that was the target of the inference to an external device.

11. An information processing system comprising an information processing device that performs inference processing using an AI model on captured images, and a server device capable of communicating with the information processing device, wherein the information processing device comprises: a score calculation unit that calculates an uncertainty score for inference based on inference result information obtained from the inference processing; a determination processing unit that determines whether or not drift has occurred based on the uncertainty score; and a transmission processing unit that, when the determination processing unit determines that drift has occurred, transmits the captured images that were the subject of inference to the server device, and the server device comprises a retraining processing unit that performs retraining processing for the AI ​​model using the captured images transmitted by the transmission processing unit as learning input images.

Images