Information processing device, information processing method

Abstract

When retraining an AI model used for object detection processing on edge devices to add identifiable classes, the aim is to improve the class identification accuracy of the AI ​​model while reducing the workload on the user. [Solution] The information processing device comprises an annotation processing unit that performs annotation processing on an input image using a large-scale AI model that performs object detection processing and is configured to output region information indicating an object detection region even for objects that cannot be classified as a class; a reception processing unit that accepts corrections from the user regarding the annotation results from the annotation processing unit; a large-scale model retraining processing unit that performs retraining processing on the large-scale AI model using the annotation information corrected by the user; and an edge model retraining processing unit that performs retraining processing on an edge model, which is an AI model used in an edge device that performs object detection processing on captured images, by knowledge distillation using the retrained large-scale AI model as the training model.

Description

Technical Field

[0001] This technology relates to an information processing apparatus and a method thereof, and particularly relates to the technical field of the relearning process of an AI (Artificial Intelligence) model.

Background Art

[0002] There is a technology for performing various image recognition processes using an AI model on a captured image by an imaging device. Here, the image recognition process referred to here broadly means a process for recognizing the image content, such as object detection processing or object recognition processing.

[0003] The result of the image recognition process performed on the captured image is used for performing various analyses on the subject. For example, a plurality of imaging devices are installed at the site of the target such as a store, and object detection processing for a person is performed on the captured images of these imaging devices using an AI model. Then, based on the result of this object detection process, it is conceivable to perform an analysis of the subject, such as a person count or a movement line analysis of the person.

[0004] As a system for realizing various analyses on the subject in this way, there is a configuration in which an AI model is installed in a device on the site side such as in the imaging device, and a server device arranged remotely from the site performs the management process of the AI model. Hereinafter, the device on the site side in such a system is referred to as an "edge device". Examples of the management process of the AI model performed by the server device include a process of deploying the AI model that can be used by the user by purchase or the like to the edge device, and a process of relearning the AI model. Note that "deployment" means a process of transmitting the AI model to the edge device side so that the AI model can be used on the edge device side. The relearning of the AI model here is performed as learning to adapt to the field environment using the captured images in the actual field in order to prevent the inference performance from deteriorating due to changes in the usage environment of the imaging device or the like.

[0005] Regarding related prior art, the following patent documents 1 and 2 can be cited. Patent Document 1 below discloses a technique for labeling (annotating) individual objects contained in captured images using a trained image recognition unit. Furthermore, Patent Document 2 discloses a technology that includes an annotation unit that adds identification information to image data using a trained learning model, and a modification unit that modifies the identification information added by the annotation unit according to a modification request, and uses the image data whose identification information has been modified by the modification unit to update the learning model (retrain the AI ​​model). [Prior art documents] [Patent Documents]

[0006] [Patent Document 1] Patent No. 7055259 [Patent Document 2] Patent No. 7390628 [Overview of the Initiative] [Problems that the invention aims to solve]

[0007] In this context, a possible change in the operating environment of the imaging device is that a new class of objects, which have not been previously targeted for imaging, may be included as the target of imaging. For example, in a case where an imaging device is used to manage product inventory in a store, this could occur when new products are displayed in that store.

[0008] In this case, the AI ​​model used on the edge device needs to be retrained to be able to identify the new class of objects. However, in this case, the user is required to perform annotation work on the captured images used for retraining (images that include the new class of objects as the target of the image), which imposes a burden of work on the user for retraining.

[0009] Furthermore, because AI models used on edge devices are relatively small due to resource constraints, directly retraining these models using annotation information that reflects the new class names as training data would make it difficult to achieve the necessary learning effect to properly identify the new classes, potentially leading to a decrease in class identification accuracy.

[0010] This technology was developed in light of the above circumstances, and aims to improve the class identification accuracy of AI models used in object detection processing on edge devices, while reducing the workload on users when retraining the AI ​​model to add identifiable classes. [Means for solving the problem]

[0011] The information processing device relating to this technology comprises: an annotation processing unit that performs annotation processing on an input image using a large-scale AI model that performs object detection processing and is configured to output region information indicating an object detection region even for objects that cannot be classified as a class; a reception processing unit that accepts corrections from the user to the annotation results from the annotation processing unit; a large-scale model retraining processing unit that performs retraining processing on the large-scale AI model using the annotation information corrected by the user; and an edge model retraining processing unit that performs retraining processing on an edge model, which is an AI model used in an edge device that performs object detection processing on captured images, by knowledge distillation using the retrained large-scale AI model as a training model. By having the reception processing unit described above, it becomes possible to have the user set the correct class name information for objects that the large-scale AI model considered unclassifiable or incorrectly classified before relearning. Then, by relearning the large-scale AI model using the corrected annotation information as described above, it becomes possible to obtain a large-scale AI model that can identify objects of new classes (objects that were considered unclassifiable or incorrectly classified before relearning). Furthermore, by relearning the edge model through knowledge distillation using this relearned large-scale AI model as the training model, it becomes possible to obtain an edge model that can identify objects of new classes. In other words, it becomes possible to retrain the AI ​​model used in object detection processing on edge devices to add classes that can be identified. Furthermore, with the above configuration, in order to retrain the AI ​​model used for object detection processing on edge devices to add identifiable classes, the user only needs to modify the annotation results by the annotation processing unit. In addition, since the retraining of the edge model is performed by knowledge distillation using a large-scale AI model that has been retrained to identify the new classes, it becomes possible to obtain an edge model capable of high-precision class identification. [Brief explanation of the drawing]

[0012] [Figure 1] This is a block diagram showing an example configuration of an information processing system as an embodiment. [Figure 2] This is a block diagram showing an example configuration of an imaging device. [Figure 3] This is a block diagram showing an example of the hardware configuration of an information processing device as an embodiment. [Figure 4] This is an example diagram showing the service's top screen. [Figure 5] This diagram illustrates the Edge Model Management Top screen. [Figure 6] This is an example diagram of the job settings screen. [Figure 7]It is a diagram illustrating an upload - learning type setting screen. [Figure 8] It is a diagram illustrating the top screen of edge model management when the relearning process is in progress. [Figure 9] It is an explanatory diagram of the evaluation result screen. [Figure 10] Similarly, it is an explanatory diagram of the evaluation result screen. [Figure 11] It is a diagram illustrating a deployment management screen. [Figure 12] It is a functional block diagram for explaining various functions related to the relearning method as an embodiment. [Figure 13] It is an explanatory diagram of the relearning method of the edge model in the case of adding classes. [Figure 14] It is a diagram illustrating a modification reception screen. [Figure 15] It is a diagram for explaining an example of modifying a class name. [Figure 16] It is an explanatory diagram of the relearning method of the edge model in the case of not adding classes. [Figure 17] It is a flowchart showing a specific example of a processing procedure to be executed to realize the relearning method as an embodiment. [Figure 18] It is a flowchart of the process related to the evaluation of the relearned edge model. [Figure 19] It is a flowchart of the process related to the deployment of the relearned edge model.

Mode for Carrying Out the Invention

[0013] Hereinafter, referring to the attached drawings, embodiments of the present technology will be described in the following order. <1. Configuration of the Information Processing System> (1 - 1. System Overview) (1 - 2. Configuration Example of the Imaging Device) (1 - 3. Configuration Example of the Information Processing Device) <2. Example of Screen Transition Related to Relearning> <3. Relearning Method as an Embodiment> <4. Processing Procedure> <5. Variation> <6. Summary of Embodiments> <7. This Technology>

[0014] <1. Configuration of the Information Processing System> (1-1. System Overview) Figure 1 is a schematic diagram illustrating an information processing system as an embodiment of the present technology, which is configured to include an information processing device. As shown in the figure, the information processing system as an embodiment comprises a server device 1, an imaging device 2, and a user terminal 3. Server device 1 and user terminal 3 are configured as computer devices each equipped with a microcomputer having a CPU (Central Processing Unit), ROM (Read Only Memory), and RAM (Random Access Memory). Server device 1 corresponds to an example of an information processing device as an embodiment of the present technology.

[0015] The server device 1 is configured to enable data communication with the imaging device 2 and the user terminal 3 via a communication network NT, such as the Internet.

[0016] The imaging device 2 captures an image of the subject and obtains an image. Herein, in this specification, "imaging" broadly means obtaining image data that captures a subject. The image data referred to here is a general term for data consisting of multiple pixel data, and the pixel data is a broad concept that includes not only data indicating the intensity of the amount of light received from the subject, but also, for example, distance to the subject, polarization information of the subject, temperature information, etc. In other words, the "image data" obtained by "imaging" (imaging image data) includes data as a grayscale image that shows the amount of light received for each pixel, data as a distance image that shows the distance to the subject for each pixel, data as a polarization image that shows the polarization information of incident light for each pixel, and data as a thermal image that shows temperature information for each pixel.

[0017] As an example, let's assume that the imaging device 2 in this example is configured to obtain the above-mentioned grayscale image as the captured image, similar to a typical digital camera. Specifically, let's assume it is configured to obtain an RGB color image as the captured image.

[0018] In the information processing system of this example, multiple imaging devices 2 are provided at the target site 100. Location 100 may vary depending on the application of the imaging device 2. For example, if the purpose is to monitor people such as customers in a store, location 100 will be the store itself, or if the purpose is to monitor vehicles in a parking lot, location 100 will be the parking lot.

[0019] In the information processing system, server device 1 is a computer device intended to be used by the provider of services using the information processing system. User terminal 3 is a computer device intended to be used by the user who receives the service.

[0020] The information processing system in this example is configured to provide a service that performs inference processing using an AI (Artificial Intelligence) model, specifically image recognition processing, on images captured by the imaging device 2, and then generates analytical information showing the analysis results of the subject based on the processing results (inference results), and presents it to the user.

[0021] In this context, image recognition processing broadly refers to the process of recognizing the content of an image. Examples of image recognition processing include object detection processing, which detects the region where an object exists; object recognition processing, which recognizes what kind of object is depicted in an image; semantic segmentation processing; and anomaly detection processing such as PatchCore. Here, object detection processing includes not only detecting the area where an object exists, but also recognizing what kind of object it is, such as YOLO (You Only Look Once) or SSD (Single Shot Multibox Detector).

[0022] In this embodiment, the image recognition processing using the AI ​​model is assumed to be an object detection process (hereinafter simply referred to as "object detection process") that recognizes not only the region where an object exists, but also what kind of object it is, as described above with YOLO and SSD.

[0023] Here, assuming a service that presents the user with analysis information of the subject based on the inference results as described above, possible uses for the imaging device 2 include, for example, monitoring indoors such as stores, offices, and residences; monitoring outdoors such as parking lots and streets (including traffic surveillance cameras); monitoring manufacturing lines in FA (Factory Automation) and IA (Industrial Automation); and monitoring inside and outside vehicles.

[0024] For example, in the case of surveillance cameras in a store, multiple imaging devices 2 could be placed at designated locations within the store, allowing users to observe customer demographics (such as gender and age group) and their behavior within the store (movement patterns). In this case, the analysis information could include data on customer demographics, movement patterns within the store, and congestion levels at checkout counters (e.g., waiting times at checkout counters). Alternatively, if the camera is used for traffic surveillance, each imaging device 2 could be placed at various locations near the road, allowing users to recognize information such as license plate numbers, vehicle color, and vehicle type of passing vehicles. In this case, the analysis information described above could include generating this information such as license plate numbers, vehicle color, and vehicle type.

[0025] Furthermore, if traffic surveillance cameras are used in the parking lot, the imaging device 2 can be positioned to monitor each parked vehicle, and it is conceivable that the camera will notify the public of the presence of suspicious individuals behaving inappropriately around each vehicle, and if a suspicious person is found, the camera will notify the public of the presence of a suspicious person and their attributes (gender, age group, clothing, etc.). Furthermore, it's conceivable that the system could monitor available parking spaces in urban areas and parking lots, and notify users of the locations where they can park their cars.

[0026] In this example, object detection processing of the captured image is performed by the imaging device 2. Specifically, the imaging device 2 is equipped with an AI model for performing object detection processing, and information indicating the results of the object detection processing using the AI ​​model (hereinafter sometimes referred to as "inference result information") is transmitted to the server device 1. Based on the inference result information transmitted from the imaging device 2 in this manner, the server device 1 performs the various analysis processes described above.

[0027] By employing this method in which the imaging device 2 performs inference processing and transmits the inference result information to the server device 1, the amount of communication data required to perform inference processing can be significantly reduced compared to the case in which the server device 1 performs inference processing on the captured images transmitted from the imaging device 2. Furthermore, since it is no longer necessary to transmit captured images from the imaging device 2 to the server device 1, it is possible to prevent the leakage of captured images containing personal information to external parties, thereby protecting privacy.

[0028] In this example, the AI ​​model used by imaging device 2 is deployed from server device 1 to imaging device 2. Deployment here refers to the process of sending the AI ​​model to the edge device so that it can be used on the edge device. In this example, the imaging device 2 is owned by the user, and the user can deploy the AI ​​model from the server device 1 to the imaging device 2 by purchasing the right to use the AI ​​model through payment of a fee to the service provider.

[0029] Furthermore, in this example of an information processing system, server device 1 also performs retraining processing for the AI ​​model used by imaging device 2. The AI ​​model used by the imaging device 2 (in this example, the AI ​​model purchased by the user) is managed by the server device 1. However, at the time of purchase, the AI ​​model is assumed to have undergone basic training for each object detection application. For example, if it is for detecting people, it is trained to detect people; if it is for detecting products, it is trained to detect products; or if it is for detecting vehicle license plates, it is trained to detect license plates. The retraining process referred to here means that the AI ​​model (hereinafter referred to as the "template model"), after basic training has been performed, undergoes further training to adapt it to the environment of Field 100. It is also conceivable that the retraining process may be performed again on the AI ​​model that has undergone such training to adapt to the environment of Field 100. In particular, the retraining process with "class additions" described later is also conceivable to be performed as a second or third training process.

[0030] The AI ​​model retraining process is performed, for example, before the service starts, by deploying the imaging device 2 at the site 100 in the same configuration as during operation, and using the images captured by the deployed imaging device 2 as training images.

[0031] Although the above example shows multiple imaging devices 2, in the information processing system as an embodiment, the number of imaging devices 2 only needs to be at least one. Furthermore, although Figure 1 shows only one user terminal 3 in the information processing system, there may be multiple user terminals 3. In other words, it is conceivable that there may be multiple users receiving services from the information processing system.

[0032] Furthermore, while the above describes an example where object detection processing using an AI model is performed in the imaging device 2, it is not mandatory for object detection processing using an AI model to be performed in the imaging device 2. For example, it is conceivable to adopt a configuration in which a computer device (information processing device) such as a fog server, which is capable of communicating with each imaging device 2 and the server device 1, has an AI model for performing object detection processing, and this computer device performs object detection processing using the AI ​​model on images captured by the imaging device 2 (one or more). In this case, the inference result information of the object detection processing will be transmitted by the computer device to the server device 1. The computer device such as a fog server referred to here could be located within the store if the purpose of the imaging device 2 is store monitoring, or it could be located in a facility different from the store, such as a data center managed by the company, if the user is a company with multiple stores.

[0033] In this context, from the perspective of server device 1 (i.e., the cloud) that manages the AI ​​model, computer devices such as imaging device 2 and the fog server mentioned above can be considered devices located on the edge side. In this specification, devices located on the edge side from the perspective of server device 1 are referred to as "edge devices." Furthermore, the AI ​​model used by the edge device for object detection processing is referred to as the "edge model."

[0034] (1-2. Example of imaging device configuration) Figure 2 is a block diagram showing an example of the configuration of the imaging device 2. As shown in the figure, the imaging device 2 comprises an imaging optical system 41, an image sensor 42, an optical system drive unit 43, a camera control unit 44, a memory unit 45, and a communication unit 46. The image sensor 42, camera control unit 44, memory unit 45, and communication unit 46 are connected via a bus BS and are capable of communicating data with each other.

[0035] In this example, the image sensor 42 is configured as a grayscale image sensor that obtains the aforementioned grayscale image. Specifically, the image sensor 42 is configured as a solid-state image sensor such as a CCD (Charge Coupled Device) type or a CMOS (Complementary Metal Oxide Semiconductor) type.

[0036] The imaging optical system 41 includes lenses such as a cover lens, zoom lens, and focus lens, as well as an aperture (iris) mechanism. This imaging optical system 41 guides light (incident light) from the subject and focuses it onto the light-receiving surface (imaging surface) of the image sensor 42.

[0037] The optical system drive unit 43 comprehensively represents the drive units for the zoom lens, focus lens, and aperture mechanism of the imaging optical system 41. Specifically, the optical system drive unit 43 includes actuators for driving the zoom lens, focus lens, and aperture mechanism, and drive circuits for said actuators.

[0038] The camera control unit 44 is configured with, for example, a microcomputer having a CPU, ROM, and RAM, and performs overall control of the imaging device 2 by having the CPU execute various processes according to a program stored in ROM or a program loaded into RAM.

[0039] Furthermore, the camera control unit 44 issues drive instructions to the optical system drive unit 43 for the zoom lens, focus lens, aperture mechanism, etc. In response to these drive instructions, the optical system drive unit 43 will perform actions such as moving the focus lens and zoom lens, and opening and closing the aperture blades of the aperture mechanism.

[0040] Furthermore, the camera control unit 44 controls the writing and reading of various data to and from the memory unit 45. The memory unit 45 is a non-volatile storage device such as a flash memory device or an HDD (Hard Disk Drive), and is used to store data used by the camera control unit 44 when performing various processes. The memory unit 45 can also be used as a storage location (recording location) for image data output from the image sensor 42.

[0041] The camera control unit 44 performs various data communications with external devices via the communication unit 46. In this example, the communication unit 46 is configured to enable communication via the network NT shown in Figure 1, and is capable of performing data communications with external devices connected to the network NT, particularly in this example, at least with the server device 1.

[0042] As shown in the figure, the image sensor 42 includes an imaging unit 51, an image signal processing unit 52, an internal sensor control unit 53, an AI processing unit 54, a memory unit 55, and a communication interface (I / F) 56, each of which is connected via a bus 57 and is capable of communicating data with one another.

[0043] The imaging unit 51 includes a pixel array unit formed by arranging pixels, each having a photoelectric conversion element (light-receiving element) such as a photodiode, in a two-dimensional manner, and a readout circuit that reads out electrical signals (receiving signals) obtained by photoelectric conversion from each pixel of the pixel array unit. In this readout circuit, electrical signals obtained by photoelectric conversion are subjected to processes such as CDS (Correlated Double Sampling) and AGC (Automatic Gain Control), and further A / D (Analog to Digital) conversion.

[0044] In the imaging unit 51 of this example, a color filter that selectively transmits light of one of the following colors (R, G, or B) is formed for each pixel, so that an RGB color image can be obtained as the captured image. In this example, the arrangement of the color filters in the imaging unit 51 is, for example, an RGGB Bayer arrangement. Note that the Bayer arrangement is merely one example, and other arrangements (mosaic arrangements) of the color filters, such as RYYB or RGBW, may be adopted.

[0045] The imaging unit 51 outputs a RAW image as the captured image. Here, a RAW image refers to a digital captured image immediately after A / D conversion of the signal read from the pixel array unit. The captured image, output as a RAW image from the imaging unit 51, is input to the image signal processing unit 52.

[0046] The image signal processing unit 62 performs preprocessing, syncing, YC generation, codec processing, and other operations on the captured image as a RAW image. Preprocessing includes clamping to set the black level to a predetermined level, and correction processing between the R, G, and B color channels. Preprocessing also includes brightness adjustments such as gamma correction, and color adjustments such as white balance adjustment and linear matrix processing. Linear matrix processing is a color reproduction error correction process that corrects the colors to suit the desired color space by performing predetermined matrix operations on RGB.

[0047] Furthermore, the simultaneous processing involves color separation so that the image data for each pixel contains all three color components: R, G, and B. For example, in the case of an image sensor using a Bayer array color filter as in this example, demosaicing is performed as the color separation process. In the YC generation process, a luminance (Y) signal and a color (C) signal are generated (separated) from the R, G, and B image data.

[0048] In codec processing, the image data that has undergone the various processing steps described above is subjected to encoding for purposes such as recording and communication, and file generation. Codec processing makes it possible to generate video files in formats such as MPEG-2 (MPEG: Moving Picture Experts Group) and H.264. It also makes it possible to generate still image files in formats such as JPEG (Joint Photographic Experts Group), TIFF (Tagged Image File Format), and GIF (Graphics Interchange Format).

[0049] The sensor control unit 53 is configured with a microcomputer, for example, which includes a CPU, ROM, RAM, etc., and comprehensively controls the operation of the image sensor 42. For example, the sensor control unit 53 issues instructions to the imaging unit 51 to control the execution of imaging operations. It also controls the execution of processing to the image signal processing unit 52.

[0050] The AI ​​processing unit 54 is configured with a programmable computing device such as a DSP (Digital Signal Processor) or FPGA (Field Programmable Gate Array), and performs inference processing (AI processing) using an AI model on captured images.

[0051] As can be understood from the previous explanation, in this example, the AI ​​model used by the AI ​​processing unit 54 is an AI model that performs object detection processing. Also, as can be understood from the previous explanation, the AI ​​model that the AI ​​processing unit 54 uses for object detection processing is an AI model that functions as an edge model.

[0052] The memory unit 55 is used to store data necessary for the AI ​​processing unit 54 to perform object detection processing. Specifically, the memory unit 55 stores data of the AI ​​model used by the AI ​​processing unit 54 for object detection processing. This AI model data, in the case of an AI model that has a neural network such as a CNN (Convolutional Neural Network), includes data such as parameters that indicate the structure of the neural network and parameters used as filter coefficients in convolution processing, etc.

[0053] The communication interface (I / F) 56 is an interface that communicates with various parts connected via the bus BS, such as the camera control unit 44 and the memory unit 45, located outside the image sensor 42. For example, the communication interface 56 communicates to acquire an AI model from an external source, based on the control of the sensor's internal control unit 53, for use by the AI ​​processing unit 54. Furthermore, it is possible to output the result information (inference result information) of the object detection processing performed by the AI ​​processing unit 54 to the outside of the image sensor 42 via the communication interface 56.

[0054] (1-3. Examples of Information Processing Device Configurations) Figure 3 is a block diagram showing an example of the hardware configuration of server device 1. Furthermore, it is conceivable that the computer device serving as user terminal 3, as shown in Figure 1, could adopt a hardware configuration similar to that shown in Figure 1.

[0055] As shown in the figure, the server device 1 is equipped with a processor 11. The processor 11 is configured to have at least a CPU and executes various processes according to a program stored in the ROM 12 or a program loaded from the storage unit 19 into the RAM 13. In this example, the server device 1 also has a GPU (Graphics Processing Unit) in addition to the CPU in order to perform various image signal processing related to the AI ​​model retraining process.

[0056] RAM13 also stores data necessary for the processor 11 to perform various processes. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output interface (I / F) 15 is also connected to this bus 14.

[0057] An input unit 16, consisting of controls or operating devices, is connected to the input / output interface 15. For example, the input unit 16 could be various controls or operating devices such as a keyboard, mouse, keys, dial, touch panel, touchpad, or remote controller. The input unit 16 detects user operations, and the signal corresponding to the input operation is interpreted by the processor 11.

[0058] Furthermore, a display unit 17, such as an LCD (Liquid Crystal Display) or an organic EL (Electro-Luminescence) panel, and an audio output unit 18, such as a speaker, are connected to the input / output interface 15, either as an integrated unit or as separate components. The display unit 17 is used for displaying various types of information and is composed of, for example, a display device provided on the casing of a computer device or a separate display device connected to a computer device.

[0059] The display unit 17 displays images for various image processing tasks, videos to be processed, etc., on the display screen based on instructions from the processor 11. The display unit 17 also displays various operation menus, icons, messages, etc., i.e., a GUI (Graphical User Interface), based on instructions from the processor 11.

[0060] The input / output interface 15 may also be connected to a storage unit 19 consisting of an HDD or solid memory, or a communication unit 20 consisting of a modem or the like.

[0061] The communications unit 20 performs communication processing via transmission lines such as the Internet, and communicates with various devices via wired / wireless communication, bus communication, etc.

[0062] The input / output interface 15 is also connected to a drive 21 as needed, and a removable recording medium 22 such as a magnetic disk, optical disk, magneto-optical disk, or semiconductor memory is appropriately mounted there.

[0063] Drive 21 allows data (including computer programs, etc.) used for various processes to be read from the removable recording medium 22. The read data is stored in the storage unit 19, or, if the read data is image data or audio data, the image or audio is output by the display unit 17 or the audio output unit 18. The computer program read from the removable recording medium 22 is installed in the storage unit 19 as needed.

[0064] In the server device 1 having the hardware configuration described above, for example, the software for processing in this embodiment can be installed via network communication by the communication unit 20 or via the removable recording medium 22. Alternatively, the software may be pre-stored in the ROM 12 or the storage unit 19, etc. The processor 11 performs processing operations based on various programs, thereby executing the necessary information processing and communication processing for the server device 1.

[0065] Furthermore, server device 1 is not limited to being a single computer device as shown in Figure 3; it may also be configured as a system of multiple computer devices. These multiple computer devices may be connected via a LAN (Local Area Network), or they may be located remotely via a VPN (Virtual Private Network) using the Internet. The multiple computer devices may also include computer devices that function as a group of servers (cloud) available through cloud computing services.

[0066] <2. Example of screen transitions related to relearning> As described above, in the information processing system of this embodiment, the server device 1 performs retraining processing on the AI ​​model as an edge model owned by the user. Here, we will first explain an example of the transitions of the screens presented to the user regarding such retraining processing. For clarification, the various screens described below are displayed on the display unit (corresponding to the display unit 17 in Figure 3) that displays various information to the user on the user terminal 3. The various screens are displayed on the display unit of the user terminal 3 based on the control of the server device 1 (in this example, the control of the processor 11).

[0067] Figure 4 illustrates the Service Top screen G1, which is the top screen for services provided to users in an information processing system. Although a diagrammatic explanation is omitted, users are required to log in and enter their account information when accessing this service's top screen.

[0068] The service top screen G1 contains various buttons related to the services available in the information processing system. Specifically, in this example, at least a device management button B1, an edge model management button B2, and a deployment management button B3 are located on the screen, as shown in the diagram. The device management button B1 is used to call up the device management screen, which displays various management information about the imaging device 2 (edge ​​device) owned by the user.

[0069] The Edge Model Management button B2 is used to access the Edge Model Management screen (Edge Model Management Top screen G2, described below), which displays various management information about the Edge Models owned by the user. Furthermore, the deployment management button B3 is used to access the screen (deployment management screen G6, described later) for deploying the edge model to the edge device (imaging device 2 in this example).

[0070] Figure 5 illustrates the Edge Model Management Top screen G2, which is displayed in response to the operation of the Edge Model Management button B2. The Edge Model Management Top screen G2 displays information about completed jobs for edge models owned by the user. In this example, there are three types of jobs that can be performed on edge models: "Retraining," "Parameter Tuning," and "Performance Evaluation." "Parameter Tuning" refers to tuning the parameters of the edge model, specifically parameters such as the threshold for likelihood (a confidence score calculated for each class) used in the determination to obtain the final class identification result.

[0071] On the Edge Model Management Top screen G2, job information is displayed as shown in the diagram, including the job name, target model information, and process information. The target model information refers to the identification information of the edge model on which the job is executed. The process information indicates the progress of the job. Here, we are illustrating a case where only job information for jobs that have been executed in the past is displayed, so as shown in the diagram, all process information is "executed". As shown in the diagram, the results display button B4 is displayed for jobs that have already been executed. The results display button B4 is a button that instructs the system to execute the performance evaluation of the target model and to display the evaluation results.

[0072] The Edge Model Management Top screen G2 has a Job Add button B5, which allows users to add new jobs.

[0073] Figure 6 shows an example of the job settings screen G3 that appears in response to the operation of the Add Job button B5. The job settings screen G3 is used to set the job name, target model, and job type for a job to be added. As shown in the diagram, it has an input box bi for entering the job name, an input box bi for entering the target model, and a checkbox cb for selecting the job type. The input box "bi" for the job name is a box into which any text information can be entered. The input box "bi" for the target model is, in this example, a dropdown box into which the user can select an edge model. The selection options are the edge models owned by the user.

[0074] Regarding the job type, users can select from the above-mentioned "retraining," "parameter tuning," and "performance evaluation" options, and a checkbox (cb) is provided for each of these options.

[0075] If the user wants to instruct the execution of a job of the selected target model according to the selected job type, they operate the Next button B7 located on the Job Settings screen G3. Furthermore, the job settings screen G3 has a back button B6, which allows the user to return to the edge model management top screen G2 shown in Figure 5 by operating the back button B6.

[0076] Although not shown in the diagram, if "Performance Evaluation" is selected as the job type and the Next button B7 is pressed, the screen transitions to the Edge Model Management Top screen G2 as exemplified in Figure 5. On this Edge Model Management Top screen G2, the corresponding job information is displayed along with process information indicating "Running". Upon completion of the process, the process information is updated to "Completed", and the Results Display button B4 is displayed for the job information. By pressing this Results Display button B4, the user can display the performance evaluation results screen for the selected edge model (details are explained in Figures 9 and 10).

[0077] Figure 7 shows an example of the Upload / Learning Type Settings screen G4, which appears when "Relearning" is selected on the Job Settings screen G3 and the Next button B7 is pressed. The Upload / Learning Type Settings screen G4 is used to configure the image data to be uploaded and the type of retraining when uploading training images to Server Device 1 for use in retraining the edge model.

[0078] As mentioned above, in this example, the training images are to be images captured by each imaging device 2 installed at the site 100. For example, the user stores the images (group of captured images) from each imaging device 2 that are to be used as training images (user terminal 3), and sends these stored images to the server device 1 from the user terminal 3, specifying them as upload images.

[0079] In this example, there are two types of retraining: "with class addition" and "without class addition." "With class addition" means retraining to add the ability to identify objects of a new class to the edge model. "Without class addition" means retraining to adapt to so-called domain changes, such as changes in the background or lighting conditions at site 100, without adding the ability to identify objects of a new class.

[0080] As shown in the diagram, the upload / learning type setting screen G4 is equipped with a select button B8, a display box di, and a checkbox cb for selecting the relearning type. When the select button B8 is operated, directory information about the data files stored on the user terminal 3 is displayed in the display box di. Based on this displayed directory information, the user can specify the directory of the data files to be uploaded (i.e., the data files consisting of the image capture data captured by the imaging device 2). Furthermore, users can select (set) the type of retraining to be performed by manipulating the checkbox cb.

[0081] The Upload / Learning Type Settings screen G4 contains the Back button B9 and the Execute button B10. The user can return to the job settings screen G3, as illustrated in Figure 6, by operating the back button B9.

[0082] When the execute button B10 is pressed, the data files (a group of captured images) in the directory specified in the display box di are uploaded from the user terminal 3 to the server device 1. The server device 1 then uses the uploaded group of captured images as training images and performs a retraining process on the edge model selected in the job setting screen G3. Details of the retraining process for "with class addition" and "without class addition" will be explained later.

[0083] It is also possible to have separate screens for setting up and executing the upload of training images, and for setting the type of retraining and executing the retraining process.

[0084] When the retraining process begins, the display unit of the user terminal 3 shows the Edge Model Management Top screen G2, as illustrated in Figure 8. In other words, it is the Edge Model Management Top screen G2 that displays job information indicating that the added retraining job is in progress. Similar to the "performance evaluation" job described earlier, for jobs that are in progress, information indicating "in progress" is displayed as process information. In this case as well, upon completion of the process, the process information is updated to "completed," and the results display button B4 is displayed for the job information. By operating the results display button B4, the user can instruct the execution of a performance evaluation of the retrained edge model and the display of the evaluation results screen G5, which shows the results of the performance evaluation.

[0085] Figures 9 and 10 are explanatory diagrams for the evaluation results screen G5. In this example, server device 1 performs performance evaluation processing for edge models, such as calculating numerical evaluation information including Accuracy, Precision, and Recall. The evaluation results screen G5 is basically a screen that displays evaluation information in numerical form, but in this example, the evaluation results screen G5 is configured to display not only numerical evaluation information but also image preview information that visually shows the object detection status (in this example, the bounding box detection status) using an image.

[0086] On the evaluation results screen G5, a results display area Ae is provided for displaying the evaluation results, and an image preview tab T1 and a numerical information tab T2 are provided for selecting whether to display image preview information or numerical evaluation information, respectively. Figure 9 shows the state when the image preview tab T1 is selected and image preview information is displayed, and Figure 10 shows the state when the numerical information tab T2 is selected and numerical evaluation information is displayed. As shown in Figure 9, in this example, the image preview information is displayed as information that shows a comparison of the ground truth and predicted bounding box information for each image.

[0087] The evaluation results screen G5 is equipped with a back button B11 and an export button B12. By using the back button B11, the user can return to the edge model management top screen G2, and by using the export button B12, they can export the evaluation information (at least numerical evaluation information) in a specified file format.

[0088] Figure 11 illustrates the deployment management screen G6, which is displayed in response to the operation of the deployment management button B3 on the service top screen G1. As shown in the diagram, the deployment management screen G6 is provided with input boxes bi for selecting the edge model to be deployed and for selecting the edge device to be deployed, as well as a back button B13 and a deploy button B14.

[0089] In this example, the input box `bi` for selecting an edge model and the input box `bi` for selecting the edge device to deploy are both dropdown selection input boxes. The former input box `bi` displays a list of edge models owned by the user, from which the user can select the edge model to deploy. The latter input box `bi` displays a list of edge devices owned by the user, from which the user can select the edge device to deploy.

[0090] On the deployment management screen G6, users can return to the service top screen G1 by using the back button B13. Furthermore, by operating the deploy button B14, the user can instruct the server device 1 to deploy the edge model specified by the selection input operation using each of the input boxes bi described above to the same specified edge device.

[0091] <3. Retraining Method as an Embodiment> Next, we will describe the retraining method as an embodiment of the system. Figure 12 is a functional block diagram illustrating the various functions of the processor 11 of the server device 1 related to the retraining method as an embodiment. In particular, this section describes the functions related to retraining with "class addition" as described above.

[0092] As shown in the figure, the processor 11 has the functions of an annotation processing unit F1, a reception processing unit F2, a large-scale model retraining processing unit F3, an edge model retraining processing unit F4, an evaluation processing unit F5, and a deployment processing unit F6.

[0093] The annotation processing unit F1 performs annotation processing on the input image using a large-scale AI model. In this context, a large-scale AI model refers to an AI model that uses larger resources than an AI model deployed on an edge device (edge ​​model), and has higher inference performance.

[0094] The reception processing unit F2 accepts user requests for modifications to the annotation results performed by the annotation processing unit F1.

[0095] The large-scale model retraining processing unit F3 performs retraining of a large-scale AI model using annotation information modified by the user.

[0096] The edge model retraining processing unit F4 performs the edge model retraining process by knowledge distillation using the retrained large-scale AI model as the training model.

[0097] The evaluation processing unit F5 performs performance evaluation on the edge model that has been retrained by the edge model retraining processing unit F4. Specifically, in this example, it calculates numerical evaluation information for the retrained edge model, as illustrated in Figure 10 above.

[0098] The deployment processing unit F6 deploys the edge model, which has been retrained by the edge model retraining processing unit F4, to the edge device. In this example, the deployment processing unit F6 performs the process of deploying the edge model specified (selected) in the deployment management screen G6, as illustrated in Figure 11, to the edge device (imaging device 2 in this example) also specified (selected) in the deployment management screen G6. In other words, the deployment processing unit F6 in this example deploys the edge model retrained by the edge model retraining processing unit F4 to the edge device according to the edge model specified in the deployment management screen G6.

[0099] Referring to Figure 13, we will explain the details of the retraining process for edge models when "classes are added," from the annotation processing unit F1 to the edge model retraining processing unit F4. In the figure, the training image set represents multiple training images uploaded by the user to server device 1. In this example, this corresponds to the images captured by each imaging device 2 that were set as upload targets in the upload / training type setting screen G4 as exemplified in Figure 7. For clarification, it should be noted that the training image set used for retraining with "class addition enabled" is assumed to include images in which objects of the class to be added are visible.

[0100] First, as shown in Figure 13A, the annotation processing unit F1 performs annotation processing on the training image set using a large-scale AI model. Here, the large-scale AI model used is a large-scale AI model that performs object detection processing and is configured to output region information indicating the object detection area even for objects whose class cannot be identified. Hereafter, a large-scale AI model that satisfies these conditions will be referred to as "Large-scale AI model ML". In this example, the large-scale AI model ML is assumed to have been trained to be able to class-identify objects of all classes that the edge model being retrained is intended to identify. For illustrative purposes, this example assumes that the edge model to be retrained is an AI model capable of identifying various products as a class. Other examples of edge models include AI models capable of identifying various individuals (age, gender, etc.).

[0101] In the annotation process, annotation information is generated for each training image using class identification results from a large-scale AI model (ML). The annotation information for each image includes information indicating the region of the detected object (e.g., information indicating the position and extent of the bounding box) and information indicating the identified class name.

[0102] Next, the reception processing unit F2 accepts corrections to the annotation information from the user.

[0103] Figure 14 illustrates a correction request screen G7 displayed by the reception processing unit F2 to accept corrections to annotation information from the user. In this example, the correction request screen G7 is displayed on the display unit of the user terminal 3 after the annotation processing by the annotation processing unit F1 has been completed following the execution of a retraining process with "class additions" for the edge model.

[0104] As shown in the diagram, the correction request screen G7 is provided with a target image display area Ap that displays the image to be processed, and a candidate image display area Ac that displays a list of thumbnail images of candidate images to be processed, i.e., images used as training images. When an image is selected from the candidate image display area Ac, the selected image is enlarged and displayed in the target image display area Ap.

[0105] In the target image display area Ap, annotation information for the image selected from the candidate image display area Ac is displayed. Specifically, the bounding box Bb of the object detected in the image and the class name display box Bc indicating the class name of the object are displayed.

[0106] Figure 14 illustrates annotation information when the large-scale AI model ML is unable to classify one of the objects detected in the selected image. In this example, the large-scale AI model ML outputs information about the bounding box Bb of the object for which class identification was not possible, as well as information indicating that the class name was unknown, such as "TargetX" as exemplified in the figure.

[0107] The user corrects the class name information for annotation information where the class name is indicated as "unknown". Specifically, in this example, as illustrated in Figure 15, the user enters the correct class name information (text information) into the corresponding class name display box Bc.

[0108] When a class name modification operation is performed in this manner, the reception processing unit F2 associates the modified class name information with the class name information of the bounding box Bb that was modified. This completes the modification of the annotation information.

[0109] The above example illustrates the correction of class name information for objects whose class identification was impossible. However, when a large-scale AI model (ML) performs object detection on an object of a new class, it is possible that it may mistakenly identify that object as belonging to an existing class. In that case, the annotation information for that object would show the incorrect class name. Users are required to correct the incorrectly assigned class names as part of the annotation information correction process on the correction request screen G7.

[0110] Since the training image set is a massive collection of images, having the user correct all of the class names that were unidentifiable or incorrectly recognized, as described above, would place a significant burden on the user and is therefore undesirable.

[0111] Therefore, the reception processing unit F2 in this example has the following functions. Specifically, when the reception processing unit F2 in this example receives a request from the user to modify a class name, it performs an automatic setting process to set the class name of the object whose modification was received as the class name of another object whose features are similar to those of the object in question. Here, "features" refers to values ​​that represent the characteristics of the object, calculated in the inference process of the large-scale AI model ML. Furthermore, "similar features" means that the error between the features is within a predetermined range.

[0112] In this case, the user's correction work is assumed to involve correcting a portion of objects, such as a dozen to several dozen, whose class names were unidentifiable or incorrectly identified. The reception processing unit F2 then performs the above-mentioned automatic setting process using the results of the class name corrections made for these portions of objects.

[0113] In this example, as a result of the automatic setting process performed by the reception processing unit F2, corrected annotation information is obtained in which the correct class name information is associated with the object of the new class.

[0114] In response to obtaining this corrected annotation information, the large-scale AI model ML is retrained by the large-scale model retraining processing unit F3, as shown in Figure 13B. This large-scale AI model ML retraining process is performed using supervised learning, where the training image set is used as training input images and the corrected annotation information is used as training data. This retraining process allows us to obtain a large-scale AI model (ML) capable of class-identifying objects of a new class. The large-scale AI model ML, which has been trained through the retraining process performed by the large-scale model retraining processing unit F3, will be referred to below as "Retrained Large-Scale AI Model MLd".

[0115] Next, as shown in Figure 13C, the edge model retraining process is performed by the edge model retraining processing unit F4. Here, the edge model (hereinafter referred to as "edge model ME") selected as the "target model" (here, the model to be retrained) in the job setting screen G3 of Figure 6 is retrained using knowledge distillation with the retrained large-scale AI model MLd as the training model. Specifically, the edge model retraining processing unit F4 has an error transmission unit F41, which calculates the error between the output of the retrained large-scale AI model MLd when given training images as input and the output of edge model ME when given the same training images as input. The calculated error is then transmitted to edge model ME to retrain edge model ME. In this context, the "output" of the AI ​​model refers to the likelihood information for each class. In other words, the example given above illustrates how knowledge distillation can transmit the error between these class-specific likelihoods (the so-called "soft target loss"). Furthermore, in knowledge distillation, it is also possible to transmit the error (the so-called "hard target loss") between the ground truth data obtained as an annotation result and the class identification result (information indicating the class as an inference result).

[0116] Through the retraining process performed by the edge model retraining processing unit F4 as described above, a new edge model capable of identifying new classes can be obtained as the edge model ME.

[0117] Here, we have described the retraining method when additional classes are added. However, if the edge model retraining processing unit F4 is instructed to perform a retraining process without adding classes, in other words, a retraining process that does not require the addition of identifiable classes, it will perform the edge model retraining process using machine learning with the annotation results from the annotation processing unit F1 as training data.

[0118] Figure 16 is an explanatory diagram of the retraining method for edge model ME without adding classes. As shown in Figure 16A, even without adding classes, the annotation processing unit F1 uses the large-scale AI model ML to perform annotation processing on the image set for retraining and obtain annotation information.

[0119] If no classes are added, the edge model retraining processing unit F4 performs retraining of the edge model ME using machine learning with the training image set as training input images and the annotation information as training data, as shown in Figure 16B. This makes it possible to retrain the edge model ME to adapt to so-called domain changes, such as the aforementioned changes in background and lighting conditions.

[0120] As explained earlier with reference to Figures 5 and 8, in this example of an information processing system, the performance evaluation process of the retrained edge model ME is performed in response to the operation of the result display button B4 on the edge model management top screen G2. This performance evaluation process is performed by the evaluation processing unit F5 mentioned above. The performance evaluation process involves having the target edge model perform an inference process using verification images (for example, images selected from the training image set that were not used for retraining) as input, and then evaluating the results based on the output. This requires an AI processor to run the edge model, and this AI processor can be a virtual processor as a simulator, or it can be a physical processor. Here, "physical device" refers to a physical processor.

[0121] In this example, the evaluation processing unit F5 uses an actual processor as an AI processor used for performance evaluation of the edge model, that is, an AI processor capable of executing object detection processing using the edge model. By performing performance evaluations using an actual AI processor, it is possible to perform more accurate performance evaluations than when using a virtual AI processor as a simulator.

[0122] Here, the AI ​​processor used by the evaluation processing unit F5 for performance evaluation may be located within the server device 4, or it may be located in an external device capable of communicating with the server device 4.

[0123] <4. Processing Procedure> Figure 17 is a flowchart showing an example of specific processing steps that the server device 1 should execute to implement the retraining method as described above. In this example, the process shown in Figure 17 is executed by the processor 11 based on a program stored in a predetermined storage device, such as the ROM 12 or the memory unit 19.

[0124] In Figure 17, the processor 11 waits for an instruction to retrain the edge model in step S101. Specifically, it waits until the execute button B10 is operated on the upload / learning type setting screen G4 (Figure 7).

[0125] In step S101, if it is determined that the execute button B10 was operated and an instruction to retrain the edge model was given, the processor 11 proceeds to step S102 to determine whether or not a class should be added. In other words, in this example, it is determined whether or not the execute button B10 was operated with the "Add class" checkbox cb checked on the upload / learning type setting screen G4.

[0126] If it is determined in step S102 that a class has been added, the processor 11 executes the processes from steps S103 to S107. First, in step S103, the processor 11 performs annotation processing on the training image set using the large-scale AI model ML, and then in the following step S104, it performs correction acceptance processing. That is, it displays the correction acceptance screen exemplified in Figure 14 on the display unit of the user terminal 3 to accept corrections from the user regarding the annotation information obtained in the annotation processing. As explained earlier, in this example, the user is only allowed to correct some of the objects that require correction.

[0127] In step S105, following step S104, the processor 11 performs an automatic class name setting process based on the correction result. That is, as explained earlier, it sets the class name of the object whose class name correction has been accepted from the user as the class name of another object whose features are similar to those of the object in question.

[0128] In step S106, following step S105, processor 11 performs retraining of the large-scale AI model ML using the corrected annotation information as training data. Specifically, it uses the corrected annotation information, which reflects the class name corrections received from the user in step S104 and the automatic setting of class names in the automatic setting process in step S105, as training data, and the training image set as training input images to perform retraining of the large-scale AI model ML. This results in the retrained large-scale AI model MLd.

[0129] In step S107, following step S106, the processor 11 performs a retraining process on the selected edge model ME using knowledge distillation with the retrained large-scale AI model MLd as the training model. That is, it performs the retraining process on the edge model ME using the method described in Figure 13C above.

[0130] Furthermore, if the processor 11 determines in step S102 that there is no class addition, it proceeds to step S108. In step S108, processor 11 performs annotation processing on the training image set using the large-scale AI model ML. Then, in the following step S109, processor 11 performs retraining processing on the edge model ME using the annotation information as training data. That is, the annotation information obtained in the annotation processing in step S108 is used as training data, and the training image set is used as training input images to retrain the edge model ME using machine learning.

[0131] The processor 11 completes the series of processes shown in Figure 17 depending on whether it has performed the process in step S107 or S109.

[0132] Figure 18 is a flowchart of the process for evaluating the retrained edge model ME. In step S201, processor 11 waits for an evaluation instruction. Specifically, on the edge model management top screen G2, when the retraining process for the edge model ME is completed, it waits until the result display button B4 displayed for the process information of the edge model ME is operated.

[0133] In step S201, if the result display button B4 is operated as described above and it is determined that an evaluation instruction has been given, the processor 11 proceeds to step S202 and executes the evaluation process. Specifically, in this example, it performs the calculation of evaluation information using the aforementioned numerical values ​​and the generation of image preview information.

[0134] In step S203, following step S202, the processor 11 performs the process of presenting the evaluation results. Specifically, it performs the process of displaying the evaluation results screen G5 on the display unit of the user terminal 3, and presents the user with the evaluation information using the numerical values ​​and image preview information described above.

[0135] In response to having performed the process in step S203, the processor 11 completes the series of processes shown in Figure 18.

[0136] Figure 19 is a flowchart of the process involved in deploying the retrained edge model ME. In step S301, the processor 11 waits for a deployment command. That is, it waits until the deploy button B14 on the deployment management screen G6 (Figure 11) is operated.

[0137] If the deploy button B14 is operated and it is determined that a deployment instruction has been given, the processor 11 proceeds to step S302 and performs the process of deploying the specified edge model ME to the specified imaging device 2. That is, it performs the process of deploying the edge model ME specified by the selection input operation to input box bi on the deployment management screen G6 to the imaging device 2 specified by the same selection input operation to input box bi. This makes it possible to use the retrained edge model ME in the imaging device 2.

[0138] In response to having performed the process in step S302, the processor 11 completes the series of processes shown in Figure 19.

[0139] <5. Variation> Although embodiments of this technology have been described above, the embodiments are not limited to the specific examples described above, and various modified configurations can be adopted. For example, the above example shows that server device 1 performs the edge model retraining process, but it is also possible that the edge model retraining process is performed on an edge device other than imaging device 2, such as the fog server mentioned above. As mentioned earlier, it is also possible to adopt a configuration where an edge device separate from the imaging device 2, such as a fog server, performs object detection processing using an edge model. It is also conceivable that such an edge device would perform retraining processing for the edge model. In that case, the object detection processing using the edge model and the retraining processing would be performed in the same device.

[0140] Furthermore, while the above example shows server device 1 handling the processing from receiving the upload of images for retraining to retraining the edge model, and the deployment of the retrained edge model, it is also conceivable to adopt a configuration in which the processing from receiving the upload of images for retraining to retraining the edge model and the deployment of the retrained edge model are handled by separate devices.

[0141] <6. Summary of Embodiments> As described above, the information processing device (server device 1) as an embodiment comprises: an annotation processing unit (F1) that performs annotation processing on an input image using a large-scale AI model (ML) that performs object detection processing and is configured to output region information indicating an object detection region even for objects that cannot be classified as a class; a reception processing unit (F2) that accepts corrections from the user regarding the annotation results from the annotation processing unit; a large-scale model retraining processing unit (F3) that performs retraining processing on the large-scale AI model using the annotation information corrected by the user; and an edge model retraining processing unit (F4) that performs retraining processing on an edge model, which is an AI model used in an edge device that performs object detection processing on captured images, by knowledge distillation using the retrained large-scale AI model (retrained large-scale AI model MLd) as a training model. By having the reception processing unit described above, it becomes possible to have the user set the correct class name information for objects that the large-scale AI model considered unclassifiable or incorrectly classified before relearning. Then, by relearning the large-scale AI model using the corrected annotation information as described above, it becomes possible to obtain a large-scale AI model that can identify objects of new classes (objects that were considered unclassifiable or incorrectly classified before relearning). Furthermore, by relearning the edge model through knowledge distillation using this relearned large-scale AI model as the training model, it becomes possible to obtain an edge model that can identify objects of new classes. In other words, it becomes possible to retrain the AI ​​model used in object detection processing on edge devices to add classes that can be identified. Furthermore, with the above configuration, in order to retrain the AI ​​model used for object detection processing on edge devices to add identifiable classes, the user only needs to modify the annotation results by the annotation processing unit. In addition, since the retraining of the edge model is performed by knowledge distillation using a large-scale AI model that has been retrained to identify the new classes, it becomes possible to obtain an edge model capable of high-precision class identification. From these points, according to this embodiment, when retraining an AI model used in object detection processing on edge devices to add identifiable classes, it is possible to improve the class identification accuracy of the AI ​​model while reducing the workload on the user.

[0142] Furthermore, in the information processing device as an embodiment, when a user requests a modification of a class name, the reception processing unit performs an automatic setting process to set the class name of the object whose modification has been received as the class name of another object whose features are similar to those of the object in question. The large-scale model retraining processing unit then uses the annotation information, for which the class name has been set by the automatic setting process, to retrain the large-scale AI model. With the above configuration, if the user modifies the class names of only some of the objects that require class name modification, the same class names will be automatically applied to the other objects that also require modification. Therefore, it is possible to further reduce the workload on users required for relearning.

[0143] Furthermore, the information processing device as an embodiment includes an evaluation processing unit that performs performance evaluation processing on the edge model retrained by the edge model retraining processing unit. This allows us to obtain performance evaluation information that enables users to confirm whether or not the retraining was performed properly.

[0144] Furthermore, in the information processing device as an embodiment, the evaluation processing unit performs performance evaluation processing using an AI processor, which is a real machine capable of executing object detection processing using an edge model. By performing performance evaluations using an actual AI processor, it is possible to perform more accurate performance evaluations than when using a virtual AI processor as a simulator.

[0145] Furthermore, the information processing device as an embodiment includes a deployment processing unit (F6) that deploys the edge model retrained by the edge model retraining processing unit to an edge device. This makes it possible to use the retrained edge model on edge devices. Therefore, the edge model used by the edge device can be adapted to changes in the field environment.

[0146] Furthermore, in the information processing device as an embodiment, if the edge model retraining processing unit is instructed to perform a retraining process that does not require the addition of identifiable classes as the edge model retraining process, it performs the edge model retraining process using machine learning with the annotation results from the annotation processing unit as training data. This makes it possible to retrain edge models to adapt to domain changes such as changes in lighting conditions or background objects at the site. Retraining edge models allows for the selective execution of learning tasks with different objectives, thereby improving user convenience.

[0147] Furthermore, in the information processing device as an embodiment, the edge model retraining processing unit performs retraining processing on the edge model selected by the user. This allows users to perform retraining only on the necessary edge models when they have multiple edge models, thereby improving convenience.

[0148] An information processing method as an embodiment involves an information processing device using a large-scale AI model that performs object detection processing, configured to output region information indicating the object detection region even for objects that cannot be classified as a class, to perform annotation processing on an input image, to accept corrections from the user to the annotation results obtained from the annotation processing, to retrain the large-scale AI model using the annotation information corrected by the user, and to retrain an edge model, which is an AI model used in an edge device that performs object detection processing on captured images, by knowledge distillation using the retrained large-scale AI model as the training model. This information processing method can also provide the same functions and effects as the information processing apparatus described in the above-described embodiment.

[0149] Here, as an embodiment, we can consider a program that implements the process described above with reference to Figure 17, etc., on, for example, a CPU, DSP, or a device including these. In other words, the program of the embodiment is a program readable by a computer device, and is a large-scale AI model that performs object detection processing, and is configured to output region information indicating the object detection region even for objects that cannot be classified into a specific class, and it performs annotation processing on an input image, accepts corrections from the user to the annotation results from the annotation processing, performs retraining processing of the large-scale AI model using the annotation information corrected by the user, and performs retraining processing of an edge model, which is an AI model used in an edge device that performs object detection processing on captured images, by knowledge distillation using the retrained large-scale AI model as the training model. Such a program allows the functions of the server device 1 as described above to be realized in a computer device.

[0150] Such programs can be pre-recorded in storage media such as HDDs and SSDs built into computer devices, or in ROMs within microcomputers with CPUs. Alternatively, the data can be temporarily or permanently stored (recorded) on removable recording media such as flexible disks, CD-ROMs (Compact Disc Read Only Memory), MO (Magneto Optical) disks, DVDs (Digital Versatile Discs), Blu-ray Discs (registered trademark), magnetic disks, semiconductor memory, and memory cards. Such removable recording media can be provided as so-called packaged software. In addition to installing such programs from removable storage media to personal computers, they can also be downloaded from download sites via networks such as LANs and the Internet.

[0151] Furthermore, such a program is suitable for providing a wide range of relearning methods as embodiments. Various forms of computer devices can be made to function as devices for implementing the relearning methods of this disclosure.

[0152] Furthermore, the effects described herein are merely illustrative and not limited to those described herein, and other effects may also occur.

[0153] <7. This Technology> Furthermore, this technology can also be configured as follows. (1) An annotation processing unit that performs annotation processing on an input image using a large-scale AI model that performs object detection processing and is configured to output region information indicating the object detection region even for objects whose class cannot be identified, Regarding the annotation results from the annotation processing unit, a receiving processing unit accepts corrections from the user, A large-scale model retraining processing unit performs retraining of the large-scale AI model using annotation information modified by the user, The system comprises an edge model retraining processing unit that performs retraining of an edge model, which is an AI model used in edge devices to perform object detection processing on captured images, by performing knowledge distillation using the large-scale AI model after retraining as the training model. Information processing device. (2) The aforementioned reception processing unit, When a user submits a class name modification request, an automatic configuration process is performed to set the modified object's class name as the class name of another object whose features are similar to those of the modified object. The aforementioned large-scale model retraining processing unit is The retraining process for the large-scale AI model is performed using the annotation information for which the class name has been set by the aforementioned automatic configuration process. The information processing device described in (1) above. (3) The system includes an evaluation processing unit that performs performance evaluation processing on the edge model retrained by the edge model retraining processing unit. The information processing device described in (1) or (2) above. (4) The evaluation processing unit is, The performance evaluation process is performed using an AI processor, which is a real machine capable of executing object detection processing using the aforementioned edge model. The information processing device described in (3) above. (5) The system includes a deployment processing unit that deploys the edge model retrained by the edge model retraining processing unit to the edge device. An information processing device as described in any of (1) to (4) above. (6) The edge model retraining processing unit, If the retraining process for the edge model is instructed to be one that does not require the addition of identifiable classes, the retraining process for the edge model will be performed using machine learning with the annotation results from the annotation processing unit as training data. An information processing device as described in any of (1) to (5) above. (7) The edge model retraining processing unit, The retraining process is executed on the edge model selected by the user. An information processing device as described in any of (1) to (6) above. (8) Information processing device, An annotation process is performed on an input image using a large-scale AI model that performs object detection processing and is configured to output region information indicating the object detection area even for objects whose class cannot be identified. The annotation results obtained through the aforementioned annotation process can be modified by the user. Using annotation information modified by the user, the large-scale AI model is retrained. The retraining process for the edge model, which is an AI model used in edge devices to perform object detection processing on captured images, is performed by knowledge distillation using the aforementioned large-scale AI model after retraining as the training model. Information processing methods. [Explanation of Symbols]

[0154] 1 Server device 2. Imaging device 3. User terminals 41 Imaging Optical System 42 Image Sensors 54 AI Processing Unit 43 Optical System Drive Unit 44 Camera Control Unit 45 Memory section 46 Communications Department 11 processors 12 ROM 13 RAM 17 Display 19 Memory section 20 Communications Department F1 Annotation Processing Unit F2 Reception Processing Unit F3 Large-scale model retraining processing unit F4 Edge Model Retraining Processing Unit F41 Error transmission section F5 Evaluation Processing Unit F6 Deployment Processing Unit ML Large-scale AI models ME Edge Model MLd Retrained Large-Scale AI Model

Claims

1. An annotation processing unit performs annotation processing on an input image using a large-scale AI model that performs object detection processing and is configured to output region information indicating the object detection region even for objects whose class cannot be identified. Regarding the annotation results from the annotation processing unit, a receiving processing unit accepts corrections from the user, A large-scale model retraining processing unit performs retraining of the large-scale AI model using annotation information modified by the user, The system comprises an edge model retraining processing unit that performs retraining of an edge model, which is an AI model used in an edge device to perform object detection processing on captured images, by knowledge distillation using the large-scale AI model after retraining as the training model. Information processing device.

2. The aforementioned reception processing unit, When a user submits a class name modification request, an automatic configuration process is performed to set the modified object's class name as the class name of another object whose features are similar to those of the modified object. The aforementioned large-scale model retraining processing unit is The retraining process for the large-scale AI model is performed using the annotation information, which has been set as a class name through the aforementioned automatic configuration process. The information processing apparatus according to claim 1.

3. The system includes an evaluation processing unit that performs performance evaluation processing on the edge model retrained by the edge model retraining processing unit. The information processing apparatus according to claim 1.

4. The evaluation processing unit is, The performance evaluation process is performed using an AI processor, which is a real machine capable of executing object detection processing using the edge model described above. The information processing apparatus according to claim 3.

5. The system includes a deployment processing unit that deploys the edge model retrained by the edge model retraining processing unit to the edge device. The information processing apparatus according to claim 1.

6. The edge model retraining processing unit, If the retraining process for the edge model is instructed to be one that does not require the addition of identifiable classes, the retraining process for the edge model will be performed using machine learning with the annotation results from the annotation processing unit as training data. The information processing apparatus according to claim 1.

7. The edge model retraining processing unit, The retraining process is executed on the edge model selected by the user. The information processing apparatus according to claim 1.

8. Information processing device, An annotation process is performed on an input image using a large-scale AI model that performs object detection processing and is configured to output region information indicating the object detection area even for objects whose class cannot be identified. The annotation results obtained through the aforementioned annotation process can be modified by the user. Using annotation information modified by the user, the large-scale AI model is retrained. The large-scale AI model, after retraining, is used as the training model for knowledge distillation, thereby retraining the edge model, which is an AI model used in edge devices that perform object detection processing on captured images. Information processing methods.

Images