Learning image selection / analysis system and method

Abstract

The objective of the present invention is to provide a technique enabling efficient improvement of inference accuracy in a system for detecting an object by using a teacher AI model and an on-site AI model.　To this end, a learning image selection / analysis system according to the present invention comprises: an on-site analysis server that selects a learning image from input images acquired by an image acquisition unit; and an annotation / AI model learning server that annotates the learning image and trains an object detection AI model. The learning image selection / analysis system is characterized in that a plurality of on-site analysis servers are installed, and one on-site analysis server includes: an object detection unit that performs object detection using the on-site AI model; a detection difficulty calculation unit that calculates a detection difficulty of the teacher AI model and the on-site AI model with respect to the input image; and a learning image selection unit that automatically selects a learning image for each object detection AI model by using an output result of the detection difficulty calculation unit.

Description

System and method for selecting and analyzing images for learning

[0001] The present invention relates to a system and method for selecting and analyzing training images.

[0002] Deep Learning (DL) and other AI technologies are increasingly being applied to tasks such as object detection and attribute classification using images and videos as input. Incorporating AI into products and systems can replace human work such as monitoring, reduce labor costs, and increase the added value of products, greatly contributing to the sales promotion of products and systems. Many companies are therefore highly interested in AI.

[0003] Known object detection methods using DL include, for example, a technique for determining the type and position of a target object from an input image using a neural network. Generally, supervised learning is used in machine learning for object detection. Supervised learning trains an AI model to output a correct output signal from input data using a set of pairs of input data for a certain problem and an expected output signal as a result of inputting the data. In this case, the pair of input data and expected output is referred to as training data or a training dataset.

[0004] In object detection, there are various phenomena that can cause a decrease in inference accuracy (detection accuracy), such as differences in the shooting environment, background, angle of view, time of day, etc. when collecting learning data, the presence of objects not included in the learning data, etc. For example, Patent Document 1 discloses a technology to address the risk of a detection target object being overlooked due to a decrease in inference accuracy caused by a detection target object being partially obscured by another object, by distinguishing between an "event occurrence" state in which an image processing unit determines that a detection target object is present in an image based on detection conditions for the detection target object, a "no event" state in which a detection target object is determined not to be present in the image based on non-detection conditions for the detection target object, and a "neutral" state in which a detection target object is determined to be occluded using correction conditions, and by setting a condition for transitioning from the "neutral" state to the "event occurrence" state to be looser than the condition for transitioning from the "no event" state to the "event occurrence" state.

[0005] Patent No. 7478306

[0006] To address the issue of reduced inference accuracy due to differences in the shooting environment, such as weather and time of day, between the training dataset and the input images during operation, it is generally desirable to acquire training data in a variety of environments and train the AI ​​model with a diverse dataset. DL and other AI model training methods allow for the construction of a general-purpose AI model that can make correct inferences for a variety of input data by training with a large amount of input data.

[0007] However, when training a general-purpose AI model using a diverse training dataset, the AI ​​model requires a massive amount of computation to extract and analyze numerous features, necessitating high-performance computers for operation. Furthermore, there are cases where high-performance computers are not available due to limitations on power consumption and installation space. A commonly used method for collecting training data involves sending data acquired in the field environment using sensors such as cameras to a high-performance, high-capacity computer and manually selecting data to be trained from the collected data. However, this method has several challenges: sending all data acquired in the field environment to the training environment increases communication traffic; the data to be trained must be carefully selected manually from the vast amount of data; and data bias occurs due to differences in the data that can be collected at each site. AI model development requires the processes of collecting training data, creating a training dataset (annotation), training, and verification. However, training data collection and annotation require manual work and account for the majority of the AI ​​model development effort. Therefore, efficient training data collection and annotation are essential for efficient AI model development.

[0008] Therefore, it is expected that object detection will be performed efficiently by combining a general-purpose AI model (also called a "teacher AI model") that requires a huge amount of calculation and manual work with an AI model (also called an "on-site AI model" or "student AI model") that requires a small amount of calculation and is suitable for operation on the field (or edge side). However, there is an issue with on-site AI models in that their inference accuracy decreases when unlearned images are input in the field environment where they are used.

[0009] Furthermore, a technique called automatic annotation is sometimes used to improve the efficiency of annotation. Automatic annotation is a method in which the inference results of data acquired in a field environment by a large-scale, general-purpose AI model (teacher AI model) trained on a diverse data set are used as training data. Automating annotation allows for efficient training of an AI model (field AI model) with low computational load. However, automatic annotation presents a challenge in that even the teacher AI model may miss or misdetect something, requiring careful manual checking. Patent Document 1 does not disclose any recognition of the issues regarding the collection of training data or the efficiency of annotation as described above.

[0010] Therefore, an object of the present invention is to provide a technology that enables efficient improvement of inference accuracy in a system that performs object detection using a teacher AI model and an on-site AI model.

[0011] In order to solve the above problems, one representative training image selection and analysis system of the present invention is a training image selection and analysis system that includes an on-site analysis server that selects training images from input images acquired by an image acquisition unit, and an annotation / AI model learning server that performs annotations on the training images to train an object detection AI model, wherein multiple on-site analysis servers are installed, and each on-site analysis server has an object detection unit that performs object detection using an on-site AI model, a detection difficulty calculation unit that calculates the detection difficulty of a teacher AI model and the on-site AI model for the input image, and a training image selection unit that automatically selects training images for each object detection AI model using the output results of the detection difficulty calculation unit.

[0012] According to the present invention, it is possible to provide a technology that efficiently improves inference accuracy in a system that performs object detection using a teacher AI model and an on-site AI model. Problems, configurations, and effects other than those described above will become clear from the description of the following embodiments of the invention.

[0013] FIG. 1 is a diagram illustrating an example of the configuration of a learning image selection and analysis system according to this embodiment. FIG. 2 is a diagram illustrating an example of the configuration of an on-site analysis server according to this embodiment. FIG. 3 is a diagram illustrating an example of the configuration of an annotation / AI model learning server according to this embodiment. FIG. 4 is a diagram illustrating an example of a learning image selection process according to this embodiment. FIG. 5 is a diagram illustrating an example of a correction process according to this embodiment. FIG. 6 is a diagram illustrating an outline of an example of a series of operations of the on-site analysis server according to this embodiment. FIG. 7 is a diagram illustrating an example of annotation start determination according to this embodiment. FIG. 8 is a diagram illustrating an outline of an example of a series of operations of the annotation / AI model learning server according to this embodiment. FIG. 9A is an example of a flowchart of the processing procedure of the learning image selection and analysis system according to this embodiment. FIG. 9B is an example of a flowchart of the processing procedure of the learning image selection and analysis system according to this embodiment.

[0014] Hereinafter, embodiments of the present invention will be described with reference to the drawings. However, the present invention is not limited to these embodiments. In the drawings, identical parts are denoted by the same reference numerals. When multiple components having the same or similar functions are present, they may be described using the same reference numerals with different subscripts. Furthermore, when it is not necessary to distinguish between these multiple components, the subscripts may be omitted. Furthermore, although terms such as "first," "second," and "third" may be used to describe various elements or components in this disclosure, it should be understood that these elements or components should not be limited by these terms. These terms are used only to distinguish one element or component from another. Therefore, a first element or component discussed below may also be referred to as a second element or component without departing from the teachings of the inventive concept. The position, size, shape, range, etc. of each component shown in the drawings may not represent the actual position, size, shape, range, etc., in order to facilitate understanding of the invention. Therefore, the present invention is not necessarily limited to the position, size, shape, range, etc. disclosed in the drawings.

[0015] <Learning Image Selection and Analysis System> First, the overall configuration of a learning image selection and analysis system according to an embodiment of the present invention will be described. FIG. 1 is a diagram showing an example configuration of a learning image selection and analysis system according to this embodiment. The learning image selection and analysis system 100 includes multiple on-site analysis servers 101 and cameras 102 installed on the on-site side (edge ​​side) at multiple sites, and an annotation / AI model training server 103 installed at a center side (when referring to an on-site analysis server or camera installed at a given site, it may be referred to as site n, on-site n analysis server, or camera n, where n: 1 to N, and N is an integer of 2 or greater). The on-site analysis server 101 and the annotation / AI model training server 103 are connected via a network 104 that allows mutual communication. The camera 102 is installed at a position (on-site) where it can capture a target image. The on-site analysis server 101 is also installed on-site, but it need only be connected to the camera 102 so as to be able to communicate with it; it does not necessarily have to be installed integrally with the camera 102. The camera 102 may be connected to the network 104 directly or indirectly via the site analysis server 101. The site analysis server 101 is equipped with a site AI model that performs object detection in the site (edge) environment. The annotation / AI model learning server 103 is equipped with a large-scale, highly versatile teacher AI model. In this disclosure, when there is no need to particularly distinguish between the teacher AI model and the site AI model, they may be collectively referred to as the object detection AI model, and when there is no need to particularly distinguish between multiple site AI models (site 1 to N AI models), they may be collectively referred to as the site AI model.

[0016] The on-site analysis server 101 and the annotation / AI model learning server 103 are configured by a computer system equipped with a CPU, but may have other configurations. For example, they may be equipped with a digital signal processor (DSP), a field-programmable gate array (FPGA), a graphics processing unit (GPU), etc. The network 104 may be any type of line, such as a dedicated line, an intranet, or an Internet IP network, as long as they are capable of communicating with each other.

[0017] <Camera> The camera 102 is an imaging device that captures a target image, and can be configured to obtain information by focusing incident light onto an imaging element via a lens and an aperture. Examples of the imaging element include a CCD (Charge-Coupled Device) image sensor and a CMOS (Complementary Metal Oxide Semiconductor) image sensor. The camera 102 captures video at, for example, 3 frames per second (3 fps) or more, and transmits the information to the site analysis server 101. Note that image information from camera n is transmitted to the site n analysis server (n: 1 to N), but can also be transmitted to other site analysis servers via the network 104. Furthermore, multiple cameras n may be installed for the same site n analysis server.

[0018] <On-site analysis server> Next, the on-site analysis server will be described. Fig. 2 is a diagram showing an example of the configuration of the on-site analysis server according to this embodiment. The on-site analysis server 101 includes an image acquisition unit 201, an object detection unit 202, a detection difficulty calculation unit 203, a learning image selection unit 204, an image selection result correction unit 205, a first output control unit 206, an image selection condition setting unit 207, and a first auxiliary storage unit 208. Each functional unit will be described below.

[0019] (Image Acquisition Unit) The image acquisition unit 201 acquires an input image from a first auxiliary storage unit 208 (described later). The input image may be in a data format such as RGB color, YUV, or monochrome depending on the application. Furthermore, the image may be reduced to a predetermined size to reduce processing costs.

[0020] (Object Detection Unit) The object detection unit 202 detects a target object included in an input image acquired by the image acquisition unit 201 using an on-site AI model installed as an object detection AI model.

[0021] (Detection Difficulty Calculation Unit) The detection difficulty calculation unit 203 calculates the detection difficulty of an object detection AI model for the input image acquired by the image acquisition unit 201. Object detection AI models for which the detection difficulty is to be determined include the on-site AI model used by the object detection unit 202, on-site AI models used by other on-site analysis servers, and teacher AI models used in automatic annotation by the annotation / AI model learning server 103. A detection difficulty evaluation AI model, described later, can be used to calculate the detection difficulty.

[0022] (Learning image selection unit) The learning image selection unit 204 determines whether the detection difficulty of the target object detection AI model for the input image satisfies the image selection conditions set by the image selection condition setting unit 207 (described later) in accordance with the output of the detection difficulty calculation unit 203. If the conditions are satisfied, the learning image selection unit 204 selects the input image as an image to be trained by the object detection AI model.

[0023] (Image selection result correction unit) The image selection result correction unit 205 corrects an image selected by the learning image selection unit 204 so that it is not selected as a learning image if it is similar to the learning image selected immediately before. As a specific example, as will be described later, the similarity is determined based on the object detection result output by the on-site AI model in the object detection unit 202 for the input image.

[0024] (First Output Control Unit) The first output control unit 206 outputs the selected learning images to the annotation / AI model learning server 103 based on the image selection results output by the learning image selection unit 204 or the image selection result correction unit 205. It can also output related information such as the input image, object detection results, image selection result information, and image selection conditions. It is also possible to output similar output information to the user on a GUI. It may also be possible to output a result notification and related information regarding input images that were not selected to the user.

[0025] (Image Selection Condition Setting Unit) The image selection condition setting unit 207 sets new image selection conditions to be used by the learning image selection unit 204 when the on-site analysis server 101 is started up or when an updated detection difficulty evaluation AI model is introduced in the annotation / AI model learning server 103. However, the timing of setting is not limited to this. In addition, the method of setting the image selection conditions can be selected from multiple methods, such as setting the maximum and minimum values ​​of the detection difficulty for several reference images as an upper threshold (first threshold) and a lower threshold (second threshold), respectively, for each object detection AI model.

[0026] (First Auxiliary Storage Unit) The first auxiliary storage unit 208 is configured, for example, by a non-volatile memory such as an HDD or flash memory, and stores programs executed by the on-site analysis server 101, data to be processed by the programs, etc. Such data includes input images captured by the camera 102 and information received from the annotation / AI model learning server 103.

[0027] <Annotation / AI Model Learning Server> Next, the annotation / AI model learning server will be described. Fig. 3 is a diagram showing an example of the configuration of the annotation / AI model learning server according to this embodiment. The annotation / AI model learning server 103 includes a teacher AI model annotation start determination unit 301, a teacher AI model annotation unit 302, a teacher AI model learning unit 303, a site AI model annotation start determination unit 304, a site AI model annotation unit 305, a site AI model learning unit 306, a second output control unit 307, and a second auxiliary memory unit 308. Each functional unit will be described below.

[0028] (Teacher AI model annotation start determination unit) The teacher AI model annotation start determination unit 301 acquires information such as the training images selected for training the teacher AI model, the detection difficulty for the training images, the acquisition date and time, and the training state (whether the images have been trained or not) from the second auxiliary memory unit 308 (described later). The training images for the teacher AI model may be selected in a data format such as RGB color, YUV, or monochrome depending on the application. Furthermore, the images may be reduced to a predetermined size to reduce processing costs. The teacher AI model annotation start determination unit 301 determines whether to start annotation for the teacher AI model depending on the frequency and number of training images selected for the teacher AI model, and whether the images are in an untrained state.

[0029] (Teacher AI model annotation unit) The teacher AI model annotation unit 302 performs annotation on the training images of the selected teacher AI model to create a training dataset. The annotation may be performed entirely manually, or a method may be used in which manual corrections are made to the results of automatic annotation by the teacher AI model.

[0030] (Teacher AI model learning unit) The teacher AI model learning unit 303 performs re-learning of the teacher AI model and the detection difficulty evaluation AI model corresponding to the teacher AI model (also referred to as the "teacher detection difficulty evaluation AI model") using the learning dataset created by the teacher AI model annotation unit 302. For re-learning, the teacher AI model may use only an unlearned learning dataset, or may use a combination of an unlearned learning dataset and a learning dataset that was previously used to train the teacher AI model.

[0031] (On-site AI model annotation start determination unit) The on-site AI model annotation start determination unit 304 acquires from the second auxiliary memory unit 308 information such as the training images selected for on-site AI model training, the detection difficulty for the training images, the acquisition date and time, and the training state (whether the images have been trained or not). It also acquires information regarding whether the teacher AI model has been retrained from the teacher AI model training unit 303. The training images for the on-site AI model may be in a data format such as RGB color, YUV, or monochrome depending on the application. Furthermore, the images may be reduced to a predetermined size to reduce processing costs. The on-site AI model annotation start determination unit 304 determines whether to start annotation for the on-site AI model based on the frequency and number of selected training images for the on-site AI model, whether the images are in an untrained state, and whether the teacher AI model has been updated.

[0032] (On-site AI model annotation unit) The on-site AI model annotation unit 305 performs annotation on the learning images of the selected on-site AI model to create a learning dataset. All annotation work is performed by automatic annotation using the most recently updated teacher AI model. However, manual corrections may be made to the results of automatic annotation using the teacher AI model.

[0033] (On-site AI model learning unit) The on-site AI model learning unit 306 performs re-learning of the on-site AI model and the detection difficulty evaluation AI model corresponding to the on-site AI model (also referred to as "on-site detection difficulty evaluation AI model") using the learning dataset created by the on-site AI model annotation unit 305. For re-learning, the on-site AI model may use only an unlearned learning dataset, or may use a combination of the unlearned learning dataset and a learning dataset that was previously used to learn the on-site AI model.

[0034] (Second output control unit) The second output control unit 307 acquires the teacher detection difficulty evaluation AI model re-learned by the teacher AI model learning unit 303, the on-site AI model re-learned by the on-site AI model learning unit 306, and the on-site detection difficulty evaluation AI model from the second auxiliary memory unit 308, and outputs them to the on-site analysis server 101 for feedback.

[0035] (Second Auxiliary Storage Unit) The second auxiliary storage unit 308 is configured, for example, by a non-volatile memory such as an HDD or flash memory, and stores programs executed by the annotation / AI model learning server 103 and data to be processed by the programs. Such data includes learning images selected and output by the on-site analysis server 101 and image selection result information.

[0036] <Selection of Learning Images Based on Detection Difficulty> Next, the selection of learning images using detection difficulty will be described. FIG. 4 is a diagram illustrating an example of a process for selecting learning images according to this embodiment. In general, a detection difficulty evaluation AI model for an object detection AI model has a function of numerically calculating the difficulty with which the AI ​​model detects an object for a certain input image. Specific details of the detection difficulty evaluation AI model are disclosed in, for example, the non-patent document "D. Yoo et al., Learning Loss for Active Learning, arXiv:1905.03677," and will not be described in detail here.

[0037] Figure 4 shows an example in which, for an input image (site 1 input image) acquired by the site 1 analysis server from the camera 1 installed at site 1 (see Figure 1), a detection difficulty evaluation AI model is used to calculate the detection difficulty for each object detection AI model, and a determination is made as to whether to select the image as a learning image according to pre-set image selection conditions. The site 1 analysis server is equipped with a detection difficulty evaluation AI model for the object detection AI model (site 1 AI model) used in the object detection unit 202 of the site 1 analysis server, the object detection AI models (site 2 to N AI models) used in the object detection units of the site analysis servers installed at other sites, and the teacher AI model used in automatic annotation of the annotation / AI model learning server 103. For the site 1 input image, the site 1 analysis server can perform one-stop calculations of the detection difficulty for all object detection AI models (site 1 to N AI models and teacher AI model) and the selection of learning images for each object detection AI model based on the calculations. However, it is also possible to configure the site 1 analysis server to be equipped with only the site 1 and teacher detection difficulty evaluation AI models (similarly, the site n analysis server to be equipped with the site n and teacher detection difficulty evaluation AI models, n: 2 to N), with the site 1 input image transmitted over the network 104 and selection for the site n AI model being performed in the site n analysis server.

[0038] In the example of Figure 4, the image selection conditions are set as follows: "(1) If the detection difficulty of the teacher AI model for the input image is between 0.5 and 2.0, it is selected as a training image for the teacher AI model; (2) If the detection difficulty of the site AI model for the input image is between 0.5 and 2.0, it is selected as a training image for the corresponding site AI model." The site 1 input image in Figure 4(a) is an example image from a time period not included in the training dataset of the site 1 AI model. In this case, the detection difficulty of the site 1 AI model, 0.7, satisfies image selection condition (2), and is therefore selected as a training image for the site 1 AI model. On the other hand, the detection difficulty of the teacher AI model, 0.1, does not satisfy image selection condition (1), and is therefore not selected as a training image for the teacher AI model. Similarly, the detection difficulty of the site N AI model, 0.3, does not satisfy selection condition (2), and is therefore not selected as a training image for the site N AI model (description of other site AI models is omitted).

[0039] 4(b) is an example of an image captured over time when an event (e.g., a gust of wind) not included in the training dataset of the teacher AI model is captured. At this time, the detection difficulty levels of the teacher AI model, site 1 AI model, and site N AI model (0.8, 1.0, and 0.7, respectively) satisfy the image selection conditions, so the site 1 input image is selected as a training image for each of the object detection AI models. On the other hand, it is not selected as a training image for other site AI models that do not satisfy the selection conditions.

[0040] Generally, a detection difficulty evaluation AI model learns the difference (loss) between the inference result of an object detection AI model and the correct answer, and therefore calculates a low detection difficulty if the features of the training dataset and the input image are similar, and a high detection difficulty if the features of the training dataset and the input image are dissimilar. As a result, by using the training image selection and analysis system 100 according to an embodiment of the present disclosure for object detection tasks in outdoor environments where the appearance (features) of objects change depending on the time of day or season, or detection tasks at untrained shooting locations, it is possible to efficiently collect images to be trained that have a high detection difficulty for each AI model.

[0041] However, if the detection difficulty for an input image is extremely high, for example, if the object to be detected is small and far away, or if the objects to be detected are densely packed, it is considered to be an image that is extremely difficult to learn. Including such an input image in the training dataset leads to a decrease in detection accuracy, so images with extremely high detection difficulty are not desirable as training images for an AI model. For this reason, the selection conditions in Figure 4 set not only a lower threshold (first threshold) but also an upper threshold (second threshold) for the detection difficulty to select training images.

[0042] 4, the input image acquired by the image acquisition unit 201 is input to the detection difficulty evaluation AI model, but various other configurations are possible, such as inputting the output result of the intermediate layer when the on-site AI model infers the input image in the object detection unit 202 to the detection difficulty evaluation AI model in the detection difficulty calculation unit 203. Furthermore, the first threshold value and the second threshold value set in FIG. 4 are merely examples, and these values ​​are determined by the object detection AI model and the detection difficulty evaluation method, and therefore should be appropriately determined depending on the level of detection difficulty of the learning data to be collected. For this reason, it is desirable to set the first threshold value and the second threshold value separately for the teacher AI model and each on-site AI model.

[0043] <Correction for Selection of Training Images> Next, correction for the selection of training images performed using the detection difficulty will be described. However, in this embodiment, whether or not to perform correction is optional. FIG. 5 is a diagram illustrating an example of the correction process according to this embodiment. FIGS. 5(a) to 5(d) show the process in which each frame image captured consecutively by a camera at an arbitrary site n is acquired by the site n analysis server as a site n input image, and after selection and correction processing, is finally selected as a training image. For the site n input image, object detection is performed by the object detection unit 202 of the site n analysis server using the site n AI model (see the square symbol in FIG. 5). Furthermore, for the site n input image, the detection difficulty calculation unit 203 of the site n analysis server calculates the detection difficulty for each site AI model using each site detection difficulty evaluation AI model (see the trapezoid symbol in FIG. 5).

[0044] Here, if the image selection condition using detection difficulty is set to "when the detection difficulty of the on-site AI model for the input image is 0.5 or more and 2.0 or less," as in Figure 4, the input image in Figure 5(a) does not satisfy the selection condition, but the input images in Figures 5(b) to (d) do.

[0045] 5, the correction condition is set as follows: "Compare the object detection results of the current frame and the immediately preceding frame, and if the number of detected objects is the same and the regions of the detected objects have an IoU (Intersection over Union: overlap rate between regions) of a threshold value of 60% or more, do not select as a learning image." The "current frame" (also referred to as the "currently selected image") refers to an input image that satisfies the image selection condition and is selected by the learning image selection unit 204, and the "immediately preceding frame" (also referred to as the "previously selected image") refers to the learning image selected by the learning image selection unit 204 immediately before the current frame and output to the annotation / AI model learning server 103.

[0046] The image selection result correction unit 205 determines whether the current frame and the immediately preceding frame are similar in accordance with the correction conditions. As a result, for the input image in Figure 5(c), the IoU between the detection area of ​​the current frame shown by the solid line and the detection area of ​​the immediately preceding frame shown by the dashed line is greater than or equal to the threshold value (70%), so the correction conditions are met, and the current frame is determined to be an image similar to the immediately preceding frame. Although the detection difficulty meets the image selection conditions, a correction is made such that the image is not selected as a learning image. In contrast, Figures 5(b) and (d) do not satisfy the correction conditions and are ultimately selected as learning images.

[0047] If the correction conditions are met, the detection result for the current frame is similar to the detection result for the immediately preceding frame, meaning that the input image for the current frame is presumably similar to the input image for the immediately preceding frame. Collecting a large number of similar input images and transmitting them to the annotation / AI model learning server 103 is likely to result in issues such as increased communication load, increased annotation effort, and overlearning due to biased variations in the learning dataset. Therefore, by using the object detection results to determine similar input images and not selecting them as learning images, the issues of communication load, annotation costs, and overlearning can be resolved.

[0048] The correction process for image selection is not limited to the method shown in Figure 5, and various forms are possible, such as using similarity with the detection result of the previous frame (SSD, SAD, NCC, ZNCC), or using inter-frame differences to calculate the similarity of the entire image and determine the difference from the previous frame. Note that when correcting the image selection for a site n input image in the site n analysis server for a site AI model other than the site n AI model (e.g., a site mAI model) or a teacher AI model, it is possible to set the server to determine whether the correction conditions are met based on the object detection results of the site n AI model. Alternatively, the site n input image may be transmitted to the site m analysis server, and the satisfaction of the correction conditions may be determined based on the object detection results of the site mAI model there. Furthermore, if correction processing is performed based on the object detection results by the teacher AI model when selecting images for teacher AI model learning, it is possible to perform correction using the object detection results by the teacher AI model in the annotation / AI model learning server 103 before the teacher AI model annotation start determination unit 301 determines whether to start annotation (described below).

[0049] <Operation of the on-site analysis server> Next, a series of operations of the on-site analysis server will be described. Fig. 6 is a diagram for explaining an outline of an example of a series of operations of the on-site analysis server according to this embodiment. In the following explanation, as in Fig. 5, an example will be described in which an input image captured by a camera at an arbitrary site n is selected as a learning image in the on-site n analysis server and output.

[0050] First, the image acquisition unit 201 acquires an input image transmitted from the camera 102 and stored in the first auxiliary storage unit 208. When the on-site analysis server 101 is started up or when an updated detection difficulty evaluation AI model is introduced in the annotation / AI model learning server 103, new image selection conditions are set in the image selection condition setting unit 207.

[0051] Next, the acquired input image is input to an object detection AI model (on-site nAI model) in the object detection unit 202 to detect the target object. The acquired input image is also input to a detection difficulty determination AI model in the detection difficulty calculation unit 203 to calculate the detection difficulties (0.3 and 0.8, respectively) of the teacher AI model and the on-site AI model for the input image.

[0052] Next, the learning image selection unit 204 determines whether the detection difficulty of the teacher AI model and the on-site AI model for the input image satisfies the image selection conditions set by the image selection condition setting unit 207, and if the conditions are met, the input image is selected as an image to be learned by the AI ​​model. In Figure 6, the conditions are set so that if the detection difficulty of the teacher AI model is between 0.5 and 4.0, the input image is selected as a learning image for the teacher AI model, and if the detection difficulty of the on-site AI model for the input image is between 0.5 and 2.0, the input image is selected as a learning image for the on-site AI model. Therefore, in the example of Figure 6, the input image is not selected as a learning image for the teacher AI model, but is selected as a learning image for the on-site AI model.

[0053] Next, the image selection result correction unit 205 performs correction on the image selection in accordance with the object detection result obtained from the on-site nAI model of the object detection unit 202. In Figure 6, the object detection results for the current frame and the immediately preceding frame do not detect the same number of objects, so the correction conditions set to the same content as in Figure 5 are not met. Therefore, the input image is selected as a learning image for the on-site AI model and output to the first output control unit 206. It is also possible to output information about the selection result and notify the user regardless of whether it is selected or not.

[0054] In Figure 6, the detection difficulty is calculated and a learning image is selected for each frame, but various other forms are possible, such as calculating the detection difficulty and selecting a learning image every few frames to reduce the calculation load on the server, or performing batch processing all at once during times when the on-site AI model is not operating.

[0055] <Determining the Start of Annotation> Next, a description will be given of a method for determining the start of annotation in the teacher AI model annotation start determination unit 301 or the on-site AI model annotation start determination unit 304 of the annotation / AI model learning server 103. Fig. 7 is a diagram illustrating an example of annotation start determination according to this embodiment.

[0056] First, the learning images (also referred to as "selected images") selected as learning images for a predetermined object detection AI model and output from the on-site analysis server 101 are stored in the second auxiliary storage unit 308 of the annotation / AI model learning server 103. The second auxiliary storage unit 308 stores, accumulates, and updates image selection result information such as acquisition date and time, detection difficulty, learning status (trained / untrained), etc. for each selected image.

[0057] If the image selection result information for the selected image satisfies the annotation start condition, annotation start is determined. The annotation start condition is set in advance for each teacher AI model and on-site AI model based on the frequency of selection as the selected image and the learning state. If the frequency of selection of learning images is low, i.e., the frequency of occurrence of images that can be retrained to improve detection accuracy is low, it is considered that the accuracy improvement effect is small compared to the AI ​​model development costs, such as annotation, learning, and verification. Therefore, it is effective not to determine annotation start when the frequency of selection of learning images is low. In the example of Figure 7, the annotation start condition is set as "200 or more untrained learning images have been selected within 10 days." Then, since the actual selection frequency was 250 times in 10 days, annotation start for the learning images is determined. Note that the annotation start condition is not limited to the condition shown in Figure 7, and various conditions are possible, such as "when the teacher AI model is retrained, annotation for the on-site AI model is always started."

[0058] <Operation of the annotation / AI model learning server> Next, a series of operations of the annotation / AI model learning server 103 will be described. Fig. 8 is a diagram outlining an example of a series of operations of the annotation / AI model learning server according to this embodiment. Fig. 8 describes the process of performing annotation in the annotation / AI model learning server 103, and relearning the teacher AI model, on-site AI model, and their detection difficulty evaluation AI models, using learning images selected for the teacher AI model and on-site AI model by the on-site analysis server from on-site input images.

[0059] First, the selected images and image selection result information output from the on-site analysis server 101 are stored in the second auxiliary memory unit 308. The teacher AI model annotation start determination unit 301 obtains image selection result information such as the images selected for teacher AI model training, the detection difficulty for the images, and the acquisition date and time from the second auxiliary memory unit 308. The teacher AI model annotation start determination unit 301 determines the start of annotation for the teacher AI model according to annotation start conditions such as the selection frequency and number of images for teacher AI model training, and whether or not they are in an untrained state. In Figure 8, a condition is set that annotation is performed on the teacher AI model training images if there are 1,000 or more untrained teacher AI model training images.

[0060] If the annotation start conditions are met, the teacher AI model annotation unit 302 performs annotation on the training images to create a training dataset. In FIG. 8 , annotation work on the teacher AI model training images is performed manually. Next, the teacher AI model training unit 303 performs re-training of the teacher AI model and the teacher detection difficulty evaluation AI model using the created training dataset. Here, the evaluation dataset for the teacher AI model may be used to verify the developed teacher AI model and the detection difficulty evaluation AI model. The teacher AI model installed in the annotation / AI model training server 103 is updated to the latest state by the re-trained teacher AI model. The re-trained teacher detection difficulty evaluation AI model is output to the on-site analysis server 101 by the second output control unit 307. The re-trained teacher detection difficulty evaluation AI model is transmitted to all on-site analysis servers (site 1 to N analysis servers).

[0061] The on-site AI model annotation start determination unit 304 also acquires image selection result information, such as images selected for on-site AI model training, the detection difficulty for the images, and the acquisition date and time, from the second auxiliary storage unit 308. The on-site AI model annotation start determination unit 304 determines the start of on-site AI model annotation according to annotation start conditions, such as the selection frequency and number of on-site AI model training images, whether they are in an untrained state, and whether relearning of the teacher AI model has been performed. In Figure 8, a condition is set that annotation of on-site AI model training images is performed when there are 500 or more untrained on-site AI model training images, or when relearning of the teacher AI model has been performed.

[0062] If the annotation start conditions are met, the on-site AI model annotation unit 305 performs automatic annotation on the training image using a teacher AI model to create a training dataset. Next, the on-site AI model training unit 306 uses the created training dataset to re-train the on-site AI model and the on-site detection difficulty evaluation AI model. Here, the evaluation dataset for the on-site AI model may be used to verify the developed on-site AI model and on-site detection difficulty evaluation AI model. The re-trained on-site AI model and on-site detection difficulty evaluation AI model are output to the on-site analysis server 101 by the second output control unit 307. The re-trained on-site n AI model is sent to the on-site n analysis server, and the re-trained on-site n detection difficulty evaluation AI model is sent to all on-site analysis servers (sites 1 to N analysis servers).

[0063] <Processing Flowchart of the Learning Image Selection and Analysis System> Next, the processing procedure of the learning image selection and analysis system will be described. Figures 9A and 9B are example flowcharts of the processing procedure of the learning image selection and analysis system according to this embodiment. Portions overlapping with the description of Figure 8 will be omitted as appropriate. First, a processor such as a CPU of the on-site analysis server 101 executes a program loaded from the first auxiliary storage unit 208 to the main storage unit to start the learning image selection and analysis system (also simply referred to as the "image analysis system"). Furthermore, a processor of the annotation / AI model learning server 103 executes a program loaded from the second auxiliary storage unit 308 to the main storage unit to start the image analysis system. The image analysis system may allow the user to check object detection results and image selection result information via a GUI, or may simply notify the user of the presence or absence of detected objects.

[0064] After the image analysis system is started, in step 901, detection parameters such as the camera that will acquire the input image, the AI ​​model to be used, the frequency with which the input image is acquired, and the size of the input image are determined. These parameters may be set by loading a pre-prepared setting file, or may be selected by the user via a GUI. The number of cameras that acquire the input image may be one or more. In the following, an example will be described in which one camera acquires the input image and sequentially acquires video in real time.

[0065] In step 902, the image selection condition setting unit 207 sets the upper and lower thresholds for the detection difficulty of the teacher AI model and the on-site AI model. The image selection conditions may be set for each object detection AI model, and may be selected from a variety of methods, such as setting the maximum and minimum values ​​of the detection difficulty for several reference images as the upper and lower thresholds, respectively. In step 903, the image acquisition unit 201 reads an input image acquired from the camera 102. In step 904, the object detection unit 202 performs target object detection on the input image read in step 903 using the on-site AI model. In step 905, the detection difficulty calculation unit 203 calculates the detection difficulty of the teacher AI model and the on-site AI model for the input image read in step 903 using the detection difficulty evaluation AI model. In step 906, the learning image selection unit 204 determines whether the detection difficulty of the teacher AI model calculated in step 905 satisfies the image selection conditions determined in step 902. If the conditions are met, the process proceeds to step 907; if not, the process proceeds to step 908. In step 907, the input image read in step 903 is selected as a learning image for the teacher AI model. In step 908, the learning image selection unit 204 determines whether the detection difficulty of the on-site AI model calculated in step 905 meets the image selection conditions determined in step 902. If the conditions are met, the process proceeds to step 909; if not, the process proceeds to step 911. In step 909, the input image read in step 903 is selected as a learning image for the on-site AI model. In step 910, the image selection result correction unit 205 corrects the image selection result using the object detection result of step 904 in accordance with the correction conditions. In step 911, it is determined whether the input image read in step 903 has been selected as a learning image. If selected, the process proceeds to step 912; if not, the process proceeds to step 913. In step 912, the first output control unit 206 outputs the object detection results obtained in step 904, the image selection results obtained in step 910, and information on the input image read in step 903 to the annotation / AI model learning server 103.In step 913, the first output control unit 206 notifies the user of the object detection results and image selection results. The notification may be for an image selected as a learning image (via 912) or not (via 911). The notification may be displayed on the GUI, or a notification that a target object has been detected may be sent to a small terminal. Once the notification is complete, the process proceeds to step 903 and the next input image is read.

[0066] In step 914, the teacher AI model annotation start determination unit 301 obtains information such as the images selected for teacher AI model training, the detection difficulty for the images, and the acquisition date and time from the second auxiliary memory unit 308. In step 915, the teacher AI model annotation start determination unit 301 determines whether the annotation start condition for the teacher AI model is met. If the condition is met, the process proceeds to step 916; if not, the process proceeds to step 918. In step 916, the teacher AI model annotation unit 302 performs annotation on the images for teacher AI model training. In step 917, the teacher AI model training unit 303 performs re-training of the teacher AI model and the teacher detection difficulty evaluation AI model using the training dataset created in step 916. In step 918, the on-site AI model annotation start determination unit 304 obtains information such as the images selected for on-site AI model training, the detection difficulty for the images, and the acquisition date and time from the second auxiliary memory unit 308. In step 919, the on-site AI model annotation start determination unit 304 determines whether the annotation start conditions for the on-site AI model are met. If the conditions are met, the process proceeds to step 920; if not, the process proceeds to step 922. In step 920, the on-site AI model annotation unit 305 performs automatic annotation using the teacher AI model on the on-site AI model training image. In step 921, the on-site AI model training unit 306 performs re-training of the on-site AI model and the on-site detection difficulty evaluation AI model using the training dataset created in step 920. In step 922, the second output control unit 307 determines whether re-training of the teacher AI model and the on-site AI model has been performed. If re-training has been performed, the process proceeds to step 923; if not, the process proceeds to step 903 and the next input image is read. In step 923, the second output control unit 307 outputs the re-trained on-site AI model to the on-site analysis server 101. The re-trained teacher and on-site detection difficulty evaluation AI models are also output to all on-site analysis servers. Thereafter, the AI ​​model and detection parameters to be used are set again in step 901 .

[0067] <Actions and Effects> The training image selection and analysis system described above installs cameras and a site analysis server at multiple sites with different environments, and selects training images from input images captured at each site using image selection conditions appropriate for the teacher AI model and each site AI model. Sharing images captured at multiple sites makes it possible to acquire a wider variety of images, increasing the probability of acquiring rare images (e.g., fires). A wider variety of images, including rare images, can be efficiently selected as training images, allowing the teacher AI model to be retrained, improving the detection accuracy of the teacher AI model. Furthermore, the improved accuracy of the teacher AI model is used to automatically annotate the training images of each site AI model, and retraining also improves the detection accuracy of each site AI model, creating a virtuous cycle of efficient accuracy improvement. Furthermore, by installing the on-site AI model, which has a relatively low computational and communication load, and the teacher / on-site detection difficulty evaluation AI model on the on-site analysis server side (edge ​​side), and installing the teacher AI model, which has a relatively high computational and communication load, on the annotation / AI model learning server side (center side), the flow of data can be smoothed and images for learning can be selected efficiently.

[0068] Although the embodiments of the present invention have been described above, the present invention is not limited to the above-described embodiments, and various modifications are possible within the scope of the gist of the present invention. For example, modifications are possible such as incorporating the imaging function of the camera into the on-site analysis server, or performing on-site analysis server processing in the camera.

[0069] The following are examples of aspects that may be included in the present invention, but the present invention is not limited to these. (Aspect 1) A training image selection and analysis system including a site analysis server that selects training images from input images acquired by an image acquisition unit, and an annotation / AI model learning server that performs annotation on the training images to train an object detection AI model, wherein a plurality of the site analysis servers are installed, and each site analysis server includes an object detection unit that performs object detection using a site AI model, a detection difficulty calculation unit that calculates the detection difficulty of a teacher AI model and the site AI model for the input image, and a training image selection unit that automatically selects training images for each object detection AI model using an output result of the detection difficulty calculation unit. (Aspect 2) The training image selection and analysis system according to Aspect 1, wherein the detection difficulty calculation unit calculates the detection difficulty of the site AI model used in the one site analysis server, and also calculates the detection difficulty of site AI models used in other site analysis servers. (Aspect 3) The training image selection and analysis system according to Aspect 1 or 2, wherein the calculation of the detection difficulty of the object detection AI model in the detection difficulty calculation unit is performed using a detection difficulty evaluation AI model for each object detection AI model. (Aspect 4) The training image selection and analysis system according to any one of Aspects 1 to 3, wherein the training image selection and analysis system further comprises an image selection result correction unit that corrects the selection by the training image selection unit based on a similarity determination between an input image selected by the training image selection unit (referred to as the "currently selected image") and a training image selected immediately before and output to the annotation / AI model training server (referred to as the "last selected image"). (Aspect 5) The training image selection and analysis system according to Aspect 4, wherein the similarity determination is performed based on the object detection results output from the on-site AI model used in the one on-site analysis server from the currently selected image and the last selected image.(Aspect 6) In the learning image selection and analysis system described in Aspect 3, the annotation and AI model learning server is characterized in having: a teacher AI model annotation unit that performs annotation on learning images of the teacher AI model; a teacher AI model learning unit that performs relearning of the teacher AI model and a detection difficulty evaluation AI model for the teacher AI model using a learning dataset created by the teacher AI model annotation unit; an on-site AI model annotation unit that performs automatic annotation by the teacher AI model on learning images of the on-site AI model; and an on-site AI model learning unit that performs relearning of the on-site AI model and a detection difficulty evaluation AI model for the on-site AI model using a learning dataset created by the on-site AI model annotation unit. (Aspect 7) In the learning image selection and analysis system described in Aspect 6, the annotation / AI model learning server is characterized in having a teacher AI model annotation start determination unit that performs an annotation start determination based on the frequency of selection of learning images for the teacher AI model, and an on-site AI model annotation start determination unit that performs an annotation start determination based on the frequency of selection of learning images for the on-site AI model or whether or not re-learning of the teacher AI model is being performed. (Aspect 8) A method for selecting and analyzing training images in which training images are selected from input images acquired by an image acquisition unit in an on-site analysis server, and an annotation / AI model learning server performs annotation on the training images to train an object detection AI model, characterized in that a plurality of the on-site analysis servers are installed, an object detection unit of one on-site analysis server performs object detection using an on-site AI model, a detection difficulty calculation unit calculates the detection difficulty of a teacher AI model and the on-site AI model for the input image, and a training image selection unit automatically selects training images for each object detection AI model using the output result of the detection difficulty calculation unit.(Aspect 9) The training image selection and analysis method according to Aspect 8, wherein the detection difficulty calculation unit calculates the detection difficulty of the on-site AI model used in the one on-site analysis server, and also calculates the detection difficulty of on-site AI models used in other on-site analysis servers. (Aspect 10) The training image selection and analysis method according to Aspect 8 or 9, wherein the detection difficulty calculation unit calculates the detection difficulty of the object detection AI model using a detection difficulty evaluation AI model for each object detection AI model. (Aspect 11) The training image selection and analysis method according to any one of Aspects 8 to 10, wherein an image selection result correction unit corrects the selection by the training image selection unit based on a similarity determination between an input image selected by the training image selection unit (referred to as the "currently selected image") and a training image selected immediately before and output to the annotation / AI model training server (referred to as the "most recently selected image"). (Aspect 12) The method for selecting and analyzing images for learning according to Aspect 11, characterized in that the similarity determination is performed based on object detection results output from the currently selected image and the immediately preceding selected image by the on-site AI model used in the one on-site analysis server. (Aspect 13) A method for selecting and analyzing learning images according to Aspect 10, characterized in that, in the annotation / AI model learning server, an annotation unit for a teacher AI model performs annotation on the learning images of the teacher AI model; a teacher AI model learning unit performs re-learning of the teacher AI model and a detection difficulty evaluation AI model for the teacher AI model using a learning dataset created by the annotation unit for the teacher AI model; an annotation unit for a site AI model performs automatic annotation by the teacher AI model on the learning images of the site AI model; and a site AI model learning unit performs re-learning of the site AI model and a detection difficulty evaluation AI model for the site AI model using the learning dataset created by the annotation unit for the site AI model.(Aspect 14) A method for selecting and analyzing learning images according to Aspect 13, characterized in that, in the annotation / AI model learning server, a teacher AI model annotation start determination unit determines whether to start annotation based on the frequency of selection of learning images for the teacher AI model, and a site AI model annotation start determination unit determines whether to start annotation based on the frequency of selection of learning images for the site AI model or whether re-learning of the teacher AI model is being performed.

[0070] 100 Learning image selection and analysis system 101 On-site analysis server 102 Camera 103 Annotation and AI model learning server 104 Network 201 Image acquisition unit 202 Object detection unit 203 Detection difficulty calculation unit 204 Learning image selection unit 205 Image selection result correction unit 206 First output control unit 207 Image selection condition setting unit 208 First auxiliary memory unit 301 Teacher AI model annotation start determination unit 302 Teacher AI model annotation unit 303 Teacher AI model learning unit 304 On-site AI model annotation start determination unit 305 On-site AI model annotation unit 306 On-site AI model learning unit 307 Second output control unit 308 Second auxiliary memory unit

Claims

1. A training image selection and analysis system comprising an on-site analysis server that selects training images from input images acquired by an image acquisition unit, and an annotation / AI model learning server that performs annotations on the training images to train an object detection AI model, wherein a plurality of the on-site analysis servers are installed, and each on-site analysis server has an object detection unit that performs object detection using an on-site AI model, a detection difficulty calculation unit that calculates the detection difficulty of a teacher AI model and the on-site AI model for the input image, and a training image selection unit that automatically selects training images for each object detection AI model using the output results of the detection difficulty calculation unit.

2. A system for selecting and analyzing learning images as described in claim 1, characterized in that the detection difficulty calculation unit calculates the detection difficulty of the on-site AI model used in one on-site analysis server, as well as the detection difficulty of the on-site AI model used in another on-site analysis server.

3. A system for selecting and analyzing images for learning as described in claim 1 or 2, characterized in that the detection difficulty of the object detection AI model in the detection difficulty calculation unit is calculated using a detection difficulty evaluation AI model for each object detection AI model.

4. A system for selecting and analyzing images for learning as described in claim 1 or 2, characterized in that it has an image selection result correction unit that corrects the selection by the image selection unit for learning based on a similarity determination between an input image selected by the image selection unit for learning (referred to as the "currently selected image") and a learning image selected immediately before and output to the annotation / AI model learning server (referred to as the "previously selected image").

5. A system for selecting and analyzing images for learning as described in claim 4, characterized in that the similarity determination is made based on the object detection results output from the currently selected image and the immediately preceding selected image by the on-site AI model used in the one on-site analysis server.

6. A system for selecting and analyzing learning images as described in claim 3, wherein the annotation and AI model learning server has: a teacher AI model annotation unit that performs annotation on learning images of the teacher AI model; a teacher AI model learning unit that performs relearning of the teacher AI model and a detection difficulty evaluation AI model for the teacher AI model using a learning dataset created by the teacher AI model annotation unit; an on-site AI model annotation unit that performs automatic annotation by the teacher AI model on learning images of the on-site AI model; and an on-site AI model learning unit that performs relearning of the on-site AI model and a detection difficulty evaluation AI model for the on-site AI model using a learning dataset created by the on-site AI model annotation unit. A system for selecting and analyzing learning images.

7. A system for selecting and analyzing images for learning as described in claim 6, wherein the annotation / AI model learning server has a teacher AI model annotation start determination unit that determines whether to start annotation based on the frequency of selection of learning images for the teacher AI model, and an on-site AI model annotation start determination unit that determines whether to start annotation based on the frequency of selection of learning images for the on-site AI model or whether re-learning of the teacher AI model is being performed.

8. A method for selecting and analyzing training images in which training images are selected from input images acquired by an image acquisition unit on an on-site analysis server, and an annotation / AI model training server performs annotation on the training images to train an object detection AI model, characterized in that a plurality of on-site analysis servers are installed, an object detection unit of one on-site analysis server performs object detection using an on-site AI model, a detection difficulty calculation unit calculates the detection difficulty of the teacher AI model and the on-site AI model for the input image, and a training image selection unit automatically selects training images for each object detection AI model using the output result of the detection difficulty calculation unit.

Images