Dataset generation system, server, and dataset generation program

Abstract

To provide a dataset generation system, a server, and a dataset generation program that enable machine learning of a DNN model for inference processing targeting captured images of cameras of numerous facilities with as few captured images as possible.SOLUTION: A system is configured to: collect frame images that meet a selection criterion set by a user, from a representative camera of each group (S15); perform inference processing (pseudo-labeling processing) for evaluation on each of the collected frame images (S17); evaluate whether a dataset including the collected frame images is suitable for a learning dataset of a learned DNN model for predetermined inference processing, on the basis of a result of the above inference processing (S18); and terminate collecting the frame images when an evaluation value of the dataset becomes equal to or more than a target value (NO in S19). The dataset collected at that time can be set as the learning dataset, so that the number of frame images included in the learning dataset can be reduced.SELECTED DRAWING: Figure 6

Description

Technical Field

[0001] The present invention relates to a dataset generation system, a server, and a dataset generation program.

Background Art

[0002] Conventionally, there is known a system that performs image analysis (object detection or object recognition) on a captured image by a camera installed in a facility such as a convenience store using a device (so-called edge-side device) arranged on the facility side where the camera is installed (see, for example, Patent Document 1). When performing inference processing for image analysis such as object detection or object recognition on such an edge-side device, a pre-trained deep neural network model (DNN model) with a small processing load (so-called "light") is implemented on the edge-side device, and inference processing on a captured image of a camera connected to the edge-side device is performed using this pre-trained DNN model. Here, due to the vulnerability of computer resources in the edge-side device, it is desirable that the pre-trained DNN model implemented on the edge-side device be an extremely light (very small processing load) DNN model. A DNN model for implementation on the edge-side device as described above is generally generated by a server connected to the edge-side device performing machine learning based on a captured image from a camera installed in the facility (see, for example, paragraphs (0037) and (0038) of Patent Document 1).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, when implementing an extremely lightweight (very low processing load) pre-trained DNN model like the one described above on edge devices deployed in numerous stores and other facilities, and performing inference processing on images captured by cameras in those facilities, the following problems arise. Specifically, when using such an extremely lightweight pre-trained DNN model, it is desirable to fine-tune the original pre-trained DNN model (a general-purpose pre-trained DNN model for inference processing such as object detection or recognition) using images captured by cameras in the relevant facilities in order to ensure accuracy. However, in the case of large chain stores (such as convenience stores), the number of stores can reach several thousand, so fine-tuning the original pre-trained DNN model using images captured by cameras deployed in all of those thousands of stores would take a long time to process, and the costs required to transfer camera images from the edge devices to the server (communication costs, electricity costs, etc.) and the costs required for the server to store the camera images received from the edge devices (costs required to secure storage space on the server) would be substantial. The above problem is common not only when fine-tuning the original pre-trained DNN model using images captured by cameras in numerous stores and other facilities, but also when performing machine learning on an inference DNN model (for example, a DNN model for object detection or recognition) using images captured by cameras in numerous stores and other facilities.

[0005] The present invention aims to solve the above problems and to provide a dataset generation system, server, and dataset generation program that enable machine learning of a neural network model for inference processing using images captured by cameras in a large number of stores and other facilities, using as few captured images as possible. [Means for solving the problem]

[0006] To solve the above problems, a dataset generation system according to a first aspect of the present invention comprises: a camera classification means for grouping a plurality of cameras; a setting operation means for performing a setting operation for selecting selection criteria for captured images; a captured image collection means for collecting captured images that match the selection criteria set by the user using the setting operation means from at least one camera in each group grouped by the camera classification means; an evaluation inference means for performing an evaluation inference process on each of the captured images collected by the captured image collection means; and a dataset evaluation means for evaluating whether the dataset composed of the captured images collected by the captured image collection means is suitable as a training dataset for a predetermined neural network model for inference processing, based on the results of the inference process by the evaluation inference means.

[0007] In this dataset generation system, the system may further include a representative camera selection means for selecting a representative camera in each group that has been grouped by the camera classification means, and the captured image collection means may collect captured images from each of the representative cameras selected by the representative camera selection means that match the selection criteria set by the user using the setting operation means.

[0008] In this dataset generation system, it is desirable for the image acquisition means to collect images in which there has been an increase or decrease in the number of recognized objects, a shift in the position of the recognized objects, or a change in the posture of the recognized objects, compared to the previous image.

[0009] In this dataset generation system, it is desirable that the dataset evaluation means further comprises a relearning means that, based on the results of the inference processing by the evaluation inference means, evaluates whether the dataset composed of images collected by the image acquisition means is suitable as a training dataset for fine-tuning or transfer learning of the pre-trained neural network model for a predetermined inference process, and when the evaluation value of the dataset by the dataset evaluation means becomes equal to or greater than a target value, performs fine-tuning or transfer learning (hereinafter referred to as "relearning") of the pre-trained neural network model for a predetermined inference process using the dataset composed of images collected by the image acquisition means as a training dataset, and an accuracy improvement evaluation means that evaluates whether the accuracy of the inference processing by the pre-trained neural network model after relearning by the relearning means has improved by a predetermined value or more compared to the accuracy of the inference processing by the pre-trained neural network model before relearning.

[0010] In this dataset generation system, it is desirable to further include a pseudo-labeling means that performs pseudo-labeling processing on each of the captured images collected by the captured image collection means, and the retraining means retrains the pre-trained neural network model for predetermined inference processing based on the captured images collected by the captured image collection means and the correct labels assigned to the captured images by the pseudo-labeling means.

[0011] In this dataset generation system, the system further comprises: a second image collection means for collecting images from each of the multiple cameras in which the object to be recognized is not visible; an image feature extraction means for extracting features from each of the images in which the object to be recognized is not visible, which are collected by the second image collection means; and an image clustering means for grouping the images in which the object to be recognized is not visible, which are collected by the second image collection means, based on the features extracted by the image feature extraction means. The camera classification means may further group the multiple cameras by grouping the cameras that captured the images based on the grouping results of the images in which the object to be recognized is not visible, which are performed by the image clustering means.

[0012] In this dataset generation system, when the evaluation value of the dataset by the dataset evaluation means reaches or exceeds the target value, the dataset consisting of images already collected by the image acquisition means at that point may be used as the training dataset for the pre-trained neural network model for the predetermined inference process.

[0013] A server according to a second aspect of the present invention is connected to a plurality of cameras via a network and comprises: a camera classification means for grouping the plurality of cameras; a setting operation means for performing selection criteria for captured images; a captured image collection means for collecting captured images from at least one camera in each group grouped by the camera classification means that match the selection criteria set by the user using the setting operation means; an evaluation inference means for performing evaluation inference processing on each of the captured images collected by the captured image collection means; and a dataset evaluation means for evaluating whether a dataset composed of captured images collected by the captured image collection means is suitable as a training dataset for a predetermined neural network model for inference processing, based on the results of the inference processing by the evaluation inference means.

[0014] A dataset generation program according to a third aspect of the present invention causes a computer to function as a camera classification means for grouping a plurality of cameras; a setting operation means for performing setting operations for selection criteria for captured images; a captured image collection means for collecting captured images from at least one camera in each group grouped by the camera classification means that match the selection criteria set by the setting operation means; an evaluation inference means for performing evaluation inference processing on each of the captured images collected by the captured image collection means; and a dataset evaluation means for evaluating whether the dataset composed of captured images collected by the captured image collection means is suitable as a training dataset for a predetermined neural network model for inference processing, based on the results of the inference processing by the evaluation inference means. [Effects of the Invention]

[0015] According to the dataset generation system according to the first aspect of the present invention, the server according to the second aspect, and the dataset generation program according to the third aspect, images captured that match selection criteria set by the user can be collected from at least one camera in each group, and it is possible to evaluate whether the dataset composed of the captured images collected at that time is suitable as a training dataset for a predetermined neural network model for inference processing. As a result, when the evaluation value of the dataset composed of the captured images collected at that time reaches or exceeds a target value, the collection of captured images is terminated, and the dataset composed of the captured images collected at that time can be used as the training dataset for the predetermined neural network model for inference processing, thereby minimizing the number of captured images included in the training dataset for the predetermined neural network model for inference processing. Therefore, machine learning of the predetermined neural network model for inference processing (for example, a neural network model for object detection or object recognition) can be performed on images captured by cameras in a large number of stores or other facilities using as few captured images as possible. Therefore, the processing time required to generate the training dataset for the neural network model for the predetermined inference process, and the processing time required for machine learning of the neural network model using this training dataset can be reduced. In addition, the costs required to transfer the camera's captured images from the camera (edge ​​side) to the server (communication costs, electricity costs, etc.) and the costs required for the server to store the camera's captured images received from the edge side (costs required to secure memory space on the server) can be reduced. [Brief explanation of the drawing]

[0016] [Figure 1] A block diagram showing the schematic configuration of a dataset generation system according to one embodiment of the present invention. [Figure 2] A block diagram showing the approximate hardware configuration of the signage in Figure 1. [Figure 3] A block diagram showing the hardware configuration of the analysis box in Figure 1. [Figure 4] Block diagram showing the hardware configuration of the learning server in FIG. 1. [Figure 5] Functional block configuration diagram of the above learning server. [Figure 6] Flowchart of the learning dataset generation process in the above dataset generation system. [Figure 7] Flowchart showing the details of the processes S11 to S13 (clustering process) in FIG. 6. [Figure 8] Diagram showing the user interface screen used when designing the subsystem of the above clustering process. [Figure 9] Diagram showing the user interface screen used when designing the subsystem of the selection and collection of frame images from the representative camera. [Figure 10] Diagram showing an example of the frame image selection algorithm according to the selection criteria set in S14 in FIG. 6. [Figure 11] Diagram showing the user interface screen used when designing the subsystem of the quality evaluation of the dataset composed of frame images collected from the above representative camera. [Figure 12] Diagram showing the user interface screen used when designing the subsystem of the evaluation through fine-tuning of the dataset determined in S19 in FIG. 6 to have reached the KPI in the comprehensive evaluation value of quality. [Figure 13] Sequence diagram showing the flow of frame image data collection in the above dataset generation system.

Mode for Carrying Out the Invention

[0017] Hereinafter, a dataset generation system, server, and dataset generation program according to embodiments of the present invention will be described with reference to the drawings. Figure 1 is a block diagram showing the schematic configuration of the dataset generation system 10 according to this embodiment. As shown in Figure 1, in this embodiment, a plurality of fixed cameras 3, which are surveillance network cameras that capture a predetermined shooting area, and an analysis box 2 that analyzes the images from each fixed camera 3 are arranged in stores such as a chain store (Sc), and signage 4 (a general term for signage 4a, 4b, etc.), which are tablet terminals for digital signage equipped with a built-in camera 5, are arranged in stores such as a chain store (Sa, Sb).

[0018] The dataset generation system 10 described above mainly consists of the above-mentioned multiple fixed cameras 3 installed in each store S (a general term for stores Sa, Sb, Sc, etc.), an analysis box 2, a digital signage 4, and a learning server 1 on the cloud C (corresponding to "server" and "computer" in the claims) connected to these analysis boxes 2 and digital signage 4 via the internet.

[0019] As shown in Figure 1, the dataset generation system 10 is equipped with a hub 7 and a router 8 in store Sc where the fixed cameras 3 and analysis boxes 2 are located. The dataset generation system 10 is also equipped with a wireless LAN router 6 in stores Sa, Sb, etc., where the signage 4 is located. The camera on the signage 4 side may be a built-in camera 5 located inside the casing of the signage 4, or a web camera attached to the signage 4. In the following description, the example where the camera on the signage 4 side is a built-in camera 5 will be described primarily. The term "multiple cameras" in the claims includes the built-in camera 5 of each signage 4 and each fixed camera 3 connected to each analysis box 2.

[0020] The fixed camera 3 described above has an IP address and can be directly connected to the network. As shown in Figure 1, the analysis box 2 is connected to multiple fixed cameras 3 via a LAN (Local Area Network) and a hub 7, and analyzes the images input from each of these fixed cameras 3. Specifically, the analysis box 2 performs object detection processing (detection of customers and their heads / faces) on the images input from each of the fixed cameras 3, and object recognition processing (recognition processing such as estimating customer attributes (gender and age (age group)) and pose) on the images of the objects detected by this object detection process.

[0021] The signage 5 is mainly installed on product shelves within the store S, and displays advertisements and other content for customers visiting the store S on its touch panel display 14 (see Figure 2). It also performs object detection processing on frame images from its built-in camera 4 (detection of customers and their heads / faces), and object recognition processing on images of objects detected by this object detection process.

[0022] The learning server 1 mentioned above is a server installed in the management department (head office, etc.) of store S. As will be explained in more detail later, the learning server 1 generates a training dataset (or a suitable dataset) for a predetermined DNN (Deep Neural Networks) model for inference processing, uses the generated training dataset to fine-tune the original pre-trained DNN model for inference processing, and then sends the fine-tuned pre-trained DNN model to each analysis box 2 and each signage 4 for installation.

[0023] Next, with reference to Figure 2, the hardware configuration of the signage 4 described above will be explained. In addition to the built-in camera 5 described above, the signage 4 includes an SoC (System-on-a-Chip) 11, a touch panel display 14, a speaker 15, a memory 16 for storing various data and programs, a communication unit 17, a secondary battery 18, and a charging terminal 19. The SoC 11 includes a CPU 12 that controls the entire device and performs various calculations, and a GPU 13 used for inference processing of various pre-trained DNN (Deep Neural Networks) models 20 for inference processing.

[0024] The memory 16 described above stores a pre-trained DNN model 20 for inference processing. This pre-trained DNN model 20 for inference processing includes multiple types of pre-trained DNN models for inference processing, such as a pre-trained DNN model for customer (person) detection (including a pre-trained DNN model for detecting the customer's face or head), and a pre-trained DNN model for object recognition processing (recognition processing such as estimating customer attributes (gender and age (age group)) and pose) on the images of the customer (person) and the customer's head / face detected in this detection process. The program stored in memory 16 includes a signage control program 29, which corresponds to the "edge-side core engine" in Figure 7.

[0025] The communication unit 17 described above is equipped with a communication IC and an antenna. The signage 4 is connected to the learning server 1 on the cloud C via the communication unit 17 and the internet. The secondary battery 18 is a rechargeable battery such as a lithium-ion battery, which stores power from the commercial power supply after it has been converted to DC power by an AC / DC converter and supplies it to various parts of the signage 4.

[0026] Next, with reference to Figure 3, the hardware configuration of the analysis box 2 will be described. The analysis box 2 includes a CPU 21 that controls the entire device and performs various calculations, a hard disk 22 that stores various data and programs, RAM (Random Access Memory) 23, inference chips (hereinafter abbreviated as "chips") 24a to 24h which are DNN (Deep Neural Networks) inference processors, and a communication control IC 25. The CPU 21 is a general-purpose CPU, or a CPU designed to enhance parallel processing performance in order to process a large number of video streams simultaneously. The data stored in the hard disk 22 includes video data after decoding the video streams (data) input from each of the fixed cameras 2. The programs stored in the hard disk 22 include trained DNN models (trained DNN models for various inference processes) for inference processing such as human detection processing, face detection processing, attribute recognition processing, and pose estimation processing, and an analysis box control program. The analysis box control program mentioned above includes a program corresponding to the "edge-side core engine" in Figure 7.

[0027] Next, with reference to Figure 4, the hardware configuration of the learning server 1 will be described. The learning server 1 comprises a CPU 31 that controls the entire device and performs various calculations, a hard disk 32 that stores various data and programs, RAM (Random Access Memory) 33, a display 34, an operation unit 35 (corresponding to the "setting operation means" in the claim), and a communication unit 36. The programs stored in the hard disk 32 include a dataset generation program 37.

[0028] Figure 5 mainly shows the functional blocks of the learning server 1 described above. The following explanation of Figure 5 describes the correspondence between each functional block in the figure and each constituent element (each means) in the claim, and the overview of the function of each of these functional blocks. The learning server 1 comprises the following functional blocks: a layout / frame image collection unit 40, an image feature extraction unit 41, an image clustering unit 42, a camera classification unit 43, a representative camera selection unit 44, a frame image collection unit 45, an evaluation inference unit 46, a dataset evaluation unit 47, a pseudo-labeling unit 48, a retraining unit 49, and an accuracy improvement evaluation unit 50. The above-mentioned layout / frame image acquisition unit 40, image feature extraction unit 41, image clustering unit 42, camera classification unit 43, representative camera selection unit 44, frame image acquisition unit 45, evaluation inference unit 46, dataset evaluation unit 47, pseudo-labeling unit 48, retraining unit 49, and accuracy improvement evaluation unit 50 correspond, respectively, to the second captured image acquisition means, image feature extraction means, image clustering means, camera classification means, representative camera selection means, captured image acquisition means, evaluation inference means, dataset evaluation means, pseudo-labeling means, retraining means, and accuracy improvement evaluation means in the claims. Furthermore, the above-mentioned layout / frame image acquisition unit 40 and frame image acquisition unit 45 are mainly realized by the communication unit 36, CPU 31, and dataset generation program 37 in Figure 4. Furthermore, the image feature extraction unit 41, image clustering unit 42, camera classification unit 43, representative camera selection unit 44, evaluation inference unit 46, dataset evaluation unit 47, pseudo-labeling unit 48, retraining unit 49, and accuracy improvement evaluation unit 50 are all implemented by the CPU 31 and dataset generation program 37 shown in Figure 4.

[0029] The layout / frame image collection unit 40, after checking the device DB (see Figure 7), finds a new data source (cameras installed in a new customer's store (including both the built-in camera 5 of the signage 4 and the fixed camera 3 connected to the analysis box 2), or cameras newly installed in an existing customer's store), and collects captured images (frame images) that do not show people from each of the new (multiple) cameras. These frame images, which do not show people and only show the background of the store, will be referred to as layout / frame images in the following description. The image feature extraction unit 41 extracts features from each of the layout / frame images (captured images that do not show people) collected by the layout / frame image collection unit 40. The image clustering unit 42 groups the layout / frame images collected by the layout / frame image collection unit 40 based on the features extracted by the image feature extraction unit 41. The camera classification unit 43 groups the cameras (built-in camera 5 and fixed camera 3) that captured the layout frame images based on the grouping results of the image clustering unit 42, thereby creating the new grouping of multiple cameras described above. The representative camera selection unit 44 selects a representative camera for each group created by the camera classification unit 43.

[0030] The frame image collection unit 45 collects frame images containing people from each of the representative cameras selected by the representative camera selection unit 44, using an operation unit 35 such as a mouse, that matches the selection criteria set by the system administrator. In this case, it is desirable for the frame image collection unit 45 to collect frame images that show an increase in the number of people, a movement of people's positions by a predetermined percentage or more, or a change in people's posture by a predetermined percentage or more compared to the previous frame image. In this embodiment, the inference processing performed by the target predetermined inference processing DNN model (the same type of inference processing as the inference processing performed by the trained DNN model for inference processing on the edge device) is an inference processing related to people (because the object to be recognized is a person (more precisely, a person, face, or head)), so the explanation will be given using the case where the frame image collection unit 45 collects frame images containing people as an example. However, the frame images collected by the frame image collection unit 45 do not need to contain people; they may also contain frame images of objects other than people, such as animals, vehicles, or products. Furthermore, if the object to be recognized is not a person, it is desirable for the frame image collection unit 45 to collect frame images in which the number of objects to be recognized has increased, the position of the objects to be recognized has shifted by a predetermined percentage or more, or the posture of the objects to have changed by a predetermined percentage or more, compared to the previous frame image. The evaluation inference unit 46 performs evaluation inference processing on each of the frame images collected by the frame image collection unit 45.

[0031] The dataset evaluation unit 47 evaluates, based on the results of the inference processing by the evaluation inference unit 46, whether the dataset consisting of frame images collected by the frame image collection unit 45 is suitable as a training dataset for a predetermined inference DNN model. More specifically, the dataset evaluation unit 47 evaluates, based on the results of the inference processing by the evaluation inference unit 46, whether the dataset consisting of frame images collected by the frame image collection unit 45 is suitable as a training dataset for fine-tuning the predetermined inference DNN model. If the evaluation value of the dataset by the dataset evaluation unit 47 is not equal to or greater than the target value (NO in S1), the CPU 31 of the learning server 1 repeats the collection of frame images containing people by the frame image collection unit 45 until the evaluation value is equal to or greater than the target value. If the system administrator deems it necessary during the repetition, they can use the operation unit 35 to add or change the frame image selection criteria, and then instruct the CPU 31 of the learning server 1 to repeat the collection of frame images containing people by the frame image collection unit 45. Furthermore, if the CPU 31 of the learning server 1 determines in the judgment of S1 above that the evaluation value of the dataset by the dataset evaluation unit 47 is not equal to or greater than the target value, and determines that there is a bias (class imbalance) in the class of each frame image in the dataset, it performs a process to remove (delete) some of the frame images belonging to the majority class from the dataset.

[0032] If the evaluation value of the dataset by the dataset evaluation unit 47 is equal to or greater than the target value (YES in S1), the retraining unit 49 uses the dataset composed of frame images collected by the frame image collection unit 45 as a training dataset to perform fine-tuning (a type of retraining) of the pre-trained DNN model for the predetermined inference process. More specifically, if the evaluation value of the dataset is equal to or greater than the target value (YES in S1), the pseudo-labeling unit 48 performs pseudo-labeling on each frame image collected by the frame image collection unit 45. Then, the retraining unit 49 performs fine-tuning of the pre-trained DNN model for the predetermined inference process based on the frame images collected by the frame image collection unit 45 and the correct labels assigned to each frame image by the pseudo-labeling unit 48.

[0033] The accuracy improvement evaluation unit 50 evaluates whether the accuracy of the inference processing by the trained DNN model after fine-tuning by the retraining unit 49 has improved by a predetermined value or more compared to the accuracy of the inference processing by the trained DNN model before fine-tuning. If the accuracy of the inference processing by the trained DNN model after fine-tuning has not improved by a predetermined value or more (NO in S2), the CPU 31 of the learning server 1 repeats the following processes until the accuracy of the inference processing improves by a predetermined value or more: the collection of frame images containing people by the frame image collection unit 45, the pseudo-labeling process by the pseudo-labeling unit 48, the fine-tuning process of the trained DNN model for inference processing by the retraining unit 49, and the accuracy evaluation process by the accuracy improvement evaluation unit 50. During the repetition described above, if deemed necessary, the system administrator uses the operation unit 35 to add or change the frame image selection criteria, and then instructs the CPU 31 of the learning server 1 to repeat the processing by the frame image acquisition unit 45, the pseudo-labeling unit 48, the retraining unit 49, and the accuracy improvement evaluation unit 50.

[0034] Furthermore, from the perspective of balancing the classes in the training dataset (frame images), if the accuracy of the inference processing by the trained DNN model after the fine-tuning described above has not improved by a predetermined value (NO in S2), it is desirable for the CPU 31 of the training server 1 to repeat the following until the accuracy of the inference processing improves by a predetermined value: collecting frame images containing people by the frame image collection unit 45 (or deleting frame images to balance the classes to which each frame image in the dataset belongs), inference processing by the evaluation inference unit 46, evaluation processing of the dataset by the dataset evaluation unit 47, determination of whether the evaluation value of the dataset in S1 is equal to or greater than the target value, pseudo-labeling processing by the pseudo-labeling unit 48, and fine-tuning processing of the trained DNN model for inference processing by the retraining unit 49.

[0035] If the evaluation value from the accuracy improvement evaluation unit 50 has improved by a predetermined value or more (YES in S2), the CPU 31 of the learning server 1 uses the communication unit 36 ​​to send the trained DNN model after fine-tuning by the retraining unit 49 to the edge devices corresponding to the cameras included in the new data source (the signage 4 corresponding to the new built-in camera 5 and the analysis box 2 connected to the new fixed camera 3) for storage. More specifically, the CPU 31 of the learning server 1 stores the trained DNN model after fine-tuning by the retraining unit 49 as a trained DNN model 20 for inference processing in the memory 16 of the signage 4 corresponding to the new built-in camera 5, and also stores it as a trained DNN model 28 for various inference processing in the hard disk 22 of the analysis box 2 connected to the new fixed camera 3.

[0036] Next, with reference to Figures 6 to 13, the overall processing flow in this dataset generation system 10 and the user interface (UI) screen used by the system administrator on the learning server 1 side will be explained. In the flowchart of Figure 6, when the CPU 31 of the learning server 1 checks the device DB 50 shown in Figure 7 and finds a new data source (a camera installed in a new customer's store (including both the built-in camera 5 of the signage 4 and the fixed camera 3 connected to the analysis box 2), or a camera newly installed in an existing customer's store) (YES in S11), it collects the above-mentioned layout frame images (images taken without people) from each of the new cameras and groups (clusters) the cameras (built-in camera 5 and fixed camera 3) that took the above-mentioned layout frame images based on the characteristics of each of the collected layout frame images (S12). Then, the CPU 31 of the learning server 1 (the representative camera selection unit 44) selects a representative camera in each of the above-mentioned groups (S13).

[0037] The details of the processes (clustering process) S11 to S13 in the flowchart of Figure 6 above will be explained with reference to the flowchart of Figure 7. As described in S11 of Figure 6 above, the CPU 31 of the learning server 1 accesses the device DB 50 at a predetermined date and time to check for new customers and cameras (cameras installed in the new customer's store (including both the built-in camera 5 and the fixed camera 3), or cameras newly installed in the existing customer's store) (S31). If there are new customers and cameras (in short, new cameras), the CPU 31 of the learning server 1 collects information on the new cameras (more precisely, information on the signage 4 with the built-in camera 5 and information on the analysis box 2 connected to the fixed camera 3) from the device DB 50 (S32). Then, the CPU 31 of the learning server 1 compiles a layout frame image acquisition request (command) for the edge devices corresponding to the new cameras (in short, the signage 4 corresponding to the new built-in camera 5 and the analysis box 2 connected to the new fixed camera 3) (S33), and sends the compiled layout frame image acquisition request to the edge devices corresponding to the new cameras using the communication unit 36 ​​(S34).

[0038] When the edge devices (signage 4 and analysis box 2) corresponding to the new cameras mentioned above receive the above-mentioned layout frame image collection request (file) from the learning server 1 (S35), they extract frame images from the images captured by each camera (fixed camera 3 or signage 4) connected to their own devices at the date and time specified in the layout frame image collection request. If a layout frame image is found among the extracted frame images from each camera (YES in S37), they send the layout frame images from each camera to the learning server 1 (S38). Here, it is desirable that the date and time specified in the above-mentioned layout frame image collection request (the date and time when the frame images from each camera were captured) be a date and time for multiple time periods, such as before opening, in the morning (after opening), noon, and evening. By collecting layout frame images found from frame images captured at multiple time periods in this way, temporal bias can be removed from the collected layout frame images.

[0039] When the learning server 1 receives layout frame images from the edge devices (signage 4 and analysis box 2) corresponding to the new cameras (S39), it stores these layout frame images in the data DB 51 (S40).

[0040] Once all the new camera layout frame images are stored in the data DB51, the CPU 31 of the learning server 1 uses a state-of-the-art AI model for image feature extraction (e.g., pretrained ResNet50) stored in the reference model DB52 to extract feature vectors (e.g., 2048-dimensional feature vectors) from each of the new camera layout frame images stored in the data DB51 (S41). For example, if there are 200 new cameras, 200 feature vectors will be extracted from each of the layout frame images (200 in total) of these cameras.

[0041] Next, the CPU 31 of the learning server 1 uses state-of-the-art clustering machine learning algorithms stored in the reference model DB 52 to cluster (group) the new camera layout frame images stored in the data DB 51 based on the feature vectors of each extracted layout frame image (S42). Examples of machine learning algorithms used for this clustering include t-SNE (t-distributed Stochastic Neighbor Embedding), GMM (Gaussian mixture model), KNN (K-Nearest Neighbor Algorithm), and PCA (Principal Component Analysis).

[0042] Next, the CPU 31 of the learning server 1 uses the most advanced representative frame image selection algorithm stored in the reference model DB 52 to select a representative frame image (hereinafter referred to as the "representative frame image") from among the layout frame images belonging to each cluster (each group) that was grouped in S42 (S43). Here, the clustering (grouping) of layout frame images in S42 is equivalent to grouping the cameras (built-in camera 5 and fixed camera 3) that captured these layout frame images. Therefore, the process of selecting a representative frame image from among the layout frame images belonging to each group in S43 is substantially the same as the process of selecting a representative camera in each grouped camera group. Examples of the algorithm for selecting the representative frame image include selecting the layout frame image located at the center of each cluster as the representative frame image of each cluster, or selecting the layout frame image with the largest number of neighboring layout frame images (number of similar layout frame images) in each cluster as the representative frame image of each cluster.

[0043] Figure 8 shows a screen (data source clustering UI 55) for the user interface (UI) used by the system administrator on the learning server 1 when designing the subsystem for clustering the new data source (new camera) described above. In the following description, "subsystem" refers to the group of software modules included in the dataset generation program 37 of the learning server 1. The data source clustering UI 55 has a toolbox section 56, a system editing section 57, and a results section 58. The toolbox section 56 displays selection buttons for the machine learning algorithm (clustering algorithm) used for clustering the layout frame image as described in S42 of Figure 7, and selection buttons for the evaluation criteria of the layout frame image received in S39 of Figure 7. In the example shown in Figure 8, it is possible to select GMM, KNN, PCA, and t-SNE as the clustering algorithm, and "detection," "pose estimation," and "face recognition" as the evaluation criteria. However, in the data source clustering UI 55, the evaluation function block (evaluation block) 65 in the system editing field 57 only evaluates the absence of people, so only the "detection" (of people) from the above evaluation criteria is used.

[0044] In the system editing section 57 of Figure 8, the following functional blocks are used by the system administrator during system design: initialization block 59, selection criterion definition block 60, filter setting 61, image data collection request block 62, collection request transmission block 63, response reception block 64, evaluation block 65, clustering block 66, and visualizer block 67.

[0045] The clustering process implemented by the above-mentioned functional blocks 59-66 is as explained in Figure 7, etc., but there are two important points in the configuration process (design) using this data source clustering UI 55. First, how to set the selection criteria for image data (frame images) (how to set the image data collection filter). For example, a system administrator can set the image data collection filter (set the selection criteria for frame images) shown in the Config column 68 by clicking the filter setting 61 on the operation unit 35. When collecting layout frame images for clustering, the selection criteria for frame images are, for example, as shown in the Config column 68 of Figure 8, (shooting) start time = 8:00 AM on May 17, 2022 (before opening), period = 10 seconds, number of people = 0. By adopting such frame image selection criteria, the learning server 1 collects at least one captured image (frame image) from each of the new (multiple) cameras that does not show people between exactly 8:00 AM and 8:00:10 AM on May 17, 2022. In practice, however, it is desirable to collect at least one layout frame image from the frame images taken by each camera at multiple time points, as described above.

[0046] Another important point in the setup process (design) using this data source clustering UI 55 is the selection process of the clustering algorithm displayed in the toolbox section 56. The left side of the results section 58 in Figure 8 shows the clustering results when the system administrator selects KNN from the clustering algorithms displayed in the toolbox section 56 using the operation unit 35. However, as explained in S41 of Figure 7, if a 2048-dimensional feature vector is extracted from each of the new camera layout frame images, in order to obtain a 2-dimensional clustering result like the one shown on the left side of the results section 58 in Figure 8, it is necessary to reduce the 2048-dimensional feature vector to 2 dimensions and then cluster it with KNN.

[0047] The Config column 69 on the right side of the data source clustering UI 55 shows the criteria for evaluating the frame images (layout frame images) received from the edge devices (signage 4 and analysis box 2), and the right side of the results column 58 shows the results of the evaluation of the frame images received from the edge devices. Since the frame images used in the clustering process are layout frame images that do not contain people, the number of people in the frame image ("nbr of p") is 0 in both the evaluation criteria shown in the Config column 69 and the evaluation results shown on the right side of the results column 58, and the number of bound boxes ("bbx / frame") (corresponding to detected people, faces, or heads) for each frame image is 0.

[0048] Once the clustering process described above is complete, the CPU 31 of the learning server 1 begins collecting frame images necessary to generate a training dataset for a predetermined DNN model for inference processing. Specifically, the CPU 31 of the learning server 1 first collects the minimum number of frame images containing people required for the training dataset, based on the mechanism edited in the data collection selection UI 72 shown in Figure 9.

[0049] The process of collecting the minimum required number of frame images is carried out as follows. First, using the data collection selection UI 72 shown in Figure 9, the system administrator can configure the image data collection filter (setting the frame image selection criteria) shown in the Config field 87 by, for example, clicking the filter setting 80 on the operation unit 35. In this example of the Config field 87 shown in Figure 9, the frame image selection criteria are expressed by a mathematical formula. In addition to writing the mathematical formula in the Config field 87, the system administrator can configure the frame image selection criteria (setting the image data collection filter) by editing the image data collection request program described in the Config field 88. Once this frame image selection criteria setting operation is completed (S14 in Figure 6), the CPU 31 of the learning server 1 executes the process edited in the data collection selection UI 72 shown in Figure 9, and collects diverse frame images containing people that match the selection criteria set in S14 from the representative camera (in each camera group) selected in S13, until the minimum required number of frame images has been collected (S15 and S16). In the example shown in Figure 9, the formula shown in Config column 87 selects frames where the number of people has increased compared to the previous frame, or where there has been movement (movement of people's positions or changes in people's postures). Also, in the example shown in Figure 9, the image data collection request program described in Config column 88 is set to detect five or more people, where heads are detected, but faces are not.

[0050] Next, referring to Figure 10, an example of a frame image selection algorithm according to the selection criteria set in S14 of Figure 6 above will be explained. The CPU 31 of the learning server 1 assigns a tracking ID to each person detected in the frame images taken by the representative camera. More specifically, the CPU 31 of the learning server 1 performs tracking processing (tracking processing) of customers (people 92) photographed by the same representative camera by assigning the same ID to the same customer across frames, for example, based on the time of shooting each frame image taken by the same representative camera and the coordinate position (or coordinate position and size of the person) of the person 92 detected by the trained DNN model for person detection for these frame images. Then, the CPU 31 of the learning server 1 uses the above tracking ID (using the tracking process) to select the first frame image (in the example of Figure 10, the frame image taken at time t_0) from among the frame images taken by a certain representative camera that satisfies the "minimum number of people" criterion (5 people in the examples of Figures 9 and 10) set in the data collection selection UI 72 above, and uses that as the frame image to be collected. In the example of Figure 10, the following explanation is given assuming that the frame rate of the frame images sent from each representative camera is 1 fps.

[0051] When the CPU 31 of the learning server 1 completes the selection process for the frame image at the above capture time t_0, it checks whether the frame image at the next capture time t_1 (=t_0+1s) satisfies the selection criteria set in the data collection selection UI 72. In this case (when the previous frame image satisfies the "minimum number of people" criterion), the frame image selection criteria are, in addition to the "minimum number of people" criterion (5 people), that the new frame image satisfies one of the following conditions from the previous frame image: (1) the number of people has increased, (2) the position of the people has moved (more precisely, 20% or more of the people have moved from their positions in the previous frame image), or (3) the posture of the people has changed (more precisely, 20% or more of the people have changed the posture of their body / head / face from the previous frame image). Since the frame image at capture time t_1 does not satisfy any of the above conditions (1) to (3), the CPU 31 of the learning server 1 does not select this frame image at capture time t_1. In contrast, the frame image taken at the next shooting time t_2 (=t_1+1s) after the shooting time t_1 does not satisfy condition (1) above, but it does satisfy condition (2), so the CPU 31 of the learning server 1 selects this frame image taken at time t_2. Also, the frame image taken at the next shooting time t_3 (=t_2+1s) after the shooting time t_2 satisfies condition (1) above, so the CPU 31 of the learning server 1 selects this frame image taken at time t_3. And the frame image taken at time t_i (=t_(i-1)+1s) does not satisfy conditions (1) and (2) above, but it satisfies condition (3), so the CPU 31 of the learning server 1 selects this frame image taken at time t_i.

[0052] To summarize the frame image selection process shown in Figure 10 above, when the previous frame image was a frame image that met the "minimum number of people" criterion, frame images that have not changed from the previous frame image (i.e., do not meet any of the conditions (1) to (3) above) are not selected. Instead, only frame images that have changed from the previous frame image (i.e., meet any of the conditions (1) to (3) above) are selected. This enables the efficient collection of diverse frame images containing people from the representative camera.

[0053] Next, the details of the data collection selection UI 72 shown in Figure 9 will be explained. This data collection selection UI 72 is a user interface (UI) screen used by the system administrator on the learning server 1 when designing the subsystem (software module group) for selecting and collecting frame images from the representative camera mentioned above. The data collection selection UI 72 has an AI toolbox section 73, a system toolbox section 74, a system editing section 75, and a results section 76. The system editing section 75 is further divided into an upper collection area 75a and a lower evaluation area 75b. The system administrator designs the subsystem for selecting and collecting frame images from the representative camera by placing functional blocks related to the selection and collection of frame images in the collection area 75a, placing functional blocks related to the evaluation of the collected frame images in the evaluation area 75b, and setting selection criteria in the Config section 87 and Config section 88.

[0054] In the system editing section 75 of the data acquisition selection UI 72, the initialization block 78, selection criterion definition block 79, filter settings 80, image data acquisition request block 81, acquisition request transmission block 82, response / data reception block 83, evaluation block 84, processing result block 85, and visualizer 86 are used, which are among the function blocks that the system administrator can select from the system toolbox section 74.

[0055] Of the functional blocks described above, functional blocks 78-82 located in the collection area 75a and the response / data reception block 83 located in the evaluation area 75b are used for the system administrator's setting of frame image selection criteria in S14 of Figure 6 and for the frame image collection process from each representative camera in S15. In addition, the evaluation block 84 located in the evaluation area 75b performs the inference processing necessary for evaluating the frame images received from each representative camera by the response / data reception block 83. In this data collection selection UI 72 (i.e., in the stage where the minimum required number of frame images are collected), the inference processing performed by the evaluation block 84 is the same type of inference processing as the human detection processing in each frame image and the inference processing performed by the predetermined inference processing DNN model that is the target (the same type of inference processing as the inference processing performed by the pre-trained DNN model for inference processing on the edge device). However, the inference processing performed by the evaluation block 84 uses a pre-trained DNN model for inference processing that is heavier and more accurate than the pre-trained DNN model for inference processing on the edge device. Furthermore, this evaluation block 84 is used to set evaluation criteria for frame images collected from each representative camera. For example, by clicking evaluation block 84 on the operation unit 35, the system administrator can input the evaluation criteria for frame images shown in the Config field 89. In the Config field 89, the criteria (for evaluation) and their range (minimum and maximum values ​​considered normal) can be set.

[0056] Furthermore, the processing result block 85 outputs the results of the inference processing for evaluation performed in the evaluation block 84 above to the evaluation value determination block in S51 and to the evaluation table 91 on the right side of the result column 76.

[0057] In the S51 judgment block in Figure 9, the CPU 31 of the learning server 1 determines whether the evaluation value for the frame image received from each representative camera by the response / data reception block 83 has reached the KPI (target). Specifically, in the system designed with this data collection selection UI 72, the objective is to collect the minimum number of frame images containing people necessary for the training dataset, so the KPIs in S51 are the following three.

[0058] The first KPI (goal) is to collect a predetermined number of frame images (the minimum number required). For example, to collect 1000 frame images from each representative camera. The second KPI (goal) is that each frame image contains a person. For example, the processing result block 85 performs person detection on all frame images received from each representative camera by the response / data reception block 83, and as a result, a person is detected in all frame images. The third KPI (goal) is that the evaluation block 84 performs the same type of inference processing on the frame images received from each representative camera by the response / data reception block 83 as the trained DNN model for inference processing on the edge device, and as a result, all frame images received from each representative camera contain content that belongs to one of the classes. Specifically, for example, if the pre-trained DNN model for inference processing on the edge device (the target pre-defined inference processing DNN model) is a pre-trained DNN model for head / face detection, the third KPI is that, as a result of the evaluation block 84 performing head / face detection processing on all frame images received from each representative camera, at least one object belonging to one of the classes that can be assigned by the pre-trained DNN model for head / face detection (i.e., "head" or "face") is detected in all frame images received from the aforementioned representative cameras. The purpose of setting this third KPI is to reduce the imbalance in the classes of objects (more precisely, people) included in the frame images within the training dataset. Such class imbalances reduce the inference accuracy for minority classes.

[0059] In the determination in S51 above, if the CPU 31 of the learning server 1 determines that the evaluation value for the frame images received from each representative camera by the response / data reception block 83 has not reached the KPI (target) (if it determines that there is a KPI among the three KPIs that has not been achieved) (NO in S51), the system administrator, if necessary, uses the operation unit 35 to add or change the frame image selection criteria in Config fields 87 and 88. The system administrator then uses the operation unit 35 to instruct the CPU 31 of the learning server 1 to repeat the processing of the selection criteria definition block 79, image data collection request block 81, collection request transmission block 82, response / data reception block 83, evaluation block 84, and processing result block 85 until the evaluation value reaches the KPI (target) (until all three KPIs are achieved).

[0060] To the right of the results column 76 in the data collection selection UI 72 described above, an evaluation table 91 is displayed, showing how the frame images received from each representative camera were evaluated against the evaluation criteria for frame images entered in the Config column 89. The evaluation results displayed in the evaluation table 91 are obtained from the inference processing performed by the evaluation block 84, except for the number of frame images received from each representative camera ("nbr of frames").

[0061] To the left of the results field 76 in the data collection selection UI 72 described above, multiple sample images 90 obtained by the visualizer 86 are displayed. These sample images 90 are frame images sampled from frame images received from each representative camera, and the results of the inference processing performed by the evaluation block 84 (e.g., bounding boxes for people or heads) are overlaid on these sample images 90 for visualization of the processing results. In other words, the visualizer 86 is a visualization tool for displaying sample frame images 90 with correct labels, and is a tool used by system administrators to check what is happening, especially when the evaluation value for frame images received from each representative camera does not reach the KPI (target).

[0062] In the determination at S51 in Figure 9 above, if the evaluation value for the frame image received from the representative camera reaches the KPI (target), that is, if the collection of the minimum required number of frame images is completed at S16 in the flowchart of Figure 6 (YES at S16), the CPU 31 of the learning server 1 performs a pseudo-labeling process for evaluation (the "inference process for evaluation" in the claim) for each of the frame images collected from each representative camera (S17). Then, based on the results of the pseudo-labeling process for these frame images, the quality of the dataset composed of the frame images collected from each representative camera is evaluated (S18). More specifically, based on the results of the pseudo-labeling process for the frame images collected from each representative camera, it is evaluated whether the dataset composed of the frame images collected from each representative camera is suitable as a training dataset for fine-tuning a predetermined DNN model for inference processing that is the target.

[0063] The evaluation of the dataset quality based on the results of the pseudo-labeling process described above is as follows: The results of the pseudo-labeling process on the frame images included in the dataset indicate whether or not the dataset has any bias. For example, if the target DNN model for inference is a DNN model for face detection, or a DNN model for inference on images of faces detected by a DNN model for face detection (e.g., a DNN model for inference such as gender and age estimation, face vector extraction, or person identification), and more than 80% of the frame images included in the dataset are frame images in which the size of the face is smaller than the predetermined size, then the dataset has a bias that the size of the faces in the frame images is small, and is therefore unsuitable as a training dataset for fine-tuning the target DNN model for inference. In order to evaluate the quality of the dataset as described above, the CPU 31 of the training server 1 performs face detection processing using a trained DNN model for face detection as a pseudo-labeling process on each frame image collected from each representative camera to obtain the size of the bounding box of the face. Then, it is determined whether the size of the bounding box of the face obtained by face detection processing (pseudo-labeling processing) for each frame image falls within the normal range (minimum to maximum value) of face size shown in Config column 111 of Figure 11. As a result of this determination, if there are many frame images in the dataset in which the face size falls within the normal range, the "overall evaluation value" (of the dataset quality) in the determination S53 of Figure 11 will be high. Conversely, if there are few frame images in which the face size falls within the normal range, the "overall evaluation value" will be low.

[0064] Furthermore, by using the current version of the pre-trained DNN model for the target inference processing DNN model to perform pseudo-labeling on the frame images included in the dataset, and comparing the results of this pseudo-labeling by the current version of the pre-trained DNN model with the results of inference processing by a more heavier and more accurate pre-trained DNN model for inference processing, if the probability of determining a false positive is high, then this dataset has a bias that contains many frame images that are likely to be determined as false positives, and is therefore unsuitable as a training dataset for fine-tuning the target inference processing DNN model. Note that even if the result of the pseudo-labeling (inference) by the current version of the pre-trained DNN model is positive, if the result of the inference processing by the above-mentioned high-accuracy pre-trained DNN model is negative, then the above-mentioned false positive determination is made. For example, if the target inference DNN model is a face detection DNN model, and for a given frame image, the current version of the trained face detection DNN model performs a pseudo-labeling process (inference process) and assigns a correct face label. However, if a more complex and accurate trained face detection DNN model performs an inference process and fails to detect a face (in the frame image), then it is judged as a False Positive. This judgment is performed for each frame image included in the dataset. If the probability of being judged as a False Positive is high, the "overall evaluation value" (of the dataset quality) in the judgment S53 in Figure 11 will be low. Conversely, if the probability of being judged as a False Positive is low, the "overall evaluation value" will be high.

[0065] Furthermore, if the target DNN model for inference is a DNN model for estimating a person's gender, and more than 80% of the frame images in the dataset are images showing only women, then this dataset has a bias that there are more women in the frame images, and is therefore unsuitable as a training dataset for fine-tuning the target DNN model for estimating a person's gender. To evaluate the quality of such a dataset, the CPU 31 of the training server 1 performs a pseudo-labeling process on each frame image collected from each representative camera, performing gender estimation processing using the trained DNN model for estimating a person's gender, and assigns the correct label of "male" or "female" to each frame image. Then, it determines whether the proportion of "male" or "female" labels in the total number of correct labels falls within the normal range. If the ratio of "males" or "females" falls within the normal range as a result of this determination, the "overall evaluation value" (of the quality of the dataset) in the S53 determination in Figure 11 will be high. Conversely, if the ratio does not fall within the normal range, the "overall evaluation value" will be low. It is desirable that the pre-trained DNN model used for gender estimation in the above pseudo-labeling process is a more heavier and more accurate inference-oriented pre-trained DNN model than the pre-trained DNN model used for gender estimation on the edge device.

[0066] The simplest method for evaluating the quality of a dataset based on the results of the pseudo-labeling process described above is as follows: When the target DNN model for inference is a DNN model for detecting people, faces, or heads, or a DNN model for object recognition of people or faces, it is preferable that the number of people ("nbr of p") in the frame image falls within a certain range (for example, between 2 and 10 people), as shown in the Config column 111 of Figure 11. In such cases, the CPU 31 of the learning server 1 performs a pseudo-labeling process on each frame image collected from each representative camera, using a trained DNN model for person detection to determine the number of people in each frame image. Then, it calculates the ratio of frame images in the dataset where the number of people in the frame image falls within the range shown in the Config column 111 of Figure 11. As a result, if the proportion of frame images in the dataset in which the number of people depicted falls within the range shown in the Config column 111 of Figure 11 is high, the "overall evaluation value" (of the dataset quality) in the S53 judgment of Figure 11 will be high. Conversely, if the proportion of frame images in which the number of people falls within the above range is low, the "overall evaluation value" will be low.

[0067] As described above, the results of the pseudo-labeling process on the frame images included in the dataset indicate whether or not the dataset has any bias. Therefore, by assigning correct labels to the frame images included in the dataset using some kind of pseudo-labeling model such as one for person detection, face detection, head detection, or pose estimation, it is possible to determine whether the current dataset falls within the normal range (minimum to maximum range) of each evaluation criterion shown in the Config column 111 of Figure 11, and from these determination results, an "overall evaluation value" (of the dataset quality) can be obtained.

[0068] If, as a result of the dataset quality evaluation in S18 above, it is determined that the "overall evaluation value" (of the dataset quality) has not reached the KPI (target) (NO in S19), the system administrator may, if necessary, use the operation unit 35 to add or change the frame image selection criteria in Config fields 109 and 110 of Figure 11, and then use the operation unit 35 to instruct the CPU 31 of the learning server 1 to improve the dataset (supplement or delete frame images) according to the results of the dataset quality evaluation above. As a result, the CPU 31 of the learning server 1 repeats the processes in S17 to S20 above until the overall evaluation value reaches the KPI (target), supplementing frame images showing people from each representative camera and deleting unnecessary frame images (S20), thereby improving the quality of the dataset. For example, if the number of frame images received from each representative camera ("nbr of frames") does not reach a predetermined number, the CPU 31 of the learning server 1 collects frame images showing people from each representative camera and supplements these frame images in the dataset. Furthermore, if the pseudo-labeling process in S17 results in a bias (class imbalance) in the labels (classes) assigned to each frame image in the dataset, the CPU 31 of the learning server 1 attempts to balance the classes in the dataset (frame images) by removing (deleting) some of the frame images belonging to the majority class from the dataset.

[0069] Next, we will explain the details of the data quality evaluation UI 93 shown in Figure 11. This data quality evaluation UI 93 is a user interface (UI) screen used by the system administrator on the learning server 1 side when designing a subsystem for evaluating the quality of a dataset composed of frame images collected from each representative camera and for supplementing (adding) frame image data to this dataset. This data quality evaluation UI 93 has an AI toolbox section 94, a system toolbox section 96, a system editing section 95, and a results section 97. Furthermore, the system editing section 95 is divided into an upper collection area 95a and a lower pseudo-labeling evaluation area 95b. The system administrator places functional blocks related to the selection and collection of frame images in the collection area 95a, and functional blocks related to the evaluation of the quality of the dataset composed of the collected frame images (via pseudo-labeling) in the pseudo-labeling evaluation area 95b. In addition, by performing setting operations to add and change selection criteria in the Config fields 110 and 111, the system administrator designs subsystems for evaluating the quality of the dataset and supplementing the dataset with frame image data.

[0070] In the system editing section 95 of the data quality evaluation UI 93, the following functional blocks are used, which can be selected by the system administrator from the system toolbox section 96: initialization block 100, selection criterion definition block 101, filter settings 102, image data collection request block 103, collection request transmission block 104, supplementary data reception block 105, pseudo-labeling block 106, processing result block 107, and visualizer 108.

[0071] Of the functional blocks described above, functional blocks 100 to 104 located in the collection area 95a and the supplemental data receiving block 105 located in the pseudo-labeling evaluation area 95b are used for the system administrator's addition and modification of frame image selection criteria and the frame image collection (supplementary) process from each representative camera, as described in the explanation of S20 in Figure 6. The pseudo-labeling block 106 located in the pseudo-labeling evaluation area 95b corresponds to the evaluation inference unit 46 in Figure 5 and performs evaluation pseudo-labeling processing (evaluation inference processing in the claim) for each frame image received from each representative camera (i.e., it performs the processing of S17 in Figure 6). This pseudo-labeling block 106 is also used to set evaluation criteria for frame images collected from each representative camera. For example, by clicking the pseudo-labeling block 106 on the operation unit 35, the system administrator can input the evaluation criteria for the quality of frame images (datasets) shown in the Config field 111. In Config section 111, you can input settings for each evaluation criterion of frame image (dataset) quality and its normal range (minimum and maximum values ​​considered normal).

[0072] Furthermore, the processing result block 107 outputs the results of the pseudo-labeling process for evaluation performed in the pseudo-labeling block 106 described above. The results of this pseudo-labeling process for evaluation are used for the judgment process in the overall evaluation value judgment block in S53 and for outputting values ​​corresponding to each criterion in the evaluation table 113 on the right side of the result column 97.

[0073] In the S53 judgment block in Figure 11, the CPU 31 of the learning server 1 determines whether the overall evaluation value of the dataset quality, obtained through pseudo-labeling processing, has reached the KPI (target).

[0074] In the determination in S53 above, if the CPU 31 of the learning server 1 determines that the overall evaluation value of the dataset quality has not reached the KPI (target) (NO in S53), the system administrator may, if necessary, use the operation unit 35 to add or change the frame image selection criteria in Config field 109 and Config field 110. Then, if, as a result of the evaluation of the dataset quality above, the number of frame images received from each representative camera ("nbr of frames") does not reach a predetermined number, the system administrator will use the operation unit 35 to instruct the CPU 31 of the learning server 1 to repeat the processing of the image data collection request block 103, the collection request transmission block 104, the supplementary data reception block 105, the pseudo-labeling block 106, the processing result block 107, and the overall evaluation value determination block in S53 until the overall evaluation value reaches the KPI (target). However, if the pseudo-labeling process results in a bias (class imbalance) in the labels (classes) assigned to each frame image in the dataset, the CPU 31 of the learning server 1 attempts to balance the classes in the dataset (frame images) by removing (deleting) some of the frame images belonging to the majority class from the dataset. In this case, after performing the process of deleting some of the frame images belonging to the majority class, the CPU 31 of the learning server 1 repeats the judgment process in S53 above.

[0075] As can be seen from the processing flow explained in Figure 6, the processing on the data quality evaluation UI93 side, which is performed immediately after the collection of the minimum number of frame images required as defined in the data collection selection UI72 is completed, consists of the processing of the pseudo-labeling block 106 and the processing result block 107, and the determination of the overall evaluation value by the determination block S53. If, in the first determination of S53, the CPU 31 of the learning server 1 determines that the overall evaluation value of the dataset quality has reached the KPI, then the processing of blocks 101 to 105 above is not performed even once.

[0076] To the right of the results column 97 in the data quality evaluation UI93 described above, an evaluation table 113 is displayed, showing how the evaluation results for the dataset, which consists of frame images received from each representative camera, matched the evaluation criteria for the dataset entered in the Config column 111. The evaluation results displayed in the evaluation table 113 are the evaluation results obtained from the pseudo-labeling process for evaluation performed by the pseudo-labeling block 106.

[0077] On the left side of the results column 97 in the data quality evaluation UI 93 described above, multiple sample images 112 obtained by the visualizer 108 are displayed. These sample images 112 are frame images sampled from frame images received from each representative camera, and the results of the pseudo-labeling process for evaluation performed by the pseudo-labeling block 106 (e.g., bounding boxes for people and heads) are overlaid on these sample images 112 for visualization of the processing results. In other words, the visualizer 108 is a visualization tool for sample displaying frame images 112 with correct labels obtained by the pseudo-labeling process, and is a tool used by system administrators to check what is happening, especially when the overall evaluation value of the dataset quality does not reach the KPI (target).

[0078] In the judgment at S53 in Figure 11 (corresponding to the judgment at S19 in Figure 6), if the overall quality evaluation value of the dataset reaches the KPI (target), that is, if the generation of a dataset deemed suitable as a training dataset for fine-tuning the target inference DNN model is completed, the CPU 31 of the training server 1 uses this dataset (a dataset consisting of frame images collected from a representative camera) as a training dataset to fine-tune the original pre-trained DNN model for inference (S21 in Figure 6). Here, the original pre-trained DNN model for inference may be the current version of the pre-trained DNN model for inference on the edge device (the predetermined target inference DNN model), or it may be a general-purpose pre-trained DNN model that has been machine-trained using a publicly available (usable by the general public) training dataset.

[0079] Once the fine-tuning of the trained DNN model described in S21 is complete, the CPU 31 of the training server 1 evaluates the trained DNN model after the fine-tuning (S22). Specifically, the CPU 31 of the training server 1 performs inference processing on frame images included in an unknown test dataset collected separately using a procedure similar to that shown in the data quality evaluation UI 93, using both the trained DNN model before fine-tuning and the trained DNN model after fine-tuning, and compares the accuracy of the inference processing by the trained DNN models before and after fine-tuning. If the accuracy of the inference processing by the trained DNN model after fine-tuning is improved by a predetermined value or more compared to the accuracy of the inference processing by the trained DNN model before fine-tuning (for example, if the F1 score of the trained DNN model after fine-tuning is improved by 5% or more compared to the F1 score of the trained DNN model before fine-tuning), the CPU 31 of the training server 1 determines that the evaluation value (F1 score) of the processing result by the trained DNN model after fine-tuning is equal to or greater than the KPI (the target has been reached). In this case (NO in S23), the CPU 31 of the learning server 1 considers that the generalization of the target inference DNN model is complete (S25) and distributes the fine-tuned, trained DNN model to the edge devices.

[0080] The test dataset described above is a dataset collected using a procedure similar to that shown in the data quality evaluation UI93 described above. In the S53 judgment of the data quality evaluation UI93, the image data collection request block 103, collection request transmission block 104, supplementary data reception block 105, pseudo-labeling block 106, processing result block 107, and the S53 overall evaluation value judgment block are repeated until the overall evaluation value of the dataset reaches the KPI (target). Note that when generating the test dataset, the evaluation criteria items for the dataset shown in the Config column 111 of Figure 11 are the same as when generating the training dataset described above, but the values ​​of each evaluation criterion item are different. For example, among the evaluation criteria for the dataset, the number of frame images ("nbr of frames") is 1000 or more per camera when generating the training dataset, but 100 or more per camera is sufficient when generating the test dataset.

[0081] Furthermore, the frame images that make up the test dataset mentioned above do not need to be frame images obtained from each representative camera, as was the case when generating the training dataset; they may be frame images obtained from cameras other than the representative camera in each camera group. By using a dataset composed of frame images obtained from cameras other than the representative camera as the test dataset, it is possible to check whether the fine-tuned trained DNN model is spatially generalized or not. Here, "generalization" of the trained DNN model means that the trained DNN model can output the correct result for unknown input data (in this case, unknown frame images). Also, "the trained DNN model is spatially generalized" in this case means that even if the trained DNN model is input with frame images obtained from cameras other than the representative camera that are not included in the training dataset, the trained DNN model can output the correct result in the same way as when it is input with frame images obtained from the representative camera.

[0082] Furthermore, the frame images constituting the above test dataset may be frame images taken by each of the above representative cameras at a different date and time than when the frame images for the training dataset were acquired. By using a dataset consisting of frame images taken by a representative camera at a different date and time than when the frame images for the training dataset were taken as the test dataset, it is possible to check whether the fine-tuned trained DNN model is temporally generalized or not. In this case, "the trained DNN model is temporally generalized" means that even when frame images taken at a different date and time than when the frame images for the training dataset were taken are input to the trained DNN model, the trained DNN model can output the correct results in the same way as when the frame images for the training dataset are input.

[0083] In contrast, if the accuracy of the inference processing by the finely tuned trained DNN model (F1 score) does not improve by a predetermined value or more compared to the accuracy of the inference processing by the pre-tuned trained DNN model (F1 score), the CPU 31 of the training server 1 determines that the evaluation value of the processing result by the finely tuned trained DNN model has not reached the KPI (target) (YES in S23). Thus, if the accuracy of the inference processing by the finely tuned trained DNN model does not improve by a predetermined value or more compared to the accuracy of the pre-tuned trained DNN model (in particular, if the accuracy of the inference processing by the finely tuned trained DNN model is lower than the accuracy of the pre-tuned trained DNN model), it means that the finely tuned trained DNN model is overfitted to the bias of the training dataset and has not generalized. In short, it means that the finely tuned trained DNN model has learned about the data in the training dataset, but has not been able to fit the data included in the unknown test dataset.

[0084] If the answer to S23 is YES (the evaluation value (F1 score) of the processing result by the fine-tuned trained DNN model has not reached the KPI (target)), the system administrator, if necessary, uses the operation unit 35 to add or change the frame image selection criteria in Config fields 137 and 138 of Figure 12, and then uses the operation unit 35 to instruct the CPU 31 of the training server 1 to collect (supplement) frame images containing people from each representative camera. As a result, the processing from S21 to S24 is repeated until the evaluation value (F1 score) reaches the KPI (target), and the supplementation of frame images containing people from each representative camera (S24) is repeated. The reason for this is that if the evaluation value (accuracy) of the processing result by the fine-tuned trained DNN model does not reach the KPI (target), it means that the size of the training dataset used for this fine-tuning is not large enough, and it is necessary to supplement the training dataset with more frame images. When the frame image replenishment process shown in S24 is completed, the CPU 31 of the learning server 1 returns to the process in S17 to perform pseudo-labeling (S17) on the frame images replenished in S24, and evaluation of the dataset quality (S18) based on the results of the pseudo-labeling process. If, as a result of the frame image replenishment process in the training dataset shown in S24, a bias (class imbalance) occurs in the labels (classes) assigned to each frame image in the dataset, the CPU 31 of the learning server 1 attempts to balance the classes in the dataset (frame images) by removing (deleting) some of the frame images belonging to the majority class from the dataset in the process in S20. However, although not highly preferred, when the frame image replenishment process shown in S24 is completed, the CPU 31 of the learning server 1 may return to the process in S21 instead of the process in S17.

[0085] Next, we will explain the data bias evaluation UI120 shown in Figure 12. This data bias evaluation UI120 is a user interface (UI) screen used when designing a subsystem for evaluating a dataset through fine-tuning, which was determined in S19 to have reached the KPI, and for supplementing (adding) frame image data to this dataset. Using the subsystem designed with the data quality evaluation UI93 shown in Figure 11, it is possible to generate a dataset that is considered suitable as a training dataset for fine-tuning a predetermined inference processing DNN model. However, when fine-tuning the original pre-trained DNN model for inference processing using this dataset, it is possible that the accuracy of the inference processing by the fine-tuned pre-trained DNN model may be lower than the accuracy of the inference processing by the (original) pre-trained DNN model, or that there may be little to no improvement in accuracy. The data bias evaluation UI120 shown in Figure 12 is a user interface screen used when designing a subsystem to supplement this dataset with the necessary frame image data and complete the generation of a training dataset for fine-tuning a predetermined inference processing DNN model, even in such cases.

[0086] This data bias evaluation UI 120 has an AI toolbox section 121, a system toolbox section 122, a system editing section 123, and a results section 124. The system editing section 123 is further divided into an upper collection area 123a and a lower fine-tuning evaluation area 123b. The system administrator places functional blocks related to the selection and collection of frame images in the collection area 123a, and functional blocks related to the evaluation of the dataset composed of the collected frame images through fine-tuning in the fine-tuning evaluation area 123b. By adding and changing selection criteria in the Config section 137 and Config section 138, the system administrator designs subsystems for evaluating the dataset through fine-tuning and supplementing the dataset with frame image data.

[0087] In the system editing section 123 of the data bias evaluation UI 120, the following functional blocks are used, which can be selected by the system administrator from the system toolbox section 122: initialization block 128, selection criterion definition block 129, filter settings 130, image data collection request block 131, collection request transmission block 132, supplemental data reception block 133, fine tuning block 134, processing result block 135, and visualizer 136.

[0088] Of the functional blocks described above, functional blocks 128 to 132 located in the collection area 123a and supplemental data reception block 133 located in the fine-tuning evaluation area 123b are used for the addition and modification of frame image selection criteria by the system administrator and the frame image collection (supplementation) process from each representative camera, as described in the explanation of S24 in Figure 6. Furthermore, the fine-tuning block 134 located in the fine-tuning evaluation area 123b uses the dataset composed of frame images collected from each representative camera as a training dataset to perform fine-tuning of the original pre-trained DNN model for inference (i.e., the process of S21 in Figure 6). This fine-tuning block 134 corresponds to the retraining unit 49 in Figure 5.

[0089] Furthermore, the processing result block 135 outputs the result of the inference processing performed by the trained DNN model after fine-tuning in the fine-tuning block 134 described above. This result of the inference processing performed by the trained DNN model after fine-tuning is used for the judgment processing in the F1 score judgment block in S71 and for outputting the F1 score in Table 141 on the right side of the result column 124.

[0090] In the S71 judgment block in Figure 12, the CPU 31 of the learning server 1 determines whether the accuracy of the inference processing by the fine-tuned trained DNN model has reached the KPI (target) (specifically, whether the accuracy of the inference processing by the fine-tuned trained DNN model (F1 score) has improved by a predetermined value or more compared to the accuracy of the inference processing by the trained DNN model before fine-tuning).

[0091] In the determination in S71 above, if the CPU 31 of the learning server 1 determines that the accuracy of the inference processing by the fine-tuned trained DNN model has not reached the KPI (target) (NO in S71), the system administrator may, if necessary, use the operation unit 35 to add or change the frame image selection criteria in Config fields 137 and 138. The system administrator then uses the operation unit 35 to instruct the CPU 31 of the learning server 1 to repeat the processing of the image data collection request block 131, the collection request transmission block 132, the supplementary data reception block 133, the fine-tuning block 134, the processing result block 135, and the F1 score (accuracy) determination block in S71 until the accuracy of the inference processing by the fine-tuned trained DNN model reaches the KPI (target).

[0092] As can be seen from the processing flow explained in Figure 6, the processing on the data bias evaluation UI 120 side, which is performed immediately after the subsystem designed (defined) in the data quality evaluation UI 93 has completed generating a dataset that is deemed suitable as a training dataset for fine-tuning the target predetermined inference DNN model (when the overall evaluation value of the dataset quality reaches the KPI), consists of the processing of the fine-tuning block 134 and the processing result block 135, and the F1 score (accuracy) determination process by the determination block S71. If, in the first determination of S71, the CPU 31 of the learning server 1 determines that the accuracy of the inference processing by the trained DNN model after fine-tuning has reached the KPI (target), then the processing of blocks 129 to 133 above is not performed even once.

[0093] To the right of the results column 124 in the data bias evaluation UI 120, Table 141 is displayed, showing the results (including the F1 score) of the inference processing performed by the trained DNN model after fine-tuning in the fine-tuning block 134.

[0094] On the left side of the results column 124 in the data bias evaluation UI 120 described above, multiple sample images 140 obtained by the visualizer 136 are displayed. These sample images 140 are frame images obtained by sampling frame images included in the above-mentioned unknown test dataset, which are used to test the inference processing by the fine-tuned trained DNN model. The results of the inference processing by the fine-tuned trained DNN model (e.g., bounding boxes for people and heads) are overlaid on these sample images 140 for visualization of the processing results. In other words, the visualizer 134 is a visualization tool for displaying sample frame images 140 with correct labels obtained by the inference processing by the fine-tuned trained DNN model. In particular, it is a tool used by system administrators to check what is happening when the accuracy of the inference processing by the fine-tuned trained DNN model does not reach the KPI (target).

[0095] Whether the accuracy (F1 score) of the inference processing performed by the finely tuned pre-trained DNN model has reached the KPI (target) can be determined, for example, by comparing the results of the inference processing performed by the finely tuned pre-trained DNN model with the results of the inference processing performed by a similar pre-trained DNN model for inference processing that is more computationally intensive and has higher accuracy than the DNN model used for inference processing. As described above, the determination of whether the accuracy (F1 score) of the finely tuned pre-trained DNN model has reached the KPI (target) is basically performed automatically by the CPU 31 of the training server 1, but the system administrator may also make a visual determination by displaying multiple sample images 140 one after another using the visualizer 134 described above.

[0096] The subsystem designed using the data bias evaluation UI120 shown in Figure 12 above determines whether the data in the training dataset, which consists of frame images collected from the data source (edge ​​device), supports the generalization of the target inference DNN model. In other words, it determines whether the pre-trained DNN model for inference, which has been fine-tuned using the training dataset consisting of frame images collected to date, is generalized in terms of time and space. For example, a pre-trained DNN model for inference, trained on data (training dataset) collected from store A (the edge device) may not be generalized to store B, even if store A and store B belong to the same group (same franchise chain, etc.) (it may not be able to output the correct result when frame images from store B are input, just as it does when frame images from store A are input). Alternatively, a pre-trained DNN model for inference, trained on data (training dataset) collected in the summer, may become less accurate in the snowy winter of Hokkaido due to changes in the customer environment caused by snow. Furthermore, the attributes of customers visiting a store change considerably; for example, the age composition of the population may have changed compared to when the training dataset data was collected. We introduce the concept of "spatial / temporal bias." This spatial / temporal bias can be automatically detected based on the accuracy of the inference results obtained by the DNN model trained on the current training dataset, applied to frame images included in the aforementioned unknown test dataset.

[0097] Next, referring to Figure 13, the flow of data collection (frame image) in this dataset generation system 10 will be explained. The data source / collector module 150 and data collector module 151 in Figure 13 are programs included in the dataset generation program 37 shown in Figure 4. Once the camera clustering (grouping) described in S12 of Figure 6 is completed (YES in S61 of Figure 13), the data source / collector module 150 of the learning server 1 selects a representative camera in each grouped group and sends a list of these representative cameras (device list) to the data collector module 151. After this, the system administrator on the learning server 1 uses the operation unit 35 and the data collection selection UI 72 in Figure 9 to define (design) a subsystem for selecting and collecting frame images from the representative cameras (hereinafter abbreviated as "selection collection subsystem"), or uses the operation unit 35 to instruct the CPU 31 of the learning server 1 to load an existing selection collection subsystem that has already been defined (S62). Then, the system administrator uses the operation unit 35 to activate the above-mentioned selective collection subsystem (S63).

[0098] Once the startup of the selective collection subsystem in S63 is complete, this selective collection subsystem (on the learning server 1 side) sends a data collection request file to the data source (edge-side device (fixed camera 3 and signage 4)) 152 using the image data collection request block 81 in Figure 9. When the data source (edge-side device) 152 receives the above data collection request file from the selective collection subsystem on the learning server 1 side, it sends the data (frame image and information file) from each representative camera to the learning server 1 side for storage in the learning server 1's DB 153. When the learning server 1 (data source / collector module 150) has finished collecting data from each representative camera from the data source (edge-side device) 152 (YES in S65), it sends a list of representative cameras (device list) in each grouped group to the data collector module 151 side, similar to when clustering (grouping) is completed in S61.

[0099] The data collector module 151 accesses the data stored in the DB 153 of the learning server 1 based on the received list of representative cameras (device list), and performs processing by the pseudo-labeling block 106 (S66) in Figure 11 above, processing by the processing result block 107 (S67), and visualization of the results by the visualizer 108 (S68) (in short, evaluation of the quality of the dataset via pseudo-labeling), and determines whether the overall evaluation value of the quality of the dataset reaches the KPI (target). If, as a result of this determination, the overall evaluation value of the dataset quality does not reach the KPI (target) and the number of iterations does not reach a predetermined limit (YES in S69), the CPU 31 of the learning server 1 performs the process of supplementing the data collected from the data source (edge ​​device) 152 (adding and storing the data in the DB 153 of the learning server 1), or the process of removing (deleting) some of the frame images belonging to the majority class from the dataset. Then, in the determination in S69, the process by the pseudo-labeling block 106, the processing by the processing result block 107, and the visualization of the results by the visualizer 108 (processing in S66 to S68 in Figure 13) are repeated until the overall evaluation value of the dataset quality reaches the KPI (target) or the number of iterations reaches a predetermined limit.

[0100] Then, in the determination in S69, if the overall evaluation value of the dataset quality reaches the KPI (target), the data collector module 151 accesses the data stored in the DB 153 of the learning server 1 based on the received list of representative cameras (device list), and performs processing by the fine-tuning block 134 (S66) in Figure 12 above (processing by the processing result block 135 (S67) and visualization of the results by the visualizer 136 (S68) (in short, evaluation of the dataset quality through fine-tuning), and determines whether the evaluation value (accuracy of inference processing by the trained DNN model after fine-tuning) reaches the KPI (target). If, as a result of this determination, the accuracy of the inference processing by the finely tuned trained DNN model has not reached the KPI (target) and the number of iterations has not reached a predetermined limit (YES in S69), the CPU 31 of the training server 1 will replenish the data collected from the data source (edge ​​device) 152 (add and store the data in the DB 153 of the training server 1), and then, in the determination of S69, it will repeat the processing by the fine-tuning block 134, the processing by the processing result block 135, and the visualization of the results by the visualizer 136 (processing in S66 to S68 in Figure 13) until the accuracy of the inference processing by the finely tuned trained DNN model reaches the KPI (target) or the number of iterations reaches a predetermined limit.

[0101] In the determination in S69 above, if the accuracy of the inference processing by the fine-tuned trained DNN model reaches the KPI (target), the generation of the training dataset by the dataset generation system 10 is considered complete, and the dataset used for fine-tuning is used as the final training dataset for the target predetermined inference processing DNN model. On the other hand, in the determination in S69, if the number of iterations reaches a predetermined limit before the evaluation value (overall evaluation value of dataset quality, or the accuracy of the inference processing by the fine-tuned trained DNN model) reaches the KPI (target), the system administrator reviews the design of each subsystem using the data source clustering UI 55 in Figure 8, the data collection selection UI 72 in Figure 9, the data quality evaluation UI 93 in Figure 11, and the data bias evaluation UI 120 in Figure 12.

[0102] From the flows shown in Figures 6 and 13 above, this dataset generation system 10 allows for the fine-tuning of a pre-trained DNN model for inference processing used in thousands or tens of thousands of stores using a training dataset consisting of as few frame images as possible. The reason for this is as follows.

[0103] (1) First, by using clustering (grouping cameras) to collect frame images only from the representative camera of each camera group, the number of frame images collected from each camera group is narrowed down. This means that, for example, even if there are many cameras belonging to a certain camera group, the frame images of each camera group are often similar, so by collecting frame images only from the representative camera of each camera group (each representative camera) and generating a training dataset, a pre-trained DNN model for inference processing that has been fine-tuned using this training dataset can perform inference with similar accuracy for frame images of any camera group. Note that the frame images used during the above clustering only need to be one layout frame image (a frame image that does not contain people) from each camera, so the number of frame images used during clustering can be small.

[0104] (2) Furthermore, by using the subsystem edited in the data collection selection UI72 to select only frame images that have changed from the previous frame image (satisfying one of the following conditions: the number of people has increased, the position of people has moved, or the posture of people has changed), it is possible to efficiently collect diverse frame images showing people from a representative camera, even with a small number of frame images to collect. In practice, frame images extracted from recorded images are repetitive (repeating similar frame images), and collecting consecutive frame images is of little use. In the subsystem edited in the data collection selection UI72, by using a pre-trained DNN model for inference processing to determine whether the number of people has increased, the position of people has moved, or the posture of people has changed from the previous frame image, and selecting only frame images that satisfy one of these conditions, it is possible to reduce the number of data (frame images) included in the training dataset while improving the quality of the data and increasing the diversity of the data.

[0105] (3) Furthermore, as shown in the flows in Figures 6 and 13, after the minimum required number of frame images have been collected (YES in S16 in Figure 6, or YES in S65 in Figure 13), the process of supplementing frame images containing people from each representative camera is repeated (S20 and S24) until the "overall evaluation value" of the quality of the dataset reaches the KPI (target) in the judgment at S19 in Figure 6, and the evaluation value (F1 score) of the processing result by the fine-tuned trained DNN model reaches the KPI (target) in the judgment at S23 (in S20, if there is a bias in the labels (classes) assigned to each frame image in the dataset, a process is also performed to remove (delete) some of the frame images belonging to the majority class from the dataset). In the above frame image supplementation process, frame images from each representative camera are added to the training dataset little by little, so the data (frame images) included in the training dataset can be increased incrementally. Then, in the S19 judgment, once the "overall evaluation value" of the dataset quality reaches the KPI (target), and in the S23 judgment, once the evaluation value (F1 score) of the processing result by the fine-tuned trained DNN model reaches the KPI (target), the collection of frame images from each representative camera can be stopped at that point, thus keeping the number of frame images included in the training dataset to a minimum.

[0106] (4) Furthermore, as described above, in the processing of S20 in Figure 6, if the pseudo-labeling process results in a bias (class imbalance) in the labels (classes) assigned to each frame image in the dataset, the CPU 31 of the learning server 1 deletes some of the frame images belonging to the majority class from the dataset. This makes it possible to remove redundant training data (frame images to be deleted and the pseudo-labels of these frame images) from the training dataset.

[0107] Therefore, without the above-mentioned ingenuity (1) to (4), it would be possible to reduce the number of frame images collected from, for example, 100,000 to 1,000.

[0108] As described above, the dataset generation system 10, learning server 1, and dataset generation program 37 of this embodiment collect frame images that match the selection criteria set by the user (in Config fields 87, 88, 109, 110, 137, and 138) using the operation unit 35 from at least one camera (fixed camera 3 or built-in camera 5) in each camera group, and evaluate whether the dataset composed of the frame images collected at that time is suitable as a training dataset for the target inference DNN model (a trained DNN model for inference processing on the edge device). As a result, when the evaluation value (overall evaluation value) of the dataset composed of the frame images collected at that time reaches or exceeds the target value (when the KPI (target) is reached), the collection of frame images is terminated, and the dataset composed of the frame images collected at that time can be used as the training dataset for the target inference DNN model, thereby minimizing the number of frame images included in the training dataset for the target inference DNN model. Therefore, it is possible to perform machine learning on a target inference DNN model using the fewest possible frame images, targeting frame images from cameras in a large number of stores and other facilities. Consequently, the processing time required to generate the training dataset for the target inference DNN model and the processing time required to perform machine learning on the inference DNN model using this training dataset can be reduced. Furthermore, the costs required to transfer camera frame images from the camera (edge ​​device) to the learning server 1 (communication costs, electricity costs, etc.) and the costs required for the learning server 1 to store the frame images received from the edge device (costs required to secure memory space on the learning server 1) can be reduced.

[0109] Furthermore, according to the dataset generation system 10 of this embodiment, a representative camera is selected from each group of cameras, and frame images that match the selection criteria set by the user are collected from each of the selected representative cameras. As a result, frame images that match the selection criteria can be collected only from the representative camera in each camera group, thereby reliably reducing the number of frame images included in the training dataset of the target inference DNN model.

[0110] Furthermore, according to the dataset generation system 10 of this embodiment, frame images are collected that show an increase in the number of people, a shift in the position of people, or a change in the posture of people, compared to the previous frame image. In this way, by selecting only frame images that show changes compared to the previous frame image, a diverse range of frame images can be efficiently collected from the camera (fixed camera 3 or built-in camera 5), ​​even if the number of frame images collected is small.

[0111] Furthermore, according to the dataset generation system 10 of this embodiment, when the "overall evaluation value" of the dataset quality reaches the KPI (target), the dataset consisting of frame images collected at this point is used as a training dataset to fine-tune the target pre-trained DNN model for inference processing, and it is evaluated whether the accuracy of the inference processing by the fine-tuned pre-trained DNN model has improved by a predetermined value or more compared to the accuracy of the inference processing by the pre-trained DNN model before fine-tuning. As described above, based on the results of the evaluation inference processing (for example, pseudo-labeling processing), it is possible to evaluate whether the dataset consisting of frame images collected from each camera is suitable as a training dataset for fine-tuning the target DNN model for inference processing. However, when the dataset evaluated as suitable as a training dataset for fine-tuning is used as the training dataset to fine-tune the target pre-trained DNN model for inference processing, it is possible that the accuracy of the inference processing by the fine-tuned pre-trained DNN model may not improve as expected. Therefore, as described above, unless a dataset whose "overall quality evaluation value" has reached the KPI (target) (i.e., evaluated as suitable as a training dataset for fine-tuning the target pre-trained DNN model for inference processing) is used as a training dataset to fine-tune the target pre-trained DNN model for inference processing, and it is confirmed whether the accuracy of the inference processing by the fine-tuned pre-trained DNN model has improved by a predetermined value or more compared to the accuracy of the pre-trained DNN model for inference processing before fine-tuning, it is not possible to make a final judgment on whether a dataset whose "overall quality evaluation value" has reached the KPI (target) should be adopted as a training dataset for fine-tuning the target pre-trained DNN model for inference processing.

[0112] Furthermore, according to the dataset generation system 10 of this embodiment, pseudo-labeling processing is performed on each frame image collected from each camera, and the fine-tuning of the target pre-trained DNN model for inference processing is performed based on the frame images collected from each camera and the ground truth labels assigned to the frame images by the pseudo-labeling processing. As described above, by assigning ground truth labels to the collected frame images using pseudo-labeling processing, the fine-tuning of the target pre-trained DNN model for inference processing can be performed automatically.

[0113] Furthermore, according to the dataset generation system 10 of this embodiment, frame images (layout frame images) that do not show people are collected from each of the multiple cameras, features are extracted from each of the collected frame images, and the collected frame images are grouped based on the extracted features. Then, based on the grouping results of the collected frame images, the cameras that captured these frame images are grouped. This makes it possible to remove the influence of people that appear in the frame images when grouping the collected frame images, so that the cameras that captured the frame images can be grouped based on the features of the frame images of stores and other facilities captured by each camera.

[0114] Variant: It should be noted that the present invention is not limited to the configurations of the embodiments described above, and various modifications are possible without changing the spirit of the invention. Next, modifications of the present invention will be described.

[0115] Variation 1: In the above embodiment, when the "overall evaluation value" of the dataset quality reaches the KPI (when the evaluation value of the dataset quality becomes equal to or greater than the target value), the dataset consisting of frame images collected at this point is used as the training dataset to fine-tune the target pre-trained DNN model for inference processing, and it is evaluated whether the accuracy of the inference processing by the fine-tuned pre-trained DNN model has improved by a predetermined value or more compared to the accuracy of the inference processing by the pre-trained DNN model before fine-tuning. However, in the present invention, it is not necessarily required to evaluate whether the accuracy of the inference processing by the fine-tuned pre-trained DNN model has improved by a predetermined value or more compared to the accuracy of the pre-trained DNN model before fine-tuning. When the evaluation value of the dataset quality becomes equal to or greater than the target value, the dataset consisting of frame images collected at this point may be used as the training dataset for fine-tuning the target pre-trained DNN model for inference processing.

[0116] Variation 2: In the above embodiment, the results of pseudo-labeling processing on the frame images collected from each camera are used to evaluate whether the dataset composed of the frame images collected from each camera is suitable as a training dataset for fine-tuning a predetermined DNN model for inference processing. However, the present invention is not limited to this, and it is sufficient to perform evaluation inference processing on each of the frame images collected from each camera, and to evaluate whether the dataset composed of the frame images collected from each camera is suitable as a training dataset for a target DNN model for inference processing based on the results of these inference processing.

[0117] Variation 3: In the above embodiment, a representative camera is selected from each group of cameras, and frame images that match the selection criteria set using the operation unit 35 are collected from each of the selected representative cameras. However, the present invention is not limited to this, and frame images that match the selection criteria set using the operation unit may be collected from multiple cameras in each group of cameras.

[0118] Variation 4: In the above embodiment, frame images were collected that show an increase in the number of people, a movement of people's positions by a predetermined percentage or more, or a change in people's posture by a predetermined percentage or more, compared to the previous frame image. However, the present invention is not limited thereto. For example, frame images may be collected that show an increase or decrease in the number of people, a movement of people's positions, or a change in people's posture compared to the previous frame image, or frame images may be collected that show an increase in the number of people or a movement of people's positions by a predetermined percentage or more compared to the previous frame image.

[0119] Variation 5: In the embodiments described above, an example was shown where "retraining" in the claims refers to fine-tuning. However, "retraining" in the present invention is not limited to this, and may also refer to transfer learning, for example. Here, transfer learning means learning only the weights of the newly added layer while keeping the weights in the original (existing) trained DNN model fixed.

[0120] Variation 6: In the above embodiment, an example was shown where the edge-side devices to which the camera is connected are the signage 4 and the analysis box 2. However, the edge-side devices are not limited to these, and may, for example, be a so-called AI camera. [Explanation of symbols]

[0121] 1. Learning Server (Server, Computer) 3. Fixed camera (camera) 5. Built-in camera (camera) 10. Dataset Generation System 20. Pre-trained DNN models for inference processing (neural network models for specified inference processing) 28. Pre-trained DNN models for various inference processes (neural network models for specified inference processes) 35 Operation unit (setting operation means) 40 Layout / frame image acquisition unit (second means for acquiring captured images) 41 Image Feature Extraction Unit (Image Feature Extraction Means) 42 Image clustering unit (image clustering means) 43 Camera Classification Unit (Camera Classification Means) 44 Representative Camera Selection Unit (Representative Camera Selection Means) 45. Frame image acquisition unit (captured image acquisition means) 46 Evaluation inference unit (evaluation inference means) 47. Dataset Evaluation Unit (Dataset Evaluation Means) 48. Pseudo-labeling section (pseudo-labeling means) 49. Retraining Unit (Retraining Method) 50. Accuracy Improvement Evaluation Unit (Accuracy Improvement Evaluation Means) 106 Pseudo-labeling block (inference means for evaluation) 134 Fine-tuning block (retraining method)

Claims

1. A camera classification method for grouping multiple cameras, A setting operation means for performing setting operations for the selection criteria of captured images, A camera image collection means collects captured images from at least one camera in each group grouped by the camera classification means that match the selection criteria set by the user using the setting operation means, An evaluation inference means performs an evaluation inference process on each of the captured images collected by the aforementioned image acquisition means, A dataset generation system comprising: a dataset evaluation means that evaluates, based on the results of inference processing by the evaluation inference means, whether or not a dataset composed of captured images collected by the captured image collection means is suitable as a training dataset for a predetermined neural network model for inference processing.

2. The system further includes a representative camera selection means for selecting a representative camera in each group that has been grouped by the camera classification means, The dataset generation system according to claim 1, characterized in that the image acquisition means collects images from each of the representative cameras selected by the representative camera selection means that match the selection criteria set by the user using the setting operation means.

3. The dataset generation system according to claim 1 or 2, characterized in that the image acquisition means acquires images in which there has been an increase or decrease in the number of recognized objects, a movement in the position of recognized objects, or a change in the posture of recognized objects, compared to the previous image.

4. The dataset evaluation means evaluates, based on the results of the inference processing by the evaluation inference means, whether the dataset composed of images collected by the image acquisition means is suitable as a training dataset for fine-tuning or transfer learning of the pre-trained neural network model for the predetermined inference processing. When the evaluation value of the dataset by the dataset evaluation means reaches or exceeds the target value, a retraining means performs fine-tuning or transfer learning (hereinafter referred to as "retraining") of the pre-trained neural network model for predetermined inference processing using the dataset composed of images collected by the image acquisition means as a training dataset, The dataset generation system according to claim 1 or 2, further comprising: an accuracy improvement evaluation means for evaluating whether the accuracy of the inference processing by the trained neural network model after retraining by the retraining means has improved by a predetermined value or more compared to the accuracy of the inference processing by the trained neural network model before retraining.

5. The system further includes a pseudo-labeling means that performs pseudo-labeling processing on each of the captured images collected by the aforementioned captured image collection means, The dataset generation system according to claim 4, characterized in that the retraining means retrains the pre-trained neural network model for predetermined inference processing based on the captured images collected by the captured image collection means and the correct labels assigned to the captured images by the pseudo-labeling means.

6. A second image acquisition means collects images from each of the aforementioned multiple cameras that do not show the object to be recognized, Image feature extraction means for extracting features from each of the captured images collected by the second captured image collection means that do not show the object to be recognized, The system further comprises an image clustering means that groups images collected by the second image acquisition means, which do not contain the object to be recognized, based on features extracted by the image feature extraction means, The dataset generation system according to claim 1 or 2, characterized in that the camera classification means groups the cameras that took the captured images based on the grouping results of the image clustering means for captured images that do not show the object to be recognized.

7. The dataset generation system according to claim 1 or 2, characterized in that when the evaluation value of the dataset by the dataset evaluation means becomes equal to or greater than the target value, the dataset composed of captured images collected by the captured image collection means at that time is used as a training dataset for the trained neural network model for the predetermined inference processing.

8. Multiple cameras are connected via a network, A camera classification means for grouping the aforementioned multiple cameras, A setting operation means for performing setting operations for the selection criteria of captured images, A camera image collection means collects captured images from at least one camera in each group grouped by the camera classification means that match the selection criteria set by the user using the setting operation means, An evaluation inference means performs an evaluation inference process on each of the captured images collected by the aforementioned image acquisition means, A server comprising: a dataset evaluation means that evaluates, based on the results of inference processing by the evaluation inference means, whether a dataset composed of captured images collected by the captured image collection means is suitable as a training dataset for a predetermined neural network model for inference processing.

9. Computers, A camera classification method for grouping multiple cameras, A setting operation means for performing setting operations for the selection criteria of captured images, A camera image collection means collects captured images from at least one camera in each group grouped by the camera classification means that match the selection criteria set by the user using the setting operation means, An evaluation inference means performs an evaluation inference process on each of the captured images collected by the aforementioned image acquisition means, A dataset generation program that functions as a dataset evaluation means for evaluating whether a dataset composed of images collected by the image acquisition means is suitable as a training dataset for a neural network model for predetermined inference processing, based on the results of the inference processing by the evaluation inference means.

Images