Image processing apparatus and image processing method
Patent Information
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- HITACHI SOLUTIONS TECH LTD
- Filing Date
- 2022-08-23
- Publication Date
- 2026-06-30
Smart Images
Figure 0007882718000001 
Figure 0007882718000002 
Figure 0007882718000003
Abstract
Description
Technical Field
[0001] The present invention relates to an image processing apparatus and an image processing method, and more particularly, to an image processing technique using AI (Artificial Intelligence).
Background Art
[0002] Patent Document 1 discloses a medical image processing apparatus capable of efficiently improving the accuracy of a learning model. The medical image processing apparatus calculates an evaluation value of the estimation result of the learning model based on a medical image input to the learning model, an estimation result output from the learning model, and teacher data, and when the evaluation value is less than or equal to a threshold value, identifies the characteristics of the medical image. Then, the medical image processing apparatus selects medical images having the identified characteristics and performs learning processing on the selected medical images.
Prior Art Documents
Patent Documents
[0003]
Patent Document 1
Summary of the Invention
Problems to be Solved by the Invention
[0004] In recent years, as a system form of AI, edge AI has been attracting attention instead of cloud AI. Cloud AI is a form in which a cloud device creates a machine learning model and performs learning to create a learned machine learning model, and an edge device performs inference using the learned machine learning model. On the other hand, edge AI is a form in which a cloud device creates a machine learning model, and an edge device performs learning on the machine learning model to create a learned machine learning model, and performs inference using the learned machine learning model.
[0005] Using edge AI allows for learning at the application site, enabling the creation of pre-trained machine learning models optimized for each specific application. Furthermore, edge AI eliminates latency associated with training on cloud devices and reduces communication costs when transmitting training data acquired by edge devices to the cloud. However, edge devices typically have lower processing power compared to cloud devices. Therefore, particularly when using edge AI, it is desirable to optimize the training process. Specifically, achieving high recognition accuracy by training with fewer images is desirable.
[0006] Therefore, one of the objectives of the present invention is to provide an image processing device that can streamline the learning process for machine learning models. [Means for solving the problem]
[0007] A brief overview of some of the representative embodiments disclosed in this application is as follows.
[0008] A typical embodiment of the present invention comprises an image processing device and an image classifier. The feature extractor extracts features from a second image data obtained by inputting the second image data into a machine learning model trained using a plurality of first image data. The image classifier pre-stores a range of features obtained by inputting a plurality of first image data into the machine learning model as a reference range, and classifies the input second image data by determining whether the features extracted by the feature extractor for the input second image data are inside or outside the reference range. [Effects of the Invention]
[0009] To briefly explain the effects obtained by a representative embodiment of the invention disclosed in this application, it becomes possible to improve the efficiency of training a machine learning model in an image processing device.
[0010] Other issues, configurations, and effects not mentioned above will be clarified by the following description of embodiments for carrying out the invention. [Brief explanation of the drawing]
[0011] [Figure 1] This is a schematic diagram showing an example of the configuration of the main parts of the image processing device according to Embodiment 1. [Figure 2] Figure 1 is a schematic diagram showing an example of the configuration of a machine learning model. [Figure 3A] Figure 1 is a schematic diagram illustrating an example of the processing steps of the image classifier. [Figure 3B] Figure 1 is a schematic diagram illustrating an example of the processing steps of the image classifier. [Figure 4] Figure 1 is a flowchart illustrating an example of the process involved in creating the reference range data. [Figure 5] Figure 1 is a flowchart illustrating an example of the processing steps involved when the controller classifies image data. [Figure 6] This is a schematic diagram showing an example of the configuration of the main parts of the image processing device according to Embodiment 2. [Figure 7] Figure 6 illustrates an example of how the image classifier works. [Figure 8] Figure 6 is a schematic diagram showing some detailed configuration examples of the machine learning model. [Figure 9] Figure 8 illustrates an example of a method for quantifying the deviation of features from the reference range for the machine learning model shown. [Figure 10] This figure illustrates a detailed example of the operation of the feature extractor and image classifier in Figure 6, assuming the use of the machine learning model shown in Figure 8. [Figure 11] This figure shows an example of the effect when additional training is performed using the image processing device shown in Figure 6. [Figure 12] A schematic diagram is shown illustrating an example of the configuration of the main components of the image processing device according to Embodiment 3. [Figure 13] Figure 12 illustrates an example of how the image classifier works. [Figure 14] It is a schematic diagram showing a configuration example of a ratio table in FIG. 12. [Figure 15] In the image processing apparatus in FIG. 12, it is a diagram for explaining an example of a setting screen for additional learning.
Embodiment for Carrying Out the Invention
[0012] Hereinafter, embodiments of the present invention will be described with reference to the drawings. The embodiments are examples for explaining the present invention, and for the sake of clarity of explanation, omissions and simplifications are made as appropriate. The present invention can also be implemented in various other forms. Unless otherwise particularly limited, each component may be singular or plural.
[0013] The positions, sizes, shapes, ranges, etc. of the respective components shown in the drawings may not represent the actual positions, sizes, shapes, ranges, etc. in order to facilitate understanding of the invention. For this reason, the present invention is not necessarily limited to the positions, sizes, shapes, ranges, etc. disclosed in the drawings.
[0014] As examples of various information, it may be described in expressions such as "table", "list", "queue", etc., but the various information may be represented by data structures other than these. For example, various information such as "XX table", "XX list", "XX queue" may be referred to as "XX information". When explaining identification information, expressions such as "identification information", "identifier", "name", "ID", "number", etc. are used, but these are mutually replaceable.
[0015] When there are a plurality of components having the same or similar functions, they may be described by attaching different subscripts to the same reference numeral. Also, when it is not necessary to distinguish these plurality of components, the subscripts may be omitted in the description.
[0016] In the embodiments described, the processing performed by executing a program may be explained. Here, the computer executes the program using a processor (e.g., CPU, GPU) and performs the processing defined in the program using memory resources (e.g., memory) and interface devices (e.g., communication ports). Therefore, the main entity performing the processing by executing the program may be the processor. Similarly, the main entity performing the processing by executing the program may be a controller, device, system, computer, or node having a processor. The main entity performing the processing by executing the program may be an arithmetic unit, and may include a dedicated circuit that performs a specific processing. Here, a dedicated circuit is, for example, an FPGA (Field Programmable Gate Array), an ASIC (Application Specific Integrated Circuit), or a CPLD (Complex Programmable Logic Device).
[0017] The program may be installed on the computer from the program source. The program source may be, for example, a program distribution server or a storage medium readable by the computer. If the program source is a program distribution server, the program distribution server includes a processor and storage resources for storing the program to be distributed, and the processor of the program distribution server may distribute the program to other computers. In addition, in this embodiment, two or more programs may be implemented as one program, or one program may be implemented as two or more programs.
[0018] (Embodiment 1) <Overview of the image processing device> For example, consider a scenario where a pre-trained machine learning model, trained on images from site A, is used to perform inference, or more specifically, object detection, at site B. In this case, differences in how objects appear at each site, such as differences in size, position, color, posture, and background, can lead to insufficient accuracy in object detection. Specifically, many false negatives (FN), representing incorrect detection or even missed detections, and false positives (FP), representing incorrect detections, may occur. Therefore, using an image processing device like the one shown in Figure 1 is beneficial.
[0019] Figure 1 is a schematic diagram showing an example of the main components of an image processing device according to Embodiment 1. The image processing device 10 shown in Figure 1 is, for example, an edge camera installed at site B. The image processing device 10 comprises a controller 15, a memory 16, a camera module 17, and a lens 18. The memory 16 is implemented by a combination of non-volatile memory such as flash memory and volatile memory such as DRAM.
[0020] The camera module 17 acquires image data PD at site B via the lens 18 and outputs the acquired image data PD to the controller 15. In this case, the image data PD may be written to memory 16 first and then read out by the controller 15. The camera module 17 can be implemented, for example, by a CCD (Charge Coupled Device) or CMOS (Complementary Metal Oxide Semiconductor) type image sensor, or by an ISP (Image Signal Processor) that processes the image data captured by the image sensor.
[0021] The controller 15 performs object detection and other operations on the image data PD acquired by the camera module 17 using AI-based image processing. The controller 15 can be implemented as, for example, an MCU (Micro Controller Unit), SoC (System on a Chip), FPGA, ASIC, or a combination thereof. Since the controller 15 performs AI-based image processing, it is desirable that it be implemented in a form that includes a processor such as a GPU.
[0022] The controller 15 comprises a feature extractor 20, an image classifier 21, a machine learning model 22, and a learning controller 23. The machine learning model 22 is implemented, for example, by a processor executing a program stored in memory 16. In practice, the controller 15 itself may also be equipped with memory such as flash memory, DRAM, or SRAM. For the sake of explanation, in this specification, the memory 16 shown in Figure 1 is assumed to include the memory provided by the controller 15.
[0023] Here, the machine learning model 22 has already been trained using multiple image data PD acquired at site A. In this specification, the image data PD from site A is referred to as image data (first image data) PD1, and the image data PD from site B is referred to as image data (second image data) PD2. The controller 15 inputs the unknown image data PD2 acquired at site B into the machine learning model 22 and detects objects contained in the image data PD2. The controller 15 then outputs the object detection result DR.
[0024] The feature extractor 20 extracts the feature vectors (FV) of the image data PD2, which are obtained by inputting the unknown image data PD2 into the machine learning model 22. Specifically, the feature vectors FV are written to memory 16 as feature maps as appropriate during the inference process by the machine learning model 22. The feature extractor 20 simply needs to read the feature vectors FV written to memory 16.
[0025] The image classifier 21 has pre-stored reference range data 26 in memory 16. The reference range data 26 defines the range of features obtained by inputting multiple image data PD1 acquired at site A into the machine learning model 22, and is, for example, created at site A. In this specification, the range of these features is called the reference range RR. The image classifier 21 classifies the unknown image data PD2 input at site B by determining whether the feature FV extracted by the feature extractor 20 is inside or outside the reference range RR. The image classifier 21 then outputs the classification result CR.
[0026] Furthermore, the image classifier 21 classifies image data PD2, whose feature FV extracted by the feature extractor 20 is outside the reference range RR, into an additional training category for storing additional training image data. Specifically, the memory 16 is provided with an additional training image area 25 corresponding to the additional training category. Based on the classification result CR, the image classifier 21 writes the target image data PD2 to the additional training image area 25.
[0027] The learning controller 23 performs additional training on the machine learning model 22 using image data PD2 classified into the additional training category, that is, image data PD2 stored in the additional training image region 25. Specifically, for example, ground truth labels are separately added to the image data PD2 stored in the additional training image region 25 before additional training is performed. The feature extractor 20, image classifier 21, and learning controller 23 are implemented, for example, by the processor executing a program stored in memory 16.
[0028] <About Features> Figure 2 is a schematic diagram showing an example of the configuration of the machine learning model in Figure 1. The machine learning model 22 generally comprises an input layer IL, several hidden layers ML1 to ML[n] placed after it, and a final layer LL, or output layer. For example, the final hidden layer ML[n] comprises several dimensions, here 5 dimensions D1 to D5. Dimensions are also called nodes, and the machine learning model 22 is constructed by connecting nodes in the preceding layers and nodes in the succeeding layers via predetermined weight coefficients.
[0029] For such a machine learning model 22, the feature extractor 20 extracts feature quantities FV for each dimension D1 to D5 in one of the intermediate layers, in this case the intermediate layer ML[n]. In other words, it extracts the values to be written to the five nodes corresponding to the five dimensions D1 to D5. On the other hand, the reference range RR is predetermined for each dimension D1 to D5 based on the features of the same layer and the same dimensions D1 to D5 as the intermediate layer ML[n] targeted by the feature extractor 20. It is desirable that the intermediate layer from which feature quantities FV are extracted is a layer closer to the final layer LL than to the input layer IL. This is because the later the intermediate layer ML, the stronger the relationship with the final result, and consequently with FN and FP.
[0030] Figures 3A and 3B are schematic diagrams illustrating an example of the processing of the image classifier in Figure 1. Figure 3A shows an example of the relationship between the reference ranges RR1, RR2, ..., RR5 for each dimension D1, D2, ..., D5 and the feature quantities FV1, FV2, ..., FV5 for each dimension D1, D2, ..., D5. Each of the reference ranges RR1 to RR5 is determined, for example, by the range from the minimum value to the maximum value of the feature quantity FV for each dimension D1 to D5 obtained from image data PD1 at site A.
[0031] In the example shown in Figure 3A, features FV1 and FV2 are outside the reference ranges RR1 and RR2, respectively, while feature FV5 is inside the reference range RR5. For example, if features FV1 to FV5 are inside the reference ranges RR1 to RR5 in all dimensions D1 to D5, the image classifier 21 will determine that the final result is inside the reference range RR. On the other hand, if feature FV is outside the reference range RR in one or more dimensions among the multiple dimensions D1 to D5, the image classifier 21 will determine that the final result is outside the reference range RR. Therefore, in the example shown in Figure 3A, the image data PD2 is determined to be outside the reference range RR and is classified into the category for additional training.
[0032] Figure 3B shows the deviation amounts Δ1 to Δ5 of the feature quantities FV1 to FV5 for each dimension D1 to D5 from the reference range RR1 to RR5 in Figure 3A. When the feature quantity FV5 is inside the reference range RR5, as in dimension D5, the deviation amount Δ5 is zero. If the deviation amounts Δ1 to Δ5 are zero for all dimensions D1 to D5, the image classifier 21 determines that the final result is inside the reference range RR. On the other hand, if the deviation amount Δ is non-zero for one or more of the multiple dimensions D1 to D5, the image classifier 21 determines that the final result is outside the reference range RR.
[0033] As mentioned earlier, if the machine learning model 22 used at site A is used for object detection at site B, many FNs and FPs may occur. In other words, the accuracy of object detection will decrease. The accuracy of object detection is determined based on P (Positive) representing detection, N (Negative) representing non-detection, and the correct label indicating whether P and N are actually correct or not. Specifically, the accuracy of object detection is determined based on FN, FP, TP (True Positive), and TN (True Negative). TP represents correct detection, and TN represents correct non-detection, and by extension, the background.
[0034] A common method for improving object detection accuracy is to pre-assign ground truth labels to all image data PD2 acquired at site B, and then perform inference using the image data with these ground truth labels to classify the image data PD2 into FN, FP, TP, and TN. In this case, the accuracy of object detection can be improved mainly by performing additional training using the image data classified as FN or FP. However, this method requires pre-assigning ground truth labels to all image data PD2, which can require a tremendous amount of work.
[0035] One possible approach is to perform inference on image data PD2 without assigned correct labels, and then select additional image data PD2 for further training using the confidence value, which is the final inference result, as an indicator. Specifically, image data PD2 with a relatively low confidence value is selected as image data PD2 that is likely to be classified as FN or FP. In this case, for example, correct labels only need to be assigned to the selected image data PD2, rather than all of the image data PD2, thus reducing the workload. The confidence value roughly represents the AI's confidence level (%) regarding P and N. A higher confidence value means greater confidence in P, and a lower confidence value means greater confidence in N.
[0036] However, this method of selecting image data PD2 for additional training based on confidence values may not be sufficient for more efficient training. One reason for this is that image data PD2 selected based on confidence values are not necessarily classified as FN or FP, and in reality, only a certain percentage are classified as FN or FP.
[0037] Therefore, as shown in Figure 1, it is beneficial to use a method that selects image data PD2 for additional training based on the feature vector FV. Using the feature vector FV allows for the selection of image data PD2 based on more information compared to using confidence values. In other words, the confidence value, which is the final inference result, is, for example, a value included in the final layer LL in Figure 2, or a value obtained by further processing the value included in the final layer LL. As a result, image data PD2 selected based on the feature vector FV is considered to have a higher probability of being classified as FN or FP compared to image data PD2 selected based on confidence values. This enables more efficient training.
[0038] <Controller operation> Figure 4 is a flowchart showing an example of the processing involved in creating the reference range data in Figure 1. The reference range data 26 can be created, for example, using the controller 15 shown in Figure 1. Furthermore, the image classifier 21 and the learning controller 23 are not required when creating the reference range data 26. The flow shown in Figure 4 is implemented, for example, by program processing by the controller 15, which includes a processor.
[0039] In Figure 4, first, the controller 15 inputs the image data PD1 from site A, not site B, into the machine learning model 22, which was trained using the image data PD1 from site A (step S101). Next, the controller 15 uses the feature extractor 20 to extract feature FV for each of the pre-defined dimensions of the intermediate layer, for each of the dimensions D1 to D5 of the intermediate layer ML[n] in the example of Figure 2 (step S102).
[0040] The controller 15 repeatedly executes the processes in steps S101 and S102 while changing the image data PD1 at site A until it reaches a preset number of images (step S103: No). When it reaches the preset number of images (step S103: Yes), the controller 15 detects the minimum and maximum values of the feature quantity FV extracted in step S102 for each dimension (step S104).
[0041] In step S104, for example, the feature quantity FV extracted in step S102 may be written to memory 16, and the minimum and maximum values may be detected by referring to it. Alternatively, each time a feature quantity FV is extracted in step S102, the extracted feature quantity FV may be compared with the minimum and maximum values stored in memory 16, and the minimum and maximum values stored in memory 16 may be updated depending on whether an update is necessary for the minimum or maximum value. In other words, it may be a method in which only the minimum and maximum values are left in memory 16.
[0042] Subsequently, the controller 15 writes the range of the minimum and maximum values of the feature quantity FV detected for each dimension in step S104 to the memory 16 as the reference range RR (step S105). This allows the controller 15 to create the reference range data 26 shown in Figure 1.
[0043] Figure 5 is a flowchart showing an example of the processing involved when the controller in Figure 1 classifies image data. The flow shown in Figure 5 is implemented, for example, by a programmed controller 15 that includes a processor. In Figure 5, first, the controller 15 inputs image data PD2 from site B into the machine learning model 22, which has been trained using image data PD1 from site A (step S201). Next, the controller 15 uses the feature extractor 20 to extract feature quantities FV for each of the pre-set dimensions of the intermediate layer, for each of the dimensions D1 to D5 of the intermediate layer ML[n] in the example in Figure 2 (step S202).
[0044] Next, the controller 15 uses the image classifier 21 to determine whether the extracted feature vector FV is inside or outside the reference range RR for each dimension (step S203). Then, if the extracted feature vector FV is inside the reference range RR in all dimensions (step S204: Yes), the controller 15 classifies the image data PD2 into the first category C1 (step S205a). On the other hand, if the extracted feature vector FV is outside the reference range RR in one or more dimensions (step S204: No), the controller 15 classifies the image data PD2 into the second category C2 (step S205b).
[0045] Image data PD2 classified as Category 1 C1 is considered suitable for object detection. On the other hand, image data PD2 classified as Category 2 C2 is considered unsuitable for object detection. Subsequently, the controller 15 stores the image data PD2 of Category 2 C2 as image data PD2 for additional training (step S206). That is, the controller 15 writes the image data PD2 of Category 2 C2 to the additional training image area 25. The controller 15 repeatedly executes the flow shown in Figure 5 until the number of images for additional training reaches the set number.
[0046] <Main effects of Embodiment 1> As described above, by using the method of Embodiment 1, image data PD can be classified based on the feature vector FV, and consequently, image data PD for additional training can be selected based on the feature vector FV. As a result, it becomes possible to improve the efficiency of training the machine learning model 22. Furthermore, by improving the efficiency of training, it becomes possible to facilitate the application of edge AI as an alternative to cloud AI. However, the method of Embodiment 1 is not necessarily limited to edge AI, but may also be applied to cloud AI.
[0047] (Embodiment 2) <Overview of the image processing device> Figure 6 is a schematic diagram showing an example of the main components of the image processing apparatus according to Embodiment 2. Figure 7 is a diagram illustrating an example of the operation of the image classifier in Figure 6. The image processing apparatus according to Embodiment 2 has the same configuration as in Figure 1. Figure 6 shows some configuration examples focusing on the differences from Figure 1. Here, we will explain focusing on the differences from Figure 1.
[0048] In Figure 6, unlike in Figure 1, memory 16 also pre-stores threshold data 40. Threshold data 40 is the data that defines the first threshold TH1. Furthermore, as described in Figure 3B, the image classifier 21a in the controller 15 calculates the deviation amount Δ from the reference range RR of the feature quantity FV extracted by the feature extractor 20. Based on the calculated deviation amount Δ, the image classifier 21a then classifies the input image data PD2 from site B.
[0049] In Embodiment 1, the image classifier 21 classified the image based on whether the deviation amount Δ was zero, i.e., whether the feature quantity FV was within the reference range RR, or whether the deviation amount Δ was non-zero, i.e., whether the feature quantity FV was outside the reference range RR. On the other hand, in Embodiment 2, the image classifier 21a further classified the image data PD2 based on a first threshold TH1 when the deviation amount Δ was non-zero. Specifically, as shown in Figure 7, the image classifier 21a classified the image data PD2 into a first category C1 where the deviation amount Δ is zero, a second category C2 which is within the range from zero to the first threshold TH1, or a third category C3 which exceeds the first threshold TH1.
[0050] The first threshold TH1, i.e., threshold data 40, is predetermined based on background image data and serves as a boundary value to distinguish between the background and the object. As a result, the image data PD2 classified as the third category C3 is considered background image data. The second category C2 is designated as an additional training category for accumulating additional training image data PD2. Specifically, the image classifier 21a writes the image data PD2 classified as the second category C2 to the additional training image area 25 in memory 16. This allows background image data to be excluded from the additional training image data, making it possible to select additional training image data with a higher learning effect.
[0051] The first threshold TH1 can be determined by one of the following methods, for example. The first method involves preparing multiple image data sets consisting only of backgrounds and image data classified as FN or FP, and experimentally determining the feature vectors that form the boundary between the background and FN / FP by extracting the features from these image data sets. The second method involves preparing multiple image data sets with a final inference result (confidence value) of 0.1% or less, extracting the features from these image data sets, and calculating the deviation amount Δ from the reference range RR for each set, and setting the smallest deviation amount Δ as the first threshold TH1. By using the smallest deviation amount Δ, more image data consisting only of backgrounds can be excluded.
[0052] <Specific examples of feature extraction and image classification> Figure 8 is a schematic diagram showing some detailed configuration examples of the machine learning model shown in Figure 6. Figure 8 shows specific examples of the final intermediate layer ML[n] and the final layer LL shown in Figure 2. The machine learning model 22 shown in Figure 8 is an example of the YOLO (You Only Look Once) model, which is widely used in object detection.
[0053] In Figure 8, the final layer LL is composed of, for example, 20 × 20 × 21 nodes ND (width × height × depth). Each node ND is also a dimension, as described in Figure 2. The 20 × 20 nodes ND located in the width and height directions correspond to the width and height directions of the image data PD, respectively. For each of the 20 × 20 nodes ND, the feature quantities of the object present at that location are represented by 21 depth-direction nodes ND. The feature quantities of the object include inferred information such as the object's center position, width and height dimensions, and object type, as well as a confidence value for that inferred information.
[0054] The intermediate layer ML[n] is composed of 20 × 20 × 1024 nodes ND (width × height × depth). By performing a convolution operation on this intermediate layer ML[n] using a 1 × 1 × 21 filter, the feature quantities of each node ND in the final layer LL are obtained. For example, the 1024 feature quantities in node space 30a of the intermediate layer ML[n] are aggregated into 21 feature quantities in node space 30b of the final layer LL through the convolution operation.
[0055] In this configuration, as shown in Figure 2, features can be extracted, for example, for each of the 20 × 20 × 10²⁴ nodes ND in the intermediate layer ML[n], or in other words, for each of the 20 × 20 × 10²⁴ dimensions. Then, for each of the 20 × 20 × 10²⁴ dimensions, the image classifier 21a can calculate the deviation amount Δ, and a comparison can be made between the deviation amount Δ and the first threshold TH1. However, as the number of dimensions increases, the processing load associated with comparing the deviation amount Δ and the first threshold TH1 can increase. Also, creating a first threshold TH1 representing the boundary with the background for each of the 20 × 20 × 10²⁴ dimensions may not always be easy. Therefore, the number of dimensions may be reduced using the following method.
[0056] Figure 9 illustrates an example of a method for quantifying the deviation of features from the reference range for the machine learning model shown in Figure 8. As an example, Figure 9 shows the features and reference range in node space 31 in Figure 8. Node space 31 has 1024 dimensions in the depth direction, and a reference range RR is predefined for each dimension. In Figure 9, the reference range RR390 for the 390th dimension and the reference range RR785 for the 785th dimension are shown as examples.
[0057] Furthermore, when the feature vector FV was extracted from the image data PD2 at site B, a non-zero deviation amount Δ390 was found in the 390th dimension of the node space 31, and a non-zero deviation amount Δ785 was found in the 785th dimension. In other dimensions, although not shown in the diagram, the deviation amount Δ is non-zero, meaning that the feature vector FV is within the reference range RR.
[0058] The image classifier 21a aggregates the deviations in the node space 31 into a single total deviation ΔALL using equation (1). The image classifier 21a then compares this total deviation ΔALL with the first threshold TH1. In this case, in the intermediate layer ML[n] shown in Figure 8, 20 × 20 first thresholds TH1 are created in advance, rather than 20 × 20 × 1024. ΔALL=(Δ1+Δ2+…+Δ1024) / 1024…(1)
[0059] Figure 10 illustrates a detailed example of the operation of the feature extractor and image classifier in Figure 6, assuming the use of the machine learning model shown in Figure 8. Figure 10 shows an example of the final inference result, i.e., the result of object detection performed on image data PD2 from site B using the YOLO model shown in Figure 8. In the YOLO model, object detection is performed by dividing the image data PD2 into regions called grid 35. In the example in Figure 10, the image data PD2 is divided into a 20x20 grid 35. Each of these 20x20 grids 35 corresponds to a 20x20 node ND in the final layer LL shown in Figure 8.
[0060] Each grid 35 contains information about the object's center position, width and height dimensions, object type, etc., along with a confidence value for that information. Object detection is performed based on the information of the grid 35 where the confidence value is at its maximum. In the example in Figure 10, an object, in this case an airplane, is detected based on the information of grid 35[10,9], where the confidence value is 0.67.
[0061] Specifically, based on the information in grid 35[10,9], it is detected that the center of the object is located at grid 35[10,9], that the object has the size indicated by the bounding box 36, and that the object is an airplane. For example, the adjacent grid 35[9,9] also contains similar information to grid 35[10,9]. However, since the confidence value of grid 35[9,9] is smaller than that of grid 35[10,9], the information contained in grid 35[9,9] is not reflected in the final inference result.
[0062] When using such a YOLO model, it is not necessary to extract the feature vector FV for all 20 × 20 × 1024 nodes ND shown in Figure 8. Instead, it is sufficient to extract the feature vector FV for one node space 31 corresponding to grid 35[10,9]. In other words, the feature extractor 20 detects the grid 35 where the confidence value is at its maximum from the final inference result, and extracts the feature vector FV for the node space 31 corresponding to the detected grid 35.
[0063] On the other hand, with respect to the reference range RR, since the position of the grid 35 with the maximum value may differ for each image data PD2, it is created in advance for each of the 20 × 20 × 1024 node NDs shown in Figure 8. In this case, for example, if a certain grid 35 is always in the background at site A, it may be difficult to create the reference range RR corresponding to that grid 35. In such cases, the reference range RR corresponding to that grid 35 may be determined based on the reference range RR of another grid 35.
[0064] The image classifier 21a determines whether the deviation amount Δ is non-zero in one or more of the 1024 dimensions extracted from a node space 31 corresponding to grid 35[10,9], and consequently whether the overall deviation amount ΔALL of the node space 31 is non-zero. If the overall deviation amount ΔALL is non-zero, the image classifier 21a further determines whether the overall deviation amount ΔALL exceeds a predetermined first threshold TH1 for the node space 31.
[0065] Then, if the total deviation amount ΔALL exceeds the first threshold TH1, the image data PD2 is classified into the third category C3, i.e., background; otherwise, the image data PD2 is classified into the second category C2, i.e., for additional learning. In this way, by extracting feature quantities FV for a specific node space 31 and performing classification using the total deviation amount ΔALL of that node space 31, it is possible to reduce the processing load on the feature extractor 20 and the image classifier 21a.
[0066] Furthermore, although the image data PD2 in the example in Figure 10 contains only one object, in reality, it may contain multiple objects. For example, if it contains two objects, the final inference result will have two grids 35 with local maxima, i.e., two grids 35 with higher confidence values than the surrounding areas, and two bounding boxes 36 will be created centered on these two grids 35. In this case, feature vectors FV can be extracted from the two node spaces 31 corresponding to these two grids 35, and the image can be classified based on the extracted feature vectors.
[0067] Thus, when classifying two node spaces 31, or in other words, one image data PD2 containing two objects, if a non-zero deviation amount Δ occurs in at least one of the objects, the image data PD2 is classified into the category for additional training. However, from the perspective of selecting images with higher training effectiveness, it is desirable to classify image data PD2 in which a non-zero deviation amount Δ occurs in both objects into the category for additional training. Furthermore, it is desirable to classify image data PD2 in which a non-zero deviation amount Δ occurs in multiple objects of different types, i.e., different classes, into the category for additional training.
[0068] Therefore, the image classifier 21a may be configured to classify image data PD2 in which a non-zero deviation amount Δ occurs in multiple objects into a category for additional training. Alternatively, the image classifier 21a may be configured to classify image data PD2 in which a non-zero deviation amount Δ occurs in multiple objects of different classes into a category for additional training. Specifically, for example, a reference range RR for each class may be created in advance. Then, the image classifier 21a classifies the image data PD2 for each class using the reference range RR for each class, and classifies the image data PD2 that has been classified into the second category C2 in multiple classes into a category for additional training.
[0069] <Other variations> As shown in Figure 7, the image classifier 21a classified image data PD2 with a deviation amount Δ of zero, i.e., image data PD2 suitable for object detection, into the first category C1. The image classifier 21a may further classify the image data PD2 classified into the first category C1 by a confidence value which is the final estimation result. In this case, image data PD2 suitable for object detection is classified, for example, into strongly suitable image data PD2 and weakly suitable image data PD2.
[0070] When performing additional training, in practice, to prevent overfitting, it is sometimes necessary to include a certain amount of image data PD2 classified as Category 1 C1. However, image data PD2 that is strongly suitable for object detection, i.e., image data PD2 with a very high confidence value, often results in image data with low training effectiveness. Therefore, it is also beneficial to select image data PD2 that is weakly suitable for object detection using this method for additional training.
[0071] <Verification Results> Figure 11 shows an example of the effect of performing additional training using the image processing device shown in Figure 6. Figure 11 shows the relationship between the number of images used for additional training and the accuracy of object detection. Figure 11 also shows verification results 43a when image data for additional training is selected using the image processing device shown in Figure 6, and verification results 43b as a comparative example when image data for additional training is selected based on the confidence value described in Embodiment 1. As can be seen from Figure 11, in verification result 43a, the number of images required to obtain the same detection accuracy is reduced to 0.4 times compared to verification result 43b, and if the number of images is the same, the detection accuracy can be increased by 9%.
[0072] <Main effects of Embodiment 2> As described above, by using the method of Embodiment 2, image data can be classified into 3 or more categories based on the feature vector FV, and consequently, background image data can be excluded to select image data for additional training. As a result, the same effects as those described in Actual Embodiment 1 can be obtained, and furthermore, by excluding background image data, it becomes possible to make the training of the machine learning model more efficient.
[0073] (Embodiment 3) <Overview of the image processing device> Figure 12 is a schematic diagram showing an example of the main components of the image processing device according to Embodiment 3. Figure 13 is a diagram illustrating an example of the operation of the image classifier in Figure 12. The image processing device shown in Figure 12 differs from the example configuration shown in Figure 6 in the following three points. The first difference is that the threshold data 40c pre-stored in memory 16 is data that defines a second threshold TH2 in addition to the first threshold TH1. The second difference is that memory 16 also pre-stores a ratio table 45.
[0074] A third difference is that a different image classifier 21c is provided compared to the case in Figure 6. As shown in Figure 13, the image classifier 21c uses a second threshold TH2 stored in memory 16 to further classify the image data PD2 from site B, which is classified as a second category C2, into either a fourth category C4 or a fifth category C5. The second threshold TH2 is a value smaller than the first threshold TH1. The fourth category C4 is a deviation amount Δ ranging from zero to the second threshold TH2. The fifth category C5 is a deviation amount Δ ranging from the second threshold TH2 to the first threshold TH1.
[0075] Furthermore, the image classifier 21c stores ratio data in the ratio table 45 within the memory 16, which represents the relationship between the ratio of the number of image data PD2 images classified into the fourth category C4 and the number of image data PD2 images classified into the fifth category C5, and the required accuracy for object detection. Based on this ratio data, the image classifier 21c classifies the input image data PD2 from site B into additional training categories so that the ratio of images corresponds to the input required accuracy.
[0076] Figure 14 is a schematic diagram showing an example of the configuration of the ratio table in Figure 12. The ratio table 45 shown in Figure 14 represents the correspondence between the required accuracy and the ratio of the number of images for additional training. In the example shown in Figure 14, the ratio of images for the first category C1 is 1 when the required accuracy is up to 75%. When the required accuracy is from 75% to 80%, the ratio of images for the first category C1 and the fourth category C4 is C1:C4=0.7:0.3. When the required accuracy is from 80% to 85%, the ratio of images for the first category C1, the fourth category C4, and the fifth category C5 is C1:C4:C5=0.5:0.3:0.2.
[0077] On the other hand, as shown in Figure 13, AI learning can achieve, for example, a required accuracy of 75% by using image data classified as TP. Furthermore, by adding image data classified as weak FP or weak FN, a required accuracy of 80% can be achieved. Furthermore, by adding image data classified as strong FP or strong FN, a required accuracy of 85% can be achieved.
[0078] However, concepts such as weak FP and strong FP are conceptual and are obtained by distinguishing FP using some kind of metric. Furthermore, using only image data classified as weak FP or weak FN may lead to overfitting. For this reason, it is desirable to appropriately add image data classified as TP as well when performing training.
[0079] In the method of Embodiment 3, image data PD2 classified as Category 4 C4 is considered to be image data classified as weak FP or weak FN. Image data PD2 classified as Category 5 C5 is considered to be image data classified as strong FP or strong FN. In other words, by providing a second threshold TH2, it is possible to distinguish between weak FP or weak FN and strong FP or strong FN.
[0080] Figure 15 illustrates an example of the additional training settings screen in the image processing device shown in Figure 12. The settings screen 50 shown in Figure 15 is displayed on a display (not shown) provided in the image processing device, or on the display of a management device such as a server connected to the image processing device via a communication network. The user selects the required object accuracy and inputs the number of images to be used for additional training via the settings screen 50 as shown in Figure 15.
[0081] Based on these settings, the image classifier 21c refers to the ratio table 45 and writes the input image data PD2 to the additional training image area 25 in memory 16, while satisfying the number ratio corresponding to the set required accuracy and until the number of input images is reached. For example, if the number of images is 1000 and the required accuracy is 75% to 80%, the image classifier 21c collects 700 image data PD2 of the first category C1 and 300 image data PD2 of the fourth category C4.
[0082] <Main effects of Embodiment 3> As described above, using the method of Embodiment 3 yields the same effects as those described in Embodiments 1 and 2. Furthermore, by providing a ratio table 45, additional training image data can be efficiently selected according to the required accuracy of the object, enabling more efficient training. [Explanation of symbols]
[0083] 10…Edge camera (image processing unit), 15…Controller, 16…Memory, 20…Feature extractor, 21…Image classifier, 25…Image region for additional training, 45…Proportion table, C1~C5…Category, D1~D5…Dimension, FV…Feature, IL…Input layer, LL…Final layer (output layer), ML…Hidden layer, PD…Image data, RR…Reference range, TH1,TH2…Threshold, Δ1~Δ1024…Deviation
Claims
1. A feature extractor that extracts features from the second image data obtained by inputting the second image data into a machine learning model that has been trained using multiple first image data, The machine learning model has a reference range for the range of features obtained by inputting the plurality of first image data, and the image classifier classifies the input second image data by determining whether the features extracted by the feature extractor for the input second image data are inside or outside the reference range. Equipped with, The aforementioned image classifier is The deviation of the features extracted by the feature extractor from the reference range is calculated, and based on the calculated deviation, the input second image data is classified into a first category where the deviation is zero, a second category where the deviation is within the range from zero to a predetermined first threshold, or a third category where the deviation exceeds the first threshold. Only the second image data classified into the second category is classified into an additional training category for accumulating additional training image data. Using a predetermined second threshold value that is smaller than the first threshold, the second image data classified into the second category is further classified into either a fourth category, which is in the range from zero to the second threshold, or a fifth category, which is in the range from the second threshold to the first threshold. The system pre-stores ratio data representing the correspondence between the number of image data images classified into the fourth category and the number of image data images classified into the fifth category, and the required accuracy for object detection. Based on this ratio data, the system classifies the second image data into the additional learning category so that the ratio corresponds to the input required accuracy. Image processing device.
2. In the image processing apparatus according to claim 1, The feature extractor extracts the feature quantities for each dimension in any of the intermediate layers of the machine learning model. The aforementioned reference range is predetermined for each dimension based on features of the same layer and dimension as the intermediate layer targeted by the feature extractor. Image processing device.
3. In the image processing apparatus according to claim 2, The intermediate layer targeted by the feature extractor is located closer to the output layer than the input layer. Image processing device.
4. In the image processing apparatus according to claim 1, The first threshold is determined based on background image data. Image processing device.
5. The range of features obtained by inputting the multiple first image data into a machine learning model trained using multiple first image data is pre-stored as a reference range. By inputting the second image data into the aforementioned machine learning model, the features of the second image data are extracted. The input second image data is classified by determining whether the feature quantities extracted from the input second image data are inside or outside the reference range. The amount of deviation of the feature quantities extracted from the second image data from the reference range is calculated, and based on the calculated amount of deviation, the input second image data is classified into a first category where the amount of deviation is zero, a second category where the deviation is within the range from zero to a predetermined first threshold, or a third category where the deviation exceeds the first threshold. Only the second image data classified into the second category is designated as an additional training category for accumulating additional training image data. Using a predetermined second threshold value that is smaller than the first threshold, the second image data classified into the second category is further classified into either a fourth category, which is in the range from zero to the second threshold, or a fifth category, which is in the range from the second threshold to the first threshold. Ratio data representing the relationship between the number of image data images classified into the fourth category and the number of image data images classified into the fifth category, and the required accuracy for object detection, is stored in advance. Based on the ratio data, the second image data is classified into the additional learning category so that the ratio of images corresponds to the input required accuracy. Image processing methods.
6. A feature extractor that extracts features from the second image data obtained by inputting the second image data into a machine learning model that has been trained using multiple first image data, The machine learning model has a reference range for the range of features obtained by inputting the plurality of first image data, and the image classifier classifies the input second image data by determining whether the features extracted by the feature extractor for the input second image data are inside or outside the reference range. Equipped with, The feature extractor extracts the feature quantities for each dimension in any of the intermediate layers of the machine learning model. The aforementioned reference range is predetermined for each dimension based on features of the same layer and dimension as the intermediate layer targeted by the feature extractor. The intermediate layer targeted by the feature extractor is defined as a layer closer to the output layer than the input layer. The image classifier classifies only the second image data in which the features extracted by the feature extractor are outside the reference range into an additional training category for accumulating additional training image data. Image processing device.