Vehicle control system and vehicle control method

Abstract

To achieve robust inference.SOLUTION: A vehicle control system comprising: a data acquisition unit configured to acquire input data input to a model; an inference unit configured to perform inference by inputting the input data to a plurality of models and output output data for each of the models; an input data feature vector extraction unit configured to extract an input data feature vector representing a feature of the input data; a learning data feature vector acquisition unit configured to acquire a learning data feature vector representing a feature of learning data used for learning of the model; a proficiency degree determination unit configured to determine, for each of the models, a proficiency degree indicating a possibility of deriving correct output data for the input data based on the input data feature vector and a plurality of learning data feature vectors for each of the models; and an output unit configured to output an inference result derived based on the proficiency degree determined by the proficiency degree determination unit and the plurality of output data.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present disclosure relates to a vehicle control system and a vehicle control method. [Background technology]

[0002] In recent years, technology for recognizing objects around a vehicle using machine learning models implemented in vehicle control devices has become an important technology supporting autonomous driving systems and driver assistance systems. Machine learning models for object recognition have been shown to have powerful inference capabilities and fast response times in evaluations using computers with abundant computational resources. However, current vehicle control devices have smaller computational resources than the computers used in such evaluations. Therefore, the models must be made lighter for implementation in vehicle control devices, and the inference accuracy of the models in actual operation deviates from the conclusions of the experimental phase. Meanwhile, machine learning models designed for operation with limited computational resources often lack the robustness required to achieve stable accuracy for a variety of inputs compared to machine learning models designed for operation on computers.

[0003] The following prior art is included as background technology in this technical field: Patent Document 1 (JP 2022-114607 A) describes a gradual reinforcement learning device that includes a storage device that stores learning data to be learned, a model learning unit that receives distribution weights corresponding to the learning data and multiple samples included in the learning data and performs learning by applying the corresponding distribution weights to the samples, learning for a model and each of the multiple samples and identifying a degree of proficiency that is an index showing the accuracy of prediction for each of the multiple samples, an ensemble unit that receives the learning data and a model related to the learning, performs ensemble learning, and outputs an ensemble model and a sample proficiency set that is a set of proficiency levels for each of the samples, a model management unit that creates model information by associating the model, the proficiency level, the ensemble model, and the sample proficiency set for each time step, and a distribution calculation unit that outputs the distribution weight for each sample based on the model information. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Publication No. 2022-114607 Summary of the Invention [Problem to be solved by the invention]

[0005] A single model can result in biased object recognition accuracy, and combining multiple models can lead to incorrect inference results due to the influence of models with mismatched characteristics. Furthermore, for each model, environmental changes or incomplete training data can cause recognition accuracy to fall below expected accuracy, potentially leading to incorrect inference results.

[0006] Although the technology described in Patent Document 1 can construct a high-performance ensemble model in a short learning time during the learning stage, it does not take into consideration how to respond if an erroneous result occurs during the inference stage of the ensemble model.

[0007] The purpose of this disclosure is to achieve robust inference using machine learning / AI models. [Means for solving the problem]

[0008] A representative example of the invention disclosed in the present application is as follows: That is, a vehicle control system includes a data acquisition unit that acquires input data to be input to a model, an inference unit that inputs the input data to a plurality of the models to perform inference and output output data for each of the models, an input data feature vector extraction unit that extracts an input data feature vector that represents features of the input data, a learning data feature vector acquisition unit that acquires a learning data feature vector that represents features of learning data used to train the models, an ability level determination unit that determines, for each of the models, an ability level that indicates the possibility of deriving correct output data for the input data, based on the input data feature vector and the learning data feature vectors for each of the plurality of models, and an output unit that outputs an inference result derived based on the ability level for each of the models determined by the ability level determination unit and the plurality of output data. [Effects of the Invention]

[0009] According to one aspect of the present invention, an object can be recognized with high accuracy by combining a plurality of trained models with different characteristics. Problems, configurations, and effects other than those described above will become clear from the following description of the embodiment. [Brief explanation of the drawings]

[0010] [Figure 1] 1 is a block diagram showing a functional configuration of a vehicle control system according to a first embodiment. [Figure 2] 10 is a flowchart of an input data output process according to the first embodiment. [Figure 3] FIG. 10 is a diagram illustrating an example of input data output processing according to the first embodiment. [Figure 4] 10 is a flowchart of an input data feature vector extraction process according to the first embodiment. [Figure 5] FIG. 10 is a diagram illustrating an example of input data feature vector extraction processing according to the first embodiment. [Figure 6] 10 is a flowchart of a learning data feature vector extraction process according to the first embodiment. [Figure 7A]FIG. 10 is a diagram illustrating an example of a learning data feature vector extraction process according to the first embodiment. [Figure 7B] FIG. 10 is a diagram illustrating an example of a learning data feature vector extraction process according to the first embodiment. [Figure 8] 10 is a flowchart of a model feature vector extraction process according to the first embodiment. [Figure 9A] FIG. 10 is a diagram illustrating an example of a model feature vector extraction process according to the first embodiment. [Figure 9B] FIG. 10 is a diagram illustrating an example of a model feature vector extraction process according to the first embodiment. [Figure 10] 10 is a flowchart of a model skill calculation process according to the first embodiment. [Figure 11A] FIG. 10 is a diagram illustrating an example of a model proficiency calculation process according to the first embodiment. [Figure 11B] FIG. 10 is a diagram illustrating an example of a model skill level calculation process according to the first embodiment. [Figure 12] 10 is a flowchart of an inference process according to the first embodiment. [Figure 13] FIG. 10 is a diagram illustrating an example of an inference process according to the first embodiment. [Figure 14] 10 is a flowchart showing a final inference result B output process according to the first embodiment. [Figure 15A] FIG. 10 is a diagram illustrating an example of weighting processing according to the first embodiment. [Figure 15B] FIG. 10 is a diagram illustrating an example of weighting processing according to the first embodiment. [Figure 16A] FIG. 10 is a diagram for explaining the derivation of an integrated inference result in the first embodiment. [Figure 16B] FIG. 10 is a diagram for explaining the derivation of an integrated inference result in the first embodiment. [Figure 17A] FIG. 10 is a diagram illustrating the derivation of the final inference result A in the first embodiment. [Figure 17B] FIG. 10 is a diagram illustrating the derivation of the final inference result A in the first embodiment. [Figure 17C] FIG. 10 is a diagram illustrating the derivation of the final inference result A in the first embodiment. [Figure 18] FIG. 10 is a diagram illustrating a final inference result B of the first embodiment. [Figure 19]10 is a flowchart of a validity determination process according to the first embodiment. [Figure 20] FIG. 10 is a diagram illustrating an example of a model classification process according to the first embodiment. [Figure 21] 10 is a flowchart of a process for determining the validity of a final inference result A in the first embodiment. [Figure 22A] FIG. 10 is a diagram illustrating an example of a final inference result A validity determination process according to the first embodiment. [Figure 22B] FIG. 10 is a diagram illustrating an example of a final inference result A validity determination process according to the first embodiment. [Figure 23A] FIG. 10 is a diagram illustrating an example of a final inference result A validity determination process according to the first embodiment. [Figure 23B] FIG. 10 is a diagram illustrating an example of a final inference result A validity determination process according to the first embodiment. [Figure 23C] FIG. 10 is a diagram illustrating an example of a final inference result A validity determination process according to the first embodiment. [Figure 24] 10 is a flowchart of an abnormality cause model determination process according to the first embodiment. [Figure 25A] FIG. 10 is a diagram illustrating an example of an abnormality cause model determination process according to the first embodiment. [Figure 25B] FIG. 10 is a diagram illustrating an example of an abnormality cause model determination process according to the first embodiment. [Figure 25C] FIG. 10 is a diagram illustrating an example of an abnormality cause model determination process according to the first embodiment. [Figure 26] 10 is a flowchart of an inference result update process according to the first embodiment. [Figure 27] FIG. 10 is a diagram illustrating an example of an integrated inference result of the first embodiment. [Figure 28] 10 is a flowchart of a final inference result B update process according to the first embodiment. [Figure 29] FIG. 10 is a diagram illustrating an example of reconference data according to the first embodiment. [Figure 30] 10 is a flowchart of a re-learning process according to the first embodiment. [Figure 31] 1 is a diagram illustrating a hardware configuration of a vehicle control device according to a first embodiment. [Figure 32]FIG. 10 is a block diagram showing a functional configuration of a vehicle control device according to a second embodiment. [Figure 33] FIG. 10 is a diagram illustrating a hardware configuration of a server according to a second embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0011] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings.

[0012] Example 1 FIG. 1 is a block diagram illustrating a functional configuration of a vehicle control device according to a first embodiment.

[0013] The vehicle control system 100 shown in this figure is an example of a vehicle control device of the present invention, and includes a data acquisition unit 101, a first input data feature vector extraction unit 102, a second input data feature vector extraction unit 103, a learning data feature vector acquisition unit 104, an inference unit 105 including multiple models, a specialty determination unit 109, an output unit 110, an output validity judgment unit 111, and a model reconstruction unit 112.

[0014] The data acquisition unit 101 acquires input data 5 to be inferred from an in-vehicle camera image 2 .

[0015] The first input data feature vector extraction unit 102 extracts vectors including image feature amounts and image complexity from the input data 5 to be inferred, and generates an input data feature vector 1001 including the extracted vectors.

[0016] The inference unit 105 includes a plurality of models 106 to 108, receives input data 5 to be inferred, and outputs a plurality of inference results. For example, when input data 5 to be inferred is input, model 1 (106), model 2 (107), and model 3 (108) included in the inference unit 105 output a model 1 inference result 1006, a model 2 inference result 1007, and a model 3 inference result 1008, respectively.

[0017] The second input data feature vector extraction unit 103 extracts image features and image complexity from integrated training data for training multiple models included in the inference unit 105, and generates a training data feature vector 1002 including the extracted image features and image complexity. For example, the second input data feature vector extraction unit 103 extracts feature vectors from model 1 training data 2, model 2 training data 3, and model 3 training data 4 to generate the training data feature vector 1002.

[0018] The training data feature vector acquisition unit 104 generates feature vectors 1003, 1004, and 1005 for each model from the training data feature vector 1002 that has been compiled to construct a plurality of models.

[0019] The skill determination unit 109 outputs the maximum similarity calculated between the input data feature vector 1001 and each model feature vector as the model skill degree. For example, the skill determination unit 109 uses the cosine similarity calculation method to calculate the maximum similarity between the input data feature vector 1001 and the model 1 feature vector 1003 and outputs it as the model 1 skill degree 1012. The skill determination unit 109 also calculates the maximum similarity between the input data feature vector 1001 and the model 2 feature vector 1004 and outputs it as the model 2 skill degree 1013. The skill determination unit 109 also calculates the maximum similarity between the input data feature vector 1001 and the model 3 feature vector 1005 and outputs it as the model 3 skill degree 1014.

[0020] The output unit 110 multiplies the model 1 inference result 1006, the model 2 inference result 1007, and the model 3 inference result 1008 by the model 1 proficiency level 1012, the model 2 proficiency level 1013, and the model 3 proficiency level 1014 input from the proficiency determination unit 109, respectively, and outputs the final inference result B1025 determined by majority voting on the multiplication results.

[0021] The output validity determination unit 111 determines the validity of the final inference result B1025 based on the model 1 proficiency level 1012, the model 2 proficiency level 1013, the model 3 proficiency level 1014, the model 1 inference result 1006, the model 2 inference result 1007, and the model 3 inference result 1008, and outputs a validity determination result 1032, model information 1039 that is the cause of the abnormality, and model information that requires relearning 1040. An abnormality in the output validity determination unit 111 occurs when the final inference result B1025 differs from the inference results of each model.

[0022] The model reconfiguration unit 112 identifies a model to be reconfigured based on the re-learning required model information 1040, and re-learns the identified target model.

[0023] The output unit 110 updates the final inference result A1024 by re-consultation, excluding abnormal inference results of the model to be reconstructed, based on information about the model to be reconstructed input from the model reconstruction unit 112, and outputs the updated final inference result B1025.

[0024] As shown in Figure 3, vehicle-mounted camera images 1 are a plurality of still images containing information about the surroundings of a vehicle captured by the vehicle-mounted camera, or a plurality of screenshots of captured video images taken at a predetermined time interval. For example, images 1A, 1B, and 1C are a portion of a plurality of images continuously captured by the vehicle-mounted camera while the vehicle is traveling.

[0025] Fig. 2 is a flowchart of input data output processing for outputting the vehicle-mounted camera image 1 acquired by the data acquisition unit 101 as input data 5. Fig. 3 is a diagram for explaining an example of the input data output processing.

[0026] In the input data output process shown in Figure 2, the data acquisition unit 101 acquires multiple vehicle-mounted camera images 1 (S101), adds processing time information to each vehicle-mounted camera image, and starts loop processing A1 (S102) to repeat the process of outputting the images as input data 5 (S103).

[0027] 3, the input data 5 includes an "input data number" 5A that records the order in which the vehicle-mounted camera images 1 were acquired, and an "input data image" 5B that stores the input data image changed from the vehicle-mounted camera image 1. For example, input data 5C, 5D, and 5E are the results of the acquisition process of the vehicle-mounted camera images 1A, 1B, and 1C, respectively.

[0028] Then, when the data acquisition unit 101 has output all of the vehicle-mounted camera images 1 as input data 5, it ends the loop process A1 (S104).

[0029] Fig. 4 is a flowchart of input data characteristic vector extraction processing for extracting a characteristic vector from input data 5. Fig. 5 is a diagram for explaining an example of input data characteristic vector extraction processing.

[0030] In the input data feature vector extraction process, a loop process B1 is started in which the processes of steps S202 to S203 shown in FIG. 4 are repeated for each piece of input data 5 acquired by the data acquisition unit 101 (S201).

[0031] As shown in FIG. 5, the first input data feature vector extraction unit 102 includes a feature vector extractor 102A.

[0032] In loop processing B1, first, feature vector extractor 102A extracts feature vector 1001B from input data 5 input from data acquisition unit 101 (S202). "Feature vector" 1001B is a vector obtained by extracting image information from input data 5 and expressing it as an array of numerical values. Image information from input data 5 includes, for example, a color histogram, texture features, complexity, edge information, etc.

[0033] Then, the first input data feature vector extraction unit 102 adds an "input data number" 1001A to the extracted feature vector 1001B to generate an input data feature vector 1001, and outputs the generated input data feature vector 1001 (S203). The extracted feature vector may be a one-dimensional vector or a multidimensional vector, but the number of elements of the vector should be the same.

[0034] Then, when the first input data characteristic vector extraction unit 102 outputs the input data characteristic vector 1001 for all input data, it ends the loop process B1 (S204).

[0035] The feature vector extractor 102A may use, for example, a known algorithm such as a convolutional neural network (CNN) or a visual geometry group (VGG), but may also use other feature extraction techniques that target image data.

[0036] Fig. 6 is a flowchart of a training data feature vector extraction process in which the second input data feature vector extraction unit 103 is used to extract a feature vector 1002 from training data provided for training of a model arranged in the inference unit 105. Figs. 7A and 7B are diagrams illustrating an example of the training data feature vector extraction process.

[0037] In the training data feature vector extraction process, first, the second input data feature vector extraction unit 103 starts a loop process C1 in which the processes of steps S302 to S307 are repeated for each of the inference models 106 to 108 arranged in the inference unit 105 (S301).

[0038] In loop processing C1, the second input data feature vector extraction unit 103 acquires training data to be used for training each of the inference models 106 to 108 arranged in the inference unit 105 (S302). For example, as shown in FIG. 7A, the training data used for training model 1 (106) is training data 2, the training data used for training model 2 (107) is training data 3, and the training data used for training model 3 (108) is training data 4. Model 1 training data 2 has a "data number" 2A and a "training data image" 2B. The structures of model 2 training data 3 and model 3 training data 4 are the same as the structure of model 1 training data 2. Since the training policies differ for each model, the training data differs. For example, model 1 (106) trains by emphasizing image data of urban areas, model 2 (107) trains by emphasizing image data of sunny days, and model 3 (108) trains by emphasizing image data of nighttime. Note that multiple models may train using the same training data.

[0039] Then, the second input data feature vector extraction unit 103 starts a loop process C2 in which the processes of steps S304 to S305 are repeated for each of the acquired learning data of each model (S303).

[0040] In loop process C2, feature vector extractor 103A of second input data feature vector extraction unit 103 extracts a feature vector for each acquired training data of each model (S304). The extracted feature vector has the same format as feature vector 1001B of input data 5 extracted by first input data feature vector extraction unit 102 in terms of the number of vector dimensions, the number of elements, etc.

[0041] Then, feature vector extractor 103A extracts a feature vector for each training data image in the training data for each model and adds a "model x data number" column to each training data. Here, feature vector extractor 103A can be the same as feature vector extractor 102A of first input data feature vector extraction unit 102. For example, second input data feature vector extraction unit 103 adds a "model x data number" column to the feature vector extracted for each model. The "model x data number" column indicates from which training data set, used for training of the corresponding model x, the feature vector was extracted. Specifically, feature vector extractor 103A extracts a feature vector 2D for each training data image in "training data image" column 2B, which stores data images of model 1 training data 2, and adds a "model 1 data number" column 2C. The same processing is performed for model 2 training data 3 and model 3 training data 4.

[0042] Then, when the second input data feature vector extraction unit 103 has executed the process of step S305 for all feature vectors extracted for each model, it ends the loop process C2 (S306). Furthermore, when the processes of steps S302 to S306 have been executed for all models, the second input data feature vector extraction unit 103 ends the loop process C1 (S307).

[0043] Then, the second input data feature vector extraction unit 103 combines the training data feature vectors 1002 used in training all models to generate a training data feature vector 1002 (S308). For example, the "feature vector" columns 2D, 3D, and 4D are set as key bindings, and the training data feature vector 1002 of model 1, the training data feature vector 1002 of model 2, and the training data feature vector 1002 of model 3, which have the same feature vector, are combined to generate a training data feature vector 1002. The training data feature vector 1002 includes a "feature vector" 1002A, a "model 1 data number" 1002B, a "model 2 data number" 1002C, and a "model 3 data number" 1002D.

[0044] Then, the second input data characteristic vector extraction unit 103 outputs the generated training data characteristic vector 1002 (S309). After the training data characteristic vector 1002 is output, the training data characteristic vector extraction process ends.

[0045] 8 is a flowchart of a model feature vector extraction process in which the training data feature vector acquisition unit 104 extracts a feature vector for each model from the training data feature vector 1002 based on the training-time verification result for each model. Figures 9A and 9B are diagrams illustrating an example of the model feature vector extraction process.

[0046] In the model feature vector extraction process, first, the training data feature vector acquisition unit 104 acquires the training data feature vector 1002 extracted by the second input data feature vector extraction unit 103 (S401). Then, a loop process D1 is started in which steps S403 to S407 are repeated for each of the inference models 106 to 108 of the inference unit 105 (S402).

[0047] In loop processing D1, the training data feature vector acquisition unit 104 acquires the verification results of the training for each of the inference models 106 to 108 of the inference unit 105 (S403). The verification results of the training for each model include, for example, as shown in FIG. 9A , data number 2A of model 1 training data 2, training data 7A with correct answers, verification data 7B with inference results, a column 7C storing IoU (Intersection over Union) as an evaluation index, and a column 7D storing F1 score as an evaluation index. The training data 7A with correct answers is training data that has already been annotated. The annotation method is not particularly limited, but includes a method of generating a bounding box (BBox) and a label. The IoU stored in column 7C and the F1 score stored in column 7D are used as evaluation indexes. However, evaluation indexes used in object detection, such as true positive (TP), may also be used, or multiple evaluation indexes may be used in combination.

[0048] Then, the training data feature vector acquisition unit 104 extracts training data numbers that have achieved high accuracy in the verification results for each model using the evaluation index threshold 10, and generates training data information 11 that has achieved high accuracy for each model (S404). For example, as shown in FIG. 9B, training data information 11 that has achieved high accuracy is generated from the model 1 verification results 7 using the evaluation index threshold 10. As shown in FIG. 9B, the evaluation index threshold 10 may be set to 0.50 for IoU and 1.00 for F1, or may be adjusted to be wider or narrower, and may be adjusted up or down depending on the evaluation criteria for the training data that has achieved high accuracy.

[0049] Then, the training data feature vector acquisition unit 104 uses the training data information 11 that obtained high accuracy for each model to extract a feature vector for each model from the training data feature vector 1002 extracted by the second input data feature vector extraction unit 103 (S405), and outputs the extracted feature vector (S406). Specifically, using the data number that obtained high accuracy for each model, the unit extracts, for each model, a feature vector 1002A corresponding to the same data number as the data number that obtained high accuracy for each model from the training data feature vector 1002. The unit then adds the data number that obtained high accuracy for each model to the feature vector 1002A corresponding to the extracted data number that obtained high accuracy, and outputs the feature vector for each model. For example, as shown in FIG. 9B, the model 1 feature vector 1003 includes a column 1003A that stores the data number that obtained high accuracy in model 1, and a "feature vector" column 1003B that stores the feature vector 1002A corresponding to the data number that obtained high accuracy.

[0050] Then, when the training data feature vector acquisition unit 104 has executed the processes of steps S403 to S406 for all models, it ends the loop process D1 (S407) and ends the model feature vector extraction process.

[0051] 10 is a flowchart of a model proficiency calculation process in which the proficiency determination unit 109 calculates the "proficiency level" of the models 106 to 108 arranged in the inference unit 105 based on the input data feature vector 1001 and the data feature vector of each model for each inference task. FIG. 11 is a diagram illustrating an example of the model proficiency calculation process.

[0052] In this embodiment, the "proficiency level" is a numerical value that indicates the likelihood that the model will derive a correct inference result, and is different from the "confidence level" that is commonly used in the field of machine learning and indicates the probability that the inference result is correct.

[0053] In the model strength calculation process, the strength determination unit 109 first starts a loop process E1 in which the processes of steps S502 to S510 are repeated for each piece of input data 5 (S501).

[0054] In loop processing E1, the specialty determination unit 109 acquires, for each piece of input data 5, an input data feature vector 1001 extracted by the first input data feature vector extraction unit 102. As shown in FIG. 11A, the acquired input data feature vector 1001 has an input data number 1001A and a column 1001B that stores the corresponding feature vector.

[0055] Then, the specialty determination unit 109 starts a loop process E2 in which the processes of steps S504 to S509 are repeated for each of the models 106 to 108 arranged in the inference unit 105 (S503).

[0056] In loop processing E2, the specialty determination unit 109 acquires a model feature vector for each of the models 106 to 108 arranged in the inference unit 105 (S504). For example, it acquires a model 1 feature vector 1003 for model 1 (106), a model 2 feature vector 1004 for model 2 (107), and a model 3 feature vector 1005 for model 3 (108). When the processing of step S504 ends, the specialty determination unit 109 starts loop processing E3 in which it repeats the processing of step S506 for each feature vector included in the model feature vector (S505).

[0057] In loop processing E3, the specialty determination unit 109 calculates the similarity between each feature vector in the acquired model feature vector and the acquired input data feature vector 1001 (S506). The similarity between feature vectors can be calculated using, for example, cosine similarity, which is expressed by equation (1). Here, the calculated similarity is a real number between -1 and 1. If the similarity is less than 0, it indicates that the two feature vectors are dissimilar, and if the similarity is greater than 0, it indicates that the two feature vectors are similar.

[0058]

number

[0059] In equation (1), vector x is the input data feature vector 1001, and vector y is the model feature vector, and their structure is expressed by equation (2).

[0060]

number

[0061] When the specialty determination unit 109 has executed the process of step S506 for all feature vectors for the input data feature vector 1001 and the model feature vector, it ends the loop process E3 (S507).

[0062] Then, the strength determination unit 109 determines the magnitude relationship of the multiple similarities calculated from the model feature vector for the input data feature vector 1001 (S508), and outputs the maximum similarity as the "strength" of the corresponding model in the input data 5 (S509).

[0063] Then, when the specialty determination unit 109 has executed the processes of steps S504 to S509 for all models, it ends the loop process E2 (S510).

[0064] Then, when the strength determination unit 109 executes the processes of steps S502 to S510 for all the input data 5, it ends the loop process E1 (S511) and ends the model strength degree calculation process.

[0065] The model proficiency calculation process described above is merely an example and is not limited to this. For example, as shown in FIGS. 11A and 11B, the proficiency determination unit 109 first obtains an input data feature vector 1001. The input data feature vector 1001 includes one input data feature vector 1001C. Next, model 1 obtains a model 1 feature vector 1003 including a data number 1003A string and a feature vector 1003B string for which high accuracy was achieved; model 2 obtains a model 2 feature vector 1004 including a data number 1004A string and a feature vector 1004B string for which high accuracy was achieved; and model 3 obtains a model 3 feature vector 1005 including a data number 1005A string and a feature vector 1005B string for which high accuracy was achieved. Then, the specialty determination unit 109 calculates the similarity between the feature vector 1001B of the input data feature vector 1001C and all the feature vectors of the model 1 feature vector 1003 (column 1003B), all the feature vectors of the model 2 feature vector 1004 (column 1004B), and all the feature vectors of the model 3 feature vector 1005 (column 1005B), and outputs the model 1 similarity 1009, the model 2 similarity 1010, and the model 3 similarity 1011, respectively. For example, the model 1 similarity 1009 has a "similarity number" column 1009A and a "vector similarity" column 1009B. Then, the magnitude relationship may be determined for all vector similarities 1009B of model 1 similarity 1009, all vector similarities 1010B of model 2 similarity 1010, and all vector similarities 1011B of model 3 similarity 1011, and the maximum similarity may be output as the "proficiency" for each model with respect to input data feature vector 1001C, thereby generating model 1 expertise 1012, model 2 expertise 1013, and model 3 expertise 1014.

[0066] Fig. 12 is a flowchart of an inference process in which the inference unit 105 uses a plurality of models 106 to 108 arranged in the inference unit 105 to output an inference result for the input data 5. Fig. 13 is a diagram illustrating an example of the inference process.

[0067] In the inference process, the inference unit 105 first acquires (S601) the input data 5. As shown in Fig. 3, the input data 5 includes a column 5A that stores input data numbers and a column 5B that stores input data images.

[0068] Then, the inference unit 105 starts a loop process F1 in which the processes of steps S603 to S604 are repeated for each of the models 106 to 108 arranged in the inference unit 105 based on the acquired input data 5 (S602).

[0069] In loop processing F1, the inference unit 105 inputs the acquired input data 5 as inference input data to models 106 to 108 arranged in the inference unit 105, and executes inference (S603). For example, as shown in FIG. 13, input data 5C is input to model 1 (106), model 2 (107), and model 3 (108) arranged in the inference unit 105, and inference is executed by each model. Here, the model can use various object detection technologies that can output a bounding box (BBox), a label indicating the recognized object class, and a confidence score for the recognized object, such as R-CNN (Region-based Convolutional Neural Network), YOLO (You Only Look Once), or SSD (Single Shot MultiBox Detector).

[0070] Inference involves recognizing objects in the input data image 5B. Then, the inference results for each model are output (S604). The inference results for each model are in the form of image data, such as model 1 inference result 1006, model 2 inference result 1007, and model 3 inference result 1008. For each recognized object, the model 1 inference result 1006 includes a bounding box (BBox) 1006A that displays the area of ​​the recognized object, a label 1006B that describes the recognized object class, and a confidence score 1006C for the recognized object.

[0071] When the processing of steps S603 to S604 is completed for all models, the inference unit 105 ends the loop processing F1 (S605), and then ends the inference processing for outputting the inference result.

[0072] 14 is a flowchart showing a final inference result B1025 output process in which the output unit 110 outputs a final inference result B1025 based on a plurality of model inference results output by the inference unit 105 and the suitability level for the input data 5 for each model output by the suitability determination unit 109. FIGS. 15A and 15B are diagrams illustrating an example of a weighting process in which the output unit 110 weights the model 1 inference result 1006 output from the inference unit 105 based on the model 1 suitability level 1012 output by the suitability determination unit 109.

[0073] In the output process for outputting the final inference result B1025, the output unit 110 first acquires the inference result for each model from the inference unit 105 (S701). Then, based on the acquired inference result, the output unit 110 starts a loop process G1 that repeats the processes of steps S703 to S706 for each inference result of the model (S702).

[0074] In loop processing G1, the output unit 110 acquires a model suitability score from the suitability determination unit 109 for each inference result of a model corresponding to the inference result acquired from the inference unit 105 (S703). For example, as shown in FIG. 15A, the output unit 110 acquires a model 1 suitability score 1012 for model 1. Then, for each recognized object in the image of the inference result, in other words, for each bounding box, the output unit 110 generates an inference result including an image of the corresponding bounding box, a label, and a confidence score (S704). The inference result 1015 for each recognized object of model 1 shown in FIG. 15A includes a "BBox Number" column 1015A that stores a BBox number for identifying the recognized object in the model 1 inference result 1006, a "BBox Image" column 1015B that stores an image within the BBox of the recognized object, a "Label" column 1015C that stores a label indicating the class of the recognized object, and a "Confidence Score" column 1015D that stores the confidence score of the recognized object.

[0075] Then, the output unit 110 performs weighting processing on the inference result for each model-recognized object using the model's suitability (S705). The weighting processing is performed by multiplying the reliability score of the inference result for each model-recognized object by a weighting coefficient to calculate a weighted reliability score. For example, the weighting processing can be performed using equation (3). In equation (3), z is the reliability score of the inference result for each model-recognized object, s indicates the model's suitability, and Z is the weighted reliability score calculated during the weighting processing.

[0076]

number

[0077] Then, the output unit 110 replaces the reliability score of the inference result for each model-recognized object with a weighted reliability score and outputs the weighted inference result 1018 for each model-recognized object (S706). For example, the weighted inference result 1018 for each model-recognized object shown in FIG. 15B includes a "BBox number" column 1018A that stores a BBox number for identifying the recognized object in the model-1 inference result 1006, a "BBox image" column 1018B that stores a BBox image of the recognized object, a "label" column 1018C that stores a label indicating the class of the recognized object, and a "weighted reliability score" column 1018D that stores the weighted reliability score of the recognized object. The "weighted reliability score" column 1018D is calculated by weighting the reliability score of the inference result for each model-recognized object by multiplying it by a weighting coefficient.

[0078] When the processing of steps S703 to S706 is completed for all model inference results, the output unit 110 ends the loop processing G1 (S707).

[0079] Thereafter, the weighted inference results 1018 for each model recognition object, including the output weighted confidence scores, are integrated (S708). For example, the weighted inference results 1018 for each model recognition object are integrated as an integrated inference result for each recognition object.

[0080] 16A and 16B are diagrams for explaining the derivation of an integrated inference result 1021 for each recognized object and an integrated inference result 1022 for each recognized object obtained by fusing the results.

[0081] 16A, the integrated inference result 1021 for each recognized object includes, for example, a "Model 1 BBox number" column 1021A storing the BBox number for identifying the recognized object of the model 1 inference result 1006, a "Model 1 BBox image" column 1021B storing the image in the BBox of the model 1 recognized object, a "Model 1 label" column 1021C storing the label representing the class of the model 1 recognized object, a "Model 1 confidence score" column 1021D storing the weighted confidence score of the model 1 recognized object, a "Model 2 BBox number" column 1021E storing the BBox number for identifying the recognized object of the model 2 inference result 1007, and a "Model 2 BBox number" column 1021F storing the image in the BBox of the model 2 recognized object. the "Model 2 BBox Image" column 1021F storing the BBox number for identifying the recognized object of the model 3 inference result 1008; the "Model 3 BBox Image" column 1021J storing the image within the BBox of the model 3 recognized object; the "Model 3 Label" column 1021K storing the label representing the class of the model 3 recognized object; and the "Model 3 Confidence Score" column 1021L storing the weighted confidence score of the model 3 recognized object.

[0082] Then, a fusion process is performed on the "Bbox number" and "Bbox image" columns of the integrated inference results for each recognized object and each model, and a final inference result A1024 is output (S709). The fusion process may use Bbox optimization and fusion methods such as NMS (Non-Maximum Suppression), Soft Voting, and WBF (Weighted Box Fusion). For example, as shown in FIG. 16A, for each row of the integrated inference result 1021 for each recognized object, the reliability scores 1021D, 1021H, and 1021L for each model are compared to determine the Bbox number and Bbox image with the highest reliability score, and a fused Bbox number 1022A and a fused Bbox image 1022B are output. As shown in FIG. 16B, an integrated inference result 1022 for each fused recognized object is created.

[0083] 17A, 17B, and 17C are diagrams illustrating how the output unit 110 derives a final inference result A 1024 based on the integrated inference result 1022 for each fused recognized object.

[0084] The integrated inference result 1022 for each fused recognized object includes, for example, as shown in FIG. 17A, a "Fused BBox Number" column 1022A that stores a BBox number for identifying the recognized object of the inference result, a "Fused BBox Image" column 1022B that stores an image within the BBox of the recognized object, a "Model 1 Label" column 1022C that stores a label representing the class of the Model 1 recognized object, a "Model 1 Confidence Score" column 1022D that stores the weighted confidence score of the Model 1 recognized object, a "Model 2 Label" column 1022E that stores a label representing the class of the Model 2 recognized object, a "Model 2 Confidence Score" column 1022F that stores the weighted confidence score of the Model 2 recognized object, a "Model 3 Label" column 1022G that stores a label representing the class of the Model 3 recognized object, and a "Model 3 Confidence Score" column 1022H that stores the weighted confidence score of the Model 3 recognized object.

[0085] As shown in FIG. 17A, the data for discussion 1023 includes, for example, a "Fused BBox Number" column 1023A that stores the BBox number for identifying the recognized object of the inference result, a "Fused BBox Image" column 1023B that stores the image within the BBox of the recognized object, a "Label 1" column 1023C that stores the label representing the class of the recognized object of model 1, a "Weight 1" column 1023D that stores the total weight of model 1, a "Label 2" column 1023E that stores the label representing the class of the recognized object of model 2, a "Weight 2" column 1023F that stores the total weight of model 2, a "Label 3" column 1023G that stores the label representing the class of the recognized object of model 3, and a "Weight 3" column 1023H that stores the total weight of model 3.

[0086] The final inference result A1024 is determined by collaborating based on the reliability scores for each model in the integrated inference result 1022 for each fused recognized object. Specifically, as shown in FIGS. 17A and 17B, the reliability scores of models that recognized the same object (same Bbox image) for each row of the integrated inference result 1022 for each fused recognized object are summed to calculate a weight to be used in collaborating on the target. Then, collaborating data 1023 is output, which combines a "fused Bbox number" 1023A, a "fused Bbox image" 1023B, and a weight 1 (1023D) obtained by summing the "fused Bbox number" 1023A and the corresponding label 1023C. Thereafter, as shown in FIG. 17C, the output unit 110 selects the label 1024C with the highest weight for the fused Bbox number and fused Bbox image based on the collaborating data 1023, and combines the "fused Bbox number" 1024A and the "fused Bbox image" 1024B to create the final inference result A1024. In the consensus process, for each fused Bbox number and fused Bbox image, the label with the maximum weight among the weights of the labels representing the multiple recognized object classes is selected. For example, methods such as majority voting or soft voting may be used.

[0087] Then, the output unit 110 reflects the final inference result A1024 in the input data 5 and outputs the final inference result B1025 in the form of an image (S709). As shown in Fig. 18, the image of the output final inference result B1025 is assigned the fused Bbox and the label decided by consensus. Thereafter, the output process of the final inference result B1025 ends.

[0088] FIG. 19 is a flowchart of a validity determination process in which the output validity determination unit 111 determines the validity of the final inference result A1024 based on the final inference result A1024 and the inference results for each model.

[0089] In the validity determination process, the output validity determination unit 111 starts a loop process H1 in which the processes of steps S802 to S813 are repeated for the final inference result A1024 for each inference task (S801).

[0090] In loop processing H1, the output validity determination unit 111 acquires the final inference result A1024 output by the output unit 110 and the inference results for each model output by the inference unit 105 (S802). Then, a model classification process is executed to classify multiple operating models allocated to the inference unit 105 corresponding to the acquired inference results for the multiple models.

[0091] In the model classification process, the output validity determination unit 111 starts a loop process H2 in which the processes of steps S804 to S805 are repeated for the multiple models 106 to 108 currently in operation arranged in the inference unit 105 (S803).

[0092] FIG. 20 is a diagram illustrating an example of a model classification process in which the output validity determination unit 111 classifies models based on the degree of suitability of each of the models 106 to 108 currently in operation and arranged in the inference unit 105 in steps S804 to S805.

[0093] In loop processing H2, first, the output validity determination unit 111 acquires the model 1 similarity 1009, model 2 similarity 1010, and model 3 similarity 1011 from the strength determination unit 109 for the models 106-108 currently in operation and arranged in the inference unit 105 (S804). Then, based on the acquired strength level for each model, the output validity determination unit 111 classifies the models 106-108 into two types, "strong models" and "poor models," and generates model classification information 1026 (S805). As shown in FIG. 20, the model classification information 1026 includes a "strong model name" column 1026A that stores strong model information and a "poor model name" column 1026B that stores poor model information. A "strong model" is a model whose strength level ranges from 0 to 1 and has high recognition accuracy for the object, while a "poor model" is a model whose strength level ranges from -1 to 0 and has low recognition accuracy for the object. 20 , since the values ​​of model 1 similarity 1009, model 2 similarity 1010, and model 3 similarity 1011 are between 0 and 1, they are classified as "good models" in the classification process, and the model names are recorded in "good model name" column 1026A of model classification information 1026. Since model 3 similarity 1014 is between -1 and 0, they are classified as "bad models" in the classification process, and the model names are recorded in "bad model name" column 1026B of model classification information 1026.

[0094] When the processing of steps S804 to S805 is completed for all models in operation, the output validity determination unit 111 ends the loop processing H2 (S806).

[0095] Then, the output validity determination unit 111 executes a final inference result A1024 validity determination process to determine the validity of the final inference result A1024 (S807).

[0096] Fig. 21 is a flowchart of the process (S807) for determining the validity of the final inference result A1024. Fig. 22A and Fig. 22B are diagrams for explaining an example of the process (S807) for determining the validity of the final inference result A1024.

[0097] In the process of determining the validity of the final inference result A1024 (S807), the output validity determination unit 111 starts a loop process I1 that repeats the processes of steps S902 to S906 for each determination process of the validity of the final inference result A1024 (S901).

[0098] In loop processing I1, the output validity determination unit 111 obtains the good model name 1026A from the model classification information 1026 (S902). Then, using the good model name 1026A, the output validity determination unit 111 extracts labels and reliability scores for all good models from the integrated inference result 1022 for each fused recognition object. Then, the output validity determination unit 111 combines the extracted labels and reliability scores for each good model, and further adds the fusion Bbox number 1022A and fusion Bbox image 1022B of the integrated inference result 1022 for each fused recognition object to create the integrated inference result 1027 for the good model (S903).

[0099] Then, for the integrated inference result 1027 of the created good model and the acquired final inference result A1024, coverage 1030 between the fusion Bbox number and the fusion Bbox image, which indicates the class of the recognized object, is calculated (S904).

[0100] Figures 23A and 23B are figures illustrating an example of a process in which the output validity judgment unit 111 calculates the coverage rate between labels indicating the class of the recognized object for the integrated inference result 1027 of the created favorite model and the acquired final inference result A1024, and judges the validity of the final inference result A1024 using coverage 1030.

[0101] Specifically, the output validity determination unit 111 extracts a column storing the fusion Bbox number and a column storing the label from the integrated inference result 1027 of the good model and the final inference result A1024, and combines the fusion Bbox number and the label as elements of the sets to create a final inference result A1024 set 1028 and an inference result set 1029 of the good model. For example, as shown in FIG. 23A, for the final inference result A1024, the fusion Bbox number 1024A and the label 1024C are combined to create the final inference result A1024 set 1028. Then, as shown in FIG. 23B, the fusion Bbox number 1027A of the integrated inference result 1027 of the good model is combined with the good model 1 label 1027C (see FIG. 22B) and the fusion Bbox number 1027A and the good model 2 label 1027E to create the inference result set 1029 of the good model.

[0102] Then, the output validity determination unit 111 calculates the coverage 1030 of the set 1028 of final inference results A 1024 in the inference result set 1029 of the good model using equation (4).

[0103]

number

[0104] In equation (4), k AB is the coverage 1030, A is the final inference result A 1024 set 1028, and B is the inference result set 1029 of the favored model.

[0105] Then, the output validity determination unit 111 compares the calculated coverage 1030 with the validity determination threshold 1031 (S905). In FIG. 23C, the validity determination threshold 1031 for the coverage 1030 is set to 1. However, the validity determination threshold 1031 may be reduced to widen the allowable range for determining the validity of the final inference result A1024.

[0106] In step S904, a method for expressing the inclusion relationship or similarity between sets, such as a subset check, a Jaccard similarity coefficient, or a Sorensen-Dice coefficient, may be used instead of the coverage 1030. The validity determination threshold 1031 that is set is also changed depending on the method used.

[0107] The output validity determination unit 111 then outputs validity determination information (S906). If the coverage 1030 is less than the validity determination threshold 1031, a validity determination result 1032 of "abnormal" is output. If the coverage 1030 is equal to or greater than the validity determination threshold 1031, a validity determination result 1032 of "no abnormality" is output. Here, "abnormal" means that the final inference result A1024 is not entirely provided by the inference result of the good model, but that part of the final inference result A1024 may be provided by the inference result of the non-good model. If the good model has a high degree of expertise but the confidence score recognized by the good model in the bounding box is lower than that of the non-good model, the output validity determination unit 111 determines an "abnormal" and outputs a determination result of "abnormal." In other words, "no abnormality" means that the final inference result A1024 is entirely provided by the inference result of the good model.

[0108] 23C, the validity determination result 1032 includes a column for storing validity information and a column for storing information on multiple problem Bbox numbers. The "problem Bbox number" is the fusion Bbox number corresponding to all elements of the final inference result A1024 set 1028 that are missing from the inference result set 1029 of the favored model when the coverage 1030 is equal to or less than the validity determination threshold 1031.

[0109] Then, when the processing of steps S902 to S906 is completed for all the determination processes, the output validity determination unit 111 ends the loop process I1 (S907).

[0110] Returning to the validity determination process (FIG. 19), the description will be continued. The output validity determination unit 111 then ends the validity determination process (S807) of the final inference result A1024.

[0111] Then, the output validity determination unit 111 determines whether the validity determination result 1032 of the final inference result A1024 includes "abnormality present" (S808).

[0112] If the validity determination result 1032 of the final inference result A1024 includes "abnormality present", an abnormality cause model determination process (S809) is executed to determine the model that is the cause of the abnormality. On the other hand, if the validity determination result 1032 of the final inference result A1024 does not include "abnormality present", the output validity determination unit 111 skips the processes of steps S809 to S813.

[0113] Fig. 24 is a flowchart of the abnormality cause model determination process (S809). Fig. 25A, Fig. 25B, and Fig. 25C are diagrams for explaining an example of the abnormality cause model determination process (S809).

[0114] In the abnormality cause model determination process (S809), the output validity determination unit 111 starts a loop process J1 that repeats the processes of steps S1002 to S1008 for each problem bounding box to be stored in the validity determination result 1032 for the final inference result A1024, as shown in Figure 24 (S1001).

[0115] In the loop process J1, the output validity determination unit 111 starts a loop process J2 in which the processes of steps S1003 to S1006 are repeated for each inference result of the model for the problem bounding box (S1002).

[0116] In loop process J2, output validity determination unit 111 extracts the inference result of the problem bounding box from the inference result for each recognized object for each model (S1003). As shown in FIG. 25A, for example, inference result 1033 for the problem bounding box of Model 1 includes a "Question Bbox Number" column 1033A, a "Bbox Image" column 1033B, a "Label" column 1033C, and a "Confidence Score" column 1033D. Inference result 1034 for the problem bounding box of Model 2 includes a "Question Bbox Number" column 1034A, a "Bbox Image" column 1034B, a "Label" column 1034C, and a "Confidence Score" column 1034D. The inference result 1035 for the problem bounding box of Model 3 includes a "Problem Bbox Number" column 1035A, a "Bbox Image" column 1035B, a "Label" column 1035C, and a "Confidence Score" column 1035D. Here, the confidence scores stored in Model 1 "Confidence Score" column 1033D, Model 2 "Confidence Score" column 1034D, and Model 3 "Confidence Score" column 1035D are confidence scores before weighting is performed.

[0117] Then, based on the information in the "Label" column from the inference results for the problem Bbox for each model, statistical information for the corresponding label is extracted from the training data for each model (S1004). For example, as shown in FIG. 25B, statistical information 1037 for "Truck" in the "Label" column 1034C of Model 2 that corresponds to "Truck" in Training Data 3 of Model 2 includes "Standard Deviation" 1037A, "Maximum Confidence Score" 1037B, and "Minimum Confidence Score" 1037C. As shown in FIG. 25C, statistical information 1038 for "Car" in the "Label" column 1035C of Model 3 that corresponds to "Car" in Training Data 4 of Model 3 includes "Standard Deviation" 1038A, "Maximum Confidence Score" 1038B, and "Minimum Confidence Score" 1038C.

[0118] Next, the output validity determination unit 111 compares the reliability score of the inference result of the extracted problem bounding box with the statistical information of the corresponding label from the training data, and determines the consistency between the reliability score of the inference result of the corresponding label and the reliability score distribution of the training data (S1005).

[0119] Thereafter, the output validity determination unit 111 determines whether or not any of the corresponding models is the cause of the anomaly according to the comparison result (S1006). If the reliability score of the inference result of the corresponding label matches the reliability score distribution of the training data, this model is not the cause of the anomaly. If the reliability score of the inference result of the corresponding label does not match the reliability score distribution of the training data, this model is the cause of the anomaly.

[0120] When the processing of steps S1003 to S1006 is completed for the inference results of all models, the output validity determination unit 111 ends the loop processing J2 (S1007).

[0121] Then, information on all models that are the cause of an abnormality is collected to generate information on models that are the cause of an abnormality 1039 (S1008). As shown in FIG. 25B, information on models that are the cause of an abnormality 1039 includes a column 1039A that stores model numbers and a column 1039B that stores model suitability scores.

[0122] Furthermore, when the processing of steps S1002 to S1008 is completed for all problem bounding boxes, the output validity determining unit 111 ends the loop process J1 (S1009) and ends the process of determining the model causing the abnormality (S809).

[0123] Returning to the validity determination process (FIG. 19), the output validity determination unit 111 determines the magnitude relationship of the suitability of the models that are the cause of anomalies to be stored in column 1039B in the model information 1039 that are the cause of anomalies (S810).

[0124] Then, the output validity determination unit 111 determines that the model causing the abnormality with the highest degree of suitability is the model requiring relearning, and outputs the model requiring relearning information (S811). For example, as shown in FIG. 25B, the magnitude relationship is determined for the model suitability degrees stored in the model suitability degree 1039B of the model information causing the abnormality 1039. The model causing the abnormality with the highest model suitability degree is model 2. Then, the output validity determination unit 111 generates the model requiring relearning information 1040 using model 2 as the model number requiring relearning.

[0125] Then, the output validity determination unit 111 determines whether the number of the model that needs to be re-learned is present in the re-learning model information 1040 (S812).

[0126] If the re-learning model information 1040 contains a model number that needs to be re-learned, an inference result update process (S813) is executed to update the inference result of the model that needs to be re-learned. On the other hand, if the re-learning model information 1040 does not contain a model number that needs to be re-learned, the output validity determination unit 111 skips the process of step S813.

[0127] FIG. 26 is a flowchart of the inference result update process (S813).

[0128] In the inference result update process, the output validity determination unit 111 starts a loop process K1 in which the processes of steps S1102 to S1103 are repeated for each model corresponding to the relearning required model number in the relearning required model information 1040 (S1101).

[0129] In the loop process K1, the output validity determination unit 111 extracts the problem Bbox number from the generated validity determination result 1032 (S1102).

[0130] Then, the output validity determination unit 111 deletes the inference result corresponding to the problem Bbox number from the integrated inference result 1022 for each fused recognition object (S1103).

[0131] When the processing of steps S1102 to S1103 is completed for all models that require relearning, the output validity determination unit 111 ends the loop processing K1 (S1104).

[0132] Then, the output validity determination unit 111 outputs the integrated inference result of the model from which all the inference results of the re-learning required models for the problem B box number have been deleted as the integrated inference result 1041 of all the updated models (S1105).

[0133] FIG. 27 is a diagram illustrating an example of an integrated inference result 1041 of all updated models.

[0134] Compared with the integrated inference result 1022 for each fused recognition object, the updated integrated inference result 1041 of all models has deleted the inference result (underlined) of Model 2, the model that requires re-learning for problem Bbox number 09 in 1041A.

[0135] Returning to the validity determination process (FIG. 19), the description will be continued. After outputting the integrated inference result 1041, the output validity determination unit 111 ends the update process (S813) for the inference result of the model requiring relearning.

[0136] Then, when the output validity determination unit 111 executes the processes of steps S802 to S813 for all the final inference results A1024 of the inference task, it ends the loop process H1 (S814) and ends the validity determination process for the final inference result A1024.

[0137] FIG. 28 is a flowchart of the process of updating the final inference result B 1025, in which the output unit 110 updates the final inference result A 1024 using the integrated inference result 1041 of all updated models.

[0138] In the process of updating the final inference result B1025, the output unit 110 starts a loop process L1 in which the processes of steps S1202 to S1204 are repeated for each inference task (S1201).

[0139] In the loop processing L1, the output unit 110 acquires the integrated inference results 1041 of all the updated models from the output validity determination unit 111 (S1202).

[0140] Then, the output unit 110 generates reconference data 1042 based on the integrated inference results 1041 of all the updated models (S1203). The process of generating the reconference data 1042 is the same as step S706 of the final inference result B 1025 output process (FIG. 14).

[0141] FIG. 29 is a diagram illustrating an example of the reconference data 1042.

[0142] The structure of the re-consultation data 1042 is the same as the structure of the conference data 1023, and includes a "Fused BBox Number" column 1042A that stores the BBox number for identifying the recognized object of the inference result, a "Fused BBox Image" column 1042B that stores the image within the BBox of the recognized object, a "Label 1" column 1042C that stores the label representing the class of the recognized object of model 1, a "Weight 1" column 1042D that stores the total weight of model 1, a "Label 2" column 1042E that stores the label representing the class of the recognized object of model 2, a "Weight 2" column 1042F that stores the total weight of model 2, a "Label 3" column 1042G that stores the label representing the class of the recognized object of model 3, and a "Weight 3" column 1042H that stores the total weight of model 3.

[0143] The output unit 110 then re-convenes based on the re-conference data 1042, generates and outputs an updated final inference result B1025 with the label determined in the re-conference (S1204). When the output unit 110 has executed the processes of steps S1202 to S1204 for all inference tasks, it ends the loop process L1 (S1205). The output unit 110 then ends the process of updating the final inference result B1025.

[0144] FIG. 30 is a flowchart of the re-learning process in which the model reconfiguration unit 112 re-learns a model that needs to be re-learned.

[0145] In the relearning process, first, the model reconfiguration unit 112 acquires the relearning required model information 1040 from the output validity determination unit 111 (S1301).

[0146] Then, the model reconfiguration unit 112 starts a loop process M1 in which the processes of steps S1303 to S1306 are repeated for the models that need to be re-learned that are stored in the re-learning model information 1040 (S1302).

[0147] In the loop process M1, the model reconfiguration unit 112 determines whether or not there is past version data 6 for the model corresponding to the re-learning required model number (S1303).

[0148] If there is past version data 6, the model reconfiguration unit 112 executes a version rollback process (S1304) for the relevant model requiring re-learning to be placed in the inference unit 105. On the other hand, if there is no past version data 6, the model reconfiguration unit 112 skips the process of step S1304.

[0149] In the version rollback process (S1304), the model reconfiguration unit 112 transmits the past version data 6 of the model that needs to be re-learned to the inference unit 105. The inference unit 105 then receives the past version data 6, deletes the existing version of the model, and replaces it with the received past version of the model.

[0150] Then, the model reconfiguration unit 112 performs re-learning on the model that needs to be re-learned (S1305). In re-learning, the model reconfiguration unit 112 adds learning data for re-learning to the model that needs to be re-learned and performs model re-learning. Then, the model reconfiguration unit 112 transmits the re-learned model data again to the inference unit 105. Then, the inference unit 105 deletes the existing version of the model and replaces it with the acquired re-learned version model, thereby updating the model (S1306). The inference unit 105 inputs the input data 5 to the model updated by the re-learning and obtains an inference result for the updated model. Alternatively, the inference unit 105 may input the input data 5 to a model other than the model that was the target of re-learning and perform inference.

[0151] Then, when the model reconfiguration unit 112 has executed the processes of steps S1303 to S1306 for all models that require relearning, it ends the loop process M1 (S1307) and terminates the relearning process.

[0152] FIG. 31 is a diagram showing the hardware configuration of an electronic control unit 30 in which the vehicle control system 100 of this embodiment is implemented.

[0153] The electronic control device 30 is an electronic control unit (ECU) 30 having a processing unit (CPU) 31, a memory 32, and a communication interface 34. The processing unit 31 is a processor (e.g., a microcomputer) that executes programs stored in the memory 32. The processing unit 31 operates as a functional block that provides various functions by executing predetermined arithmetic processing. The memory 32 includes a non-volatile storage area and a volatile storage area. The non-volatile storage area is accessible by the processing unit 31 and includes a program area that stores programs executed by the processing unit 31, and a data area that stores data used by the processing unit 31 when executing the programs. The volatile storage area temporarily stores data used by the processing unit 31 when executing the programs. The communication interface 34 connects to other electronic control units via a network such as a CAN or Ethernet.

[0154] As described above, according to the first embodiment of the present invention, the vehicle control system 100 acquires an in-vehicle camera image 1 using the data acquisition unit 101 and generates input data 5. The first input data feature vector extraction unit 102 extracts an input data feature vector 1001 from the input data 5. The inference unit 105 does not perform inference on the input data 5 using a single model, but performs inference using multiple models taking into account the characteristics of each model, and generates multiple inference results. The second input data feature vector extraction unit 103 extracts training data feature vectors 1002 for all models. The training data feature vector acquisition unit 104 acquires the feature vector for each model. The suitability determination unit 109 calculates a model suitability score for each model using the extracted input data feature vector 1001 and the multiple model feature vectors. The output unit 110 outputs a final inference result B1025 by consensus based on the inference results of the multiple models and the multiple model suitability scores. The output validity determination unit 111 determines the validity of the final inference result A1024, determines the model that is the cause of the abnormality, and outputs re-learning required model information 1040. The model reconfiguration unit 112 re-learns the model based on the re-learning required model information 1040.

[0155] Therefore, the vehicle control system 100 of the first embodiment can improve the robustness of inference compared to a vehicle control device that detects an object using a single model. Also, the validity of the final inference result A1024 can be accurately determined. Even if an abnormality is determined in the final inference result A1024, the vehicle control device does not stop the next inference, but can continue inferring new data.

[0156] The present embodiment includes a system and a method for executing the above information processing.

[0157] As described above, according to the first embodiment of the present invention, an object can be recognized with high accuracy by combining multiple trained models with different characteristics. Furthermore, the model that is the cause of an abnormality is determined based on the degree of suitability of each model for the input data 5, the inference results of each model, and the final inference result A1024, so that re-learning can be performed by narrowing down the models that are most effective for re-learning.

[0158] <Example 2> Although the vehicle control device described in the first embodiment is realized by an on-board electronic control unit (ECU), it may be configured by combining a server (cloud) and an electronic control unit. In the second embodiment, a configuration in which a server and an electronic control unit are combined will be described.

[0159] FIG. 32 is a block diagram showing the functional configuration of a vehicle control system 200 that combines the server 20 and the electronic control unit 30 according to the second embodiment.

[0160] The vehicle control system 200 of the second embodiment is composed of a server 20 and an electronic control unit 30 connected via a network. The server 20 has a second input data feature vector extraction unit 103, a model reconstruction unit 112, and an output validity determination unit 111. The server 20 stores learning data 2, 3, and 4 for each model, verification results 7, 8, and 9 for each model, and previous version data 6 of the model. The electronic control unit 30 has a data acquisition unit 101, a first input data feature vector extraction unit 102, a learning data feature vector acquisition unit 104, an inference unit 105, a specialty determination unit 109, and an output unit 110.

[0161] In the vehicle control system 200 on the server 20 side, the second input data feature vector extraction unit 103 extracts the learning data feature vectors 1002 of all models. In addition, the output validity determination unit 111 determines the validity of the final inference result A1024, performs processing to determine the model that is the cause of the abnormality, and outputs re-learning required model information 1040. The model reconstruction unit 112 executes the re-learning processing of the model based on the re-learning required model information 1040.

[0162] In the vehicle control system 200 on the electronic control device 30 side, the vehicle control system 200 acquires the in-vehicle camera image 1 using the data acquisition unit 101, and generates input data 5. The first input data feature vector extraction unit 102 extracts an input data feature vector 1001 from the input data 5. The inference unit 105 does not perform inference on the input data 5 using a single model, but performs inference using multiple models taking into account the characteristics of each model, and generates multiple inference results. The training data feature vector acquisition unit 104 acquires the feature vector for each model. The suitability determination unit 109 calculates a model suitability score for each model using the extracted input data feature vector 1001 and the multiple model feature vectors. The output unit 110 outputs a final inference result B1025 by consensus based on the inference results of the multiple models and the multiple model suitability scores.

[0163] FIG. 33 is a diagram showing the hardware configuration of the server 20. As shown in FIG.

[0164] The server 20 is configured by a computer having a processor (CPU) 21, a memory 22, an auxiliary storage device 23, and a communication interface 24. The server 20 may also have an input interface 25 and an output interface .

[0165] The processor 21 is a computing device that executes programs stored in the memory 22. The processor 21 executes various programs to realize each functional unit of the server 20. Note that part of the processing performed by the processor 21 by executing the programs may be executed by another computing device (for example, hardware such as an ASIC or FPGA, or a GPU).

[0166] The memory 22 includes a ROM, which is a non-volatile storage element, and a RAM, which is a volatile storage element. The ROM stores unchanging programs (e.g., BIOS) and the like. The RAM is a high-speed, volatile storage element such as a DRAM (Dynamic Random Access Memory), and temporarily stores programs executed by the processor 21 and data used when the programs are executed.

[0167] The auxiliary storage device 23 is a large-capacity, non-volatile storage device such as a magnetic storage device (HDD) or a flash memory (SSD). The auxiliary storage device 23 also stores data used by the processor 21 when executing a program and the program executed by the processor 21. That is, the program is read from the auxiliary storage device 23, loaded into the memory 22, and executed by the processor 21 to realize each function of the server 20.

[0168] The communication interface 24 is a network interface device that controls communication with other devices in accordance with a predetermined protocol.

[0169] The input interface 25 is an interface to which input devices such as a keyboard 27 and a mouse 28 are connected and which receives input from an operator. The output interface 26 is an interface to which output devices such as a display device 29 and a printer (not shown) are connected and which outputs the results of program execution in a format that can be viewed by an operator.

[0170] The programs executed by the processor 21 are provided to the server 20 via removable media (CD-ROM, flash memory, etc.) or a network, and are stored in a non-volatile auxiliary storage device 23, which is a non-transitory storage medium. For this reason, the server 20 should preferably have an interface for reading data from removable media.

[0171] The server 20 is a computer system configured on one physical computer or on multiple logically or physically configured computers, and may operate on a virtual computer constructed on multiple physical computer resources. For example, each functional unit may operate on a separate physical or logical computer, or multiple functional units may be combined to operate on a single physical or logical computer.

[0172] As described above, according to the second embodiment of the present invention, processes with high processing loads are executed by the server 20 with high computing power, so that a computing device with low computing power can be used for the on-board electronic control device 30, thereby reducing the cost of the electronic control device 30.

[0173] The present invention is not limited to the above-described embodiments, but includes various modifications and equivalent configurations within the spirit and scope of the appended claims. For example, the above-described embodiments have been described in detail to clearly explain the present invention, and the present invention is not necessarily limited to configurations including all of the described configurations. Furthermore, part of the configuration of one embodiment may be replaced with the configuration of another embodiment. Furthermore, the configuration of another embodiment may be added to the configuration of one embodiment. Furthermore, part of the configuration of each embodiment may be added, deleted, or replaced with other configurations.

[0174] Furthermore, the aforementioned configurations, functions, processing units, processing means, etc. may be realized in part or in whole in hardware, for example by designing them as integrated circuits, or may be realized in software by having a processor interpret and execute a program that realizes each function.

[0175] Information such as programs, tables, and files that realize each function can be stored in a storage device such as a memory, a hard disk, or an SSD (Solid State Drive), or in a recording medium such as an IC card, an SD card, or a DVD.

[0176] In addition, the control lines and information lines shown are those that are considered necessary for explanation, and do not necessarily represent all the control lines and information lines that are necessary for implementation. In reality, it can be assumed that almost all components are interconnected. [Explanation of symbols]

[0177] 100: Vehicle control device, 101: Data acquisition unit, 102: First input data feature vector extraction unit, 103: Second input data feature vector extraction unit, 104: Learning data feature vector acquisition unit, 105: Inference unit, 106: Model 1, 107: Model 2, 108: Model 3, 109: Advantage determination unit, 110: Output unit, 111: Output validity determination unit, 112: Model reconstruction unit

Claims

1. A vehicle control system, comprising: a data acquisition unit that acquires input data to be input into the model; an inference unit that inputs the input data to a plurality of the models, performs inference, and outputs output data for each of the models; an input data feature vector extraction unit that extracts an input data feature vector representing a feature of the input data; a learning data feature vector acquisition unit that acquires a learning data feature vector representing features of the learning data used in training the model; a suitability determination unit that determines, for each model, a suitability indicating a possibility that the model will derive correct output data for the input data, based on the input data feature vector and the training data feature vector for each of the plurality of models; an output unit that outputs an inference result derived based on the degree of suitability for each model determined by the degree of suitability determination unit and the plurality of output data.

2. 2. The vehicle control system according to claim 1, the input data is an image captured by a camera mounted on a vehicle, A vehicle control system, wherein the input data feature vector and the training data feature vector are vectors configured with feature quantities of the image.

3. 2. The vehicle control system according to claim 1, The vehicle control system, wherein the proficiency level determination unit determines, for each model, a similarity between the input data feature vector and the training data feature vector as the proficiency level.

4. 2. The vehicle control system according to claim 1, The output unit derives the inference result by summing up output data for each of the models weighted according to the degree of suitability for each of the models determined by the degree of suitability determination unit.

5. 2. The vehicle control system according to claim 1, A vehicle control system comprising a validity determination unit that compares the output data output from the good model with the high degree of goodness with the inference result output from the output unit to determine the validity of the inference result.

6. 6. The vehicle control system according to claim 5, When the validity of the output data is low, the validity determination unit compares the reliability score for the inference object contained in the output data output from the plurality of models with the reliability score for the inference object contained in the learning data used to learn the model, and identifies a model to be re-learned.

7. 7. The vehicle control system according to claim 6, The inference unit inputs the input data to a model other than the re-learning target model, performs inference, and outputs output data for each of the models.

8. 7. The vehicle control system according to claim 6, a model reconfiguration unit that performs relearning of the relearning target model, The inference unit inputs the input data into a re-learned model re-learned by the model reconstruction unit, performs inference, and outputs output data.

9. A vehicle control method executed by a vehicle control system, comprising: the vehicle control system includes a computing device that executes arithmetic processing and a storage device that is accessible by the computing device; The vehicle control method includes: a data acquisition procedure for acquiring input data to be fed into the model; an inference procedure in which the input data is input to a plurality of the models, and inference is performed, and output data for each of the models is output; an input data feature vector extraction step of extracting an input data feature vector representing a feature of the input data; a learning data feature vector acquisition step of acquiring a learning data feature vector representing the features of the learning data used in training the model; a preference level determination step of determining, for each of the models, a preference level indicating a possibility of deriving correct output data for the input data based on the input data feature vector and the training data feature vector for each of the plurality of models; an output step of outputting an inference result derived based on the degree of suitability for each model determined in the degree of suitability determination step and the plurality of output data.

Images