Vehicle control system and vehicle control method

Abstract

A vehicle control system comprising: a data acquisition unit that acquires input data that is inputted to a model; an inference unit that inputs the input data to a plurality of models to perform inference, and outputs output data for each model; an input data feature vector extraction unit that extracts an input data feature vector representing the feature of the input data; a training data feature vector acquisition unit that acquires a training data feature vector representing the feature of the training data used for training the model; an expertise degree determination unit that, on the basis of the input data feature vector and the training data feature vector for each of the plurality of models, determines, for each of the models, an expertise degree indicating the possibility of deriving correct output data for the input data; and an output unit that outputs an inference result derived on the basis of the expertise degree for each model determined by the expertise degree determination unit and the plurality of output data.

Description

Vehicle control system and vehicle control method Incorporation by Reference

[0001] This application claims priority from Japanese Patent Application No. 2024-122636, filed on July 29, 2024, the contents of which are incorporated herein by reference.

[0002] The present disclosure relates to a vehicle control system and a vehicle control method.

[0003] In recent years, technology for recognizing objects around a vehicle using machine learning models implemented in vehicle control devices has become an important technology supporting autonomous driving systems and driver assistance systems. Machine learning models for object recognition have been shown to have powerful inference capabilities and fast response times in evaluations using computers with abundant computational resources. However, current vehicle control devices have smaller computational resources than the computers used in such evaluations. Therefore, the models must be made lighter for implementation in vehicle control devices, and the inference accuracy of the models in actual operation deviates from the conclusions of the experimental phase. Meanwhile, machine learning models designed for operation with limited computational resources often lack the robustness required to achieve stable accuracy for a variety of inputs compared to machine learning models designed for operation on computers.

[0004] The following prior art exists as background art in this technical field: Patent Document 1 (JP 2022-114607 A) describes a gradual reinforcement learning device that includes a storage device that stores learning data to be learned, a model learning unit that receives distribution weights corresponding to the learning data and multiple samples included in the learning data and performs learning by applying the corresponding distribution weights to the samples, learning a model and each of the multiple samples and identifying a degree of proficiency that is an index showing the accuracy of prediction for each of the multiple samples, an ensemble unit that receives the learning data and a model related to the learning, performs ensemble learning, and outputs an ensemble model and a sample proficiency set that is a set of proficiency degrees for each of the samples, a model management unit that creates model information by associating the model, the proficiency degrees, the ensemble model, and the sample proficiency set for each time step, and a distribution calculation unit that outputs the distribution weight for each sample based on the model information.

[0005] A single model can result in biased object recognition accuracy, and combining multiple models can lead to incorrect inference results due to the influence of models with mismatched characteristics. Furthermore, for each model, environmental changes or incomplete training data can cause recognition accuracy to fall below expected accuracy, potentially leading to incorrect inference results.

[0006] Although the technology described in Patent Document 1 can construct a high-performance ensemble model in a short learning time during the learning stage, it does not take into consideration how to respond if an erroneous result occurs during the inference stage of the ensemble model.

[0007] The purpose of the present disclosure is to achieve robust inference using machine learning / AI models.

[0008] A representative example of the invention disclosed in the present application is as follows: That is, a vehicle control system includes a data acquisition unit that acquires input data to be input to a model, an inference unit that inputs the input data to a plurality of the models to perform inference and output output data for each of the models, an input data feature vector extraction unit that extracts an input data feature vector that represents features of the input data, a learning data feature vector acquisition unit that acquires a learning data feature vector that represents features of learning data used to train the models, an ability level determination unit that determines, for each of the models, an ability level that indicates the possibility of deriving correct output data for the input data, based on the input data feature vector and the learning data feature vectors for each of the plurality of models, and an output unit that outputs an inference result derived based on the ability level for each of the models determined by the ability level determination unit and the plurality of output data.

[0009] According to one aspect of the present invention, an object can be recognized with high accuracy by combining a plurality of trained models with different characteristics. Problems, configurations, and effects other than those described above will become clear from the following description of the embodiment.

[0010] 1 is a block diagram showing a functional configuration of a vehicle control system according to a first embodiment; FIG. 2 is a flowchart of input data output processing according to the first embodiment; FIG. 3 is a diagram illustrating an example of input data output processing according to the first embodiment; FIG. 4 is a flowchart of input data feature vector extraction processing according to the first embodiment; FIG. 5 is a diagram illustrating an example of input data feature vector extraction processing according to the first embodiment; FIG. 6 is a flowchart of training data feature vector extraction processing according to the first embodiment; FIG. 7 is a diagram illustrating an example of training data feature vector extraction processing according to the first embodiment; FIG. 8 is a diagram illustrating an example of model feature vector extraction processing according to the first embodiment; FIG. 9 is a diagram illustrating an example of model feature vector extraction processing according to the first embodiment; FIG. 10 is a diagram illustrating an example of model feature vector extraction processing according to the first embodiment; FIG. 11 is a diagram illustrating an example of model proficiency level calculation processing according to the first embodiment; FIG. 12 is a diagram illustrating an example of model proficiency level calculation processing according to the first embodiment; FIG. 13 is a diagram illustrating an example of model proficiency level calculation processing according to the first embodiment; FIG. 14 is a flowchart of inference processing according to the first embodiment; FIG. 15 is a diagram illustrating an example of inference processing according to the first embodiment; FIG. 16 is a flowchart showing a final inference result B output processing according to the first embodiment; FIG. 17 is a diagram illustrating an example of weighting processing according to the first embodiment; FIG. 18 is a diagram illustrating an example of weighting processing according to the first embodiment; FIG. 19 is a diagram illustrating derivation of an integrated inference result according to the first embodiment; FIG. 1 is a diagram illustrating derivation of an integrated inference result in the first embodiment. FIG. 2 is a diagram illustrating derivation of a final inference result A in the first embodiment. FIG. 3 is a diagram illustrating derivation of a final inference result A in the first embodiment. FIG. 4 is a diagram illustrating derivation of a final inference result B in the first embodiment. FIG. 5 is a flowchart of a validity determination process in the first embodiment. FIG. 6 is a diagram illustrating an example of a model classification process in the first embodiment. FIG. 7 is a flowchart of a final inference result A validity determination process in the first embodiment. FIG. 8 is a diagram illustrating an example of a final inference result A validity determination process in the first embodiment. FIG. 9 is a diagram illustrating an example of a final inference result A validity determination process in the first embodiment. FIG. 10 is a diagram illustrating an example of a final inference result A validity determination process in the first embodiment. FIG. 11 is a diagram illustrating an example of a final inference result A validity determination process in the first embodiment.FIG. 1 is a diagram illustrating an example of an abnormality cause model determination process according to a first embodiment. FIG. 2 is a diagram illustrating an example of an abnormality cause model determination process according to a first embodiment. FIG. 3 is a flowchart of an inference result update process according to a first embodiment. FIG. 4 is a diagram illustrating an example of an integrated inference result according to a first embodiment. FIG. 5 is a flowchart of a final inference result B update process according to a first embodiment. FIG. 6 is a diagram illustrating an example of data for re-consultation according to a first embodiment. FIG. 7 is a flowchart of a re-learning process according to a first embodiment. FIG. 8 is a diagram illustrating a hardware configuration of a vehicle control device according to a first embodiment. FIG. 9 is a block diagram illustrating a functional configuration of a vehicle control device according to a second embodiment. FIG. 10 is a diagram illustrating a hardware configuration of a server according to a second embodiment.

[0011] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings.

[0012] First Embodiment FIG. 1 is a block diagram showing the functional configuration of a vehicle control device according to a first embodiment.

[0013] The vehicle control system 100 shown in this figure is an example of a vehicle control device of the present invention, and has a data acquisition unit 101, a first input data feature vector extraction unit 102, a second input data feature vector extraction unit 103, a learning data feature vector acquisition unit 104, an inference unit 105 including multiple models, a specialty determination unit 109, an output unit 110, an output validity judgment unit 111, and a model reconstruction unit 112.

[0014] The data acquisition unit 101 acquires input data 5 to be inferred from an in-vehicle camera image 2 .

[0015] The first input data feature vector extraction unit 102 extracts vectors including image feature amounts and image complexity from the input data 5 to be inferred, and generates an input data feature vector 1001 including the extracted vectors.

[0016] The inference unit 105 includes a plurality of models 106 to 108, receives input data 5 to be inferred, and outputs a plurality of inference results. For example, when input data 5 to be inferred is input, model 1 (106), model 2 (107), and model 3 (108) included in the inference unit 105 output a model 1 inference result 1006, a model 2 inference result 1007, and a model 3 inference result 1008, respectively.

[0017] The second input data feature vector extraction unit 103 extracts image features and image complexity from integrated training data for training multiple models included in the inference unit 105, and generates a training data feature vector 1002 including the extracted image features and image complexity. For example, the second input data feature vector extraction unit 103 extracts feature vectors from model 1 training data 2, model 2 training data 3, and model 3 training data 4 to generate the training data feature vector 1002.

[0018] The training data feature vector acquisition unit 104 generates feature vectors 1003, 1004, and 1005 for each model from the training data feature vector 1002 compiled to construct a plurality of models.

[0019] The skill determination unit 109 outputs the maximum similarity calculated between the input data feature vector 1001 and each model feature vector as the model skill degree. For example, the skill determination unit 109 uses the cosine similarity calculation method to calculate the maximum similarity between the input data feature vector 1001 and the model 1 feature vector 1003 and outputs it as the model 1 skill degree 1012. The skill determination unit 109 also calculates the maximum similarity between the input data feature vector 1001 and the model 2 feature vector 1004 and outputs it as the model 2 skill degree 1013. The skill determination unit 109 also calculates the maximum similarity between the input data feature vector 1001 and the model 3 feature vector 1005 and outputs it as the model 3 skill degree 1014.

[0020] The output unit 110 multiplies the model 1 inference result 1006, the model 2 inference result 1007, and the model 3 inference result 1008 by the model 1 proficiency level 1012, the model 2 proficiency level 1013, and the model 3 proficiency level 1014 input from the proficiency determination unit 109, respectively, and outputs the final inference result B1025 determined by majority voting on the multiplication results.

[0021] The output validity determination unit 111 determines the validity of the final inference result B1025 based on the model 1 proficiency level 1012, the model 2 proficiency level 1013, the model 3 proficiency level 1014, the model 1 inference result 1006, the model 2 inference result 1007, and the model 3 inference result 1008, and outputs a validity determination result 1032, model information 1039 that is the cause of the abnormality, and model information requiring relearning 1040. An abnormality in the output validity determination unit 111 occurs when the final inference result B1025 differs from the inference results of each model.

[0022] The model reconfiguration unit 112 identifies a model to be reconfigured based on the re-learning required model information 1040, and re-learns the identified target model.

[0023] The output unit 110 updates the final inference result A1024 by re-consultation, excluding abnormal inference results of the model to be reconstructed, based on information about the model to be reconstructed input from the model reconstruction unit 112, and outputs the updated final inference result B1025.

[0024] 3, the vehicle-mounted camera images 1 are a plurality of still images containing information about the surroundings of the vehicle captured by the vehicle-mounted camera, or a plurality of screenshots of captured moving images taken at a predetermined time interval. For example, 1A, 1B, and 1C are a portion of a plurality of images continuously captured by the vehicle-mounted camera while the vehicle is traveling.

[0025] Fig. 2 is a flowchart of the input data output process for outputting the vehicle-mounted camera image 1 acquired by the data acquisition unit 101 as input data 5. Fig. 3 is a diagram for explaining an example of the input data output process.

[0026] In the input data output process shown in Figure 2, the data acquisition unit 101 acquires multiple vehicle-mounted camera images 1 (S101), adds processing time information to each vehicle-mounted camera image, and starts a loop process A1 (S102) that repeats the process of outputting the images as input data 5 (S103).

[0027] 3, the input data 5 includes an "input data number" 5A that records the order in which the vehicle-mounted camera images 1 were acquired, and an "input data image" 5B that stores the input data image that was changed from the vehicle-mounted camera image 1. For example, input data 5C, 5D, and 5E are the results of the acquisition process of the vehicle-mounted camera images 1A, 1B, and 1C, respectively.

[0028] Then, when the data acquisition unit 101 has output all of the vehicle-mounted camera images 1 as input data 5, it ends the loop process A1 (S104).

[0029] Fig. 4 is a flowchart of an input data feature vector extraction process for extracting feature vectors from input data 5. Fig. 5 is a diagram for explaining an example of the input data feature vector extraction process.

[0030] In the input data feature vector extraction process, a loop process B1 is started in which the processes of steps S202 to S203 shown in FIG. 4 are repeated for each piece of input data 5 acquired by the data acquisition unit 101 (S201).

[0031] As shown in FIG. 5, the first input data feature vector extraction unit 102 includes a feature vector extractor 102A.

[0032] In loop process B1, first, feature vector extractor 102A extracts feature vector 1001B from input data 5 input from data acquisition unit 101 (S202). "Feature vector" 1001B is a vector expressed as an array of numerical values ​​by extracting image information from input data 5. Image information of input data 5 includes, for example, color histogram, texture feature amount, complexity, edge information, etc.

[0033] Then, the first input data feature vector extraction unit 102 adds an "input data number" 1001A to the extracted feature vector 1001B to generate an input data feature vector 1001, and outputs the generated input data feature vector 1001 (S203). The extracted feature vector may be a one-dimensional vector or a multidimensional vector, but the number of elements of the vector should be equal.

[0034] Then, when the first input data characteristic vector extraction unit 102 has output the input data characteristic vector 1001 for all the input data, it ends the loop process B1 (S204).

[0035] The feature vector extractor 102A may use, for example, a known algorithm such as a convolutional neural network (CNN) or a visual geometry group (VGG), but may also use other feature extraction techniques that target image data.

[0036] 6 is a flowchart of a training data feature vector extraction process in which the second input data feature vector extraction unit 103 is used to extract a feature vector 1002 from training data provided for training of a model arranged in the inference unit 105. Figures 7A and 7B are diagrams illustrating an example of the training data feature vector extraction process.

[0037] In the training data feature vector extraction process, first, the second input data feature vector extraction unit 103 starts a loop process C1 in which steps S302 to S307 are repeated for each of the inference models 106 to 108 placed in the inference unit 105 (S301).

[0038] In loop processing C1, the second input data feature vector extraction unit 103 acquires training data to be used for training each of the inference models 106-108 arranged in the inference unit 105 (S302). For example, as shown in FIG. 7A , the training data used for training model 1 (106) is training data 2, the training data used for training model 2 (107) is training data 3, and the training data used for training model 3 (108) is training data 4. Model 1 training data 2 has a "data number" 2A and a "training data image" 2B. The structures of model 2 training data 3 and model 3 training data 4 are the same as those of model 1 training data 2. Since the training policies differ for each model, the training data differs. For example, model 1 (106) is trained with an emphasis on image data of urban areas, model 2 (107) is trained with an emphasis on image data of sunny days, and model 3 (108) is trained with an emphasis on image data of nighttime scenes. Note that multiple models may be trained using the same training data.

[0039] Then, the second input data feature vector extraction unit 103 starts a loop process C2 in which the processes of steps S304 to S305 are repeated for each of the acquired learning data of each model (S303).

[0040] In loop process C2, feature vector extractor 103A of second input data feature vector extraction unit 103 extracts a feature vector for each acquired training data of each model (S304). The extracted feature vector has the same format as feature vector 1001B of input data 5 extracted by first input data feature vector extraction unit 102 in terms of the number of vector dimensions, the number of elements, etc.

[0041] Then, the feature vector extractor 103A extracts a feature vector for each training data image in the training data for each model, and adds a "model x data number" column to each training data. Here, the feature vector extractor 103A can be the same as the feature vector extractor 102A of the first input data feature vector extraction unit 102. For example, the second input data feature vector extraction unit 103 adds a "model x data number" column to the feature vector extracted for each model. The "model x data number" column indicates from which training data set the feature vector was extracted. Specifically, the feature vector extractor 103A extracts a feature vector 2D for each training data image in the "training data image" column 2B, which stores the data image of model 1 training data 2, and adds a "model 1 data number" column 2C. The same processing is performed for model 2 training data 3 and model 3 training data 4.

[0042] Then, the second input data feature vector extraction unit 103 executes the process of step S305 for all feature vectors extracted for each model, and then ends the loop process C2 (S306). Furthermore, when the processes of steps S302 to S306 have been executed for all models, the second input data feature vector extraction unit 103 ends the loop process C1 (S307).

[0043] Then, the second input data feature vector extraction unit 103 combines the training data feature vectors 1002 used in training all of the models to generate a training data feature vector 1002 (S308). For example, the "feature vector" columns 2D, 3D, and 4D are set as key bindings, and the training data feature vector 1002 of model 1, the training data feature vector 1002 of model 2, and the training data feature vector 1002 of model 3, which have the same feature vector, are combined to generate a training data feature vector 1002. The training data feature vector 1002 includes a "feature vector" 1002A, a "model 1 data number" 1002B, a "model 2 data number" 1002C, and a "model 3 data number" 1002D.

[0044] Then, the second input data characteristic vector extraction unit 103 outputs the generated training data characteristic vector 1002 (S309). After the training data characteristic vector 1002 is output, the training data characteristic vector extraction process ends.

[0045] 8 is a flowchart of a model feature vector extraction process in which the training data feature vector acquisition unit 104 extracts a feature vector for each model from the training data feature vector 1002 based on the training-time verification result for each model. Figures 9A and 9B are diagrams illustrating an example of the model feature vector extraction process.

[0046] In the model feature vector extraction process, first, the training data feature vector acquisition unit 104 acquires the training data feature vector 1002 extracted by the second input data feature vector extraction unit 103 (S401), and then starts a loop process D1 in which steps S403 to S407 are repeated for each of the inference models 106 to 108 in the inference unit 105 (S402).

[0047] In loop processing D1, the training data feature vector acquisition unit 104 acquires the verification results of the training for each of the inference models 106-108 of the inference unit 105 (S403). For example, as shown in FIG. 9A , the verification results of the training for each model include data number 2A for model 1 training data 2, training data 7A with correct answers, verification data 7B with inference results, a column 7C storing IoU (Intersection over Union) as an evaluation index, and a column 7D storing F1 score as an evaluation index. The training data 7A with correct answers is training data that has already been annotated. The annotation method is not particularly limited, but includes, for example, a method of generating a bounding box (BBox) and a label. The IoU stored in column 7C and the F1 score stored in column 7D are used as evaluation indices, but the evaluation indices may be evaluation indices used in object detection such as true positive (TP), or multiple evaluation indices may be used in combination.

[0048] Then, the training data feature vector acquisition unit 104 extracts training data numbers that have achieved high accuracy in the verification results for each model using the evaluation index threshold 10, and generates training data information 11 that has achieved high accuracy for each model (S404). For example, as shown in FIG. 9B , training data information 11 that has achieved high accuracy is generated from the model 1 verification results 7 using the evaluation index threshold 10. As shown in FIG. 9B , the evaluation index threshold 10 may be set to 0.50 for IoU and 1.00 for F1, or may be adjusted to be wider or narrower, and may be adjusted up or down depending on the evaluation criteria for the training data that have achieved high accuracy.

[0049] Then, the training data feature vector acquisition unit 104 uses the training data information 11 that has achieved high accuracy for each model to extract a feature vector for each model from the training data feature vector 1002 extracted by the second input data feature vector extraction unit 103 (S405), and outputs the extracted feature vector (S406). Specifically, using the data number that has achieved high accuracy for each model, the unit 104 extracts, for each model, a feature vector 1002A corresponding to the same data number as the data number that achieved high accuracy for each model from the training data feature vector 1002. The unit 104 then adds the data number that achieved high accuracy for each model to the feature vector 1002A corresponding to the extracted data number that achieved high accuracy, and outputs the feature vector for each model. For example, as shown in FIG. 9B , the model 1 feature vector 1003 includes a column 1003A that stores the data number that achieved high accuracy for model 1, and a "feature vector" column 1003B that stores the feature vector 1002A corresponding to the data number that achieved high accuracy.

[0050] Then, when the training data feature vector acquisition unit 104 has executed the processes of steps S403 to S406 for all models, it ends the loop process D1 (S407) and ends the model feature vector extraction process.

[0051] 10 is a flowchart of a model proficiency calculation process in which the proficiency determination unit 109 calculates the "proficiency level" of the models 106 to 108 arranged in the inference unit 105 for each inference task based on the input data feature vector 1001 and the data feature vector for each model. FIG. 11 is a diagram illustrating an example of the model proficiency calculation process.

[0052] In this embodiment, the "proficiency level" is a numerical value that indicates the likelihood that the model will derive a correct inference result, and is different from the "confidence level" that is commonly used in the field of machine learning and indicates the probability that the inference result is correct.

[0053] In the model strength calculation process, the strength determination unit 109 first starts a loop process E1 in which the processes of steps S502 to S510 are repeated for each input data 5 (S501).

[0054] In loop processing E1, the specialty determination unit 109 acquires, for each piece of input data 5, an input data feature vector 1001 extracted by the first input data feature vector extraction unit 102. As shown in FIG. 11A , the acquired input data feature vector 1001 has an input data number 1001A and a column 1001B that stores the corresponding feature vector.

[0055] Then, the strength determination unit 109 starts a loop process E2 in which the processes of steps S504 to S509 are repeated for each of the models 106 to 108 arranged in the inference unit 105 (S503).

[0056] In loop process E2, the specialty determination unit 109 acquires a model feature vector for each of models 106 to 108 arranged in the inference unit 105 (S504). For example, it acquires a model 1 feature vector 1003 for model 1 (106), a model 2 feature vector 1004 for model 2 (107), and a model 3 feature vector 1005 for model 3 (108). When the process of step S504 ends, the specialty determination unit 109 starts loop process E3 in which it repeats the process of step S506 for each feature vector included in the model feature vector (S505).

[0057] In loop process E3, the expertise determination unit 109 calculates the similarity between each feature vector in the acquired model feature vector and the acquired input data feature vector 1001 (S506). The similarity between feature vectors can be calculated using, for example, cosine similarity, which is expressed by equation (1). Here, the calculated similarity is a real number ranging from -1 to 1. A similarity smaller than 0 indicates that the two feature vectors are dissimilar, and a similarity greater than 0 indicates that the two feature vectors are similar.

[0058]

[0059] In equation (1), vector x is the input data feature vector 1001, and vector y is the model feature vector, and their structure is expressed by equation (2).

[0060]

[0061] When the specialty determination unit 109 has executed the process of step S506 for all feature vectors for the input data feature vector 1001 and the model feature vector, it ends the loop process E3 (S507).

[0062] Then, the expertise determination unit 109 determines the magnitude relationship of the multiple similarities calculated from the model feature vector for the input data feature vector 1001 (S508), and outputs the maximum similarity as the "expertise" of the corresponding model in the input data 5 (S509).

[0063] Then, when the specialty determination unit 109 has executed the processes of steps S504 to S509 for all models, it ends the loop process E2 (S510).

[0064] Then, when the strength determination unit 109 has executed the processes of steps S502 to S510 for all of the input data 5, it ends the loop process E1 (S511) and ends the model strength calculation process.

[0065] The model proficiency calculation process described above is merely an example and is not limited to this. For example, as shown in FIGS. 11A and 11B , the proficiency determination unit 109 first obtains an input data feature vector 1001. The input data feature vector 1001 includes one input data feature vector 1001C. Next, model 1 obtains a model 1 feature vector 1003 including a data number sequence 1003A and a feature vector sequence 1003B for which model 1 achieved high accuracy, model 2 obtains a model 2 feature vector 1004 including a data number sequence 1004A and a feature vector sequence 1004B for which model 2 achieved high accuracy, and model 3 obtains a model 3 feature vector 1005 including a data number sequence 1005A and a feature vector sequence 1005B for which model 3 achieved high accuracy. The specialty determination unit 109 then calculates the similarity between the feature vector 1001B of the input data feature vector 1001C and all of the feature vectors of the model 1 feature vector 1003 (column 1003B), all of the feature vectors of the model 2 feature vector 1004 (column 1004B), and all of the feature vectors of the model 3 feature vector 1005 (column 1005B), and outputs the model 1 similarity 1009, the model 2 similarity 1010, and the model 3 similarity 1011, respectively. For example, the model 1 similarity 1009 has a "similarity number" column 1009A and a "vector similarity" column 1009B. Then, the magnitude relationship may be determined for all vector similarities 1009B of model 1 similarity 1009, all vector similarities 1010B of model 2 similarity 1010, and all vector similarities 1011B of model 3 similarity 1011, and the maximum similarity may be output as the "proficiency" for each model with respect to input data feature vector 1001C, thereby generating model 1 expertise 1012, model 2 expertise 1013, and model 3 expertise 1014.

[0066] 12 is a flowchart of an inference process in which the inference unit 105 uses a plurality of models 106 to 108 arranged in the inference unit 105 to output an inference result for the input data 5. FIG. 13 is a diagram illustrating an example of the inference process.

[0067] In the inference process, the inference unit 105 first acquires (S601) input data 5. As shown in Fig. 3, the input data 5 includes a column 5A that stores input data numbers and a column 5B that stores input data images.

[0068] Then, the inference unit 105 starts a loop process F1 in which the processes of steps S603 to S604 are repeated for each of the models 106 to 108 arranged in the inference unit 105 based on the acquired input data 5 (S602).

[0069] In loop processing F1, the inference unit 105 inputs the acquired input data 5 as inference input data to models 106 to 108 arranged in the inference unit 105, and executes inference (S603). For example, as shown in FIG. 13 , input data 5C is input to model 1 (106), model 2 (107), and model 3 (108) arranged in the inference unit 105, and inference is executed using each model. Here, the models can use various object detection technologies that can output a bounding box (BBox), a label representing the recognized object class, and a confidence score for the recognized object, such as R-CNN (Region-based Convolutional Neural Network), YOLO (You Only Look Once), or SSD (Single Shot MultiBox Detector).

[0070] Inference involves recognizing objects in the input data image 5B. Then, an inference result for each model is output (S604). The model inference results are in the form of image data, such as a model 1 inference result 1006, a model 2 inference result 1007, and a model 3 inference result 1008. The model 1 inference result 1006 includes, for each recognized object, a bounding box (BBox) 1006A that displays the area of ​​the recognized object, a label 1006B that describes the recognized object class, and a confidence score 1006C for the recognized object.

[0071] When the processing of steps S603 to S604 is completed for all models, the inference unit 105 ends the loop processing F1 (S605), and then ends the inference processing for outputting the inference result.

[0072] 14 is a flowchart showing a final inference result B1025 output process in which the output unit 110 outputs a final inference result B1025 based on multiple model inference results output by the inference unit 105 and the degree of suitability for input data 5 for each model output by the suitability determination unit 109. Figures 15A and 15B are diagrams illustrating an example of weighting process performed by the output unit 110 on the model 1 inference result 1006 output from the inference unit 105 based on the model 1 suitability degree 1012 output by the suitability determination unit 109.

[0073] In the output process for outputting the final inference result B1025, the output unit 110 first acquires the inference result for each model from the inference unit 105 (S701).Then, based on the acquired inference result, the output unit 110 starts a loop process G1 in which the processes of steps S703 to S706 are repeated for each inference result of the model (S702).

[0074] In loop processing G1, the output unit 110 acquires a model suitability score from the suitability determination unit 109 for each inference result of a model corresponding to an inference result acquired from the inference unit 105 (S703). For example, as shown in FIG. 15A , the output unit 110 acquires a model 1 suitability score 1012 for model 1. Then, for the inference result of each model, the output unit 110 generates an inference result including an image of the corresponding bounding box, a label, and a confidence score for each recognized object in the image of the inference result, in other words, for each bounding box (S704). The inference result 1015 for each recognized object of Model 1 shown in Figure 15A includes a "BBox Number" column 1015A that stores the BBox number for identifying the recognized object of Model 1 inference result 1006, a "BBox Image" column 1015B that stores the image inside the BBox of the recognized object, a "Label" column 1015C that stores a label representing the class of the recognized object, and a "Confidence Score" column 1015D that stores the confidence score of the recognized object.

[0075] Then, the output unit 110 performs a weighting process on the inference result for each model-recognized object using the model's suitability (S705). The weighting process calculates a weighted confidence score by multiplying the confidence score of the inference result for each model-recognized object by a weighting coefficient. For example, the weighting process can be performed using equation (3). In equation (3), z is the confidence score of the inference result for each model-recognized object, s indicates the model's suitability, and Z is the weighted confidence score calculated during the weighting process.

[0076]

[0077] Then, the output unit 110 replaces the confidence score of the inference result for each model-recognized object with a weighted confidence score and outputs the weighted inference result 1018 for each model-recognized object (S706). For example, the weighted inference result 1018 for each model-recognized object shown in FIG. 15B includes a "B Box Number" column 1018A that stores a B Box number for identifying the recognized object in the model-1 inference result 1006, a "B Box Image" column 1018B that stores a B Box image of the recognized object, a "Label" column 1018C that stores a label representing the class of the recognized object, and a "Weighted Confidence Score" column 1018D that stores the weighted confidence score of the recognized object. The "Weighted Confidence Score" column 1018D is calculated by weighting the confidence score of the inference result for each model-recognized object by multiplying it by a weighting coefficient.

[0078] When the processing of steps S703 to S706 is completed for all model inference results, the output unit 110 ends the loop processing G1 (S707).

[0079] Thereafter, the weighted inference results 1018 for each model recognition object, including the output weighted confidence scores, are integrated (S708). For example, the weighted inference results 1018 for each model recognition object are integrated as an integrated inference result for each recognition object.

[0080] 16A and 16B are diagrams for explaining the derivation of an integrated inference result 1021 for each recognized object and an integrated inference result 1022 for each fused recognized object.

[0081] As shown in FIG. 16A, the integrated inference result 1021 for each recognized object includes, for example, a “Model 1 BBox Number” column 1021A storing the BBox number for identifying the recognized object of the model 1 inference result 1006, a “Model 1 BBox Image” column 1021B storing the image in the BBox of the model 1 recognized object, a “Model 1 Label” column 1021C storing the label representing the class of the model 1 recognized object, a “Model 1 Confidence Score” column 1021D storing the weighted confidence score of the model 1 recognized object, a “Model 2 BBox Number” column 1021E storing the BBox number for identifying the recognized object of the model 2 inference result 1007, and a “Model 2 BBox Image” column 1021F storing the image in the BBox of the model 2 recognized object. the "Model 2 BBox Image" column 1021F storing the image in the BBox of the Model 3 recognized object; the "Model 2 Label" column 1021G storing the label representing the class of the Model 2 recognized object; the "Model 2 Confidence Score" column 1021H storing the weighted confidence score of the Model 2 recognized object; the "Model 3 BBox Number" column 1021I storing the BBox number for identifying the recognized object of the Model 3 inference result 1008; the "Model 3 BBox Image" column 1021J storing the image in the BBox of the Model 3 recognized object; the "Model 3 Label" column 1021K storing the label representing the class of the Model 3 recognized object; and the "Model 3 Confidence Score" column 1021L storing the weighted confidence score of the Model 3 recognized object.

[0082] Then, a fusion process is performed on the "B box number" and "B box image" columns of the integrated inference results for each recognized object and each model, and a final inference result A1024 is output (S709). The fusion process may use B box optimization and fusion techniques such as Non-Maximum Suppression (NMS), Soft Voting, and Weighted Boxes Fusion (WBF). For example, as shown in FIG. 16A , for each row of the integrated inference result 1021 for each recognized object, the reliability scores 1021D, 1021H, and 1021L for each model are compared to determine the B box number and B box image with the highest reliability score. The fused B box number 1022A and fused B box image 1022B are output, and an integrated inference result 1022 for each fused recognized object is created as shown in FIG. 16B.

[0083] 17A, 17B, and 17C are diagrams illustrating how the output unit 110 derives a final inference result A 1024 based on the integrated inference result 1022 for each fused recognized object.

[0084] The integrated inference result 1022 for each fused recognized object includes, for example, as shown in FIG. 17A, a "Fused BBox Number" column 1022A that stores a BBox number for identifying the recognized object of the inference result, a "Fused BBox Image" column 1022B that stores an image within the BBox of the recognized object, a "Model 1 Label" column 1022C that stores a label representing the class of the model 1 recognized object, a "Model 1 Confidence Score" column 1022D that stores the weighted confidence score of the model 1 recognized object, a "Model 2 Label" column 1022E that stores a label representing the class of the model 2 recognized object, a "Model 2 Confidence Score" column 1022F that stores the weighted confidence score of the model 2 recognized object, a "Model 3 Label" column 1022G that stores a label representing the class of the model 3 recognized object, and a "Model 3 Confidence Score" column 1022H that stores the weighted confidence score of the model 3 recognized object.

[0085] As shown in FIG. 17A, the data for discussion 1023 includes, for example, a "Fusion BBox Number" column 1023A that stores the BBox number for identifying the recognized object of the inference result, a "Fusion BBox Image" column 1023B that stores the image inside the BBox of the recognized object, a "Label 1" column 1023C that stores the label representing the class of the recognized object of model 1, a "Weight 1" column 1023D that stores the total weight of model 1, a "Label 2" column 1023E that stores the label representing the class of the recognized object of model 2, a "Weight 2" column 1023F that stores the total weight of model 2, a "Label 3" column 1023G that stores the label representing the class of the recognized object of model 3, and a "Weight 3" column 1023H that stores the total weight of model 3.

[0086] The final inference result A1024 is determined by deliberation based on the reliability scores for each model in the integrated inference result 1022 for each fused recognized object. Specifically, as shown in FIGS. 17A and 17B , the reliability scores of models that recognized the same object (same B box image) for each row of the integrated inference result 1022 for each fused recognized object are summed to calculate a weight to be used in deliberation for the target. Then, deliberation data 1023 is output, which combines a "fused B box number" 1023A, a "fused B box image" 1023B, and a weight of 1 (1023D) obtained by summing the corresponding label 1023C. Then, as shown in FIG. 17C , the output unit 110 selects the label 1024C with the highest weight for the fused B box number and fused B box image based on the deliberation data 1023, and combines the "fused B box number" 1024A and the "fused B box image" 1024B to create the final inference result A1024. In the consensus process, for each fusion Bbox number and fusion Bbox image, a label with the maximum weight among the weights of labels representing multiple recognized object classes is selected. For example, a majority vote, soft voting, or other methods may be used.

[0087] The output unit 110 then reflects the final inference result A1024 in the input data 5 and outputs the final inference result B1025 in the form of an image (S709). As shown in Figure 18, the image of the output final inference result B1025 is assigned the fused B box and the label decided by consensus. Thereafter, the output process of the final inference result B1025 ends.

[0088] FIG. 19 is a flowchart of a validity determination process in which the output validity determination unit 111 determines the validity of the final inference result A1024 based on the final inference result A1024 and the inference results for each model.

[0089] In the validity determination process, the output validity determination unit 111 starts a loop process H1 in which the processes of steps S802 to S813 are repeated for the final inference result A1024 for each inference task (S801).

[0090] In loop processing H1, the output validity determination unit 111 acquires the final inference result A1024 output by the output unit 110 and the inference results for each model output by the inference unit 105 (S802), and then executes model classification processing to classify multiple operating models allocated to the inference unit 105 that correspond to the acquired inference results for the multiple models.

[0091] In the model classification process, the output validity determination unit 111 starts a loop process H2 in which the processes of steps S804 to S805 are repeated for the multiple models 106 to 108 currently in operation arranged in the inference unit 105 (S803).

[0092] FIG. 20 is a diagram illustrating an example of a model classification process in which the output validity determination unit 111 classifies models based on the degree of expertise of each of the models 106 to 108 currently in operation and placed in the inference unit 105 in steps S804 to S805.

[0093] In loop process H2, the output validity determination unit 111 first obtains the model 1 similarity 1009, model 2 similarity 1010, and model 3 similarity 1011 from the strength determination unit 109 for the models 106-108 currently in operation and allocated to the inference unit 105 (S804). Then, based on the obtained strength level for each model, the output validity determination unit 111 classifies the models 106-108 into two types, "strong models" and "poor models," and generates model classification information 1026 (S805). As shown in FIG. 20 , the model classification information 1026 includes a "strong model name" column 1026A that stores strong model information and a "poor model name" column 1026B that stores poor model information. A "strong model" is a model whose strength level ranges from 0 to 1 and has high recognition accuracy for the object. A "poor model" is a model whose strength level ranges from -1 to 0 and has low recognition accuracy for the object. 20 , since the values ​​of model 1 similarity 1009, model 2 similarity 1010, and model 3 similarity 1011 are between 0 and 1, they are classified as "good models" in the classification process, and the model names are recorded in "good model name" column 1026A of the model classification information 1026. Since model 3 similarity 1014 is between -1 and 0, they are classified as "bad models" in the classification process, and the model names are recorded in "bad model name" column 1026B of the model classification information 1026.

[0094] When the processing of steps S804 to S805 is completed for all models in operation, the output validity determination unit 111 ends the loop processing H2 (S806).

[0095] Then, the output validity determination unit 111 executes a final inference result A1024 validity determination process to determine the validity of the final inference result A1024 (S807).

[0096] Fig. 21 is a flowchart of the process (S807) for determining the validity of the final inference result A1024. Fig. 22A and Fig. 22B are diagrams for explaining an example of the process (S807) for determining the validity of the final inference result A1024.

[0097] In the process of determining the validity of the final inference result A1024 (S807), the output validity determination unit 111 starts a loop process I1 that repeats the processes of steps S902 to S906 for each determination process of the validity of the final inference result A1024 (S901).

[0098] In loop processing I1, the output validity determination unit 111 obtains the good model name 1026A from the model classification information 1026 (S902). The output validity determination unit 111 then uses the good model name 1026A to extract labels and reliability scores for all good models from the integrated inference result 1022 for each fused recognition object. The output validity determination unit 111 then combines the extracted labels and reliability scores for each good model, and adds the fusion B box number 1022A and fusion B box image 1022B of the integrated inference result 1022 for each fused recognition object to create the good model integrated inference result 1027 (S903).

[0099] Then, for the created integrated inference result 1027 of the good model and the acquired final inference result A1024, coverage 1030 between the labels indicating the class of the recognized object is calculated for the fusion Bbox number and the fusion Bbox image (S904).

[0100] Figures 23A and 23B are figures illustrating an example of a process in which the output validity judgment unit 111 calculates the coverage rate between labels indicating the class of the recognized object for the integrated inference result 1027 of the created favorite model and the acquired final inference result A1024, and judges the validity of the final inference result A1024 using coverage 1030.

[0101] Specifically, the output validity determination unit 111 extracts a column storing the fusion B box number and a column storing the label from the integrated inference result 1027 of the good model and the final inference result A1024, and combines the fusion B box number and label as set elements to create a final inference result A1024 set 1028 and a good model inference result set 1029. For example, as shown in FIG. 23A , for the final inference result A1024, the fusion B box number 1024A and label 1024C are combined to create the final inference result A1024 set 1028. Then, as shown in FIG. 23B , the fusion B box number 1027A of the good model integrated inference result 1027 is combined with the good model 1 label 1027C (see FIG. 22B ), and the fusion B box number 1027A and the good model 2 label 1027E are combined to create the good model inference result set 1029.

[0102] Then, the output validity determining unit 111 calculates the coverage 1030 of the set 1028 of final inference results A 1024 in the inference result set 1029 of the good model using equation (4).

[0103]

[0104] In formula (4), k AB is the coverage 1030, A is the final inference result A 1024 set 1028, and B is the inference result set 1029 of the good model.

[0105] Then, the output validity determination unit 111 compares the calculated coverage 1030 with a validity determination threshold 1031 (S905). In Fig. 23C, the validity determination threshold 1031 for the coverage 1030 is set to 1. However, the validity determination threshold 1031 may be reduced to widen the allowable range for determining the validity of the final inference result A1024.

[0106] In step S904, a method for expressing the inclusion relationship or similarity between sets, such as a subset check, a Jaccard similarity coefficient, or a Sorensen-Dice coefficient, may be used instead of the coverage 1030. The validity determination threshold 1031 that is set is also changed depending on the method used.

[0107] The output validity determination unit 111 then outputs validity determination information (S906). If the coverage 1030 is less than the validity determination threshold 1031, a validity determination result 1032 of "abnormal" is output. If the coverage 1030 is equal to or greater than the validity determination threshold 1031, a validity determination result 1032 of "no abnormality" is output. Here, "abnormal" means that the final inference result A1024 is not entirely provided by the inference result of the good model, and that part of the final inference result A1024 may be provided by the inference result of the non-good model. If the good model has a high degree of goodness but the confidence score recognized by the good model in the bounding box is lower than that of the non-good model, the output validity determination unit 111 determines an "abnormal" and outputs a determination result of "abnormal." In other words, "no abnormality" means that the final inference result A1024 is entirely provided by the inference result of the good model.

[0108] 23C, the validity determination result 1032 includes a column for storing validity information and a column for storing information on multiple problem B box numbers. The "problem B box number" is the fusion B box number corresponding to all elements of the final inference result A1024 set 1028 that are missing from the inference result set 1029 of the favored model when the coverage 1030 is equal to or less than the validity determination threshold 1031.

[0109] Then, when the processing of steps S902 to S906 is completed for all the determination processes, the output validity determination unit 111 ends the loop process I1 (S907).

[0110] Returning to the validity determination process (FIG. 19), the description will be continued. The output validity determination unit 111 then ends the validity determination process (S807) of the final inference result A1024.

[0111] Then, the output validity determination unit 111 determines whether the validity determination result 1032 of the final inference result A1024 includes "abnormality present" (S808).

[0112] If the validity determination result 1032 of the final inference result A1024 includes "abnormality present", an abnormality cause model determination process (S809) is executed to determine the model that is the cause of the abnormality. On the other hand, if the validity determination result 1032 of the final inference result A1024 does not include "abnormality present", the output validity determination unit 111 skips the processes of steps S809 to S813.

[0113] Fig. 24 is a flowchart of the abnormality cause model determination process (S809). Fig. 25A, Fig. 25B, and Fig. 25C are diagrams for explaining an example of the abnormality cause model determination process (S809).

[0114] In the abnormality cause model determination process (S809), the output validity determination unit 111 starts a loop process J1 that repeats the processes of steps S1002 to S1008 for each problem bounding box to be stored in the validity determination result 1032 for the final inference result A1024, as shown in Figure 24 (S1001).

[0115] In the loop process J1, the output validity determination unit 111 starts a loop process J2 in which the processes of steps S1003 to S1006 are repeated for each inference result of the model for the problem bounding box (S1002).

[0116] In loop process J2, the output validity determination unit 111 extracts the inference result of the question bounding box from the inference result for each recognized object for each model (S1003). As shown in FIG. 25A , for example, the inference result 1033 for the question bounding box of Model 1 includes a "Question B box number" column 1033A, a "B box image" column 1033B, a "label" column 1033C, and a "confidence score" column 1033D. The inference result 1034 for the question bounding box of Model 2 includes a "Question B box number" column 1034A, a "B box image" column 1034B, a "label" column 1034C, and a "confidence score" column 1034D. The inference result 1035 for the problem bounding box of Model 3 includes a "Question B box number" column 1035A, a "B box image" column 1035B, a "Label" column 1035C, and a "Confidence score" column 1035D. Here, the confidence scores stored in Model 1 "Confidence score" column 1033D, Model 2 "Confidence score" column 1034D, and Model 3 "Confidence score" column 1035D are confidence scores before weighting is performed.

[0117] Then, based on the information in the "Label" column from the inference results for Problem Box B for each model, statistical information for the corresponding label is extracted from the training data for each model (S1004). For example, as shown in FIG. 25B , statistical information 1037 for "Truck" in the "Label" column 1034C of Model 2, which corresponds to "Truck" in Training Data 3 for Model 2, includes a "Standard Deviation" 1037A, a "Maximum Confidence Score" 1037B, and a "Minimum Confidence Score" 1037C. As shown in FIG. 25C , statistical information 1038 for "Car" in the "Label" column 1035C of Model 3, which corresponds to "Car" in Training Data 4 for Model 3, includes a "Standard Deviation" 1038A, a "Maximum Confidence Score" 1038B, and a "Minimum Confidence Score" 1038C.

[0118] Next, the output validity determination unit 111 compares the reliability score of the inference result of the extracted problem bounding box with the statistical information of the corresponding label from the training data, and determines the consistency between the reliability score of the inference result of the corresponding label and the reliability score distribution of the training data (S1005).

[0119] Thereafter, the output validity determination unit 111 determines whether or not any of the relevant models is the cause of the anomaly, based on the comparison result (S1006). If the reliability score of the inference result for the relevant label matches the reliability score distribution of the training data, this model is not the cause of the anomaly. If the reliability score of the inference result for the relevant label does not match the reliability score distribution of the training data, this model is the cause of the anomaly.

[0120] When the processing of steps S1003 to S1006 is completed for the inference results of all models, the output validity determination unit 111 ends the loop processing J2 (S1007).

[0121] Then, information on all models that are the cause of an abnormality is collected to generate information on models that are the cause of an abnormality 1039 (S1008). As shown in Fig. 25B, information on models that are the cause of an abnormality 1039 includes a column 1039A that stores model numbers and a column 1039B that stores model suitability scores.

[0122] Furthermore, when the processing of steps S1002 to S1008 is completed for all problem bounding boxes, the output validity determination unit 111 ends the loop process J1 (S1009) and ends the process of determining the model causing the abnormality (S809).

[0123] Returning to the validity determination process (FIG. 19), the output validity determination unit 111 determines the magnitude relationship of the degrees of suitability of the models that are the cause of anomalies stored in column 1039B in the model information 1039 that are the cause of anomalies (S810).

[0124] The output validity determination unit 111 then determines that the model causing the abnormality with the highest degree of proficiency is the model requiring relearning, and outputs the model requiring relearning information (S811). For example, as shown in FIG. 25B , the magnitude relationship is determined for the model proficiency degrees stored in the model proficiency degree 1039B of the model information causing the abnormality 1039. The model causing the abnormality with the highest model proficiency degree is model 2. The output validity determination unit 111 then generates the model requiring relearning information 1040 using model 2 as the model number requiring relearning.

[0125] Then, the output validity determining unit 111 determines whether the number of the model that needs to be re-learned is included in the re-learning model information 1040 (S812).

[0126] If the relearning-needed model information 1040 contains a relearning-needed model number, an inference result update process (S813) is executed to update the inference result of the relearning-needed model. On the other hand, if the relearning-needed model information 1040 does not contain a relearning-needed model number, the output validity determination unit 111 skips the process of step S813.

[0127] FIG. 26 is a flowchart of the inference result update process (S813).

[0128] In the inference result update process, the output validity determination unit 111 starts a loop process K1 in which the processes of steps S1102 to S1103 are repeated for each model corresponding to the relearning required model number in the relearning required model information 1040 (S1101).

[0129] In the loop process K1, the output validity determination unit 111 extracts the question B-box number from the generated validity determination result 1032 (S1102).

[0130] Then, the output validity determination unit 111 deletes the inference result corresponding to the question B-box number from the integrated inference result 1022 for each fused recognition object (S1103).

[0131] When the processing of steps S1102 to S1103 is completed for all models requiring relearning, the output validity determination unit 111 ends the loop processing K1 (S1104).

[0132] Then, the output validity determination unit 111 outputs the integrated inference result of the model from which all inference results of the re-learning required models for the question B box number have been deleted as the integrated inference result 1041 of all the updated models (S1105).

[0133] FIG. 27 is a diagram illustrating an example of an integrated inference result 1041 of all updated models.

[0134] When compared with the integrated inference result 1022 for each fused recognition object, the updated integrated inference result 1041 of all models has deleted the inference result (underlined) of model 2, the model that requires re-learning for problem B box number 09 in 1041A.

[0135] Returning to the validity determination process (FIG. 19), the description will be continued. After outputting the integrated inference result 1041, the output validity determination unit 111 ends the update process (S813) for the inference result of the model requiring relearning.

[0136] Then, when the output validity determination unit 111 has executed the processes of steps S802 to S813 for all final inference results A1024 of the inference task, it ends the loop process H1 (S814) and ends the validity determination process for the final inference result A1024.

[0137] FIG. 28 is a flowchart of the process of updating the final inference result B 1025, in which the output unit 110 updates the final inference result A 1024 using the integrated inference result 1041 of all updated models.

[0138] In the process of updating the final inference result B1025, the output unit 110 starts a loop process L1 in which the processes of steps S1202 to S1204 are repeated for each inference task (S1201).

[0139] In the loop processing L1, the output unit 110 acquires the integrated inference results 1041 of all the updated models from the output validity determination unit 111 (S1202).

[0140] Then, the output unit 110 generates reconference data 1042 based on the integrated inference results 1041 of all the updated models (S1203). The process of generating the reconference data 1042 is the same as step S706 of the final inference result B1025 output process (FIG. 14).

[0141] FIG. 29 is a diagram illustrating an example of the reconference data 1042.

[0142] The structure of the re-consultation data 1042 is the same as the structure of the consultation data 1023, and includes a "Fused BBox Number" column 1042A that stores the BBox number for identifying the recognized object of the inference result, a "Fused BBox Image" column 1042B that stores the image inside the BBox of the recognized object, a "Label 1" column 1042C that stores the label representing the class of the recognized object of model 1, a "Weight 1" column 1042D that stores the total weight of model 1, a "Label 2" column 1042E that stores the label representing the class of the recognized object of model 2, a "Weight 2" column 1042F that stores the total weight of model 2, a "Label 3" column 1042G that stores the label representing the class of the recognized object of model 3, and a "Weight 3" column 1042H that stores the total weight of model 3.

[0143] The output unit 110 then conducts another consultation based on the re-consultation data 1042, and generates and outputs an updated final inference result B1025 with the label determined in the re-consultation (S1204). After executing steps S1202 to S1204 for all inference tasks, the output unit 110 ends loop processing L1 (S1205). The output unit 110 then ends the process of updating the final inference result B1025.

[0144] FIG. 30 is a flowchart of the re-learning process in which the model reconfiguration unit 112 re-learns a model that needs to be re-learned.

[0145] In the relearning process, first, the model reconfiguration unit 112 acquires the relearning required model information 1040 from the output validity determination unit 111 (S1301).

[0146] Then, the model reconfiguration unit 112 starts a loop process M1 in which the processes of steps S1303 to S1306 are repeated for the models that need to be retrained stored in the retraining model information 1040 (S1302).

[0147] In the loop process M1, the model reconfiguration unit 112 determines whether or not there is previous version data 6 for the model corresponding to the re-learning required model number (S1303).

[0148] If there is past version data 6, the model reconfiguration unit 112 executes a version rollback process (S1304) for the relevant model requiring re-learning to be placed in the inference unit 105. On the other hand, if there is no past version data 6, the model reconfiguration unit 112 skips the process of step S1304.

[0149] In the version rollback process (S1304), the model reconfiguration unit 112 transmits the past version data 6 of the model that needs to be re-learned to the inference unit 105. The inference unit 105 then receives the past version data 6, deletes the existing version of the model, and replaces it with the received past version of the model.

[0150] The model reconfiguration unit 112 then performs re-learning on the model requiring re-learning (S1305). In re-learning, the model reconfiguration unit 112 adds learning data for re-learning to the model requiring re-learning and performs model re-learning. The model reconfiguration unit 112 then transmits the re-learned model data to the inference unit 105 again. The inference unit 105 then deletes the existing version of the model and replaces it with the acquired re-learned version model, thereby updating the model (S1306). The inference unit 105 inputs input data 5 to the model updated by re-learning and obtains an inference result for the updated model. Alternatively, the inference unit 105 may input input data 5 to a model other than the model being re-learned and perform inference.

[0151] Then, when the model reconfiguration unit 112 has executed the processes of steps S1303 to S1306 for all models that require relearning, it ends the loop process M1 (S1307) and terminates the relearning process.

[0152] FIG. 31 is a diagram showing the hardware configuration of the electronic control unit 30 in which the vehicle control system 100 of this embodiment is implemented.

[0153] The electronic control unit 30 is an electronic control unit (ECU) 30 having a processing unit (CPU) 31, a memory 32, and a communication interface 34. The processing unit 31 is a processor (e.g., a microcomputer) that executes programs stored in the memory 32. The processing unit 31 operates as a functional block that provides various functions by executing predetermined arithmetic processing. The memory 32 includes a non-volatile storage area and a volatile storage area. The non-volatile storage area is accessible by the processing unit 31 and includes a program area that stores programs executed by the processing unit 31, and a data area that stores data used by the processing unit 31 when executing the programs. The volatile storage area temporarily stores data used by the processing unit 31 when executing the programs. The communication interface 34 connects to other electronic control units via a network such as a CAN or Ethernet.

[0154] As described above, according to the first embodiment of the present invention, the vehicle control system 100 acquires an in-vehicle camera image 1 using the data acquisition unit 101 and generates input data 5. The first input data feature vector extraction unit 102 extracts an input data feature vector 1001 from the input data 5. The inference unit 105 does not perform inference on the input data 5 using a single model, but performs inference using multiple models taking into account the characteristics of each model, and generates multiple inference results. The second input data feature vector extraction unit 103 extracts training data feature vectors 1002 for all models. The training data feature vector acquisition unit 104 acquires the feature vector for each model. The suitability determination unit 109 calculates a model suitability score for each model using the extracted input data feature vector 1001 and the multiple model feature vectors. The output unit 110 outputs a final inference result B1025 by consensus based on the inference results of the multiple models and the multiple model suitability scores. The output validity determination unit 111 determines the validity of the final inference result A1024, determines the model that is the cause of the abnormality, and outputs re-learning required model information 1040. The model reconfiguration unit 112 re-learns the model based on the re-learning required model information 1040.

[0155] Therefore, the vehicle control system 100 of the first embodiment can improve the robustness of inference compared to a vehicle control device that detects an object using a single model. Furthermore, the validity of the final inference result A1024 can be accurately determined. Even if an abnormality is determined in the final inference result A1024, the vehicle control device can continue inferring new data without stopping the next inference.

[0156] The present embodiment includes a system and a method for executing the above information processing.

[0157] As described above, according to the first embodiment of the present invention, an object can be recognized with high accuracy by combining multiple trained models with different characteristics. Furthermore, the model that is the cause of the abnormality is determined based on the degree of suitability of each model for the input data 5, the inference results of each model, and the final inference result A1024, so that re-learning can be performed by narrowing down the models that will be most effective for re-learning.

[0158] Although the vehicle control device described in the first embodiment is realized by an on-board electronic control unit (ECU), it may be configured by combining a server (cloud) and the electronic control unit. In the second embodiment, a configuration in which the server and the electronic control unit are combined will be described.

[0159] FIG. 32 is a block diagram showing the functional configuration of a vehicle control system 200 that combines the server 20 and the electronic control unit 30 of the second embodiment.

[0160] The vehicle control system 200 of the second embodiment is composed of a server 20 and an electronic control unit 30 connected via a network. The server 20 has a second input data feature vector extraction unit 103, a model reconstruction unit 112, and an output validity determination unit 111. The server 20 stores learning data 2, 3, and 4 for each model, verification results 7, 8, and 9 for each model, and previous version data 6 of the model. The electronic control unit 30 has a data acquisition unit 101, a first input data feature vector extraction unit 102, a learning data feature vector acquisition unit 104, an inference unit 105, a specialty determination unit 109, and an output unit 110.

[0161] In the vehicle control system 200 on the server 20 side, the second input data feature vector extraction unit 103 extracts the learning data feature vectors 1002 of all models. In addition, the output validity determination unit 111 determines the validity of the final inference result A1024, performs processing to determine the model causing the abnormality, and outputs re-learning required model information 1040. The model reconfiguration unit 112 performs model re-learning processing based on the re-learning required model information 1040.

[0162] In the vehicle control system 200 on the electronic control unit 30 side, the vehicle control system 200 acquires the in-vehicle camera image 1 using the data acquisition unit 101, and generates input data 5. The first input data feature vector extraction unit 102 extracts an input data feature vector 1001 for the input data 5. The inference unit 105 does not perform inference on the input data 5 using a single model, but performs inference using multiple models taking into account the characteristics of each model, and generates multiple inference results. The training data feature vector acquisition unit 104 acquires the feature vector for each model. The ability determination unit 109 calculates a model ability score for each model using the extracted input data feature vector 1001 and the multiple model feature vectors. The output unit 110 outputs a final inference result B1025 by consensus based on the inference results of the multiple models and the multiple model ability scores.

[0163] FIG. 33 is a diagram showing the hardware configuration of the server 20.

[0164] The server 20 is configured by a computer having a processor (CPU) 21, a memory 22, an auxiliary storage device 23, and a communication interface 24. The server 20 may also have an input interface 25 and an output interface 26.

[0165] The processor 21 is an arithmetic device that executes programs stored in the memory 22. The processor 21 executes various programs to realize each functional unit of the server 20. Note that part of the processing performed by the processor 21 by executing the programs may be executed by another arithmetic device (for example, hardware such as an ASIC or FPGA, or a GPU).

[0166] The memory 22 includes a ROM, which is a non-volatile storage element, and a RAM, which is a volatile storage element. The ROM stores unchanging programs (e.g., BIOS), etc. The RAM is a high-speed, volatile storage element such as a DRAM (Dynamic Random Access Memory), and temporarily stores programs executed by the processor 21 and data used when the programs are executed.

[0167] The auxiliary storage device 23 is a large-capacity, non-volatile storage device such as a magnetic storage device (HDD) or a flash memory (SSD). The auxiliary storage device 23 also stores data used by the processor 21 when executing a program and the program executed by the processor 21. That is, the program is read from the auxiliary storage device 23, loaded into the memory 22, and executed by the processor 21 to realize each function of the server 20.

[0168] The communication interface 24 is a network interface device that controls communication with other devices in accordance with a predetermined protocol.

[0169] The input interface 25 is an interface to which input devices such as a keyboard 27 and a mouse 28 are connected and which receives input from an operator. The output interface 26 is an interface to which output devices such as a display device 29 and a printer (not shown) are connected and which outputs the results of program execution in a format that can be viewed by an operator.

[0170] The programs executed by the processor 21 are provided to the server 20 from removable media (such as a CD-ROM or flash memory) or via a network, and are stored in a non-volatile auxiliary storage device 23, which is a non-transitory storage medium. For this reason, the server 20 should preferably have an interface for reading data from removable media.

[0171] The server 20 is a computer system configured on a single physical computer or on multiple logically or physically configured computers, and may operate on a virtual computer constructed on multiple physical computer resources. For example, each functional unit may operate on a separate physical or logical computer, or multiple functional units may be combined to operate on a single physical or logical computer.

[0172] As described above, according to the second embodiment of the present invention, processes with high processing loads are executed by the server 20 with high computing power, so that a computing device with low computing power can be used for the on-board electronic control unit 30, thereby reducing the cost of the electronic control unit 30.

[0173] The present invention is not limited to the above-described embodiments, and includes various modifications and equivalent configurations within the spirit and scope of the appended claims. For example, the above-described embodiments have been described in detail to clearly explain the present invention, and the present invention is not necessarily limited to configurations including all of the described configurations. Furthermore, part of the configuration of one embodiment may be replaced with the configuration of another embodiment. Furthermore, the configuration of another embodiment may be added to the configuration of one embodiment. Furthermore, part of the configuration of each embodiment may be added, deleted, or replaced with other configurations.

[0174] Furthermore, the aforementioned configurations, functions, processing units, processing means, etc. may be realized in part or in whole in hardware, for example by designing them as integrated circuits, or may be realized in software by a processor interpreting and executing a program that realizes each function.

[0175] Information such as programs, tables, and files that realize each function can be stored in a storage device such as a memory, hard disk, or SSD (Solid State Drive), or in a recording medium such as an IC card, SD card, or DVD.

[0176] In addition, the control lines and information lines shown are those that are considered necessary for explanation, and do not necessarily represent all the control lines and information lines that are necessary for implementation. In reality, it can be assumed that almost all components are interconnected.

Claims

1. A vehicle control system comprising: a data acquisition unit that acquires input data to be input to a model; an inference unit that inputs the input data to a plurality of the models, performs inference, and outputs output data for each of the models; an input data feature vector extraction unit that extracts an input data feature vector that represents the features of the input data; a training data feature vector acquisition unit that acquires a training data feature vector that represents the features of the training data used to train the models; an ability determination unit that determines, for each of the models, an ability level that indicates the likelihood that the model will derive correct output data for the input data, based on the input data feature vector and the training data feature vectors for each of the plurality of models; and an output unit that outputs an inference result derived based on the ability level for each of the models determined by the ability determination unit and the plurality of output data.

2. A vehicle control system as described in claim 1, wherein the input data is an image captured by a camera mounted on a vehicle, and the input data feature vector and the learning data feature vector are vectors composed of features of the image.

3. A vehicle control system according to claim 1, wherein the proficiency level determination unit determines the proficiency level for each model as the similarity between the input data feature vector and the training data feature vector.

4. A vehicle control system as described in claim 1, wherein the output unit derives the inference result by summing up weighted output data for each model according to the degree of expertise for each model determined by the degree of expertise determination unit.

5. A vehicle control system as described in claim 1, comprising a validity determination unit that compares the output data output from the good model with the high degree of goodness with the inference result output from the output unit, and determines the validity of the inference result.

6. A vehicle control system as described in claim 5, wherein, when the validity of the output data is low, the validity determination unit compares the reliability score for the inference target contained in the output data output from the plurality of models with the reliability score for the inference target contained in the learning data used to learn the model, and identifies a model to be re-learned.

7. A vehicle control system according to claim 6, wherein the inference unit inputs the input data to a model other than the re-learning target model, performs inference, and outputs output data for each of the models.

8. A vehicle control system as described in claim 6, comprising a model reconfiguration unit that performs re-learning of the re-learning target model, wherein the inference unit inputs the input data into the re-learned model re-learned by the model reconfiguration unit, performs inference, and outputs output data.

9. A vehicle control method executed by a vehicle control system, the vehicle control system having a calculation device that performs calculation processing and a storage device accessible by the calculation device, the vehicle control method comprising: a data acquisition step of acquiring input data to be input to a model; an inference step of inputting the input data to a plurality of the models to perform inference and outputting output data for each of the models; an input data feature vector extraction step of extracting input data feature vectors that represent features of the input data; a learning data feature vector acquisition step of acquiring learning data feature vectors that represent features of learning data used to train the models; a skill determination step of determining, for each of the models, a skill level that indicates the possibility of deriving correct output data for the input data, based on the input data feature vector and the learning data feature vectors for each of the plurality of models; and an output step of outputting an inference result derived based on the skill level for each of the models determined in the skill determination step and the plurality of output data.

Images