Edge terminal ai inference system

Abstract

To provide an edge terminal AI inference system which provides a learned model for accurately performing automatic operation of a traveling object.SOLUTION: An edge terminal AI inference system 1 includes: a meta-information generation unit which generates meta-information MD including information regarding a class of an object by using a first inference model on the basis of input sensor information SD; a meta-information extension unit which generates extended meta-information ED being information extended by using a meta-information extension model on the basis of the meta-information; and a prediction information generation unit which generates prediction information regarding the meta-information by using a second inference model on the basis of the meta-information or the extended meta-information. The first inference model includes a machine learning model using known sensor information and meta-information, the meta-information extension model includes a machine learning model using known meta-information and extended meta-information acquired from an external language model, and the second inference model includes a machine learning model using known meta-information, the extended meta-information, and the prediction information. The prediction information generation unit generates the prediction information on the basis of the generated meta-information or reliability of the prediction information.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] The present invention relates to an edge terminal AI inference system, and more particularly to an AI inference system for edge terminals in a hierarchical local edge network that performs tasks such as controlling the movement of mobile objects. [Background technology]

[0002] In recent years, Mobility as a Service (MaaS) technology has been rapidly spreading, and technological development is underway to utilize huge amounts of time-series real data, both temporally and spatially, acquired from numerous vehicles connected to wide-area networks such as the Internet, for the automated driving or operation of vehicles, efficient traffic control, etc. In order to efficiently process so-called big data from vehicles distributed and connected to a wide-area network, a local edge network is effective, which includes a hierarchical vehicle communication system and database consisting of an upper-level cloud server covering a wide area, an intermediate-level local area server covering a more limited area, and a terminal-level edge terminal operated by the vehicle driving system installed in each vehicle.

[0003] Meanwhile, neural networks, or so-called AI (artificial intelligence), are being used in various fields. For example, so-called object detection technology, which is a technology that detects objects in camera images based on trained neural networks based on deep learning, has attracted attention. Object detection technology can estimate one or more object candidate regions in a captured image and identify the class of the candidate object in each of the one or more candidate regions. An autonomous driving system for a vehicle that uses such object detection technology can, for example, detect obstacles captured in an image based on images obtained by an on-board camera, and capture road conditions to control the steering and acceleration / deceleration of the vehicle.

[0004] In such an automated driving system for a vehicle, in order to perform appropriate driving control of the vehicle, the movements of moving objects around the vehicle are predicted from camera images (see, for example, Japanese Patent Application Laid-Open No. 2023-106554). Specifically, the moving objects around the vehicle and their types are distinguished, and the vehicle's travel route is determined so as to satisfy the safety distance that should be maintained between each moving object according to the movement prediction.

[0005] The automatic driving device of Patent Document 1 uses a machine-learned neural network that calculates the optimal driving route using inputs such as a safe distance and the driving environment. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] Japanese Patent Application Publication No. 2023-106554 Summary of the Invention [Problem to be solved by the invention]

[0007] In Patent Document 1, the AI ​​only determines the driving path, and conventional methods not relying on AI, such as pattern matching, are used for the processes of, for example, recognizing the type of obstacle from a camera image, determining whether the object is moving or stationary, and calculating the safe distance to be maintained between the obstacle and the vehicle depending on the type of obstacle. Furthermore, when conventional AI is used to recognize the type of obstacle in the automated driving of a mobile vehicle, meta-information related to the class of obstacle is limited to learned keywords, and the amount of information about the recognized object is therefore limited, resulting in a problem of low prediction accuracy using a learned model when predicting the movement of an object from sensor information.

[0008] Here, "meta information" includes data describing attributes and related information that represent certain data. For example, "meta information about an object" includes the object's class (type), size, position in the image, and time of photograph.

[0009] The present invention has been made in consideration of these points, and its purpose is to provide a trained model that can perform automatic driving of mobile objects, including edge terminals, with high accuracy based on sensor information in a local edge network. [Means for solving the problem]

[0010] To achieve the above objectives, we targeted an edge terminal AI inference system in a local edge network including edge terminals and implemented the following solutions.

[0011] That is, in the first invention, the edge terminal AI inference system a first generation unit that generates meta information including information about a class of identifiable objects using a first estimation model based on sensor information input to the edge terminal; a data decompression unit that generates decompressed meta information by decompressing the information based on the meta information using a meta information decompression model; a second generation unit that generates prediction information about the meta information using a second estimation model based on the meta information or the decompressed meta information; the first estimation model includes a model machine-learned using known sensor information as input and known meta-information including information about the object class as output; the meta information decompression model includes a model that is machine-learned using known meta information of the edge terminal as input and decompressed meta information acquired from an external language model as output, the second estimation model includes a model machine-learned using known meta information or decompressed meta information as input and known prediction information as output, The second generation unit generates the prediction information based on reliability of the generated meta information or prediction information.

[0012] In the edge terminal AI inference system, the data decompression unit uses a meta information decompression model trained using an external language model to generate decompressed meta information in which information is decompressed to include additional detailed information about the classes contained in the meta information.Therefore, the second generation unit can generate more detailed prediction information from the decompressed meta information, which is more detailed than the meta information, thereby providing a trained model with high prediction accuracy.

[0013] In a second invention, in the first invention, the first generation unit generates meta information certainty data indicating a reliability of the meta information, the second generation unit generates prediction information confidence level data indicating a reliability of the prediction information; The second generation unit generates the prediction information based on the meta information certainty data or the prediction information certainty data.

[0014] The first generation unit and the second generation unit generate meta information certainty data and predicted information certainty data that indicate the reliability of the meta information and predicted information generated when generating the meta information and predicted information. Therefore, data indicating the reliability of the meta information and predicted information can be easily obtained, making it possible to easily provide a trained model with high prediction accuracy.

[0015] In a third aspect of the present invention, in the first aspect of the present invention, a database unit is further provided, the database unit includes a plurality of the first inference models, a plurality of the meta information decompression models, and a plurality of the second inference models that are compatible with a plurality of scenarios; The first guess model, the meta information decompression model, and the second guess model are each selected based on the scenario.

[0016] The edge terminal AI inference system includes in its database section a plurality of first inference models, a plurality of meta information extension models, and a plurality of second inference models that are adapted to a plurality of scenarios that reflect, for example, external environmental conditions such as road conditions, weather, and time during travel, or internal environmental conditions such as the remaining fuel level as determined by sensors on the mobile body, or the physical condition and fatigue level of the occupant as determined by cameras, etc., and therefore a scenario model that is adapted to the scenario of the sensor information is selected from the database section, making it possible to provide a trained model with high predictive accuracy.

[0017] In a fourth invention, in the first to third inventions, the first estimation model, the meta information extension model, and the second estimation model are re-learned using newly acquired sensor information, newly generated meta information, extended meta information, and prediction information, and the corresponding correct answer data.

[0018] The first estimation model is retrained using newly acquired sensor information and correct answer data corresponding to newly generated meta information, the meta information extension model is retrained using newly generated meta information and correct answer data corresponding to newly generated extended meta information, and the second estimation model is retrained using newly generated meta information or extended meta information and correct answer data corresponding to newly generated prediction information.As a result of retraining, the error between the meta information, extended meta information, and prediction information generated by each model and the correct answer data becomes smaller, so that each model generates meta information, extended meta information, and prediction information with a small error from the correct answer data, and therefore it is possible to provide a trained model with high prediction accuracy. [Effects of the Invention]

[0019] According to the present invention, it is possible to provide a trained model that can perform automatic driving of a mobile body including an edge terminal with high accuracy based on sensor information. [Brief explanation of the drawings]

[0020] [Figure 1] FIG. 1 shows the overall configuration of a vehicle communication system according to an embodiment of the present invention. [Figure 2] FIG. 2 shows a schematic configuration of a vehicle driving system including an edge terminal AI inference system according to an embodiment of the present invention. [Figure 3] FIG. 3 shows the functional configuration of an edge terminal AI inference system according to an embodiment of the present invention. [Figure 4] FIG. 4 is a diagram illustrating the functions of the edge terminal AI inference system according to the embodiment of the present invention. [Figure 5] FIG. 5 is a diagram illustrating the configuration of a meta information generation model database unit according to an embodiment of the present invention. [Figure 6] FIG. 6 is a diagram illustrating the configuration of the prediction information generation model database unit according to the embodiment of the present invention. [Figure 7] FIG. 7 is a diagram for explaining the operation of the edge terminal AI inference system according to the embodiment of the present invention. [Figure 8] FIG. 8 is a schematic flowchart of an edge terminal AI inference system according to an embodiment of the present invention. [Figure 9] FIG. 9 is a detailed flowchart of the meta information decompression unit according to the embodiment of the present invention. [Figure 10] FIG. 10 shows an example of the operation of the meta information and prediction information generating unit according to the embodiment of the present invention. [Figure 11] FIG. 11 shows another example of the operation of the meta information and prediction information generating unit according to the embodiment of the present invention. [Figure 12] FIG. 12 shows an example of the hardware configuration of an edge terminal AI inference system according to an embodiment of the present invention. [Figure 13] FIG. 13 shows an example of a database structure of an edge terminal AI inference system according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0021] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Preferred embodiments of the present invention will now be described in detail with reference to the accompanying drawings. It should be noted that the following description of the preferred embodiments is merely exemplary in nature.

[0022] As shown in FIG. 4, an edge-terminal AI inference system 1 according to an embodiment of the present invention can be applied to a system that performs sequential step-by-step processing, such as a vehicle driving system 71 that automatically drives a vehicle 70. Specifically, the edge-terminal AI inference system 1 receives sensor information SD from a sensor unit 2, such as an on-board camera. A meta-information generation unit 17 generates meta-information MD based on the sensor information SD. A meta-information decompression unit 21 generates decompressed meta-information ED based on the generated meta-information MD. A prediction information generation unit 23 generates prediction information PD based on the meta-information MD or the decompressed meta-information ED. A judgment information generation unit 25 generates judgment information JD based on the generated prediction information PD. A behavior information generation unit 27 generates behavior information AD based on the generated judgment information JD, and finally outputs a control signal to a vehicle control ECU. In this way, the edge-terminal AI inference system 1 is suitable for a system that generates data containing more abstract information in a step-by-step manner based on sensor information, which is physical information.

[0023] Below, the first and second generation units of the present invention will be described when applied to the meta information generation unit 17 and prediction information generation unit 23 of the edge terminal AI inference system 1 applied to the vehicle driving system 71, but the scope of application of the present invention is not limited to this, and for example, the first and second generation units of the present invention may be applied to any consecutive generation units that perform sequential step processing, such as the prediction information generation unit 23 and judgment information generation unit 25.

[0024] (Vehicle communication system) 1, the vehicular communication network 100, which is a local edge network, includes a network 90 and a plurality of vehicles 70 connected to the network 90. ​​The network 90 includes a plurality of base stations 80, and the base stations 80 include a main base station 81 in an upper hierarchy that communicates with the vehicles 70 in a wide area, and a local area base station 83 in an intermediate hierarchy that communicates with the vehicles 70 in a more limited area that is a part of the wide area. The vehicles 70 include a vehicle driving system 71.

[0025] (network) The network 90 includes a plurality of base stations 80 and a plurality of vehicles 70, which are connected to each other via a common wireless interface, and are configured to enable direct or indirect communication between any two of them.

[0026] Wireless interfaces that can be used include 5G, local 5G, 4G, DSRC (Dedicated Short Range Communications), LPWA (Low Power Wide Area), WiFi, IEEE802.11 series, LTE (Long Term Evolution), and ITS (Intelligent Transport System) communication interfaces.

[0027] (base station) The base station 80 may include one or more main base stations 81 and multiple local area base stations 83, and may be configured to be able to perform direct communication between any two of the multiple vehicles 70. Furthermore, communication may be performed using the vehicles 70 as repeaters, for example, from one vehicle 70 to the base station 80 via another vehicle 70. Note that in FIG. 1, not all of these aspects are shown in order to avoid cluttering the drawing.

[0028] (Main base station) The main base station 81 is a base station that manages and controls the geographical area in which wireless communication is possible in the vehicle communication network 100, and performs data management processing and distribution control processing for the multiple vehicles 70. The main base station 81 also communicates with base stations 80 of other vehicle communication networks 100 or with a cloud data center (not shown), forming a communication backbone network.

[0029] (Local area base station) Furthermore, the local area base station 83 belongs to a lower layer than the main base station 81 and subdivides and controls the geographical area controlled by the main base station 81. However, if there is no need to subdivide the geographical area, it is possible to omit the local area base station 83. In that case, the base station 80 is composed of only the main base station 81.

[0030] (Vehicles and vehicle driving systems) The vehicle 70 includes a vehicle driving system 71 for controlling automatic driving, etc., and as shown in Figure 2, the vehicle driving system 71 includes a sensor unit 2 that collects sensor information SD for monitoring the surroundings of the vehicle while driving, an edge terminal AI inference system 1 that generates steering and acceleration / deceleration information for the vehicle 70 from the collected sensor information SD, a vehicle control ECU 3 that outputs control signals for driving control of the vehicle 70 from the steering and acceleration / deceleration information generated by the edge terminal AI inference system 1 to control an ECU (Engine Control Unit), a locator 4, and a touch panel 5 as an input / output device.

[0031] (sensor part) The sensor information SD detected by the sensor unit 2 may be any information that can be detected by a sensor, such as information on objects on the road, time, moving speed, position, and approach of people to the vehicle, as well as any natural phenomenon such as weather or temperature, or artificial phenomenon such as traffic congestion, and human vital information such as heart rate, respiratory rate, and blood pressure of passengers in the vehicle 70. In this embodiment, the sensor unit 2 includes a camera that captures road conditions and an inertial sensor such as an acceleration sensor that detects the moving speed and direction of an object.

[0032] (Vehicle control ECU) The vehicle control ECU 3 is an electronic control device that performs driving control by outputting control signals to each driving control device of the vehicle 70, such as an electronically controlled throttle, a brake actuator, and an EPS (Electric Power Steering) motor. The vehicle control ECU 3 causes each driving control device of the vehicle 70 to perform driving control in accordance with steering and acceleration / deceleration information generated by the edge terminal AI inference system 1. In other words, the vehicle control ECU 3 performs driving control of the vehicle 70 equipped with the vehicle driving system 71, based on the steering and acceleration / deceleration information generated by the edge terminal AI inference system 1. The vehicle control ECU 3 controls the vehicle in accordance with the steering and acceleration / deceleration information generated by the edge terminal AI inference system 1 and the position of the vehicle 70 as determined by the locator 4, thereby enabling the vehicle 70 to automatically perform evasive action and drive to avoid approaching nearby objects.

[0033] (locator) The locator 4 includes a GPS (Global Positioning System) or GNSS (Global Navigation Satellite System) receiver and an inertial sensor. The GNSS receiver receives positioning signals from multiple positioning satellites. The inertial sensor includes, for example, a gyro sensor and an acceleration sensor. The locator 4 sequentially determines the position of the vehicle 70 equipped with the locator 4 by combining the positioning signals received by the GNSS receiver with the measurement results of the inertial sensor. The vehicle position can be expressed, for example, in latitude and longitude coordinates.

[0034] (input / output devices) The touch panel 5 of the input / output device is a device for receiving input operations from passengers of the vehicle 70 to the edge terminal AI inference system 1, and for presenting output information of the edge terminal AI inference system 1 to the passengers.

[0035] (Edge terminal AI inference system) 3 and 4 show the functional configuration of the edge terminal AI inference system 1. As shown in Fig. 3, the edge terminal AI inference system 1 functionally comprises an input / output unit 1a, a communication interface unit 1b, a memory unit 1c, and a control unit 1d.

[0036] For simplicity, interface may be abbreviated as I / F or IF, and database as DB.

[0037] (input / output section) The input / output unit 1a is connected to input / output devices such as a keyboard, touch panel, display, and speaker, as well as the GPS 4, and has a mechanism for inputting and outputting data and control between the edge terminal AI inference system 1 and external devices. Interfaces that can be used include, but are not limited to, USB (registered trademark), HDMI (registered trademark), Bluetooth (registered trademark), a PCI (Peripheral Component Interconnect / Interface) bus, and the Universal Chip Interconnect Express (UCIe) standard. The input / output unit 1a is also connected to the camera 2 and vehicle control ECU 3 of the vehicle 70. As described below, the input / output unit 1a outputs and accumulates sensor information SD input from the camera 2 in the sensor information database unit 32, and outputs steering and acceleration / deceleration information generated by the behavior information generation unit 27 based on the sensor information SD to the vehicle control ECU 3.

[0038] (Communication interface section) The communication interface unit 1b includes the wireless interface and the like, and has a mechanism for communicating with a base station 80 or other vehicles 70, etc., and transmitting and receiving the current positions of the vehicle and other vehicles using a GPS 4 or the like, the latest traffic information, etc. The communication interface unit 1b can also transmit and receive data to and from an external language model server including chatGPT or the like in order to build a trained meta information extension model database unit 37b, which will be described later.

[0039] (Storage part) The memory unit 1c includes, for example, memories such as flash memory, DRAM (Dynamic Random Access Memory), SRAM (Static Random Access Memory), and magnetic storage devices such as hard disk drives, and includes an application program 31 used by the edge terminal AI inference system 1, a sensor information database unit 32 in which sensor information SD input from the input / output unit 1a is stored, generation and extension database units (35a, 37a, 39a, 41a, 43a) in which generated data is stored, and model database units (35b, 37b, 39b, 41b, 43b) in which each generation and extension model for generating that data is held.

[0040] The application program 31 is executed by the processor of the edge terminal AI inference system 1, causing the processor to function as each control unit described below. Data generated by each control unit using the generation and decompression models of the model database units (35b, 37b, 39b, 41b, 43b) is output to and stored in the corresponding generation and decompression database units (35a, 37a, 39a, 41a, 43a).

[0041] Details of the model database units (35b, 37b, 39b, 41b, 43b) and the generation and decompression database units (35a, 37a, 39a, 41a, 43a) will be described later in the sections for the relevant control units.

[0042] (Scenario model database section) 4, the model database unit (35b, 37b, 39b, 41b, 43b) that holds generation and decompression models for various scenes of the sensor information SD input from the camera 2 includes a scenario model database unit 33 (33a, 33b, 33c, 33d, 33e) that holds models corresponding to specific scenarios for various scenes such as general roads, expressways, and country roads. This embodiment includes scenario models #0, #1, and #2 that correspond to general road scenario #0, expressway scenario #1, and country road scenario #2, respectively, but the types of scenario models are not limited to these.

[0043] For example, the scenario model database unit 33 may hold a model that reflects external environmental conditions such as road conditions, weather, and time of day during travel, or internal environmental conditions such as the amount of fuel remaining as determined by the vehicle's 70 sensor 2 or the driver's physical condition and fatigue level as determined by the camera 2.

[0044] (Control unit) The control unit 1d includes a processor, etc., and when the application program 31 in the memory unit 1c is executed by the processor, it functions as an input / output control unit 11, a communication control unit 13, a scenario control unit 15, a meta information generation unit 17, a model control unit 19, a meta information expansion unit 21, a prediction information generation unit 23, a judgment information generation unit 25, a behavioral information generation unit 27, and a re-learning unit 29.

[0045] (Input / output control unit) The input / output control unit 11 controls the input / output processing of the input / output unit 1a.

[0046] (Communication control unit) The communication control unit 13 controls the communication processing of the communication interface unit 1b.

[0047] (Scenario control section) The scenario control unit 15 selects the scenario model that is appropriate for the sensor information SD from among the scenario models #0, #1, and #2 that are compatible with each scenario contained in the scenario model database unit 33. As will be described later, the scenario control unit 15 is configured to select a scenario model based on confidence data that indicates the reliability of the generated data. In other words, the meta information generation model, prediction information generation model, extension prediction information generation model, judgment information generation model, and behavior information generation model, which will be described later, are each selected from the scenario models #0, #1, and #2 based on the scenario corresponding to the sensor information SD. Alternatively, a passenger in the vehicle 70 may directly select a scenario model via an input device such as the touch panel 5.

[0048] (Meta information generation unit) The meta information generation unit 17 extracts physical information features from the sensor information SD and generates meta information MD containing information about the object class of a detected obstacle, etc. That is, the meta information generation unit 17 uses a meta information generation model to generate, from the sensor information SD stored in the sensor information database unit 32, meta information MD having a class hierarchical structure indicating the hierarchical relationships between the classes constituting the identifiable class group based on information about the classes constituting the class group, and meta information certainty data MC indicating its reliability, and outputs these to the meta information generation database unit 35a. Note that the meta information certainty data MC is a decimal value between 0 and 1, with the closer to 1 the data, the higher the reliability of the generated meta information MD. The same applies to the certainty data output by each generation unit or extension unit described later.

[0049] (Meta-information generation model) 4 and 5, the meta information generation model is machine-learned for each scenario using known sensor information as input and known meta information as output, and a plurality of models corresponding to the class hierarchical structure of the generated meta information are stored in the meta information generation model database unit 35b and the scenario model database unit 33a. That is, the meta information generation model includes, for each scenario, a meta information generation model MM0 corresponding to object class hierarchy 0, which is the highest hierarchy included in the meta information, a meta information generation model MM1 corresponding to object class hierarchy 1, and so on.

[0050] The meta information MD estimates one or more object candidate regions based on input sensor information, identifies the class of the object detected in each of the one or more object candidate regions, and includes region and class information of the identified objects.

[0051] As shown in Fig. 5, the meta information generation model MM0 is configured to generate meta information including class information of class hierarchy 0 among a group of classes having a hierarchical structure of an object identified from the input sensor information SD. The meta information generation model MM1 is configured to generate meta information including class information of class hierarchy 1, which is one hierarchy lower than the top hierarchy, among the group of classes having a hierarchical structure of an object identified from the input sensor information SD.

[0052] For example, in the example of Figure 5, the meta-information generation model MM0 identifies the hierarchical classes of "person" in class hierarchy 0, "male / female" in class hierarchy 1, "adult / child" in class hierarchy 2, "facing forward / facing backward" in class hierarchy 3, and "talking / playing with smartphone" in class hierarchy 4 for the attributes of the object in the input known sensor information, and outputs meta-information including the class information "person" in class hierarchy 0. In addition, the meta-information generation model MM1 outputs meta-information including the identified class information "male" in class hierarchy 1.

[0053] The meta information generation models MM0, MM1, etc. used by the meta information generation unit 17 are selected by the model control unit 19, which will be described later. Note that the number of class hierarchical structures corresponding to the meta information generation model is not limited to two, and may be any number depending on the identified object.

[0054] In addition, for machine learning, models such as CNN (Convolutional Neural Network), DNN (Deep Neural Network), GAN (Generative Adversarial Network), GPT, diffusion model, etc. The same applies to the machine learning of each generative model described below.

[0055] (Model control section) The model control unit 19 selects a model to be used by each generation and decompression unit in each model database unit for scenario models #0, #1, and #2. That is, for a specific scenario model selected by the scenario control unit 15, the model control unit 19 selects a meta information generation model to be used by the meta information generation unit 17, a meta information decompression model to be used by the meta information decompression unit 21 (described later), and a prediction information generation model to be used by the prediction information generation unit 23, based on the reliability of the generated prediction information, etc.

[0056] Specifically, the model control unit 19 selects, for a specific scenario model, the models MM0, MM1, etc. used by the meta information generation unit 17 so that prediction information confidence data PC indicating the reliability of prediction information PD generated by the prediction information generation unit 23 (described later) satisfies a predetermined threshold. In other words, the meta information generation models MM0, MM1, etc. are selected by the model control unit 19 for the scenario model identified by the scenario control unit 15 based on the prediction information confidence data PC indicating the reliability of the prediction information PD generated.

[0057] Alternatively, the metainformation generation models MM0, MM1, etc. may be selected based on the metainformation certainty data MC to satisfy a predetermined threshold. In this case, the metainformation generation models MM0, MM1, etc. are selected based on the metainformation certainty data MC, which eliminates the need for a control signal line between the prediction information generator 23 and the metainformation generator 17, simplifying the connection between them and facilitating implementation.

[0058] (Meta information expansion section) The meta information decompression unit 21 decompresses the physical information extracted by the meta information generation unit 17, and generates decompressed meta information ED to which detailed information about the extracted obstacles, etc. is added using a language model. That is, the meta information decompression unit 21 receives the meta information MD generated by the meta information generation unit 17 using the meta information decompression model, and generates decompressed meta information ED obtained by decompressing the meta information MD using the associative function of the language model, as well as decompressed meta information certainty data EC indicating its reliability, and outputs the decompressed meta information ED to the meta information decompression database unit 37a.

[0059] (Metadata expansion model) The metainformation decompression model is a model that has been machine-learned for each scenario using an external language model such as chatGPT, with known metainformation as input and known decompressed metainformation as output, and is stored in the metainformation decompression model database unit 37b and the scenario model database unit 33b. These database units are also stored as pre-trained databases in a ROM (Read Only Memory) or the like of firmware included in the edge terminal AI inference system 1 to enable high-speed database searches. The metainformation decompression model used by the metainformation decompression unit 21 is selected from scenario models #0, #1, and #2 by the scenario control unit 15.

[0060] Language models include, for example, GPT (Generative Pretrained Transformer)-1, GPT-2, GPT-3, GPT-3.5, GPT-4, T5 (Text-to-Text Transfer Transformer), ELMo (Embeddings from Language Models), BERT (Bidirectional Encoder Representations from Transformers), LaMDA (Large language Model Meta AI), PaLM (Pathways Language Model), PaLM2, NEMO LLM, Claude, Alpaca 7B, Vicuna 13B, OpenFlamingo, Transformer, or language models or large language models (LLMs) that have been modified, improved, transferred, or additionally trained on these.

[0061] The meta information decompression model may be, for example, a model having a certainty tree type database structure described later, or may be a pre-trained model on a cloud server or a separate offline training model.

[0062] (Prediction information generation unit) The prediction information generation unit 23 extracts semantic information based on the physical information extracted by the meta information generation unit 17, and generates prediction information PD including a future movement path for the extracted obstacle, etc. That is, using a prediction information generation model, the prediction information generation unit 23 receives the meta information MD or the expanded meta information ED generated by the meta information generation unit 17 or the meta information expansion unit 21 as input, generates prediction information PD for the meta information, and prediction information certainty factor data PC indicating its reliability, and outputs the prediction information PD to the prediction information generation database unit 39a.

[0063] Here, the predicted information PD includes information predicted about the movement path of one or more identified objects based on the sensor information SD of the sensor information database unit 32 and the meta information MD generated by the meta information generation unit 17, etc.

[0064] (Predictive information generation model) As shown in FIG. 6, the prediction information generation model includes a keyword prediction information generation model KPM and an extension prediction information generation model EPM (hereinafter, both may be simply referred to as prediction information generation models). The keyword prediction information generation model KPM is machine-learned using, as input, known meta information before information decompression, such as limited keywords, and known prediction information as output. The extension prediction information generation model EPM is machine-learned using, as input, known extension meta information after keyword information is decompressed using a language model acquired by chat GPT or the like, and known prediction information as output. That is, the prediction information generation unit 23 uses the keyword prediction information generation model KPM to generate prediction information PD and prediction information certainty data PC from meta information MD and output them to the database unit 39a. The prediction information generation unit 23 also uses the extension prediction information generation model EPM to generate prediction information PD and prediction information certainty data PC from decompressed meta information ED and output them to the database unit 39a.

[0065] The prediction information generation models KPM and EPM are machine-learned for each scenario and stored in the prediction information generation model database unit 39b and the scenario model database unit 33c, and are selected by the model control unit 19 based on the prediction information confidence data PC for the scenario model identified by the scenario control unit 15.

[0066] Specifically, the model control unit 19 selects the keyword prediction information generation model KPM when the input information is meta information MD, and selects the extended prediction information generation model EPM when the input information is extended meta information ED.

[0067] The prediction information generation models KPM and EPM may be selected based on the meta information confidence data MC. In this way, the prediction information generation models KPM and EPM are selected when the meta information generation unit 17 generates the meta information confidence data MC, and therefore the prediction information generation unit 23 can operate faster than when a model is selected based on the prediction information confidence data PC.

[0068] (Judgment information generation unit) The judgment information generation unit 25 extracts value information based on the semantic information extracted by the prediction information generation unit 23, and generates judgment information JD including a judgment of the danger of the extracted obstacles, etc., on the travel path. That is, the judgment information generation unit 25 uses the judgment information generation model to receive the prediction information PD generated by the prediction information generation unit 23, and generates judgment information JD based on the prediction information PD and judgment information certainty data JC indicating its reliability, and outputs them to the judgment information generation database unit 41a.

[0069] Here, the judgment information JD includes information that judges the degree of danger posed to the traveling vehicle 70 based on the sensor information SD of the sensor information database unit 32 and the prediction information PD generated by the prediction information generation unit 23, and judges the degree of danger posed to the traveling vehicle 70 when one or more identified objects take a predicted movement path.

[0070] (Judgment information generation model) The judgment information generation model is machine-learned for each scenario using known prediction information as input and known judgment information as output, and is stored in the judgment information generation model database unit 41b and the scenario model database unit 33d. The judgment information generation model used by the judgment information generation unit 25 is selected by the scenario control unit 15 from among scenario models #0, #1, and #2.

[0071] (Behavior information generation department) The behavior information generation unit 27 generates behavior information AD including behavior to avoid the danger determined for the extracted obstacle, etc., based on the value information extracted by the judgment information generation unit 25. That is, the behavior information generation unit 27 receives the judgment information JD generated by the judgment information generation unit 25 as input, uses the behavior information generation model, generates behavior information AD based on the judgment information JD, and behavior information certainty data AC indicating its reliability, and outputs them to the behavior information generation database unit 43a.

[0072] Here, the behavior information AD determines the steering and acceleration / deceleration of the vehicle 70 based on the sensor information SD of the sensor information database unit 32 and the judgment information JD generated by the judgment information generation unit 25, and includes steering and acceleration / deceleration information that is output to the vehicle control ECU 3 according to the degree of danger to the vehicle 70 that occurs when one or more identified objects take a predicted movement path.

[0073] (Behavioral information generation model) The behavior information generation model is machine-learned for each scenario using known judgment information as input and known behavior information as output, and is stored in the behavior information generation model database unit 43b and the scenario model database unit 33e. The behavior information generation model used by the behavior information generator 27 is selected by the scenario control unit 15 from among scenario models #0, #1, and #2.

[0074] (Relearning section) The re-learning unit 29 re-learns the meta information generation models MM0, MM1, etc. by inverse reinforcement learning (IRL) or the like using the sensor information SD newly acquired and stored in the sensor information database unit 32 and the correct answer data of the corresponding meta information MD stored in the meta information generation database unit 35a, thereby updating the meta information generation model database unit 35b and the scenario model database unit 33a.

[0075] In addition, the re-learning unit 29 re-learns the meta information decompression model by inverse reinforcement learning (IRL) or the like using the meta information MD newly stored in the meta information generation database unit 35a and the correct answer data of the corresponding decompressed meta information ED stored in the meta information decompression database unit 37a, thereby updating the meta information decompression model database unit 37b and the scenario model database unit 33b.

[0076] Similarly, the re-learning unit 29 re-learns the keyword prediction information generation model KPM and the extended prediction information generation model EPM using the meta information MD and the extended meta information ED newly stored in the meta information generation database unit 35a or the meta information extension database unit 37a, and the correct answer data of the corresponding prediction information PD stored in the prediction information generation database unit 39a, respectively, to update the prediction information generation model database unit 39b and the scenario model database unit 33c. The judgment information generation model database unit 41b and the behavior information generation model database unit 43b are also configured to be re-learned using inverse reinforcement learning (IRL) or the like.

[0077] (Certainty tree database structure) In the edge terminal AI inference system 1, any database unit may have a certainty tree structure as shown in Fig. 13. Fig. 13 shows examples of meta information MD, expanded meta information ED, and predicted information PD stored in the meta information generation DB unit 35a, meta information expansion DB unit 37a, and predicted information generation DB unit 39a. That is, the meta information generation DB unit 35a stores registered data X1 to X3 of the meta information MD and their respective certainty data f X1 , f X2 , f X3 The registered data X1 to X3 are, for example, a traffic cone, a bicycle, a pedestrian, etc. in FIG. 7. The meta information decompression DB unit 37a also stores the registered data Y1 to Y3 of the decompressed meta information and their respective confidence data f Y1 , f Y2 , f Y3 The predicted information generating DB unit 39a stores Z1 to Z3, which are registered data of the predicted information PD, and the respective confidence data fZ1 , f Z2 , f Z3 and are maintained.

[0078] In addition, there are links E that show the relationship between each registered data. X1Y1 For example, link E in Figure 13 X1Y1 includes line segment data indicating that the registered data X1 of the meta information MD contributed to the generation of the registered data Y1 of the decompressed meta information ED by the meta information decompression unit 21, and the link E Y1Z1 includes line segment data indicating that the registered data Y1 of the decompressed meta information ED contributed to the generation of the registered data Z1 of the prediction information PD by the prediction information generating unit 23.

[0079] In such a certainty tree database structure, for each registered data, a "pruning" operation (described later) is performed to delete links with low relevance and the registered data on both ends of those links based on the certainty between the registered data located on both ends of the line segment data of the link. This allows only highly relevant links and registered data to be added sequentially, thereby constructing a database that can compactly hold only highly reliable registered data.

[0080] (Pruning) Link E in Figure 13 XiYj The pruning of the registered data Xi and Yj (i, j are integers from 1 to 3) at both ends of the link is performed by, for example, using the confidence data f Xi and f Yj The evaluation function can be based on the following: Xi and f Yj The link E may be a sum, a product, a maximum value, a minimum value, an arithmetic mean value, a geometric mean value, etc. XiYj is the evaluation function f Xi and f YjWhen the evaluation function value is a monotonically increasing function of , it is preferable to delete data when it is equal to or less than a predetermined value. In this case, when the value of the evaluation function is small, the likelihood that registered data Xi and Yj will be generated simultaneously is low. By performing such pruning every time new data is registered, it is possible to always store only highly reliable registered data in a compact database.

[0081] (Operation of edge terminal AI inference system) The operation of the edge terminal AI inference system 1 will be described below with reference to FIGS.

[0082] 7 and 8, an edge-terminal AI inference system 1 of a vehicle driving system 71 mounted on a vehicle 70 traveling on a public road sequentially generates meta information MD and decompressed meta information ED about obstacles on the road recognized by the vehicle 70, generates prediction information PD about the movement path of the obstacles, generates judgment information JD about dangers posed to the vehicle 70 for each prediction information PD, and generates steering and acceleration / deceleration behavior information AD for the vehicle 70 to avoid the judged dangers. However, this does not mean that the scope of application of the edge-terminal AI inference system 1 is limited to this embodiment.

[0083] (Operation flowchart of the edge terminal AI inference system) FIG. 8 shows an operation flowchart of the edge terminal AI inference system 1 of the vehicle driving system 71.

[0084] In step S101, the input / output control unit 11 of the edge terminal AI inference system 1 receives the sensor information SD from the sensor unit 2 via the input / output unit 1a, and outputs it to the sensor information database unit 32 for storage.

[0085] In step 102, the scenario control unit 15 sets the scenario number (#) of the scenario model to be used to the initial value of 0. This sets, for example, scenario model #0 for a general road scenario.

[0086] In step 103, the model control unit 19 sets the class hierarchy number of the meta information generation model to be used to the initial value of 0. This sets the meta information generation model MM0 for class hierarchy 0.

[0087] In step S104, the meta information generation unit 17 inputs the sensor information SD from the sensor information database unit 32, and generates meta information MD and meta information certainty data MC using the corresponding meta information generation model MM0 in the meta information generation model database unit 35b, and outputs them to the meta information generation database unit 35a.

[0088] In step S105, the meta information decompression unit 21 receives the meta information MD from the meta information generation database unit 35a, and generates decompressed meta information ED and decompressed meta information certainty data EC using the meta information decompression model in the meta information decompression model database unit 37b, and outputs them to the meta information decompression database unit 37a. The detailed operation of the meta information decompression unit 21 will be described later with reference to FIG.

[0089] In step S106, the prediction information generation unit 23 inputs the decompressed meta information ED from the meta information decompression database unit 37a, and uses the corresponding decompressed prediction information generation model EPM in the prediction information generation model database unit 39b to generate prediction information PD and prediction information confidence data PC, and output them to the prediction information generation database unit 39a.

[0090] In step S107, if the model control unit 19 determines that the output prediction information confidence data PC does not satisfy a predetermined threshold, the class hierarchy number of the meta information generation model to be used is incremented by 1 if it is not equal to the maximum value (S107a, S107b), and if the class hierarchy number is equal to the maximum value, the scenario control unit 15 increments the scenario number by 1 (S107a, S107c, S107d) and re-executes the meta information generation of step 104. If the scenario number is already equal to the maximum value at this time, an error is output via an input / output device such as the touch panel 5 connected to the input / output unit 1a, and an end determination is made (S107a, S107c, S107e, S111).

[0091] In step S108, if the judgment information generation unit 25 determines that the output prediction information certainty data PC is greater than or equal to a predetermined threshold, it inputs prediction information PD from the prediction information generation database unit 39a, and uses the judgment information generation model in the judgment information generation model database unit 41b to generate judgment information JD and judgment information certainty data JC, and outputs them to the judgment information generation database unit 41a.

[0092] In step S109, the behavioral information generation unit 27 inputs the judgment information JD from the judgment information generation database unit 41a, and uses the behavioral information generation model in the behavioral information generation model database unit 43b to generate behavioral information AD and behavioral information certainty data AC, and outputs them to the behavioral information generation database unit 43a.

[0093] In step S110, the input / output control unit 11 outputs the behavior information AD output from the behavior information generation unit 27 to the vehicle control ECU 3 via the input / output unit 1a.

[0094] In step S111, the edge terminal AI inference system 1 repeats the above steps S101 to S110 until the operation is completed.

[0095] (Operation of meta information decompression unit) FIG. 9 shows a flowchart of the operation of the meta information decompression unit 21.

[0096] In step S105a, the meta information decompression unit 21 receives the meta information MD from the meta information generating database unit 35a.

[0097] In step S105b, the meta information decompression unit 21 inputs the meta information MD into the meta information decompression model in the meta information decompression model database unit 37b.

[0098] In step S105c, the meta information decompression unit 21 acquires the decompressed meta information ED from the meta information decompression model.

[0099] In step S105d, the meta information decompression unit 21 outputs the acquired decompressed meta information ED to the meta information decompression database unit 37a.

[0100] In step S105e, the meta information decompression unit 21 repeats the above steps S105a to S105d until the operation ends.

[0101] (Example of operation of edge terminal AI inference system) 10 and 11 show examples of the meta information MD, the decompressed meta information ED, and the prediction information PD generated by the operation of the edge terminal AI inference system 1 in the case of FIG.

[0102] 10(A) shows sensor information SD acquired by camera 2 while vehicle 70 is traveling on an open road. The sensor information SD is stored in sensor information database unit 32. The sensor information SD captures images of pedestrian P, bicycle B, traffic cones C, and roadside edge S on the road ahead in the direction of travel of vehicle 70, and also detects acceleration sensor information X0 of pedestrian P traveling diagonally down the road from roadside edge S toward traffic cone C, and acceleration sensor information Y0 of bicycle B traveling straight toward traffic cone C.

[0103] (Example of operation of meta-information generation model MM1 of class hierarchy 1) Figures 10(B) to (C) show a case where sensor information SD is input from the sensor information database unit 32 in step S101 of Figure 8, and in steps S102 to S107, a general road scenario model #0 and a meta information generation model MM1 of class hierarchy 1 are selected based on the predicted information confidence data PC.

[0104] Fig. 10(B) shows meta information MD of class hierarchy 1 generated from sensor information SD in step S104 of Fig. 8. The meta information generation unit 17 detects object P included in the sensor information SD, and classifies the detected objects into "human (P)," "bicycle (B)," "traffic cone (C)," and "road edge (S)" of class hierarchy 0, and further classifies "human (P)" into "male" of class hierarchy 1 based on predicted information certainty data PC, and outputs meta information MD including "male (P)," which is class information of class hierarchy 1 of object P.

[0105] Fig. 10(C) shows the decompressed meta information ED generated from the meta information MD in step S105 of Fig. 8. The meta information decompression unit 21 outputs decompressed meta information ED from the meta information MD "man (P), bicycle (B), traffic cone (C), road edge (S)" and the acceleration sensor information X0, for example, "On an open road, a man (P) is attempting to cross from the road edge (S) toward the traffic cone (C) in front of the traveling bicycle (B).", in which the information about the meta information "man (P)" is further decompressed to "attempting to cross."

[0106] 10(D) shows prediction information PD generated from the decompressed meta information ED in step S106 of FIG. 8. The prediction information generation unit 23 outputs prediction information PD about the movement paths of the "man (P)," "bicycle (B)," and "triangle cone (C)" from the decompressed meta information ED and the acceleration sensor information X0 and Y0. For example, the prediction information PD is output indicating that object P has a "male" walking speed that is higher than the average walking speed of "people" including women, and is intentionally "attempting to cross" rather than unexpectedly jumping out from the "roadside (S)," and therefore is highly likely to move in the future according to acceleration vector X1 perpendicular to the "roadside (S)," which makes the crossing path the shortest. Furthermore, prediction information PD is output indicating that "bicycle (B)" is highly likely to move in the future according to acceleration vector Y1 in order to avoid "man (P)" moving according to acceleration vector X1 and the host vehicle 70 traveling from behind. The edge terminal AI inference system 1 outputs steering and acceleration / deceleration information for the host vehicle 70 to avoid the "man (P)," the "bicycle (B)," and the "triangular cone (C)" based on the prediction information PD.

[0107] (Example of operation of meta-information generation model MM0 for class hierarchy 0) 11B and 11C show a case where the meta information generation model MM0 of class hierarchy 0 is selected from the sensor information SD in FIG. 10 in steps S102 to S107 in FIG.

[0108] Fig. 11(B) shows meta information MD of class hierarchy 0 generated from sensor information SD in step S104 of Fig. 8. The meta information generation unit 17 identifies "human" of class hierarchy 0 based on predicted information certainty data PC, and outputs meta information MD including "person (P)".

[0109] In Figure 11 (C), in the meta information expansion in step S105 of Figure 8, expanded meta information ED is output in which information about the meta information "person (P)" is expanded from the meta information MD "person (P), bicycle (B), traffic cone (C), road edge (S)" and acceleration sensor information X0, for example, to say, "On a public road, a person (P) is attempting to cross from the road edge (S) toward the traffic cone (C) in front of the moving bicycle (B)."

[0110] 11(D), in the generation of prediction information in step S106 in FIG. 8, prediction information PD is output from the decompressed meta information ED, for example, that object P is a "person," and therefore has the average walking speed of a "human," and that object P is intentionally "attempting to cross" rather than suddenly jumping out from the "roadside (S)," and therefore is highly likely to move in the future according to acceleration vector X2 in a direction perpendicular to the "roadside (S)," which will provide the shortest crossing path; and that bicycle B is highly likely to move in the future according to acceleration vector Y2 in order to avoid person P moving according to acceleration vector X2 and host vehicle 70 traveling from behind. Based on the prediction information PD, edge terminal AI inference system 1 outputs steering and acceleration / deceleration information for host vehicle 70 to avoid person P, bicycle B, and traffic cone C.

[0111] According to this embodiment, it is possible to obtain an edge terminal AI inference system 1 having a trained model that can perform automatic driving of a vehicle 70 with high accuracy from sensor information SD.

[0112] The advantages of the configuration of this embodiment will be described below.

[0113] (Separation of AI and DB departments) In the edge terminal AI inference system 1, sensor information SD input from the camera 2 is stored in the sensor information DB unit 32, and meta information MD, decompressed meta information ED, prediction information PD, judgment information JD, and behavior information AD generated or decompressed by the meta information generation unit 17, meta information decompression unit 21, prediction information generation unit 23, judgment information generation unit 25, and behavior information generation unit 27 (hereinafter referred to as AI units) are stored in the corresponding meta information generation DB unit 35a, meta information decompression DB unit 37a, prediction information generation DB unit 39a, judgment information generation DB unit 41a, and behavior information generation DB unit 43a, respectively. Furthermore, prior scenarios and each model are stored in the meta information generation model DB unit 35b, meta information decompression model DB unit 37b, prediction information generation model DB unit 39b, judgment information generation model DB unit 41b, and behavior information generation model DB unit 43b, and are input to the corresponding generation and decompression units of the AI ​​units, respectively. In other words, in the edge terminal AI inference system 1, the input / output data and generative models of the AI ​​unit are stored in the DB unit (hereinafter simply referred to as the DB unit), and all data input / output with the AI ​​unit or external cloud servers, etc. is performed via the DB unit.

[0114] (Database security) In this configuration, all data and generative models are input and output via the DB unit, enabling a security method that protects only the DB unit from external attacks. In other words, since the DB unit manages trained models and data after AI processing, lightweight security processing can be applied to the data management layer, and security processing that generates overhead, such as increasing AI processing, such as encryption or decryption, can be omitted in the AI ​​unit.

[0115] (Digital Twin and Security) That is, in the edge terminal AI inference system 1, data stored in the DB unit can be transferred to the AI ​​unit at any time. For example, for sensor information SD in the physical space, meta information MD can be generated and transferred to the DB unit and then erased from the AI ​​unit without performing security processing in the AI ​​unit. Alternatively, when meta information MD, etc. is transferred or overwritten from the DB unit to the AI ​​unit at regular intervals, there is no need to store the meta information MD, etc. in the AI ​​unit, eliminating the need for security processing in the AI ​​unit while maintaining a certain level of security strength in the data management layer of the DB unit. In other words, the edge terminal AI inference system 1 can configure a cyberspace in which the DB unit reproduces data in the AI ​​unit using meta information MD, etc., and provides a bridge function between the AI ​​unit and the DB unit, eliminating the need for security processing in the AI ​​unit, making security implementation easy.

[0116] (Data Integrity) Furthermore, the above-described configuration of the edge-terminal AI inference system 1 facilitates mutual data monitoring between multiple edge-terminal AI inference systems 1. In other words, the edge-terminal AI inference systems 1 installed in multiple vehicles 70 can configure a blockchain in the DB section of the edge-terminal AI inference system 1 of any vehicle 70 from the DB section of the edge-terminal AI inference system 1 of each vehicle 70, thereby easily realizing mutual data monitoring without the need to add a new processing mechanism. In other words, the blockchain configured in the DB section of the edge-terminal AI inference system 1 of one vehicle 70 can collectively manage the data of the edge-terminal AI inference systems 1 of multiple vehicles 70. Therefore, by monitoring the blockchain in the edge-terminal AI inference system 1 of one vehicle 70, data consistency of the edge-terminal AI inference systems 1 of multiple vehicles 70 can be easily ensured. It is advantageous to provide the database configuring the above-described blockchain in a higher-level base station 80 that communicates with multiple vehicles 70.

[0117] (tamper-resistant) Furthermore, even if data in the DB section of the edge terminal AI inference system 1 of a vehicle 70 is tampered with from outside, the original data in the DB section of the edge terminal AI inference system 1 of the vehicle 70 that is the target of tampering can be reproduced from the DB section of the edge terminal AI inference system 1 of a vehicle 70 other than the vehicle 70 that is the target of tampering, so data tampering can be easily detected by comparing it with the reproduced data.

[0118] (fault tolerant operation) Furthermore, as described above, in the edge terminal AI inference system 1 of the vehicle 70, data from the AI ​​part in the physical space can be reproduced from the DB part that constitutes the cyberspace. As a result, even if part of the AI ​​part of the edge terminal AI inference system 1 of the vehicle 70 becomes unable to operate normally due to a malfunction, soft error, environmental noise, or the like, the data from the DB part that constitutes the cyberspace can be transferred and used again to reproduce the data from the AI ​​part, allowing the vehicle 70 to continue to perform its control operations.

[0119] (Energy saving) Furthermore, in the edge terminal AI inference system 1 of the vehicle 70, if there are no data updates for the physical space from sensor information SD or the like and the most recent data is provided only from the DB unit that constitutes cyberspace, it is possible to stop AI processing operations for the physical space, including sensor operations. The above-mentioned bridge function of the edge terminal AI inference system 1 can reduce the AI ​​processing operation rate for the physical space, thereby contributing to energy savings.

[0120] Each DB unit may include, for example, an environment such as SQL, a cache unit that handles data, and a mechanism that manages them.

[0121] In addition, the edge terminal AI inference system 1 may configure a type of GAN between the trained model of the AI ​​part and the trained model of the DB part, which form a digital twin.

[0122] (Summary of the embodiment) In the edge terminal AI inference system 1, the meta information decompression unit 21 (data decompression unit) uses a meta information decompression model trained using an external language model to generate decompressed meta information ED in which information is decompressed to include additional detailed information regarding the classes contained in the meta information MD.Therefore, the prediction information generation unit 23 (second generation unit) can generate prediction information PD with higher detail from the decompressed meta information ED, which has a higher detail than the meta information MD, and therefore can provide a trained model with high prediction accuracy.

[0123] The meta information generation unit 17 (first generation unit) and the prediction information generation unit 23 (second generation unit) generate meta information certainty data MC and prediction information certainty data PC indicating the reliability of the meta information and prediction information generated when generating the meta information MD and prediction information PD, respectively. Therefore, data indicating the reliability of the meta information and prediction information can be easily obtained, making it possible to easily provide a trained model with high prediction accuracy.

[0124] The edge terminal AI inference system 1 includes in its scenario model database unit 33 (database unit) a plurality of meta information generation models MM0, MM1, etc. (first guess models), a plurality of meta information extension models, and a plurality of keyword prediction information generation models KPM and extension prediction information generation models EPM (second guess models and extension second guess models) that are compatible with a plurality of scenarios that reflect various scenes of public roads, expressways, country roads, etc. Therefore, scenario models #0, #1, #2 that are compatible with the scenario of the sensor information SD are selected from the scenario model database unit 33, making it possible to provide a trained model with high prediction accuracy.

[0125] The meta information generation model (first guess model) is retrained using newly acquired sensor information SD and correct answer data corresponding to the newly generated meta information MD, the meta information extension model is retrained using newly generated meta information MD and correct answer data corresponding to the newly generated extended meta information ED, and the keyword prediction information generation model KPM or extended prediction information generation model EPM (second guess model or extended second guess model) is retrained using newly generated meta information MD or extended meta information ED and correct answer data corresponding to the newly generated prediction information PD.As a result of retraining, the error between the meta information MD, extended meta information ED, and prediction information PD generated by each model and the correct answer data becomes smaller, so that each model generates meta information, extended meta information, and prediction information with a small error from the correct answer data, and therefore, a trained model with high prediction accuracy can be provided.

[0126] According to this embodiment, a trained model can be provided that enables automatic driving of the vehicle 70 to be performed with high accuracy based on sensor information.

[0127] (Example of hardware configuration) Next, a hardware configuration example of an edge-terminal AI inference system 1 according to an embodiment of the present invention will be described. Below, an example of the hardware configuration of an information processing device 50 will be described as an example of the hardware configuration of the edge-terminal AI inference system 1 according to an embodiment of the present invention. Note that the hardware configuration example of the information processing device 50 described below is merely one example of the hardware configuration of the edge-terminal AI inference system 1. Therefore, the hardware configuration of the edge-terminal AI inference system 1 may be such that unnecessary components are deleted from the hardware configuration example of the information processing device 50 described below, or new components are added.

[0128] 12 is a diagram showing the hardware configuration of an information processing device 50 as an example of an edge terminal AI inference system 1 according to an embodiment of the present invention. The information processing device 50 is a chiplet in which multiple chips are integrated into a single package, and includes processors 51a and 51b, memories ROM 52 and RAM 53, a storage device 56, a host bus 58, a bridge chip 59, and an external bus 61.

[0129] (Processor) The processor 51a functions as an arithmetic processing unit and a control unit, and controls the overall operation of the information processing device 50 in accordance with various programs. The processor 51b is a search engine for performing high-speed database searches. Since AI such as neural networks makes heavy use of database calculations, a dedicated accelerator capable of high-speed database searches is effective in increasing the speed of the information processing device 50. The processors 51a and 51b may also be microprocessors or FPGAs (Field Programmable Gate Arrays).

[0130] (Memory) The ROM 52 stores programs such as firmware used by the processors 51a and 51b, calculation parameters, databases, etc. The RAM 53 temporarily stores programs used in the execution of the processors 51a and 51b, parameters that change as appropriate during the execution, cache tags, generated information, confidence data, etc. The RAM 53 may be a cache memory, a tag memory, etc. The memory may also be an on-chip memory integrated into a semiconductor integrated circuit such as a processor. These memories are connected to each other by a host bus 58 that includes a processor bus, etc.

[0131] (storage device) The storage device 56 is a device for holding a database including meta information and confidence data having a tree-like hierarchical structure, models, etc. The storage device 56 may include a storage medium, a recording device for recording data in the storage medium, a reading device for reading data from the storage medium, and a deletion device for deleting data recorded in the storage medium. The storage device 56 includes, for example, a magnetic disk device such as an HDD (Hard Disk Drive), a flash storage device using an SSD (Solid State Drive), etc. The storage device 56 holds programs executed by the processors 51a, 51b, etc., and various data.

[0132] (host bus) The host bus 58 is connected to an external bus 61 via a bridge chip 59, which will be described later. Separating the host bus 58, which connects multiple chips that operate at high speed within the package, from the external bus 61, which is connected to external input / output devices that operate at low speeds, enables the host bus 58 to operate at high speed, and is therefore effective in increasing the speed of the information processing device 50. Note that, for ease of implementation, the functions of the host bus 58, bridge chip 59, and external bus 61 may be implemented on a single bus rather than being separated.

[0133] (Bridge chip) The bridge chip 59 has a function that allows the information processing device 50 to send and receive hierarchical meta-information, confidence factor data, etc. to and from an external AI. The bridge chip 59 includes a UCIe interface unit 59a for interconnecting chiplets within a package, a meta-information generation unit 59b including the meta-information generation unit 17, a DDR interface and file system controller unit 59c, and an input / output unit and communication interface unit 59d. The DDR interface and file system controller unit 59c controls data read and write to the DDR memory included in the RAM 53. The input / output unit and communication interface unit 59d includes various interfaces, for example, composed of input / output units and communication devices, for connecting the information processing device 50 to the external bus 61 and a network, and may support either wireless or wired protocols. Note that these functions may be implemented separately without using the bridge chip 59.

[0134] (external bus) The external bus 61 includes a PCI bus and the like, and is connected to external input / output devices and the like to the vehicle control ECU 3 of the vehicle 70. Input devices connected to the external bus 61 include various sensors, a GNSS or GPS device, a keyboard, a touch panel, a touch pad, a pointing device such as a mouse, and the like. Output devices include display devices such as a touch panel display, a CRT (Cathode Ray Tube) display, a liquid crystal display (LCD), an OLED (Organic Light Emitting Diode), a lamp, and the like, and audio output devices such as a speaker, for example.

[0135] Modifications of embodiments of the present invention will be described below.

[0136] (About the scenario model) In the edge terminal AI inference system 1 according to the embodiment of the present invention, the scenario models held in the scenario model database unit 33 are not limited to scenario models for general roads, expressways, country roads, etc. For example, by providing various scenario models that reflect external environmental conditions such as weather, time, and the region in which the vehicle is traveling, or internal environmental conditions such as the remaining fuel level as measured by a sensor on the vehicle or the driver's physical condition and fatigue level as measured by a camera, the model can be adapted to more scenarios of the sensor information SD, thereby improving the prediction accuracy of the trained model.

[0137] (About the predictive information generation model) In the edge terminal AI inference system 1 according to an embodiment of the present invention, the prediction information generation model used by the prediction information generator 23 is not limited to the keyword prediction information generation model KPM and the extended prediction information generation model EPM, but may be one, or may be three, four, or more models depending on the level of detail of the extended meta information to be input. For example, the prediction information generation model can be simplified by using only the keyword prediction information generation model KPM as a model, or by using a single prediction information generation model obtained by machine learning that inputs known meta information and / or extended meta information and outputs known prediction information.

[0138] (About confidence data) In the edge terminal AI inference system 1 according to an embodiment of the present invention, the certainty data for selecting the meta information and the prediction information generation model (first and second generation models) is not limited to the prediction information certainty data PC. For example, the judgment information certainty data JC or the action information certainty data AC may be used. By using the certainty data of data generated later in the sequential stage processing to select the generation model, a wider range of optimization is achieved, from the meta information MD to the generation section of the judgment information JD or the action information AD, thereby improving the prediction accuracy of the trained model over a wider range.

[0139] Furthermore, the certainty data for selecting the meta information and the prediction information generation model (first and second generation models) may be determined by a multi-frame analysis based on the sensor information SD. The certainty data based on the multi-frame analysis allows the reliability of the prediction information PD, etc. to be determined over a plurality of frame analyses, and therefore the prediction accuracy of the selected meta information and prediction information generation model can be improved stably over a plurality of frames.

[0140] (Regarding the re-learning section) In the edge-terminal AI inference system 1 according to an embodiment of the present invention, the re-learning unit 29 may perform re-learning, such as inverse reinforcement learning, on the judgment information generation model and the behavior information generation model. This improves the prediction accuracy of each generation model. Alternatively, the re-learning unit 29 may be omitted. By omitting the re-learning unit 29, the edge-terminal AI inference system 1 can easily implement connections between each generation database, and can reduce the implementation area and power consumption, allowing it to be provided at low cost.

[0141] (About the meta information expansion model) In the edge terminal AI inference system 1 according to an embodiment of the present invention, the metainformation decompression unit 21 may directly acquire decompressed metainformation ED, etc. from an external language model server including chatGPT, etc. via the communication interface unit 1b, and output the decompressed metainformation ED, etc. to the metainformation decompression database unit 37a, without using the metainformation decompression model in the metainformation decompression model database unit 37b. Because the edge terminal AI inference system 1 does not require a ROM, etc. for storing the metainformation decompression model as firmware for the metainformation decompression model database unit 37b, the mounting area and power consumption can be reduced, and the system can be provided at low cost.

[0142] The present invention is not limited to the above-described embodiment, and includes various modifications to the above-described embodiment without departing from the spirit of the present invention. [Explanation of symbols]

[0143] 1. Edge terminal AI inference system 2 Sensor section 17 Meta information generation unit (first generation unit) 19 Model control section 21 Meta information decompression unit (data decompression unit) 23 Prediction information generation unit (second generation unit) 33 Scenario model database section (database section) 35b Meta information generation model database section 37b Meta information expansion model database section 39b Prediction information generation model database section ED extended meta information EPM Growth Prediction Information Generation Model (Growth Second Estimation Model) KPM Keyword Prediction Information Generation Model (Second Guessing Model) MC Meta information confidence data MD Meta Information MM0, MM1 Meta-information generation model (first guess model) PC prediction information confidence data PD forecast information SD sensor information

Claims

1. An edge terminal AI inference system in a local edge network including an edge terminal, a first generation unit that generates meta information including information about a class of identifiable objects using a first estimation model based on sensor information input to the edge terminal; a data decompression unit that generates decompressed meta information by decompressing the information based on the meta information using a meta information decompression model; a second generation unit that generates prediction information about the meta information using a second estimation model based on the meta information or the decompressed meta information; the first estimation model includes a model machine-learned using known sensor information as input and known meta-information including information about the object class as output; the meta information decompression model includes a model that has been machine-trained using known meta information as input and decompressed meta information acquired from an external language model as output, the second estimation model includes a model machine-learned using known meta information or decompressed meta information as input and known prediction information as output, The second generation unit generates the prediction information based on the reliability of the generated meta information or prediction information, an edge terminal AI inference system.

2. the first generation unit generates meta information certainty data indicating reliability of the meta information; the second generation unit generates prediction information certainty data indicating reliability of the prediction information; The edge terminal AI inference system according to claim 1 , wherein the second generation unit generates the prediction information based on the meta information certainty data or the prediction information certainty data.

3. Further comprising a database unit, the database unit includes a plurality of the first inference models, a plurality of the meta information decompression models, and a plurality of the second inference models that are compatible with a plurality of scenarios; The edge terminal AI inference system of claim 1 , wherein the first inference model, the meta-information decompression model, and the second inference model are each selected based on the scenario.

4. The edge terminal AI inference system according to any one of claims 1 to 3, wherein the first inference model, the meta information extension model, and the second inference model are re-learned using the newly acquired sensor information, the newly generated meta information, the extension meta information, and the prediction information, as well as the corresponding correct answer data.

Images