Information processing device and program

Abstract

To provide a technique capable of efficiently controlling a vehicle.SOLUTION: An information processing device includes a first system and a plurality of second systems. The first system has a device information acquisition section for acquiring information output from a first device installed in a prescribed moving body as device information, and an instruction information generation section for generating instruction information to determine control information for controlling a second device of the moving body on the basis of the device information. Each of the plurality of second systems has an instruction information acquisition section for acquiring at least a part of the instruction information generated by the instruction information generation section, and a control information generation section for generating the control information using at least a part of the instruction information.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present disclosure relates to an information processing device and a program. [Background technology]

[0002] Autonomous driving technology for automobiles has been one of the technologies that has attracted attention in recent years, and many methods related to the autonomous driving technology have been attempted. For example, Patent Document 1 discloses one aspect of such a method, in which a machine learning algorithm is applied to the control of an autonomous vehicle (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2020-083309 Summary of the Invention [Problem to be solved by the invention]

[0004] Here, in controlling an autonomous vehicle, it is necessary to move safely in an environment with a large amount of information including various uncertain factors such as the road traffic environment, and therefore, unless highly accurate calculation results are output in the above-mentioned environment, there is a possibility that the operation of the vehicle will be hindered. In this regard, many of the conventional technologies, including the above-mentioned Patent Document 1, simply apply machine learning algorithms and machine learning models to functional parts such as recognition and planning that are necessary to configure an autonomous driving function, and there is room for improvement in the system configuration.

[0005] The present disclosure has been made in light of the above circumstances, and aims to provide a technology that can control a moving object more accurately and efficiently.

[0006] In order to achieve the above object, a first aspect of the present disclosure provides: An information processing device including a first system and a plurality of second systems, The first system comprises: a device information acquisition unit that acquires information output from a first device installed in a predetermined mobile object as device information; an instruction information generation unit that generates instruction information for determining control information for controlling a second device of the moving object based on the device information; Equipped with Each of the plurality of second systems includes: an instruction information acquisition unit that acquires at least a part of the instruction information generated by the instruction information generation unit; a control information generating unit that generates the control information by using at least a part of the instruction information; The information processing device includes:

[0007] A program according to an aspect of the present disclosure is also provided as a program corresponding to an information processing device according to an aspect of the present disclosure. [Effects of the Invention]

[0008] According to the present disclosure, it is possible to provide a technology that can efficiently control a moving object. [Brief explanation of the drawings]

[0009] [Figure 1] 1 is a diagram illustrating an example of an information processing system including a vehicle system according to an embodiment of the present disclosure. [Figure 2] FIG. 1 is a diagram illustrating an example of a hardware configuration of an in-vehicle device that constitutes an information processing system according to an embodiment of the present disclosure. [Figure 3] FIG. 1 is a diagram illustrating an example of a functional configuration of an in-vehicle device that configures an information processing system according to an embodiment of the present disclosure. [Figure 4] FIG. 2 is a diagram schematically illustrating processing executed in the information processing system of the present disclosure. [Figure 5] FIG. 2 is a diagram illustrating an example of the flow of a Drive AGI process among the processes executed by an in-vehicle device constituting an information processing system according to an embodiment of the present disclosure. [Figure 6]1 is a diagram illustrating an example of the flow of a decoder process among processes executed by an in-vehicle device constituting an information processing system according to an embodiment of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0010] [Embodiment] FIG. 1 is a diagram showing an example of an information processing system (hereinafter referred to as "this system") including a vehicle system according to an embodiment of a moving body of the present disclosure.

[0011] 1, a vehicle system S according to an embodiment of the present disclosure includes an on-board device 1, a vehicle sensor 10, an HMI (Human Machine Interface) 20, and a control ECU (Electronic Control Unit) 30. These devices and equipment are connected via a predetermined network such as a Controller Area Network (CAN) or Ethernet. Here, the vehicle on which the vehicle system S is mounted may include any vehicle, moving object, etc., such as an automobile powered by electricity or gasoline, or an autonomous vehicle.

[0012] The vehicle sensor 10 is a device installed in a vehicle (mobile body) and includes various sensors for detecting the external environment around the vehicle (an environment that may include other vehicles, pedestrians, structures, road shapes, etc.) and on-board instruments that are various sensors for detecting the operating status of traveling equipment related to the vehicle's traveling (the behavior of the mobile body), such as the steering wheel, wheels, and motor. Here, the external environment around the vehicle may include, for example, traffic participants (other vehicles, pedestrians, etc.), buildings such as commercial facilities, traffic signs installed on the side of the road, road markings formed on the road surface, dividing lines, traffic lights, utility poles, guardrails, animals, fallen objects, private property such as parking lots, etc. The external environment around the vehicle may also include, for example, weather, information about the road surface (roads, sidewalks, etc.) on which the mobile body can travel, and its condition (e.g., whether the road surface is wet or uneven). 1, the vehicle sensor 10 includes a camera (front camera) that is installed to be able to capture an image in front of the vehicle, a camera (side camera) that is installed to be able to capture an image on the side of the vehicle, a camera (rear camera) that is installed to be able to capture an image behind the vehicle, millimeter-wave radar, ultrasonic radar, a LiDAR (Light Detection and Ranging) sensor, an acceleration sensor, a GNSS (Global Navigation Satellite System), and an external microphone (sound collection device). Furthermore, the operating status of the vehicle's running equipment refers to the operating information of each device necessary for vehicle movement, such as the rotation speed of the engine, the steering angle, and the operating status of lights, and the vehicle sensor 10 includes on-board instruments and the like that acquire this information.

[0013] Here, the camera is configured, for example, with a camera using a CCD (Charge Coupled Device), a CMOS (Complementary Metal Oxide Semiconductor), etc. In the present disclosure, a total of multiple cameras may be installed in the front, sides, rear, and interior of the vehicle.

[0014] In addition, the external microphone (sound collection device) is composed of a general-purpose microphone, etc., and is used to acquire information about sounds emitted by objects outside the vehicle, including, for example, sirens of ambulances and police cars, human voices, etc.

[0015] The HMI 20 is a device that presents various information to the driver and passengers of the vehicle and accepts various input operations. Specifically, as shown in Fig. 1, the HMI 20 includes, for example, a display, operation buttons, a microphone, various navigation systems, a speaker, etc. The HMI 20 is, for example, an IVI (In-Vehicle Infotainment) system, but is not limited to this.

[0016] The control ECU 30 is connected to the in-vehicle device 1, transmits and receives various information, and controls driving devices related to the driving of the vehicle (movement of the moving body). Specifically, as shown in Fig. 1, the control ECU 30 includes, for example, individual ECUs that perform various controls, and controls driving devices such as the brake, accelerator, steering, lights such as turn signals and lights, power unit, transmission, and suspension.

[0017] 1, the system may include a vehicle system S (on-board device 1) managed by a driver of the vehicle or the like, and a server 2 managed by an administrator of the system or the like. The vehicle system S and the server 2 may be connected to each other via a predetermined network N such as the Internet. The server 2 acquires various information related to vehicle operation periodically transmitted from the vehicle system S (particularly the on-board device 1) and is used to manage the various information.

[0018] FIG. 2 is a diagram illustrating an example of a hardware configuration of an in-vehicle device 1 that constitutes an information processing system according to an embodiment of the present disclosure.

[0019] As shown in FIG. 2, the in-vehicle device 1 includes a control unit 41, a ROM (Read Only Memory) 42, a RAM (Random Access Memory) 43, a bus 44, an input / output interface 45, a storage unit 46, and a communication unit 47.

[0020] The control unit 41 is configured by a microcomputer or the like including a CPU, a GPU, an FPGA (Field Programmable Gate Array), an NPU (Neural Processing Unit), a semiconductor memory, etc. The control unit 41 executes various processes according to a program recorded in a ROM 42 or a program loaded from the storage unit 46 to a RAM 43. The RAM 43 also stores information and the like necessary for the control unit 41 to execute various processes, as appropriate.

[0021] The control unit 41, ROM 42, and RAM 43 are interconnected via a bus 44. An input / output interface 45 is also connected to this bus 44. The input / output interface 45 is connected to the vehicle sensors 10, the HMI 20, the control ECU 30, a storage unit 46, a communication unit 47, and the like.

[0022] The storage unit 46 is configured with a hard disk drive (HDD), a solid state drive (SSD), etc., and stores various information. For example, the storage unit 46 stores various programs and the like required for executing various processes related to this system.

[0023] The communication unit 47 controls communication between other hardware and the like via a network N including the Internet.

[0024] 3 is a diagram illustrating an example of a functional configuration of an in-vehicle device constituting an information processing system according to an embodiment of the present disclosure. Specifically, the functions of the control unit 41 of the in-vehicle device 1 in this embodiment are roughly divided into two functions: a Drive AGI 100, and a Language Decoder 140a, a Visual Decoder 140b, and a Control Decoder 140c (hereinafter collectively referred to as "Decoder 140").

[0025] Drive AGI 100 performs calculations and inference processing related to large-scale multimodal models (trained models), such as large-scale language models, which perform integrated recognition and behavioral decisions for the vehicle. Specifically, Drive AGI 100 makes comprehensive decisions based on information acquired by external sensors such as cameras installed in the vehicle and on-board instruments, speech information and operational input information from people such as the vehicle's driver and passengers, and background knowledge about the road traffic environment, and provides appropriate instruction information to Decoder 140.

[0026] In response to this, Decoder 140 executes arithmetic processing that executes calculations using a machine learning model based on limited information. Decoder 140 outputs control information for controlling vehicle devices based on instruction information output by Drive AGI 100. In this system, the arithmetic processing and output processing by Decoder 140 are configured to be lightweight compared to the arithmetic processing and output processing by Drive AGI 100, and each Decoder outputs control information to each device in the vehicle, so that vehicle control can be achieved in the time (short time) necessary and sufficient for controlling an autonomous vehicle.

[0027] As shown in FIG. 3 , the control unit 41 of the in-vehicle device 1 executes various programs, causing the Drive AGI 100 and the Decoder 140 to function. A model information DB 300 and a map information DB 400 are provided in one area of ​​the storage unit 46 of the in-vehicle device 1. The model information DB 300 stores trained models for outputting instruction information (described later) in response to input information from various sensors, and trained models for outputting control information (described later) in response to input information from the various sensors. The map information DB 400 stores map information, etc. Map information is, for example, information displaying information such as latitude and longitude related to roads, buildings, etc. in a two-dimensional or three-dimensional format. It may be general-purpose map information used for navigation, or a map such as a high-definition map (HD map) that includes legal information such as traffic light locations, the number of lanes, speed limits, and stop line locations. In this system, the map information stored in the map information DB 400 is used as appropriate in situations where various processes (described later) are required.

[0028] The following describes the functional configuration related to the Drive AGI 100. The Drive AGI 100 is composed of a device information acquisition unit 120 and an instruction information generation unit 121.

[0029] The device information acquisition unit 120 acquires and manages device information output from the vehicle's devices. The device information includes sensor information acquired from the sensors constituting the vehicle sensor 10 and HMI information output from the HMI 20. The HMI information includes information about video, audio, map information, information about objects on the road, route information, traffic information, information about questions for the occupant, and information about answers to the questions from the occupant. The device information acquisition unit 120 may acquire all or part of the device information. It is preferable that the device information acquisition unit 120 acquires at least information necessary for the vehicle's driving, i.e., information acquired by on-board instruments from the vehicle's driving devices (vehicle speed, steering angle, etc.), and sensor information detecting the environment around the vehicle (video data, radar distance measurement data, etc.). Note that the device information (e.g., video data) may be tokenized information so that it can be processed by a trained model stored in the model information DB 300.

[0030] The instruction information generation unit 121 generates information (hereinafter referred to as "instruction information") to be processed by the Decoder 140 based on the device information acquired by the device information acquisition unit 120 and the trained model stored in the model information DB 300. Here, the instruction information is typically information related to a medium- to long-term action plan or driving instructions for the vehicle, and is an instruction related to the vehicle's behavior. Furthermore, the instruction information may include, for example, the reason for the decision to issue the instruction. For example, the instruction information may be instruction information having meanings such as "The traffic light is red, please stop the vehicle," "There is a sharp curve ahead, so please slow down gradually," "You will turn left ahead, please turn on your blinker," or "There is an intersection ahead, please go right," output as an embedding vector as a latent representation. Note that, according to this example of instruction information, for example, information "Please stop the vehicle" is an instruction related to the vehicle's behavior, and information "The traffic light is red" is the reason for the decision to issue the instruction. The instruction information may include, for example, information of the above content. In this way, the present system can realize more appropriate and efficient operation control in Decoder 140 by outputting instructions regarding medium- to long-term action plans or operation plans based on various information obtained by the various sensors that make up vehicle sensor 10 and outputting them to Decoder 140.

[0031] Here, the content of the instruction information will be explained in more detail. Specifically, for example, while the vehicle is traveling on a lane of a highway, the instruction information output by Drive AGI 100 may include information such as "move to the left lane to exit the highway 1 km ahead." Then, the Control Decoder 140c outputs specific control information, including information regarding the action the vehicle should take in the future, as a result of processing based on the instruction information, such as "(1) flash the left turn signal, (2) change lanes 3 seconds after the flashing, (3) decelerate to 90 km / h after changing lanes." In other words, by including information indicating that the vehicle is exiting the highway in the instruction information during processing by the Decoder 140, a processing result such as (3) decelerate to 90 km / h after changing lanes can be derived, which contributes to more comfortable and safer travel for the occupants.

[0032] This instruction information includes, for example, information on a more accurate mid- to long-term action plan or operation instructions obtained by Decoder 140 over a longer calculation time based on a larger amount of information (e.g., device information, described later) compared to the information processed by Decoder 140. In contrast, the processing by Decoder 140, described later, generates more specific control instructions, etc. regarding vehicle operation and device control for a shorter period than the mid- to long-term action plan or operation plan generated by Drive AGI 100, based on limited information (e.g., part of the device information, described later) and / or at least part of the generated instruction information. By combining and executing these two control processes with different roles, it is possible to safely and efficiently achieve autonomous driving control and device control for a vehicle.

[0033] Here, the information acquired by the device information acquisition unit 120 and the information generated by the instruction information generation unit 121 will be described in a series of steps. As an example, assume that a vehicle is traveling straight toward a crossroads located 100 meters ahead, and a user inside the vehicle has requested to turn right. In this situation, the device information acquisition unit 120 acquires, from the vehicle sensors, video image information in front of the vehicle from the front camera, vehicle speed information, operation information of safety devices such as lights, and output value information of the engine, and also acquires, from the HMI 20, voice information related to the above-mentioned instruction from the user. Based on the various information acquired by the device information acquisition unit 120, the instruction information generation unit 121 generates instructions such as "slow down the vehicle so that its speed is less than 5 km / h 100 m ahead," "activate the right turn lights," "perform a right turn if there are no approaching vehicles in the oncoming lane and no obstacles in the direction of travel of the vehicle," "output a voice message to the user indicating that a right turn will be made," and "display light activation information to the user when the lights are activated," and outputs these instructions to the Decoder 140.

[0034] The functional configuration related to the Language Decoder 140a will be described. The Language Decoder 140a controls the voice and text for the occupant output from the HMI 20 (vehicle equipment) based on at least a part of the instruction information. The Language Decoder 140a includes an instruction information acquisition unit 160a, a control information generation unit 161a, and a control information output unit 162a.

[0035] The instruction information acquisition unit 160a acquires and manages at least a part of the instruction information generated by the instruction information generation unit 121 of the Drive AGI 100. The instruction information acquisition unit 160a may acquire all of the instruction information, or may acquire a part of the instruction information that is pre-assigned to the Language Decoder 140a.

[0036] The control information generation unit 161a generates information (hereinafter referred to as "control information a") on specific control instructions for controlling the voice for the occupant to be output from the HMI 20 (vehicle equipment) based on the instruction information acquired by the instruction information acquisition unit 160a and the contents of the trained model stored in the model information DB 300. Here, the control information a is, for example, control information for the HMI 20. Specifically, the control information a is information indicating the content of the voice for the occupant to be output from the HMI 20, and in the vehicle of this embodiment, it is information about objects on the road, information about the route, traffic information, etc. Furthermore, the control information a may include, for example, information about questions for the occupant and information about answers to the questions from the occupant. The HMI 20 can achieve appropriate voice control by outputting the voice based on the control information a.

[0037] The control information output unit 162a provides the HMI 20 of the vehicle system S with the control information a generated by the control information generation unit 161a.

[0038] The Language Decoder 140a may acquire at least a portion of the device information (for example, HMI information) and generate the control information a based on the acquired device information and instruction information. In this case, it is preferable that the Language Decoder 140a acquires only a portion of the device information that is effective for generating the control information a, so that the amount of information used for generating the control information a is reduced.

[0039] Furthermore, the language decoder 140a may not only output the control information a to the HMI 20, but may also directly control the HMI 20 using the control information a. In other words, the language decoder 140a may function as a control unit for the HMI 20.

[0040] The functional configuration related to the visual decoder 140b will be described. The visual decoder 140b controls the video images output from the HMI 20 (vehicle equipment) based on at least a part of the instruction information. The visual decoder 140b includes an instruction information acquisition unit 160b, a control information generation unit 161b, and a control information output unit 162b.

[0041] The instruction information acquisition unit 160b acquires and manages at least a part of the instruction information generated by the instruction information generation unit 121 of the Drive AGI 100. The instruction information acquisition unit 160b may acquire all of the instruction information, or may acquire a part of the instruction information that is pre-assigned to the Visual Decoder 140b.

[0042] The control information generation unit 161b generates control information b based on the instruction information acquired by the instruction information acquisition unit 160b and the contents of the trained model stored in the model information DB 300. Here, the control information b is, for example, control information for the HMI 20. Specifically, the control information b is information indicating the contents of the moving image for the occupant output from the HMI 20. In the vehicle of this embodiment, the control information b includes map information, information on moving images outside the vehicle acquired from the vehicle sensor 10, the vehicle's operating status along the travel route, the recognition status of surrounding objects, and vehicle dynamics / telemetry information such as vehicle speed and fuel efficiency. Furthermore, the control information b may include, for example, information on questions to the occupant and information on answers to the questions from the occupant. The HMI 20 can realize appropriate control of the moving image by outputting the moving image based on the control information b.

[0043] The control information output unit 162b provides the HMI 20 of the vehicle system S with the control information b generated by the control information generation unit 161b.

[0044] The Visual Decoder 140b may acquire at least a portion of the device information (e.g., HMI information) and generate the control information b based on the acquired device information and instruction information. In this case, it is preferable that the Visual Decoder 140b acquires only a portion of the device information that is effective for generating the control information b, so that the amount of information used for generating the control information b is reduced.

[0045] Furthermore, the Visual Decoder 140b may not only output the control information b to the HMI 20, but also directly control the HMI 20 using the control information b. That is, the Visual Decoder 140b may function as a control unit for the HMI 20.

[0046] The functional configuration related to the Control Decoder 140c will be described. The Control Decoder 140c controls devices (such as brakes, accelerators, and steering) related to the movement of the vehicle (the behavior of the moving object) based on at least a part of the instruction information. The Control Decoder 140c includes an instruction information acquisition unit 160c, a control information generation unit 161c, and a control information output unit 162c.

[0047] The instruction information acquisition unit 160c acquires and manages at least a portion of the instruction information generated by the instruction information generation unit 121 of the Drive AGI 100. The instruction information acquisition unit 160c may acquire all of the instruction information, or may acquire a portion of the instruction information that is pre-allocated to the Drive Decoder 140c.

[0048] The control information generation unit 161c generates control information c based on the instruction information acquired by the instruction information acquisition unit 160c and the contents of the trained model stored in the model information DB 300. Specifically, the control information c is information for controlling devices related to the movement of the moving object. In the case of the vehicle in this embodiment, the control information c includes vehicle speed information, vehicle acceleration / deceleration information, information related to a behavior plan such as a movement trajectory to be followed by the vehicle, and information related to the direction in which the vehicle should travel. The control information c may also include information related to driving instructions for the driver, including, for example, a predicted route of the vehicle and instructions related to special actions. The predicted route is, for example, a predicted future route of the vehicle calculated or planned based on the device information. The special action is, for example, specific instruction information related to operations such as activation of left and right turn signals (including flashing hazard lights), sudden deceleration, a gear shift instruction, and a horn instruction. Each control ECU 30 controls the vehicle based on the control information c, thereby achieving appropriate autonomous driving of the vehicle.

[0049] The control information output unit 162c provides the control information c generated by the control information generation unit 161c to the control ECU 30 of the vehicle system S and the like.

[0050] The control information generating unit 161c may acquire at least a portion of the device information (for example, sensor information) and generate the control information c based on the acquired device information and instruction information. In this case, it is preferable that the control information generating unit 161c acquires only a portion of the sensor information that is effective for generating the control information c, so that the amount of information used for generating the control information c is reduced. Specifically, it is preferable that the control information generating unit 161c acquires at least information necessary for the vehicle to travel, that is, information acquired by on-board instruments from the vehicle's travel devices (vehicle speed, steering angle, etc.) and sensor information that detects the environment around the vehicle (video data, distance measurement data by radar, etc.).

[0051] In this case, the output cycle (e.g., 20 Hz) at which Drive AGI 100 outputs instruction information is set longer than the output cycle (e.g., 60 Hz) at which Control Decoder 140c outputs control information. This allows the Control Decoder 140c, which controls the vehicle's nearest devices and solves operational issues, to maintain the processing speed for near-term vehicle control, while Drive AGI 100 performs highly accurate recognition and judgment in the medium to long term, thereby ensuring overall high accuracy and processing time that can withstand autonomous driving control.

[0052] Furthermore, if the control decoder 140c is unable to acquire instruction information, it may generate the control information c based only on the acquired device information. This allows the control decoder 140c to control the vehicle's driving even when instruction information cannot be acquired. In particular, it is preferable that the control decoder 140c generates the control information c based only on the device information so as to enable short-term autonomous driving control, such as automatic braking, emergency hazard warning, lane / road departure prevention steering, and ACC. For this reason, it is preferable that the control decoder 140c acquires sensor information (e.g., camera images) necessary for autonomous driving control.

[0053] Furthermore, the Control Decoder 140c may not only output the control information c to the control ECU 30, but also directly control devices related to the movement of the vehicle (such as an accelerator) using the control information c. In other words, the Control Decoder 140c may function as the ECU 30.

[0054] As described above, each Decoder 140 may acquire all of the instruction information and generate control information, or may acquire only a portion of the instruction information and generate control information. In the latter case, the instruction information acquired by each Decoder 140 may be different from the others, or may share a portion of the instruction information. For example, the instruction information may have four parts, with the first part being input to three Decoders 140 in common, the second part being input to Language Decoder 140a, the third part being input to Visual Decoder 140b, and the fourth part being input to Control Decoder 140c. The part of the instruction information acquired by each Decoder 140 may be preset or may be specified by a special token. In this way, by inputting only a portion of the instruction information to each Decoder 140, the Decoder 140 does not need to perform unnecessary calculations, thereby ensuring sufficient calculation speed.

[0055] In addition, in this embodiment, the in-vehicle device 1 includes three decoders 140, but may include two or four or more decoders 140. For example, another decoder 140 may be a decoder 140 that outputs a control signal for controlling the in-vehicle interior (air conditioner, audio, seat reclining, etc.) based on instruction information.

[0056] The hardware configuration of the server 2 is not described here because it can be basically the same as the hardware configuration of the in-vehicle device 1. Cooperation between such various hardware and software enables the in-vehicle device 1 and the like to execute various processes described below.

[0057] Here, the processing executed in the information processing system according to one embodiment of the present disclosure will be described. Fig. 4 is a diagram schematically illustrating the processing executed in the information processing system. Fig. 5 is a diagram illustrating an example of the flow of Drive AGI processing, which is one of the processing executed in the in-vehicle device constituting the information processing system according to one embodiment of the present disclosure.

[0058] In step S1, the device information acquisition unit 120 acquires and manages device information (sensor information and HMI information) from the vehicle sensor 10 and the HMI 20.

[0059] In step S2, the instruction information generation unit 121 generates and outputs instruction information based on the device information acquired by the device information acquisition unit 120 and the contents of the trained model stored in the model information DB 300. This completes the Drive AGI processing of the in-vehicle device 1.

[0060] FIG. 6 is a diagram illustrating an example of the flow of a decoder process among the processes executed by an in-vehicle device constituting an information processing system according to an embodiment of the present disclosure.

[0061] In step S21, the instruction information acquisition unit 160a acquires and manages at least a portion of the instruction information a assigned to the Language Decoder 140a from among the instruction information generated by the instruction information generation unit 121. The instruction information acquisition unit 160b acquires and manages at least a portion of the instruction information b assigned to the Visual Decoder 140b from among the instruction information generated by the instruction information generation unit 121. The instruction information acquisition unit 160c acquires and manages at least a portion of the instruction information c assigned to the Control Decoder 140c from among the instruction information generated by the instruction information generation unit 121.

[0062] In step S22, the control information generation units 161a, 161b, and 161c generate control information a, b, and c regarding specific control instructions for controlling the vehicle's equipment based on the instruction information acquired by the instruction information acquisition units 160a, 160b, and 160c and the contents of the trained model stored in the model information DB300.

[0063] In step S23, the control information output units 162a and 162b provide the control information a and b generated by the control information generation units 161a and 161b, respectively, to the HMI 20. Furthermore, the control information output unit 162c provides the control information c generated by the control information generation unit 161c to the control ECU 30 of the vehicle system S, etc. This completes the decoder process of the in-vehicle device 1.

[0064] The above describes one embodiment of the present disclosure, but the present disclosure is not limited to the above-described embodiment, and modifications and improvements within the scope of achieving the object of the present disclosure are included in the present disclosure.

[0065] [Other embodiments] In the above-described embodiment, the vehicle is described as a general-purpose autonomous vehicle, but this is not limited thereto. Vehicles to which the present system can be applied may include any type of mobile object, regardless of shape or power source, such as automobiles, trucks, motorcycles, railroad cars, bicycles, robots, AGVs (Automatic Guided Vehicles), and drones. Furthermore, the information processing device or information processing system according to the present system does not need to function independently as an information processing device, and may be provided, for example, as an integrated part of a vehicle (mobile object).

[0066] In the above embodiment, the device information acquisition unit 120 has been described as acquiring only information related to a camera installed to capture an image ahead of the vehicle, a millimeter-wave radar or ultrasonic radar installed to acquire information about the front, and on-board instruments as sensor information. However, this is not limited to this. The administrator of the system can freely design which types of sensor information to use as sensor information. The same applies to HMI information. Specifically, for example, when the vehicle is backing up, the sensor information may include all or part of an image acquired by a camera capable of capturing an image behind the vehicle, various types of information acquired by a millimeter-wave radar or ultrasonic radar installed to acquire information about the rear, and an image acquired by a camera capable of capturing an image to the side.

[0067] Furthermore, the present system may change the selection of sensors to acquire as sensor information depending on the vehicle situation. Specifically, for example, the present system normally acquires only sensor information from a front camera, but may acquire images from left, right, or rear cameras (e.g., in the direction of the lane change) as sensor information when changing lanes. The lane change referred to here may include vehicle operations such as simple lateral movement, left or right turns, etc. The advantage of the Decoder 140 of the present system acquiring only instruction information rather than device information and executing various processes is that by acquiring information only from instruction information and performing processing related to the vehicle operation, the Decoder 140 can process faster than when acquiring and processing device information.

[0068] Although not described in the above embodiment, the trained model used by Drive AGI 100 or Decoder 140 may be updated as appropriate before or after generating various types of information. For example, the in-vehicle device 1 may acquire a newly trained trained model using output instruction information, control information, etc., and update the content of the trained model stored in the model information DB 300. Note that the training method used here may be, for example, a method classified into various deep learning methods such as DNN (Deep Neural Network), or a combination thereof.

[0069] Although the above-described embodiment has been described simply, it is assumed that the information stored in the model information DB 300 has undergone learning processing in advance by the in-vehicle device 1 or other hardware, etc. Specifically, the information stored in the model information DB 300 is model information that has been sufficiently adjusted for application to autonomous driving by, for example, learning a wide range of information including various videos, text, natural language, etc., and labeling separately collected images of the vehicle traveling with information related to vehicle control, supplementary information, etc. Note that the learning data, etc. used for these learnings do not necessarily have to be acquired by the vehicle to which autonomous driving is applied, but may be learning data, etc., from another vehicle or data generated by a method such as simulation.

[0070] Furthermore, in the learning of this system, for example, a general-purpose large language model represented by GPT-4 or Llama (Language Large Models Meta AI), or a large-scale multimodal model including a large-scale language model trained specifically for autonomous vehicle control may be used. Note that even when using such a large-scale multimodal model, for example, various information acquired regarding the movement of the vehicle may be further trained to generate a trained model.

[0071] Furthermore, although not described in the above embodiment, the control information generation unit 161c of the Decoder 140c may have a function of setting stricter constraints on vehicle operation (autonomous behavior) when it is unable to acquire instruction information generated by the instruction information generation unit 121 compared to when instruction information is acquired. For example, the control information generation unit 161c may adjust the so-called autonomous driving level between 1 and 5, such as restricting the level of autonomous driving to autonomous driving level 1, which requires the driver to hold the steering wheel and monitor the road, or to autonomous driving level 2, which requires the driver to only monitor the road ahead and allows for immediate transfer of steering control to the driver. In this case, when it is unable to acquire instruction information, the control information generation unit 161c may, for example, not generate control information or may output control information according to the constraints of the autonomous driving level described above.

[0072] In the above embodiment, the external environment around the vehicle is described as an example of the environment in which the moving body is placed. However, the environment in which the moving body is placed is not necessarily limited to the external environment around the vehicle.

[0073] Although not described in the above embodiment, the present system may output instruction information as information expressed as a latent vector (intermediate representation data) of X tokens (X is a predetermined number). That is, the instruction information may be tokenized instructions such as "slow down the vehicle so that its speed is less than 5 km / h 100 m ahead," "activate right-turn lights," "perform a right turn if there are no approaching vehicles in the oncoming lane and no obstacles in the vehicle's direction of travel," "output a voice message indicating a right turn to the user," or "display light activation information to the user when the lights are activated." For example, the instruction information may be set to four parts, each of which is X / 4 tokens, and the first part (X / 4 token) may be input to three Decoders 140 in common, the second part (X / 4 token) to Language Decoder 140a, the third part (X / 4 token) to Visual Decoder 140b, and the fourth part (X / 4 token) to Control Decoder 140c. The portion of the instruction information acquired by each Decoder 140 may be preset or may be specified by a special token. By inputting a portion of the instruction information to each Decoder 140 in this way, the Decoder 140 does not perform unnecessary calculation processing, and each Decoder can mainly acquire the tokens necessary to output its own control information, thereby ensuring sufficient calculation processing speed. In addition, the system may output the instruction information as information in a format that can be understood by humans as natural language, for example.

[0074] Although the above-described embodiment has been merely a simplified explanation, the types and number of various sensors included in the vehicle sensor 10 are at the discretion of the administrator of the system. For example, the system may include any sensor different from the above-described sensors as part of the configuration of the vehicle sensor 10, or may omit unnecessary sensors from the configuration of the vehicle sensor 10. Furthermore, since the number of various sensors in the vehicle sensor 10 is also arbitrary, the system may, for example, install multiple cameras, microphones, etc. at any position on the vehicle.

[0075] Furthermore, the above-described series of processes can be executed by hardware or software. In other words, the functional configurations shown in Figure 3 and other figures are merely examples and are not particularly limited. That is, it is sufficient for the information processing system to be provided with a function capable of executing the above-described series of processes as a whole, and the type of functional block used to realize this function is not limited to the example shown in Figure 3 and other figures. Furthermore, the location of the functional block is not limited to the example shown in Figure 3 and other figures and may be arbitrary. Furthermore, one functional block may be configured as a single piece of hardware, a single piece of software, or a combination thereof.

[0076] Furthermore, the number of various hardware components constituting this system and the number of users are optional, and the system may also include other hardware components.

[0077] Furthermore, when a series of processes is executed by software, the programs that make up the software are installed into a computer or the like from a network or a recording medium.

[0078] The computer may also be a computer that is built into dedicated hardware, or a computer that can execute various functions by installing various programs.

[0079] Furthermore, the recording medium containing such a program may be configured not only as a removable medium (not shown) provided separately from the device main body in order to provide the program to the user, but also as a recording medium provided to the user in a state where it is pre-installed in the device main body.

[0080] In addition, in this specification, the term "system" refers to an overall device that is made up of a plurality of devices or a plurality of means.

[0081] The effects of this embodiment can be achieved even when these other embodiments are adopted. Furthermore, this embodiment can be combined with other embodiments, and other embodiments can be combined with each other as appropriate.

[0082] To summarize the above, the information processing system applied to the present disclosure can take various forms having the following configurations. An information processing device (vehicle-mounted device 1) including a first system (Drive AGI 100) and a plurality of second systems (Decoders 140), The first system comprises: a device information acquisition unit (device information acquisition unit 120) that acquires information output from a first device (vehicle sensor 10, HMI 20) installed in a predetermined moving body as device information; an instruction information generation unit (121) that generates instruction information for determining control information for controlling a second device (traveling device, HMI 20) of the moving object based on the device information; Equipped with Each of the plurality of second systems includes: an instruction information acquisition unit (140) that acquires at least a part of the instruction information generated by the instruction information generation unit (121); a control information generating unit (161) that uses at least a part of the instruction information to output the control information; Any information processing device having the above is sufficient.

[0083] <Additional Notes> The present embodiment includes the following disclosure.

[0084] (Appendix 1) An information processing device including a first system and a plurality of second systems, The first system comprises: a device information acquisition unit that acquires information output from a first device installed in a predetermined mobile object as device information; an instruction information generation unit that generates instruction information for determining control information for controlling a second device of the moving object based on the device information; Equipped with Each of the plurality of second systems includes: an instruction information acquisition unit that acquires at least a part of the instruction information generated by the instruction information generation unit; a control information generating unit that generates the control information by using at least a part of the instruction information; An information processing device comprising:

[0085] (Appendix 2) The plurality of second systems each acquire a predetermined portion of the instruction information. 2. The information processing device according to claim 1.

[0086] (Appendix 3) The first device of the mobile body includes a sensor installed on the mobile body, a device related to the behavior of the mobile body, and an HMI (Human Machine Interface) mounted on the mobile body. 2. The information processing device according to claim 1.

[0087] (Appendix 4) The second device of the moving object includes a device related to the behavior of the moving object and an HMI mounted on the moving object. 2. The information processing device according to claim 1.

[0088] (Appendix 5) The plurality of second systems output the control information relating to audio, video, and behavior of a moving object, respectively. 2. The information processing device according to claim 1.

[0089] (Appendix 6) the second system outputs the control information related to audio, the second system outputs the control information related to video images, and the second system outputs the control information related to behavior of a moving object. 2. The information processing device according to claim 1.

[0090] (Appendix 7) The instruction information is a latent vector 2. The information processing device according to claim 1.

[0091] (Appendix 8) the instruction information generation unit generates, as the instruction information, information including an instruction regarding an action of the moving object and a reason for determining to issue the instruction; 2. The information processing device according to claim 1.

[0092] (Appendix 9) an output period of the instruction information generation unit is longer than an output period of the control information output unit; 2. The information processing device according to claim 1.

[0093] (Appendix 10) The device information acquisition unit acquires the device information including information acquired from a camera provided so as to be able to observe directions including front, rear, left, and right of the moving body, and information acquired from a sound collection device installed outside the moving body. 2. The information processing device according to claim 1.

[0094] (Appendix 11) The sound collection device installed outside the moving body is provided in plurality on the left and right sides of the moving body. 10. The information processing device according to claim 9.

[0095] (Appendix 12) At least a part of the control information generating unit acquires sensor information acquired from a part of sensors installed in the mobile object as device information. 2. The information processing device according to claim 1.

[0096] (Appendix 13) At least a part of the control information generating unit generates the control information for controlling a device related to the behavior of the mobile object. 13. The information processing device according to claim 12.

[0097] (Appendix 14) the control information generation unit acquires, as the device information, information acquired by an imaging device capable of capturing an image of a forward direction of travel of the moving body; 2. The information processing device according to claim 1.

[0098] (Appendix 15) the control information generation unit generates the control information including information on a predicted path of the moving object and a special action in controlling the moving object. 2. The information processing device according to claim 1.

[0099] (Appendix 16) The instruction information generation unit further includes a trained model management unit, The trained model management unit executes a process for outputting the instruction information by using a large-scale multimodal model. 2. The information processing device according to claim 1.

[0100] (Appendix 17) the instruction information includes information regarding the behavior of the moving object at a time after a control time of the moving object based on the control information generated by the control information generation unit; 2. The information processing device according to claim 1.

[0101] (Appendix 18) When the instruction information acquisition unit is unable to acquire the instruction information, the mobile body imposes stricter restrictions on the autonomous behavior of the mobile body than when the instruction information acquisition unit is able to acquire the instruction information. 2. The information processing device according to claim 1.

[0102] (Appendix 19) The device information acquisition unit acquires, as the device information, information from a traveling device of the vehicle and sensor information that detects an environment around the vehicle. 2. The information processing device according to claim 1.

[0103] (Appendix 20) A program executed by an information processing device including a first system and a plurality of second systems, The first system comprises: a first acquisition step of acquiring information output from a first device installed in a predetermined mobile object as device information; a first output step of outputting instruction information for determining control information for controlling a second device of the moving object based on the device information; Equipped with Each of the plurality of second systems includes: a second acquisition step of acquiring at least a part of the instruction information output by the first output step; a second output step of outputting the control information by using at least a part of the device information or the instruction information; A program that executes control processing including: [Explanation of symbols]

[0104] S Vehicle System 1 In-vehicle device 100 Drive AGI 120 Device information acquisition section 121 Instruction information generation unit 140 Decoder 160 Instruction information acquisition unit 161 Control information generation unit 162 Control information output unit 300 Model Information DB 400 Map Information DB 10 Vehicle Sensors 20 HMI 30 Control ECU 41 Control Unit 2 Server

Claims

1. An information processing apparatus including a first system and a plurality of second systems, The first system is a device information acquisition unit that acquires, as device information, information output from a first device installed in a predetermined mobile object; an instruction information generation unit that generates instruction information for determining control information for controlling a second device of the moving object based on the device information; Equipped with Each of the plurality of second systems includes: an instruction information acquisition unit that acquires at least a part of the instruction information generated by the instruction information generation unit; a control information generating unit that generates the control information by using at least a part of the instruction information; An information processing device comprising:

2. The plurality of second systems each acquire a predetermined portion of the instruction information. The information processing device according to claim 1 .

3. The first device of the mobile body includes a sensor installed in the mobile body and an HMI (Human Machine Interface) mounted on the mobile body. The information processing device according to claim 1 .

4. The second device of the mobile object includes a device related to the behavior of the mobile object and an HMI mounted on the mobile object. The information processing device according to claim 1 .

5. The plurality of second systems output the control information relating to audio, video, and behavior of a moving object, respectively. The information processing device according to claim 1 .

6. the second system outputs the control information related to audio, the second system outputs the control information related to video images, and the second system outputs the control information related to behavior of a moving object. The information processing device according to claim 1 .

7. The instruction information is a latent vector The information processing device according to claim 1 .

8. the instruction information generation unit generates, as the instruction information, information including an instruction regarding an action of the moving object and a reason for determining to issue the instruction; The information processing device according to claim 1 .

9. an output period of the instruction information generation unit is longer than an output period of the control information generation unit; The information processing device according to claim 1 .

10. The device information acquisition unit acquires the device information including information acquired from a camera provided so as to be able to observe directions including front, rear, left, and right of the moving body, and information acquired from a sound collection device installed outside the moving body. The information processing device according to claim 1 .

11. The sound collection device installed outside the moving body is provided in plurality on the left and right sides of the moving body. The information processing device according to claim 9 .

12. At least a part of the control information generating unit acquires sensor information acquired from a part of sensors installed in the mobile object as device information. The information processing device according to claim 1 .

13. At least a part of the control information generating unit generates the control information for controlling a device related to the behavior of the mobile object. The information processing device according to claim 12.

14. the control information generation unit acquires, as the device information, information acquired by an imaging device capable of capturing an image of a forward direction of travel of the moving body; The information processing device according to claim 1 .

15. the control information generation unit generates the control information including information on a predicted path of the moving object and a special action in controlling the moving object. The information processing device according to claim 1 .

16. The instruction information generation unit further includes a trained model management unit, The trained model management unit executes a process for outputting the instruction information by using a large-scale multimodal model. The information processing device according to claim 1 .

17. the instruction information includes information regarding the behavior of the moving object at a time after a control time of the moving object based on the control information generated by the control information generation unit; The information processing device according to claim 1 .

18. When the instruction information acquisition unit is unable to acquire the instruction information, the mobile body imposes stricter restrictions on the autonomous behavior of the mobile body than when the instruction information acquisition unit is able to acquire the instruction information. The information processing device according to claim 1 .

19. The device information acquisition unit acquires, as the device information, information from a traveling device of the vehicle and sensor information that detects an environment around the vehicle. The information processing device according to claim 1 .

20. A program executed by an information processing device including a first system and a plurality of second systems, The first system is a first acquisition step of acquiring information output from a first device installed in a predetermined mobile object as device information; a first output step of outputting instruction information for determining control information for controlling a second device of the moving object based on the device information; Equipped with Each of the plurality of second systems includes: a second acquisition step of acquiring at least a part of the instruction information output by the first output step; a second output step of outputting the control information by using at least a part of the device information or the instruction information; A program that executes control processing including:

Images