Information processing device and program

Abstract

The present invention provides technology enabling efficient control of a vehicle. This information processing device includes a first system and a plurality of second systems. The first system comprises: a device information acquisition unit that acquires, as device information, information output from a first device installed in a predetermined mobile body; and an instruction information generation unit that, on the basis of the device information, generates instruction information for determining control information for controlling a second device of the mobile body. Each of the plurality of second systems comprises: an instruction information acquisition unit that acquires at least a portion of the instruction information generated by the instruction information generation unit; and a control information generation unit that generates the control information by using at least a portion of the instruction information.

Description

Information processing device and program

[0001] The present disclosure relates to an information processing device and a program.

[0002] Autonomous driving technology for automobiles has been one of the technologies that has attracted attention in recent years, and many methods related to the autonomous driving technology have been attempted. For example, Patent Document 1 discloses one aspect of such technology, which is a method of applying a machine learning algorithm to the control of an autonomous vehicle (see, for example, Patent Document 1).

[0003] Japanese Patent Application Laid-Open No. 2020-083309

[0004] Here, in controlling an autonomous vehicle, safe movement is required in an environment with a large amount of information including various uncertain factors such as the road traffic environment, and therefore, unless highly accurate calculation results are output in the above-mentioned environment, there is a possibility that the operation of the vehicle will be hindered. In this regard, many of the conventional technologies including the above-mentioned Patent Document 1 merely apply machine learning algorithms and machine learning models to functional parts such as recognition and planning required to configure an autonomous driving function, and there is room for improvement in the system configuration.

[0005] The present disclosure has been made in light of the above circumstances, and aims to provide a technology that can control a moving object more accurately and efficiently.

[0006] In order to achieve the above-mentioned object, a first aspect of the present disclosure is an information processing device including a first system and a plurality of second systems, wherein the first system comprises: a device information acquisition unit that acquires information output from a first device installed in a specified mobile body as device information; and an instruction information generation unit that generates instruction information for determining control information for controlling a second device of the mobile body based on the device information; and each of the plurality of second systems comprises: an instruction information acquisition unit that acquires at least a portion of the instruction information generated by the instruction information generation unit; and a control information generation unit that generates the control information using at least a portion of the instruction information.

[0007] A program according to an aspect of the present disclosure is also provided as a program corresponding to an information processing device according to an aspect of the present disclosure.

[0008] According to the present disclosure, it is possible to provide a technology that can efficiently control a moving object.

[0009] FIG. 1 is a diagram illustrating an example of an information processing system including a vehicle system according to an embodiment of the present disclosure. FIG. 2 is a diagram illustrating an example of a hardware configuration of an in-vehicle device constituting the information processing system according to an embodiment of the present disclosure. FIG. 3 is a diagram illustrating an example of a functional configuration of an in-vehicle device constituting the information processing system according to an embodiment of the present disclosure. FIG. 4 is a diagram illustrating an example of a flow of a Drive AGI process among the processes executed by the in-vehicle device constituting the information processing system according to an embodiment of the present disclosure. FIG. 5 is a diagram illustrating an example of a flow of a Decoder process among the processes executed by the in-vehicle device constituting the information processing system according to an embodiment of the present disclosure.

[0010] [Embodiment] FIG. 1 is a diagram showing an example of an information processing system (hereinafter referred to as "this system") including a vehicle system according to an embodiment of a moving body of the present disclosure.

[0011] 1 , a vehicle system S according to an embodiment of the present disclosure includes an on-board device 1, a vehicle sensor 10, an HMI (Human Machine Interface) 20, and a control ECU (Electronic Control Unit) 30. These devices and equipment are connected via a predetermined network such as a Controller Area Network (CAN) or Ethernet. The vehicle on which the vehicle system S is mounted may include any vehicle, moving object, etc., including automobiles powered by electricity or gasoline, as well as self-driving cars.

[0012] The vehicle sensor 10 is a device installed in a vehicle (mobile body) and includes various sensors for detecting the external environment around the vehicle (an environment that may include other vehicles, pedestrians, structures, road shapes, etc.) and on-board instruments that detect the operating status of driving equipment related to the vehicle's driving (the behavior of the mobile body), such as the steering wheel, wheels, and motor. Here, the external environment around the vehicle may include, for example, traffic participants (other vehicles, pedestrians, etc.), buildings such as commercial facilities, traffic signs installed on the side of the road, road markings formed on the road surface, lane markings, traffic lights, utility poles, guardrails, animals, fallen objects, private property such as parking lots, etc. The external environment around the vehicle may also include, for example, weather, information about the road surface (road, sidewalk, etc.) on which the mobile body can move, and its condition (e.g., whether the road surface is wet, uneven, etc.). 1, the vehicle sensor 10 includes a camera (front camera) that is installed to be able to capture an image in front of the vehicle, a camera (side camera) that is installed to be able to capture an image on the side of the vehicle, a camera (rear camera) that is installed to be able to capture an image behind the vehicle, millimeter-wave radar, ultrasonic radar, a LiDAR (Light Detection and Ranging) sensor, an acceleration sensor, a GNSS (Global Navigation Satellite System), and an external microphone (sound collection device). Furthermore, the operating status of the vehicle's running equipment refers to the operating information of each device necessary for vehicle movement, such as the rotational speed of the engine, the steering angle, and the operating status of lights, and the vehicle sensor 10 includes on-board instruments and the like that acquire this information.

[0013] Here, the camera is configured using, for example, a camera that uses a CCD (Charge Coupled Device), a CMOS (Complementary Metal Oxide Semiconductor), etc. In the present disclosure, a total of multiple cameras may be installed in the front, sides, rear, and interior of the vehicle.

[0014] In addition, the external microphone (sound collection device) is composed of a general-purpose microphone, etc., and is used to acquire information about sounds emitted by objects outside the vehicle, including, for example, sirens of ambulances and police cars, human voices, etc.

[0015] The HMI 20 is a device that presents various information to the driver and passengers of the vehicle and accepts various input operations. Specifically, as shown in Fig. 1, the HMI 20 includes, for example, a display, operation buttons, a microphone, various navigation systems, a speaker, etc. The HMI 20 is, for example, an IVI (In-Vehicle Infotainment), but is not limited to this.

[0016] The control ECU 30 is connected to the in-vehicle device 1, transmits and receives various information, and controls driving devices related to the driving of the vehicle (movement of the moving body). Specifically, as shown in Fig. 1, the control ECU 30 includes individual ECUs that perform various controls, and controls driving devices such as the brake, accelerator, steering, lights such as turn signals and lights, power unit, transmission, and suspension.

[0017] 1, the system may include a vehicle system S (on-board device 1) managed by a driver of the vehicle or the like, and a server 2 managed by an administrator of the system or the like. The vehicle system S and the server 2 may be connected to each other via a predetermined network N such as the Internet. The server 2 acquires various types of information related to vehicle operation periodically transmitted from the vehicle system S (particularly the on-board device 1) and is used to manage the various types of information.

[0018] FIG. 2 is a diagram illustrating an example of a hardware configuration of the in-vehicle device 1 that constitutes the information processing system according to an embodiment of the present disclosure.

[0019] As shown in FIG. 2 , the in-vehicle device 1 includes a control unit 41, a ROM (Read Only Memory) 42, a RAM (Random Access Memory) 43, a bus 44, an input / output interface 45, a storage unit 46, and a communication unit 47.

[0020] The control unit 41 is configured by a microcomputer or the like including a CPU, a GPU, an FPGA (Field Programmable Gate Array), an NPU (Neural Processing Unit), a semiconductor memory, etc. The control unit 41 executes various processes according to a program recorded in a ROM 42 or a program loaded from a storage unit 46 to a RAM 43. The RAM 43 also stores information, etc., necessary for the control unit 41 to execute various processes, as appropriate.

[0021] The control unit 41, ROM 42, and RAM 43 are interconnected via a bus 44. An input / output interface 45 is also connected to this bus 44. The input / output interface 45 is connected to the vehicle sensors 10, the HMI 20, the control ECU 30, a storage unit 46, a communication unit 47, and the like.

[0022] The storage unit 46 is configured with a hard disk drive (HDD), a solid state drive (SSD), etc., and stores various types of information. For example, the storage unit 46 stores various programs and the like required for executing various processes related to this system.

[0023] The communication unit 47 controls mutual communications with other hardware and the like via a network N including the Internet.

[0024] 3 is a diagram illustrating an example of the functional configuration of an in-vehicle device constituting an information processing system according to an embodiment of the present disclosure. Specifically, the functions of the control unit 41 of the in-vehicle device 1 in this embodiment are broadly divided into two functions: a Drive AGI 100, and a Language Decoder 140a, a Visual Decoder 140b, and a Control Decoder 140c (hereinafter collectively referred to as "Decoder 140").

[0025] Drive AGI 100 performs calculations and inference processing related to large-scale multimodal models (trained models), such as large-scale language models, which perform integrated recognition and behavioral decisions for the vehicle. Specifically, Drive AGI 100 makes comprehensive decisions based on information acquired by external sensors such as cameras installed in the vehicle and on-board instruments, speech information and operational input information from people such as the vehicle's driver and passengers, and background knowledge about the road traffic environment, and can provide appropriate instruction information to Decoder 140.

[0026] In response to this, Decoder 140 executes arithmetic processing that executes calculations using a machine learning model based on limited information. Decoder 140 outputs control information for controlling vehicle devices based on the instruction information output by Drive AGI 100. In this system, the arithmetic processing and output processing by Decoder 140 are configured to be lightweight compared to the arithmetic processing and output processing by Drive AGI 100, and each Decoder outputs control information to each device in the vehicle, thereby enabling vehicle control to be achieved in a time (short time) necessary and sufficient for controlling an autonomously driven vehicle.

[0027] As shown in FIG. 3 , the control unit 41 of the in-vehicle device 1 executes various programs, thereby enabling the Drive AGI 100 and the Decoder 140 to function. A model information DB 300 and a map information DB 400 are provided in one area of ​​the storage unit 46 of the in-vehicle device 1. The model information DB 300 stores trained models for outputting instruction information (described later) in response to input information from various sensors, and trained models for outputting control information (described later) in response to input information from the various sensors. The map information DB 400 stores map information, etc. Map information is, for example, information that displays information such as latitude and longitude regarding roads, buildings, etc. in a two-dimensional or three-dimensional format. It may be general-purpose map information used for navigation, or a map such as a high-precision map (HD map) that includes legal information such as traffic light locations, the number of lanes, speed limits, and stop line locations. In this system, the map information stored in the map information DB 400 is used as appropriate in situations where various processes (described later) are required.

[0028] The following describes the functional configuration related to the Drive AGI 100. The Drive AGI 100 is composed of a device information acquisition unit 120 and an instruction information generation unit 121.

[0029] The device information acquisition unit 120 acquires and manages device information output from the vehicle's devices. The device information includes sensor information acquired from the sensors constituting the vehicle sensor 10 and HMI information output from the HMI 20. The HMI information includes information about video images, audio, map information, information about objects on the road, route information, traffic information, information about questions for the occupant, and information about responses to the questions from the occupant. The device information acquisition unit 120 may acquire all or part of the device information. Preferably, the device information acquisition unit 120 acquires at least information necessary for vehicle driving, i.e., information acquired by onboard instruments from the vehicle's driving devices (e.g., vehicle speed, steering angle, etc.), and sensor information detecting the environment around the vehicle (e.g., video data, radar ranging data, etc.). Note that the device information (e.g., video data) may be tokenized information that can be processed by a trained model stored in the model information DB 300.

[0030] The instruction information generation unit 121 generates information (hereinafter referred to as "instruction information") to be processed by the Decoder 140 based on the device information acquired by the device information acquisition unit 120 and the trained model stored in the model information DB 300. Here, the instruction information is typically information regarding a medium- to long-term vehicle behavior plan or driving instructions, i.e., an instruction regarding the vehicle's behavior. Furthermore, the instruction information may also include, for example, the reason for the decision to issue the instruction. For example, the instruction information may be instruction information having meanings such as "The traffic light is red, please stop the vehicle," "There is a sharp curve ahead, so please slow down gradually," "You will turn left ahead, please turn on your blinker," or "There is an intersection ahead, please go right," output as an embedded vector as a latent representation. In this example of instruction information, for example, information such as "Please stop the vehicle" is an instruction regarding the vehicle's behavior, and information such as "The traffic light is red" is the reason for the decision to issue the instruction. The instruction information may include, for example, information of the above content. In this way, the present system can realize more appropriate and efficient operation control in Decoder 140 by outputting instructions regarding medium- to long-term action plans or operation plans based on various information obtained by the various sensors that make up vehicle sensor 10 and outputting them to Decoder 140.

[0031] The content of the instruction information will now be described in more detail. Specifically, for example, while a vehicle is traveling on a lane of a highway, the instruction information output by the Drive AGI 100 may include information indicating, "Move to the left lane to exit the highway 1 km ahead." Then, the Control Decoder 140c outputs specific control information, including information regarding the actions the vehicle should take in the future, as a result of processing based on the instruction information, such as, "(1) Flash the left turn signal, (2) Change lanes 3 seconds after the flashing, and (3) After changing lanes, decelerate to 90 km / h." That is, by including information indicating exiting the highway in the instruction information during processing by the Decoder 140, a processing result such as (3) Decelerate to 90 km / h after changing lanes can be derived, which contributes to more comfortable and safer travel for the occupants.

[0032] This instruction information includes, for example, information regarding a more accurate medium- to long-term action plan or operation instructions obtained by Decoder 140 over a longer calculation time based on a larger amount of information (e.g., device information described later) compared to the information processed by Decoder 140. In contrast, the processing by Decoder 140 described later generates more specific control instructions, etc. regarding vehicle operation and device control for a shorter period of time than the medium- to long-term action plan or operation plan generated by Drive AGI 100, based on limited information (e.g., part of the device information described later) and / or at least part of the generated instruction information. By combining and executing these two control processes with different roles, it is possible to safely and efficiently achieve autonomous driving control and device control for a vehicle.

[0033] Here, the information acquired by the device information acquisition unit 120 and the information generated by the instruction information generation unit 121 will be described in a series of steps. As an example, assume that a vehicle is traveling straight toward a crossroads located 100 meters ahead and a user inside the vehicle has requested to turn right. In this situation, the device information acquisition unit 120 acquires, from the vehicle sensors, video image information in front of the vehicle from the front camera, vehicle speed information, operation information of safety devices such as lights, and output value information of the engine, and also acquires, from the HMI 20, voice information related to the above-mentioned instruction from the user. Based on the various information acquired by the device information acquisition unit 120, the instruction information generation unit 121 generates instructions such as "slow down the vehicle so that its speed is less than 5 km / h 100 m ahead," "activate the right turn lights," "perform a right turn if there are no approaching vehicles in the oncoming lane and no obstacles in the direction of travel of the vehicle," "output a voice message to the user indicating that a right turn will be made," and "display light activation information to the user when the lights are activated," and outputs these instructions to the Decoder 140.

[0034] The functional configuration related to the Language Decoder 140a will be described. The Language Decoder 140a controls the voice and text for the occupant output from the HMI 20 (vehicle equipment) based on at least a part of the instruction information. The Language Decoder 140a includes an instruction information acquisition unit 160a, a control information generation unit 161a, and a control information output unit 162a.

[0035] The instruction information acquisition unit 160a acquires and manages at least a portion of the instruction information generated by the instruction information generation unit 121 of the Drive AGI 100. The instruction information acquisition unit 160a may acquire all of the instruction information, or may acquire a portion of the instruction information that is pre-assigned to the Language Decoder 140a.

[0036] The control information generation unit 161a generates information (hereinafter referred to as "control information a") regarding specific control instructions for controlling the voice for the occupant to be output from the HMI 20 (vehicle equipment) based on the instruction information acquired by the instruction information acquisition unit 160a and the contents of the trained model stored in the model information DB 300. Here, the control information a is, for example, control information for the HMI 20. Specifically, the control information a is information indicating the content of the voice for the occupant to be output from the HMI 20, and in the vehicle of this embodiment, it is information about objects on the road, information about the route, traffic information, etc. Furthermore, the control information a may include, for example, information about questions for the occupant and information about answers to the questions from the occupant. The HMI 20 can achieve appropriate voice control by outputting voice based on the control information a.

[0037] The control information output unit 162a provides the control information a generated by the control information generation unit 161a to the HMI 20 of the vehicle system S.

[0038] The language decoder 140a may acquire at least a portion of the device information (e.g., HMI information) and generate the control information a based on the acquired device information and instruction information. In this case, it is preferable that the language decoder 140a acquires only a portion of the device information that is effective for generating the control information a, so that the amount of information used for generating the control information a is reduced.

[0039] Furthermore, the language decoder 140a may not only output the control information a to the HMI 20, but may also directly control the HMI 20 using the control information a. In other words, the language decoder 140a may function as a control unit for the HMI 20.

[0040] The functional configuration related to the visual decoder 140b will be described. The visual decoder 140b controls the video output from the HMI 20 (vehicle equipment) based on at least a part of the instruction information. The visual decoder 140b includes an instruction information acquisition unit 160b, a control information generation unit 161b, and a control information output unit 162b.

[0041] The instruction information acquisition unit 160b acquires and manages at least a portion of the instruction information generated by the instruction information generation unit 121 of the Drive AGI 100. The instruction information acquisition unit 160b may acquire all of the instruction information, or may acquire a portion of the instruction information that has been pre-assigned to the Visual Decoder 140b.

[0042] The control information generation unit 161b generates control information b based on the instruction information acquired by the instruction information acquisition unit 160b and the contents of the trained model stored in the model information DB 300. Here, the control information b is, for example, control information for the HMI 20. Specifically, the control information b is information indicating the contents of the moving image for the occupant output from the HMI 20. In the vehicle of this embodiment, the control information b includes map information, information on moving images outside the vehicle acquired from the vehicle sensor 10, the vehicle's operating status along the travel route, the recognition status of surrounding objects, and vehicle dynamics / telemetry information such as vehicle speed and fuel efficiency. Furthermore, the control information b may include, for example, information on questions to the occupant and information on answers to the questions from the occupant. The HMI 20 can realize appropriate control of the moving image by outputting the moving image based on the control information b.

[0043] The control information output unit 162b provides the control information b generated by the control information generation unit 161b to the HMI 20 of the vehicle system S.

[0044] The Visual Decoder 140b may acquire at least a portion of the device information (e.g., HMI information) and generate the control information b based on the acquired device information and instruction information. In this case, it is preferable that the Visual Decoder 140b acquires only a portion of the device information that is effective for generating the control information b, so that the amount of information used for generating the control information b is reduced.

[0045] Furthermore, the visual decoder 140b may not only output the control information b to the HMI 20, but also directly control the HMI 20 using the control information b. In other words, the visual decoder 140b may function as a control unit for the HMI 20.

[0046] The functional configuration related to the Control Decoder 140c will be described. The Control Decoder 140c controls devices (such as brakes, accelerators, and steering) related to the movement of the vehicle (the behavior of the moving object) based on at least a part of the instruction information. The Control Decoder 140c includes an instruction information acquisition unit 160c, a control information generation unit 161c, and a control information output unit 162c.

[0047] The instruction information acquisition unit 160c acquires and manages at least a portion of the instruction information generated by the instruction information generation unit 121 of the Drive AGI 100. The instruction information acquisition unit 160c may acquire all of the instruction information, or may acquire a portion of the instruction information that has been previously allocated to the Drive Decoder 140c.

[0048] The control information generation unit 161c generates control information c based on the instruction information acquired by the instruction information acquisition unit 160c and the contents of the trained model stored in the model information DB 300. Specifically, the control information c is information for controlling equipment related to the movement of the moving object. In the vehicle of this embodiment, the control information c includes vehicle speed information, vehicle acceleration / deceleration information, information related to a behavior plan such as a movement trajectory to be followed by the vehicle, and information related to the direction in which the vehicle should travel. The control information c may also include information related to driving instructions for the driver, including, for example, a predicted route of the vehicle and instructions related to special actions. The predicted route is, for example, a predicted future route of the vehicle calculated or planned based on the equipment information. The special action is, for example, specific instruction information related to operations such as activation of left and right turn signals (including flashing hazard lights), sudden deceleration, gear shift instructions, and horn instructions. Each control ECU 30 can achieve appropriate autonomous driving of the vehicle by controlling the vehicle based on the control information c.

[0049] The control information output unit 162c provides the control information c generated by the control information generation unit 161c to the control ECU 30 of the vehicle system S and the like.

[0050] The control information generator 161c may acquire at least a portion of the device information (e.g., sensor information) and generate the control information c based on the acquired device information and instruction information. In this case, it is preferable that the control information generator 161c acquire only a portion of the sensor information that is effective for generating the control information c, so as to reduce the amount of information used for generating the control information c. Specifically, it is preferable that the control information generator 161c acquires at least information necessary for vehicle driving, i.e., information acquired by on-board instruments from the vehicle's driving equipment (vehicle speed, steering angle, etc.), and sensor information that detects the environment around the vehicle (video data, radar ranging data, etc.).

[0051] In this case, the output cycle (e.g., 20 Hz) at which Drive AGI 100 outputs instruction information is set longer than the output cycle (e.g., 60 Hz) at which Control Decoder 140c outputs control information. This allows Control Decoder 140c, which controls the vehicle's nearest devices and solves operational issues, to maintain the processing speed for near-term vehicle control, while Drive AGI 100 performs highly accurate recognition and judgment in the medium to long term, thereby ensuring overall processing time that is highly accurate and can withstand autonomous driving control.

[0052] Furthermore, if the control decoder 140c cannot acquire instruction information, it may generate the control information c based only on the acquired device information. This allows the control decoder 140c to control the vehicle's driving even when it cannot acquire instruction information. In particular, it is preferable that the control decoder 140c generates the control information c based only on the device information so as to enable short-term autonomous driving control, such as automatic braking, emergency hazard warning, lane / road departure prevention steering, and ACC. For this reason, it is preferable that the control decoder 140c acquires sensor information (e.g., camera images) necessary for autonomous driving control.

[0053] Furthermore, the control decoder 140c may not only output the control information c to the control ECU 30, but also directly control devices related to the movement of the vehicle (such as the accelerator) using the control information c. In other words, the control decoder 140c may function as the ECU 30.

[0054] As described above, each Decoder 140 may acquire all of the instruction information and generate control information, or may acquire only a portion of the instruction information and generate control information. In the latter case, the instruction information acquired by each Decoder 140 may be different from the others, or may share some of the instruction information. For example, the instruction information may have four parts, with the first part being input to three Decoders 140 in common, the second part being input to Language Decoder 140a, the third part being input to Visual Decoder 140b, and the fourth part being input to Control Decoder 140c. The part of the instruction information acquired by each Decoder 140 may be preset or may be specified by a special token. In this way, by inputting only a portion of the instruction information to each Decoder 140, the Decoder 140 does not need to perform unnecessary computation, thereby ensuring sufficient computation speed.

[0055] Furthermore, in this embodiment, the in-vehicle device 1 includes three decoders 140, but may include two or four or more decoders 140. For example, another possible decoder 140 may be a decoder 140 that outputs control signals for controlling the in-vehicle interior (air conditioner, audio, seat reclining, etc.) based on instruction information.

[0056] The hardware configuration of the server 2 is not described here because it can be basically the same as the hardware configuration of the in-vehicle device 1. The cooperation of such various hardware and software enables the in-vehicle device 1 and the like to execute various processes described below.

[0057] Here, processing executed in the information processing system according to an embodiment of the present disclosure will be described. Fig. 4 is a diagram schematically illustrating processing executed in the information processing system. Fig. 5 is a diagram illustrating an example of the flow of Drive AGI processing, which is one of the processing executed by an in-vehicle device constituting the information processing system according to an embodiment of the present disclosure.

[0058] In step S1, the device information acquisition unit 120 acquires and manages device information (sensor information and HMI information) from the vehicle sensor 10 and the HMI 20.

[0059] In step S2, the instruction information generation unit 121 generates and outputs instruction information based on the device information acquired by the device information acquisition unit 120 and the contents of the trained model stored in the model information DB 300. This completes the Drive AGI processing of the in-vehicle device 1.

[0060] FIG. 6 is a diagram illustrating an example of the flow of a decoder process among the processes executed by an in-vehicle device constituting an information processing system according to an embodiment of the present disclosure.

[0061] In step S21, the instruction information acquisition unit 160a acquires and manages at least a portion of the instruction information a assigned to the Language Decoder 140a from among the instruction information generated by the instruction information generation unit 121. The instruction information acquisition unit 160b acquires and manages at least a portion of the instruction information b assigned to the Visual Decoder 140b from among the instruction information generated by the instruction information generation unit 121. The instruction information acquisition unit 160c acquires and manages at least a portion of the instruction information c assigned to the Control Decoder 140c from among the instruction information generated by the instruction information generation unit 121.

[0062] In step S22, the control information generation units 161a, 161b, and 161c generate control information a, b, and c regarding specific control instructions for controlling the vehicle's equipment based on the instruction information acquired by the instruction information acquisition units 160a, 160b, and 160c and the contents of the trained model stored in the model information DB 300.

[0063] In step S23, the control information output units 162a and 162b provide the control information a and b generated by the control information generation units 161a and 161b, respectively, to the HMI 20. In addition, the control information output unit 162c provides the control information c generated by the control information generation unit 161c to the control ECU 30 of the vehicle system S, etc. This completes the decoder process of the in-vehicle device 1.

[0064] The above describes one embodiment of the present disclosure, but the present disclosure is not limited to the above-described embodiment, and modifications and improvements within the scope of achieving the object of the present disclosure are included in the present disclosure.

[0065] [Other Embodiments] In the above-described embodiments, the vehicle is described as a general-purpose autonomous vehicle, but this is not limited thereto. Vehicles to which the present system can be applied may include any type of mobile object, regardless of shape or power source, such as automobiles, trucks, motorcycles, railroad cars, bicycles, robots, AGVs (Automatic Guided Vehicles), and drones. Furthermore, the information processing device or information processing system according to the present system does not need to function independently as an information processing device, and may be provided, for example, as an integrated part of a vehicle (mobile object).

[0066] In the above embodiment, the device information acquisition unit 120 has been described as acquiring only information related to a camera installed to capture an image ahead of the vehicle, a millimeter-wave radar or ultrasonic radar installed to acquire information about the front, and on-board instruments as sensor information. However, this is not limited to this. The administrator of the system can freely design which types of sensor information to use as sensor information. The same applies to HMI information. Specifically, for example, when the vehicle is backing up, the sensor information may include all or part of an image acquired by a camera capable of capturing an image behind the vehicle, various types of information acquired by a millimeter-wave radar or ultrasonic radar installed to acquire information about the rear, and an image acquired by a camera capable of capturing an image to the side.

[0067] Furthermore, the present system may change the selection of sensors to acquire as sensor information depending on the vehicle situation. Specifically, for example, the present system normally acquires only sensor information from a front camera, but may acquire sensor information from left, right, or rear cameras (e.g., in the direction of the lane change) when changing lanes. The lane change referred to here may include vehicle operations such as simple lateral movement, left or right turns, etc. The advantage of the Decoder 140 of the present system acquiring only instruction information rather than device information and performing various processes is that by acquiring information only from instruction information and performing processes related to the vehicle operation, the Decoder 140 can perform processes faster than when acquiring and processing device information.

[0068] Although not described in the above embodiment, the trained model used by Drive AGI 100 or Decoder 140 may be updated as appropriate before or after generating various types of information. For example, the in-vehicle device 1 may acquire a newly trained trained model using output instruction information, control information, etc., and update the content of the trained model stored in the model information DB 300. Note that the learning method used here may be, for example, a method classified into various deep learning methods such as DNN (Deep Neural Network), or a combination thereof.

[0069] Although the above-described embodiment has been described simply, it is assumed that the information stored in the model information DB 300 has undergone a learning process in advance by the in-vehicle device 1 or other hardware. Specifically, the information stored in the model information DB 300 is model information that has been sufficiently adjusted for application to autonomous driving by learning a wide range of information including various videos, text, natural language, etc., as well as by labeling separately collected images of the vehicle traveling with information related to vehicle control, supplementary information, etc. Note that the learning data used for these learnings does not necessarily have to be acquired by the vehicle to which autonomous driving is applied, and may be learning data from another vehicle or data generated by a method such as simulation.

[0070] Furthermore, in the learning of this system, for example, a general-purpose large language model represented by GPT-4 or Llama (Language Large Models Meta AI), or a large-scale multimodal model including a large-scale language model trained specifically for autonomous vehicle control may be used. Note that even when using such a large-scale multimodal model, for example, various information acquired regarding the movement of the vehicle may be further trained to generate a trained model.

[0071] Furthermore, although not described in the above embodiment, the control information generation unit 161c of the Decoder 140c may have a function of setting stricter constraints on vehicle operation (autonomous behavior) when the instruction information generated by the instruction information generation unit 121 cannot be acquired compared to when the instruction information can be acquired. For example, the control information generation unit 161c may adjust the so-called autonomous driving level between 1 and 5, such as restricting the level of autonomous driving to autonomous driving level 1, which requires the driver to hold the steering wheel and monitor the road, or to autonomous driving level 2, which requires the driver to only monitor the road ahead and allows immediate transfer of steering control to the driver. In this case, when the control information generation unit 161c cannot acquire instruction information, for example, the control information generation unit 161c may not generate control information, or may output control information according to the constraints of the autonomous driving level described above.

[0072] In the above embodiment, the external environment around the vehicle is described as an example of the environment in which the moving body is placed. However, the environment in which the moving body is placed is not necessarily limited to the external environment around the vehicle.

[0073] Although not described in the above embodiment, the present system may output instruction information as information expressed as a latent vector (intermediate representation data) of X tokens (X is a predetermined number). That is, the instruction information may be tokenized instructions such as "slow down the vehicle so that its speed is less than 5 km / h 100 m ahead," "activate right-turn lights," "perform a right turn if there are no approaching vehicles in the oncoming lane and no obstacles in the vehicle's direction of travel," "output a voice message indicating a right turn to the user," or "display light activation information to the user when the lights are activated." For example, the instruction information may be configured with four parts, each consisting of X / 4 tokens, and the first part (X / 4 token) may be input to three Decoders 140 in common, the second part (X / 4 token) to Language Decoder 140a, the third part (X / 4 token) to Visual Decoder 140b, and the fourth part (X / 4 token) to Control Decoder 140c. The portion of the instruction information acquired by each Decoder 140 may be preset or may be specified by a special token. By inputting a portion of the instruction information to each Decoder 140 in this manner, the Decoder 140 does not need to perform unnecessary calculations, and each Decoder can mainly acquire the tokens necessary to output its own control information, thereby ensuring sufficient calculation speed. Furthermore, the system may output the instruction information as information in a format that can be understood by humans as natural language, for example.

[0074] Although the above-described embodiment has been merely a simplified explanation, the types and number of various sensors included in the vehicle sensor 10 are at the discretion of the administrator of the system. For example, the system may include any sensor different from the above-described sensors as part of the configuration of the vehicle sensor 10, or may omit unnecessary sensors from the configuration of the vehicle sensor 10. Furthermore, since the number of various sensors in the vehicle sensor 10 is also arbitrary, the system may, for example, install multiple cameras, microphones, etc. at any position on the vehicle.

[0075] Furthermore, the above-described series of processes can be executed by hardware or software. In other words, the functional configurations shown in Figure 3 and other figures are merely examples and are not particularly limited. That is, it is sufficient for the information processing system to be provided with the functionality to execute the above-described series of processes as a whole, and the type of functional block used to realize this functionality is not limited to the example shown in Figure 3 and other figures. Furthermore, the location of the functional block is not limited to the example shown in Figure 3 and other figures and may be arbitrary. Furthermore, one functional block may be configured as hardware alone, software alone, or a combination thereof.

[0076] Furthermore, the number of various hardware components constituting this system and the number of users are optional, and the system may also include other hardware components.

[0077] Furthermore, when a series of processes is executed by software, the programs that make up the software are installed into a computer or the like from a network or a recording medium.

[0078] The computer may also be a computer that is built into dedicated hardware, or a computer that can execute various functions by installing various programs.

[0079] Furthermore, the recording medium containing such a program may be configured not only as a removable medium (not shown) provided separately from the device main body in order to provide the program to the user, but also as a recording medium provided to the user in a state where it is pre-installed in the device main body.

[0080] In addition, in this specification, the term "system" refers to an overall device that is made up of a plurality of devices or a plurality of means.

[0081] The effects of this embodiment can be achieved even when these other embodiments are adopted. Furthermore, this embodiment can be combined with other embodiments, and other embodiments can be combined with each other as appropriate.

[0082] To summarize the above, the information processing system applicable to the present disclosure can take various forms having the following configurations: An information processing device (on-vehicle device 1) including a first system (Drive AGI 100) and a plurality of second systems (Decoders 140), wherein the first system includes: a device information acquisition unit (device information acquisition unit 120) that acquires, as device information, information output from a first device (vehicle sensor 10, HMI 20) installed in a predetermined mobile object; and an instruction information generation unit 121 that generates, based on the device information, instruction information for determining control information for controlling a second device (traveling device, HMI 20) of the mobile object; and each of the plurality of second systems includes: an instruction information acquisition unit 140 that acquires at least a portion of the instruction information generated by the instruction information generation unit 121; and a control information generation unit 161 that uses at least a portion of the instruction information to output the control information.

[0083] <Additional Notes> This embodiment includes the following disclosure.

[0084] (Supplementary Note 1) An information processing device including a first system and a plurality of second systems, wherein the first system comprises: a device information acquisition unit that acquires information output from a first device installed in a predetermined mobile body as device information; and an instruction information generation unit that generates instruction information for determining control information for controlling a second device of the mobile body based on the device information; and each of the plurality of second systems comprises: an instruction information acquisition unit that acquires at least a portion of the instruction information generated by the instruction information generation unit; and a control information generation unit that generates the control information using at least a portion of the instruction information.

[0085] (Supplementary Note 2) The information processing device according to Supplementary Note 1, wherein the plurality of second systems each acquire a predetermined portion of the instruction information.

[0086] (Supplementary Note 3) The information processing device according to Supplementary Note 1, wherein the first device of the mobile body includes a sensor installed on the mobile body, a device related to the behavior of the mobile body, and an HMI (Human Machine Interface) mounted on the mobile body.

[0087] (Supplementary Note 4) The information processing device according to Supplementary Note 1, wherein the second device of the moving object includes a device related to the behavior of the moving object and an HMI mounted on the moving object.

[0088] (Supplementary Note 5) The information processing device according to Supplementary Note 1, wherein the plurality of second systems output the control information relating to audio, video, and behavior of a moving object, respectively.

[0089] (Supplementary Note 6) The information processing device according to Supplementary Note 1, comprising: the second system that outputs the control information related to audio; the second system that outputs the control information related to moving images; and the second system that outputs the control information related to behavior of a moving object.

[0090] (Supplementary Note 7) The information processing device according to Supplementary Note 1, wherein the instruction information is a latent vector.

[0091] (Supplementary Note 8) The information processing device according to Supplementary Note 1, wherein the instruction information generation unit generates, as the instruction information, information including an instruction regarding an action of the moving object and a reason for arriving at a decision to issue the instruction.

[0092] (Supplementary Note 9) The information processing device according to Supplementary Note 1, wherein an output period of the instruction information generation unit is longer than an output period of the control information output unit.

[0093] (Supplementary Note 10) The information processing device according to Supplementary Note 1, wherein the device information acquisition unit acquires the device information including information acquired from a camera provided to be able to observe directions including front, rear, left, and right of the moving body, and information acquired from a sound collection device installed outside the moving body.

[0094] (Supplementary Note 11) The information processing device according to Supplementary Note 9, wherein a plurality of the sound collecting devices installed outside the moving body are provided on the left and right sides of the moving body.

[0095] (Supplementary Note 12) The information processing device according to Supplementary Note 1, wherein at least a part of the control information generating unit acquires, as device information, sensor information acquired from a part of sensors installed in the mobile object.

[0096] (Supplementary Note 13) The information processing device according to Supplementary Note 12, wherein at least a part of the control information generation unit generates the control information for controlling a device related to an action of the moving object.

[0097] (Supplementary Note 14) The information processing device according to Supplementary Note 1, wherein the control information generation unit acquires, as the device information, information acquired by an imaging device capable of capturing an image of a front area in a traveling direction of the moving body.

[0098] (Supplementary Note 15) The information processing device according to Supplementary Note 1, wherein the control information generation unit generates the control information including information on a predicted path of the moving object and a special action in controlling the moving object.

[0099] (Supplementary Note 16) The information processing device according to Supplementary Note 1, wherein the instruction information generation unit further includes a trained model management unit, and the trained model management unit executes processing for outputting the instruction information by utilizing a large-scale multimodal model.

[0100] (Supplementary Note 17) The information processing device according to Supplementary Note 1, wherein the instruction information includes information relating to a behavior of the moving object at a time after a control time of the moving object based on the control information generated by the control information generation unit.

[0101] (Supplementary Note 18) The information processing device according to Supplementary Note 1, wherein, when the instruction information acquisition unit is unable to acquire the instruction information, the mobile body imposes stricter restrictions on the autonomous behavior of the mobile body than when the instruction information acquisition unit is able to acquire the instruction information.

[0102] (Supplementary Note 19) The information processing device according to Supplementary Note 1, wherein the device information acquisition unit acquires, as the device information, information from a traveling device of the vehicle and sensor information that detects an environment around the vehicle.

[0103] (Supplementary Note 20) A program executed by an information processing device including a first system and a plurality of second systems, wherein the first system comprises: a first acquisition step of acquiring information output from a first device installed in a predetermined mobile body as device information; and a first output step of outputting instruction information for determining control information for controlling a second device of the mobile body based on the device information; and wherein the plurality of second systems each comprise: a second acquisition step of acquiring at least a part of the instruction information output by the first output step; and a second output step of outputting the control information using at least a part of the device information or the instruction information.

[0104] S Vehicle system 1 In-vehicle device 100 Drive AGI 120 Device information acquisition unit 121 Instruction information generation unit 140 Decoder 160 Instruction information acquisition unit 161 Control information generation unit 162 Control information output unit 300 Model information DB 400 Map information DB 10 Vehicle sensor 20 HMI 30 Control ECU 41 Control unit 2 Server

Claims

1. An information processing device including a first system and a plurality of second systems, wherein the first system comprises: a device information acquisition unit that acquires information output from a first device installed in a predetermined mobile body as device information; and an instruction information generation unit that generates instruction information for determining control information for controlling a second device of the mobile body based on the device information; and each of the plurality of second systems comprises: an instruction information acquisition unit that acquires at least a portion of the instruction information generated by the instruction information generation unit; and a control information generation unit that generates the control information using at least a portion of the instruction information.

2. The information processing device according to claim 1, wherein the plurality of second systems each acquire a predetermined portion of the instruction information.

3. The information processing device according to claim 1, wherein the first device of the mobile body includes a sensor installed on the mobile body and an HMI (Human Machine Interface) mounted on the mobile body.

4. The information processing device according to claim 1, wherein the second device of the mobile object includes a device related to the behavior of the mobile object and an HMI mounted on the mobile object.

5. The information processing device according to claim 1, wherein the plurality of second systems output the control information relating to audio, video, and the behavior of a moving object, respectively.

6. An information processing device as described in claim 1, comprising: the second system that outputs the control information related to audio; the second system that outputs the control information related to moving images; and the second system that outputs the control information related to the behavior of a moving object.

7. The information processing device according to claim 1, wherein the instruction information is a latent vector.

8. The information processing device according to claim 1, wherein the instruction information generating unit generates, as the instruction information, information including an instruction regarding the behavior of the moving object and a reason for the decision to issue the instruction.

9. The information processing device according to claim 1, wherein the output period of the instruction information generating section is longer than the output period of the control information generating section.

10. The information processing device according to claim 1, wherein the device information acquisition unit acquires the device information including information acquired from a camera installed so as to be able to observe directions including the front, back, left and right of the moving body, and from a sound collection device installed outside the moving body.

11. The information processing device according to claim 9, wherein a plurality of the sound collecting devices installed outside the moving body are provided on the left and right sides of the moving body.

12. The information processing device according to claim 1, wherein at least a part of the control information generating unit acquires, as device information, sensor information acquired from a part of sensors installed in the mobile object.

13. The information processing device according to claim 12, wherein at least a part of the control information generating unit generates the control information for controlling a device related to the behavior of the mobile object.

14. The information processing device according to claim 1, wherein the control information generating unit acquires, as the device information, information acquired by an imaging device capable of capturing an image ahead in the traveling direction of the moving body.

15. The information processing device according to claim 1, wherein the control information generation unit generates the control information including information on a predicted path of the moving object and special actions in controlling the moving object.

16. The information processing device according to claim 1, wherein the instruction information generation unit further includes a trained model management unit, and the trained model management unit executes processing for outputting the instruction information using a large-scale multimodal model.

17. The information processing device according to claim 1, wherein the instruction information includes information relating to the behavior of the mobile object at a time after the control time of the mobile object based on the control information generated by the control information generating unit.

18. The information processing device according to claim 1, wherein, when the instruction information acquisition unit is unable to acquire the instruction information, the mobile body imposes stricter restrictions on the autonomous behavior of the mobile body than when the instruction information acquisition unit is able to acquire the instruction information.

19. The information processing device according to claim 1, wherein the device information acquisition unit acquires, as the device information, information from the vehicle's running equipment and sensor information that detects the environment around the vehicle.

20. A program executed by an information processing device including a first system and a plurality of second systems, wherein the first system comprises: a first acquisition step of acquiring information output from a first device installed in a predetermined mobile body as device information; and a first output step of outputting instruction information for determining control information for controlling a second device of the mobile body based on the device information; and wherein each of the plurality of second systems comprises: a second acquisition step of acquiring at least a portion of the instruction information output by the first output step; and a second output step of outputting the control information using at least a portion of the device information or the instruction information.

Images