Information processing device and program

Abstract

The first sensor information acquisition unit 120 acquires and manages all sensor information from all sensors constituting the vehicle sensor 10. The auxiliary information generation unit 121 generates auxiliary information based on all sensor information and the contents of the trained model stored in the model information DB300. The second sensor information acquisition unit 160 acquires and manages partial sensor information including image information acquired by a camera installed in front of the vehicle, various information acquired by a millimeter wave radar or ultrasonic radar installed in the front, and various information acquired by on-board instruments. The control information generation unit 161 generates control information based on the partial sensor information, auxiliary information, and the contents of the trained model stored in the model information DB300.

Description

[Technical field]

[0001] The present disclosure relates to an information processing device and a program. [Background technology]

[0002] Autonomous driving technology for automobiles has been one of the technologies that has been attracting attention in recent years, and many methods related to the autonomous driving technology have been attempted. As one aspect of such an approach, for example, Patent Document 1 discloses a method of applying a machine learning algorithm to decision-making for controlling an autonomous vehicle (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent Publication No. 2020-083309 Summary of the Invention [Problem to be solved by the invention]

[0004] When applying machine learning technology to the control of autonomous vehicles, the amount of information to be processed and the speed of calculations are very important. In other words, the learning process related to machine learning and the application process of the learning results require a lot of information and calculations, which take time. On the other hand, in the control of autonomous vehicles, the time available for calculations is limited, and unless calculations are performed effectively in an extremely short time, there is a possibility that the operation of the vehicle will be hindered. In this regard, many of the conventional techniques, including the above-mentioned Patent Document 1, merely apply machine learning algorithms.

[0005] The present disclosure has been made in consideration of the above circumstances, and aims to provide a technology capable of efficiently controlling a moving object.

[0006] In order to achieve the above object, a first aspect of the present disclosure provides: An information processing apparatus including a first system and a second system, The first system comprises: A first acquisition unit that acquires, as sensor information, information about an environment in which a predetermined moving object is located, the information being acquired by a sensor installed in the moving object; a first output unit that outputs auxiliary information for assisting in determining control information for controlling a behavior of the moving object based on the sensor information; Equipped with The second system comprises: A second acquisition unit that acquires some sensor information from some of the sensors installed in the moving object; a third acquisition unit that acquires the auxiliary information output by the first output unit; a second output unit that outputs the control information by using at least a part of the sensor information or the auxiliary information; Equipped with It is an information processing device.

[0007] In addition, in the first aspect, the first output unit may output, as the auxiliary information, information including an instruction regarding an action of the moving object and a reason for arriving at a decision to issue the instruction.

[0008] In the first aspect, an output period of the first output section may be longer than an output period of the second output section.

[0009] In addition, in a first aspect, the first acquisition unit may acquire the sensor information including information acquired from a camera configured to observe directions including front, back, left and right of the moving body, and a sound collection device installed outside the moving body.

[0010] In the first aspect, the sound collecting device installed outside the moving body may be a plurality of devices provided on the left and right sides of the moving body.

[0011] In addition, in the first aspect, the second acquisition unit may acquire, as the partial sensor information, information acquired by an imaging device capable of capturing an image of a front area in a traveling direction of the moving object.

[0012] In addition, in the first aspect, when an action is planned in which the moving body moves in a direction different from a direction in which the moving body is currently traveling, the second acquisition unit may acquire information regarding the different direction as the partial sensor information.

[0013] In addition, in the first aspect, the first acquisition unit may acquire the sensor information including information acquired from a sensor related to millimeter wave or sonar information.

[0014] In addition, in the first aspect, the second output unit may output the control information including information on a predicted path of the moving object and a special action in controlling the moving object.

[0015] In addition, in the first aspect, the first output unit further includes a trained model management unit, The learned model management unit may execute a process for outputting the auxiliary information by utilizing a general-purpose large-scale language model.

[0016] In addition, in the first aspect, the auxiliary information may include information on a behavior of the moving object at a time after a control time of the moving object based on the control information output by the second output unit.

[0017] Also, in the first aspect, the first system may be connected to a navigation system mounted on the moving body, and the second system may not be connected to the navigation system mounted on the moving body.

[0018] In addition, in the first aspect, when the auxiliary information cannot be acquired by the third acquisition unit, the mobile body may impose stricter constraints on the autonomous behavior of the mobile body compared to when the third acquisition unit is able to acquire the auxiliary information.

[0019] A second aspect of the present disclosure is A program executed by an information processing device including a first system and a second system, The first system comprises: A first acquisition step of acquiring information on an environment in which a predetermined moving object is located, the information being acquired by a sensor installed in the moving object, as sensor information; a first output step of outputting auxiliary information for assisting in determining control information for controlling a behavior of the moving object based on the sensor information; Equipped with The second system comprises: A second acquisition step of acquiring some sensor information from some of the sensors installed in the moving object; a third acquisition step of acquiring the auxiliary information output by the first output step; a second output step of outputting the control information by using at least a part of the part of the sensor information or the auxiliary information; It is a program that executes control processing including the above.

[0020] A program according to an embodiment of the present disclosure is also provided as a program corresponding to an information processing device according to an embodiment of the present disclosure. Effect of the Invention

[0021] According to the present disclosure, it is possible to provide a technique capable of efficiently controlling a moving object. [Brief description of the drawings]

[0022] [Figure 1] 1 is a diagram illustrating an example of an information processing system including a vehicle system according to an embodiment of the present disclosure. [Diagram 2] 1 is a diagram illustrating an example of a hardware configuration of an in-vehicle device constituting an information processing system according to an embodiment of the present disclosure. [Diagram 3] 1 is a diagram illustrating an example of a functional configuration of an in-vehicle device constituting an information processing system according to an embodiment of the present disclosure. [Figure 4] 2 is a diagram illustrating an example of a flow of a Navigator process among processes executed by an in-vehicle device constituting an information processing system according to an embodiment of the present disclosure. FIG. [Diagram 5] 2 is a diagram illustrating an example of the flow of a Driver process among the processes executed by an in-vehicle device constituting an information processing system according to an embodiment of the present disclosure. FIG. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0023] [Embodiment] FIG. 1 is a diagram illustrating an example of an information processing system including a vehicle system according to an embodiment of the present disclosure (hereinafter, referred to as "this system").

[0024] As shown in FIG. 1, a vehicle system S according to an embodiment of the present disclosure includes an in-vehicle device 1, a vehicle sensor 10, and an HMI (Human Machine Interface) 20. and a control ECU 30 (Electronic Control Unit). These devices and equipment are connected to each other via a predetermined network such as a Controller Area Network (CAN) or Ethernet. Here, the vehicle on which the vehicle system S is mounted may include any vehicle, moving object, etc., including, for example, an automobile powered by electricity or gasoline.

[0025] The vehicle sensor 10 is composed of various sensors for detecting the external environment around the vehicle (an environment that may include other vehicles, pedestrians, structures, road shapes, etc.). Here, the external environment around the vehicle may include, for example, traffic participants (other vehicles, pedestrians, etc.), buildings such as commercial facilities, traffic signs installed on the side of the road, road markings formed on the road surface, dividing lines, traffic lights, utility poles, guardrails, animals, fallen objects, etc. In addition, the external environment around the vehicle may include, for example, weather, information about the road surface (roads, sidewalks, etc.) on which the mobile object can move and the condition of the road surface (wet road surface, uneven road surface, etc.). Specifically, as shown in FIG. 1, the vehicle sensor 10 includes a camera (front camera) installed to capture an image in front of the vehicle, a camera (side camera) installed to capture an image on the side of the vehicle, a camera (rear camera) installed to capture an image behind the vehicle, a millimeter wave radar, an ultrasonic radar, a LiDAR (Light Detection And Ranging), an acceleration sensor, a GNSS (Global Navigation Satellite System), an external microphone (sound collection device), on-board instruments, and the like.

[0026] Here, the camera is, for example, a camera using a charge coupled device (CCD) or a complementary metal oxide semiconductor (CMOS), etc. In the present disclosure, a total of multiple cameras may be installed in the front, sides, and rear of the vehicle.

[0027] In addition, the external microphone (sound collection device) is composed of a general-purpose microphone, etc., and is used to obtain information about sounds made by objects outside the vehicle, including, for example, ambulance and police car sirens, human voices, etc.

[0028] The HMI 20 presents various information to the driver and passengers of the vehicle and accepts various input operations. Specifically, as shown in Fig. 1, the HMI 20 includes, for example, a display, operation buttons, a microphone, various navigation systems, a speaker, and the like.

[0029] The control ECU 30 is connected to the in-vehicle device 1, transmits and receives various information, and executes various controls related to the operation of the vehicle. Specifically, as shown in Fig. 1, the control ECU includes individual ECUs that execute various controls, and executes various controls such as brake control, accelerator control, steering control, lights such as turn signals and lights, power unit, transmission, suspension, etc.

[0030] As shown in FIG. 1, the system may include a vehicle system S (vehicle-mounted device 1) managed by a driver of the vehicle or the like, and a server 2 managed by an administrator of the system or the like. The vehicle system S and the server 2 may be connected to each other via a predetermined network N such as the Internet. However, the network N is not a required component, and for example, NFC (Near Field Communication), Bluetooth (registered trademark), LAN (Local Area Network), etc. may be used. The server 2 acquires various types of information related to vehicle operation that are periodically transmitted from the vehicle system S (particularly the in-vehicle device 1) and is used to manage the various types of information.

[0031] FIG. 2 is a diagram illustrating an example of a hardware configuration of an in-vehicle device constituting an information processing system according to an embodiment of the present disclosure.

[0032] As shown in FIG. 2, the in-car device 1 includes a control unit 41, a read only memory (ROM) 42, a random access memory (RAM) 43, a bus 44, an input / output interface 45, a storage unit 46, and a communication unit 47.

[0033] The control unit 41 is configured with a microcomputer including a CPU, a GPU, an FPGA (Field-Programmable Gate Array), a semiconductor memory, etc. The control unit 41 executes various processes according to a program recorded in a ROM 42 or a program loaded from a storage unit 46 to a RAM 43. The RAM 43 also stores information necessary for the control unit 41 to execute various processes as appropriate.

[0034] The control unit 41, the ROM 42, and the RAM 43 are connected to one another via a bus 44. An input / output interface 45 is also connected to the bus 44. The input / output interface 45 is connected to the vehicle sensor 10, the HMI 20, the control ECU 30, a storage unit 46, a communication unit 47, and the like.

[0035] The storage unit 46 is configured with a hard disk drive (HDD), a solid state drive (SSD), etc., and stores various information. For example, the storage unit 46 stores various programs and the like required for executing various processes related to the present system.

[0036] The communication unit 47 controls mutual communications with other hardware and the like via a network N including the Internet.

[0037] The hardware configuration of the server 2 is not described here because it can be basically the same as the hardware configuration of the in-vehicle device 1. The cooperation of such various hardware and software enables the in-vehicle device 1 to execute various processes, which will be described later.

[0038] FIG. 3 is a diagram illustrating an example of a functional configuration of an in-vehicle device constituting an information processing system according to an embodiment of the present disclosure. Specifically, the functions of the control unit 41 of the in-car device 1 in this embodiment are roughly divided into two functions called a Navigator model unit 100 and a Driver model unit 140.

[0039] The Navigator model unit 100 executes processing related to a large-scale language model (trained model) that performs integrated cognition and decision-making. Specifically, the Navigator model unit 100 can provide appropriate instruction information to the Driver model by making a comprehensive judgment based on human language, input operations, background knowledge, and the like.

[0040] On the other hand, the Driver model unit 140 is a high-speed and lightweight calculation processing unit that executes calculations and inference processing based on limited information. It outputs control information based on limited sensor information and auxiliary information output by the Navigator model unit 100, which will be described later. That is, in the case of automatic driving control of a vehicle, it is usually desirable to realize a large amount of information and highly accurate recognition and judgment. On the other hand, such highly accurate information processing takes a long time. Therefore, in this system, the Driver model unit 140, which solves the immediate operational issues of the vehicle, maintains the processing speed, while the Navigator model unit 100 assists in mid- to long-term highly accurate recognition and judgment, thereby ensuring a processing time that is highly accurate overall and can withstand the control of an automatic driving vehicle. For this reason, in this system, the output period in which the Navigator model unit 100 outputs various information is usually set to be longer than the output period in which the Driver model unit 140 outputs various information. In this system, the arithmetic processing and output processing by the Driver model unit 140 are configured to be lightweight compared to the arithmetic processing and output processing by the Navigator model unit 100, and are configured to realize control of the vehicle in a time (short time) necessary and sufficient for control of an autonomous vehicle.

[0041] As shown in FIG. 3, in the control unit 41 of the in-car device 1, a Navigator model unit 100 and a Driver model unit 140 function by executing various programs. In addition, a model information DB 300 and a map information DB 400 are stored in one area of ​​the storage unit 46 of the vehicle-mounted device 1 . The model information DB300 stores trained models (programs, etc. that specify the results of learning that have been subjected to statistical processing using the in-vehicle device 1 or other large-scale language models, etc.) for outputting auxiliary information, described below, in response to input information from various sensors, and trained models (programs, etc. that specify the results of learning that have been subjected to statistical processing using the in-vehicle device 1 or other hardware, etc.) for outputting control information, described below, in response to input information from various sensors. Moreover, general-purpose map information and the like are stored in the map information DB 400. The map information is, for example, information such as latitude and longitude information related to roads, buildings, etc. displayed in a two-dimensional or three-dimensional format. In this system, the map information stored in the map information DB 400 is appropriately used in situations where various processes, which will be described later, are required.

[0042] The functional configuration related to the Navigator model unit 100 will be described. The navigator model unit 100 includes a first sensor information acquisition unit 120 and an auxiliary information generation unit 121 .

[0043] The first sensor information acquisition unit 120 acquires and manages various pieces of information acquired by all of the sensors constituting the vehicle sensor 10 (hereinafter, referred to as “all sensor information”).

[0044] The auxiliary information generation unit 121 generates information (hereinafter referred to as "auxiliary information") to assist the calculation processing of the driver model unit 140 based on all sensor information acquired by the first sensor information acquisition unit 120 and the contents of the trained model stored in the model information DB300. Here, the auxiliary information is typically information related to a medium- to long-term action plan or driving instructions for the vehicle, and includes instructions regarding the vehicle's actions and the reasons for the decision to issue those instructions. Specifically, for example, auxiliary information is instruction information similar to language, such as "The traffic light is red, please stop the vehicle," "There is a sharp curve ahead, please slow down gradually," "You will turn left ahead, please turn on your blinker," and "There is an intersection ahead, please go right." According to this example of auxiliary information, for example, information such as "Please stop the vehicle" is an instruction regarding the behavior of the vehicle, and information such as "The traffic light is red" is the reason for the decision to issue that instruction. Auxiliary information includes, for example, information with such content. In this way, by including information on medium- to long-term action plans or operation plans, the system can realize more appropriate and efficient operation control in the Driver model unit 140.

[0045] Here, the contents of the auxiliary information will be described in more detail. Specifically, for example, while the vehicle is traveling on a lane of a highway, the auxiliary information output by the Navigator model unit 100 is information of the content "move to the left lane to get off the highway 1 km ahead". Then, including information on the action the vehicle should take in the future, the Driver model unit 140 outputs specific control information, for example, "(1) blink the left turn signal, (2) change lanes 3 seconds after blinking, (3) decelerate to 90 km / h after changing lanes", as a result of processing based on the auxiliary information and some sensor information. That is, by including information on getting off the highway in the auxiliary information during processing by the Driver model unit 140, it is possible to derive a processing result such as (3) decelerate to 90 km / h after changing lanes, which can contribute to more comfortable and safer travel for the occupants.

[0046] This auxiliary information includes, for example, information on a more accurate mid- to long-term action plan or operation instructions obtained by the Driver model unit 140 over a long calculation time based on a large amount of information (for example, all sensor information described later) compared to the information processed by the Driver model unit 140. In contrast, the processing by the Driver model unit 140 described later generates more specific control instructions and the like regarding vehicle operation in a short term than the mid- to long-term action plan or operation plan generated by the Navigator model unit 100 based on limited information (for example, some sensor information described later) and the generated auxiliary information. By combining and executing these two control processes with different roles, automatic driving control of the vehicle can be realized safely and efficiently.

[0047] The functional configuration related to the driver model unit 140 will be described. The driver model unit 140 includes a second sensor information acquisition unit 160, a control information generation unit 161, and a control information output unit 162.

[0048] The second sensor information acquisition unit 160 acquires and manages some or all of the information (hereinafter referred to as "some sensor information") including image information acquired by a camera installed to capture an image of the area in front of the vehicle from among the sensors constituting the vehicle sensor 10, various information acquired by a millimeter wave radar or ultrasonic radar installed to acquire information about the area in front of the vehicle, and various information acquired by on-board instruments.

[0049] The control information generation unit 161 generates information (hereinafter referred to as "control information") regarding specific control instructions for controlling the operation of the vehicle based on some of the sensor information acquired by the second sensor information acquisition unit 160, the auxiliary information generated by the auxiliary information generation unit 121, and the contents of the trained model stored in the model information DB300. Here, the control information is, for example, instruction information for each control ECU 30. Specifically, the control information is information for controlling the movement of a moving body, and in the case of the vehicle in this embodiment, the control information is information on the vehicle speed, information on the acceleration / deceleration of the vehicle, information on the action plan such as the movement trajectory that the vehicle should follow, information on the direction in which the vehicle should proceed, and the like. The control information may also include information on driving instructions to the driver, including, for example, a predicted route of the vehicle and instructions on special actions. The predicted route is, for example, a predicted future route of the vehicle calculated or planned based on some sensor information. The special action is, for example, specific instruction information on operations such as activation of left and right turn signals (including flashing hazard lights), sudden deceleration, gear shift instructions, and horn instructions. Each control ECU 30 can realize appropriate automatic driving of the vehicle by executing control of the vehicle based on the control information.

[0050] The control information output unit 162 provides the control information generated by the control information generation unit 161 to the control ECU 30 of the vehicle system S and the like.

[0051] FIG. 4 is a diagram illustrating an example of the flow of Navigator processing among the processing executed by an in-vehicle device constituting an information processing system according to an embodiment of the present disclosure.

[0052] In step S1, the first sensor information acquisition unit 120 acquires and manages all sensor information from all sensors constituting the vehicle sensor 10.

[0053] In step S2, the auxiliary information generating unit 121 generates auxiliary information based on all the sensor information acquired by the first sensor information acquiring unit 120 and the contents of the trained model stored in the model information DB 300. This ends the Navigator process of the in-vehicle device 1.

[0054] FIG. 5 is a diagram illustrating an example of the flow of Driver processing among the processing executed by an in-vehicle device constituting an information processing system according to an embodiment of the present disclosure.

[0055] In step S21, the second sensor information acquisition unit 160 acquires and manages some or all of the sensor information, including image information acquired by a camera installed to capture an image of the area in front of the vehicle among the sensors that constitute the vehicle sensor 10, various information acquired by a millimeter wave radar or ultrasonic radar installed to acquire information about the area in front of the vehicle, and various information acquired by on-board instruments.

[0056] In step S22, the control information generation unit 161 generates control information regarding specific control instructions for controlling the operation of the vehicle based on the partial sensor information acquired by the second sensor information acquisition unit 160, the auxiliary information generated by the auxiliary information generation unit 121, and the contents of the trained model stored in the model information DB300.

[0057] In step S23, the control information output unit 162 provides the control information generated by the control information generation unit 161 to the control ECU 30 of the vehicle system S. This causes the Driver process of the in-vehicle device 1 to end.

[0058] Although one embodiment of the present disclosure has been described above, the present disclosure is not limited to the above-described embodiment, and modifications and improvements within the scope that can achieve the object of the present disclosure are included in the present disclosure.

[0059] [Other embodiments] In the above embodiment, the vehicle is described as a general-purpose autonomous vehicle, but is not limited thereto. Vehicles to which the present system is applicable may include any type of moving body, regardless of shape or power source, such as automobiles, trucks, motorcycles, railroad cars, bicycles, robots, AGVs (Automatic Guided Vehicles), drones, etc. Furthermore, the information processing device or information processing system according to the present system does not need to function independently as an information processing device, and may be provided as an integrated part with a vehicle (mobile body), for example.

[0060] In the above embodiment, the second sensor information acquisition unit 160 has been described as acquiring only the information on the camera installed to be able to capture an image of the front of the vehicle, the millimeter wave radar or ultrasonic radar installed to be able to acquire information of the front, and the on-board instrument as the partial sensor information, but is not limited thereto. The administrator of the system can arbitrarily design which types of sensor information are to be the partial sensor information. Specifically, for example, when the vehicle is backing up or reversing, some of the sensor information may include all or part of images acquired from a camera capable of capturing images of the rear, various information acquired from a millimeter wave radar or ultrasonic radar installed so as to be able to acquire information about the rear, images acquired from a camera capable of capturing images of the side, etc.

[0061] Furthermore, the system may change the selection of sensors to be acquired as the partial sensor information depending on the vehicle situation. Specifically, for example, the system normally acquires only the front camera as the partial sensor information, but when changing lanes, the system may acquire images from the left, right, or rear cameras (e.g., in the direction of the lane change) as the partial sensor information. The lane change referred to here may include vehicle operations such as simple lateral movement, left turn, right turn, etc. The advantage of the Driver model unit 140 of this system acquiring only some of the sensor information rather than all of the sensor information to execute various processes is that the Driver model unit 140 can process information related to vehicle operation by acquiring information only from some of the sensor information, and thus can process faster than when acquiring and processing information from all of the sensors.

[0062] In addition, although not described in the above-described embodiment, the trained model used in the Navigator model unit 100 or the Driver model unit 140 may be updated as appropriate before or after generating various information. For example, the in-vehicle device 1 may acquire a newly trained trained model using the output auxiliary information, control information, etc., and update the contents of the trained model stored in the model information DB 300. Note that the learning method used here may be, for example, a method classified into various deep learning methods such as DNN (Deep Neural Network) or a combination thereof.

[0063] Although the above embodiment has been described simply, the information stored in the model information DB300 is assumed to have undergone learning processing in advance by the in-vehicle device 1 or other hardware, etc. Specifically, the information stored in the model information DB300 is model information that has been sufficiently adjusted for application to autonomous driving by, for example, learning a wide range of information including various videos, texts, natural languages, etc., and labeling separately collected images of the vehicle traveling with information related to vehicle control, supplementary information, etc. The learning data used for these learning processes does not necessarily have to be acquired from a vehicle to which autonomous driving is applied, but may be learning data from another vehicle or data generated by a method such as simulation.

[0064] Furthermore, in the learning related to this system, for example, a general-purpose large language model such as ChatGPT or BERT (Bidirectional Encoder Representations from Transformers) may be used. Note that even when using these large language models, for example, various information acquired regarding the movement of the vehicle may be further trained to generate a trained model.

[0065] Although not described in the above embodiment, the present system may, for example, connect only the navigation system installed in the vehicle to the Navigator model unit 100, and prohibit the connection between the navigation system and the Driver model unit 140. This allows the Driver model unit 140 to perform only the calculations required for controlling the behavior of the vehicle without performing unnecessary calculations, thereby ensuring a sufficient calculation speed.

[0066] Although not described in the above embodiment, the control information generating unit 161 of the driver model unit 140 may have a function of setting heavier constraints on vehicle operation (autonomous behavior) when the auxiliary information generated by the auxiliary information generating unit 121 cannot be acquired, compared to when the auxiliary information can be acquired. For example, the control information generating unit 161 may adjust the so-called autonomous driving level between 1 and 5, such as restricting the autonomous driving level to autonomous driving level 1, which requires the driver to hold the steering wheel and monitor it, or restricting it to autonomous driving level 2, which requires the driver to only monitor the road ahead and allows for immediate transfer of steering to the driver. In this case, if the control information generator 161 is unable to acquire the auxiliary information, for example, it may not be necessary to generate the control information, or may output the control information in accordance with the constraints of the autonomous driving level described above.

[0067] In the above embodiment, the external environment around the vehicle has been described as an example of the environment in which the moving object is located. However, the environment in which the moving object is located is not necessarily limited to the external environment around the vehicle.

[0068] Although not described in the above embodiment, the system may output the auxiliary information as information in a form in which external world information is expressed as a vector close to natural language. The system may output the auxiliary information as information in a form that can be understood by humans as natural language.

[0069] Although the above embodiment has been described simply, the types and number of various sensors included in the vehicle sensor 10 can be determined at the discretion of the administrator of the system, etc. In the present system, for example, any sensor different from the above-mentioned sensors may be included as part of the configuration of the vehicle sensor 10, or unnecessary sensors may be omitted from the configuration of the vehicle sensor 10. Furthermore, since the number of various sensors in the vehicle sensor 10 is arbitrary, the present system may, for example, install a plurality of cameras, microphones, etc. at arbitrary positions on the vehicle.

[0070] Furthermore, the above-described series of processes can be executed by hardware or software. In other words, the functional configurations in FIG. 3 and the like are merely examples and are not particularly limited. That is, it is sufficient that the information processing system is provided with a function capable of executing the above-mentioned series of processes as a whole, and the type of functional block used to realize this function is not limited to the example in Fig. 3, etc. Furthermore, the location of the functional block is not limited to the example in Fig. 3, etc., and may be arbitrary. Furthermore, one functional block may be configured as a single piece of hardware, may be configured as a single piece of software, or may be configured as a combination of both.

[0071] Furthermore, the number of different hardware components constituting this system and the number of users are optional, and the system may also be configured to include other hardware.

[0072] Furthermore, when the series of processes is executed by software, the programs constituting the software are installed into a computer or the like from a network or a recording medium.

[0073] The computer may be a computer that is built into dedicated hardware, or a computer that is capable of executing various functions by installing various programs.

[0074] Furthermore, the recording medium containing such a program may not only be constituted by a removable medium (not shown) provided separately from the device main body in order to provide the program to a user, etc., but may also be constituted by a recording medium that is provided to the user in a state in which it is pre-installed in the device main body.

[0075] In addition, in this specification, the term "system" refers to an overall device that is composed of a plurality of devices or a plurality of means.

[0076] Even when these other embodiments are adopted, the effects of the present embodiment are exhibited. In addition, the present embodiment can be appropriately combined with other embodiments, and other embodiments can be appropriately combined with each other.

[0077] In summary, the information processing system applied to the present disclosure can take various forms having the following configurations. An information processing apparatus including a first system and a second system, The first system comprises: A first acquisition unit (e.g., a first sensor information acquisition unit 120) that acquires, as sensor information, information about an environment in which a predetermined moving object is located, the information being acquired by a sensor installed in the moving object; a first output unit (e.g., an auxiliary information generating unit 121) that outputs auxiliary information for assisting in determining control information for controlling a behavior of the moving object based on the sensor information; Equipped with The second system comprises: A second acquisition unit (e.g., a second sensor information acquisition unit 160) that acquires some sensor information from some of the sensors installed in the moving object; a third acquisition unit (for example, a control information generation unit 161) that acquires the auxiliary information output by the first output unit; a second output unit (for example, a control information output unit 162) that outputs the control information by using at least a part of the partial sensor information or the auxiliary information; Equipped with Any information processing device will suffice. [Explanation of symbols]

[0078] S Vehicle System 1 In-vehicle device 100 Navigator model part 120 First sensor information acquisition unit 121 Auxiliary information generation section 140 Driver model section 160 Second sensor information acquisition unit 161 Control information generation unit 162 Control information output unit 300 Model Information DB 400 Map Information DB 10 Vehicle Sensors 20 Human Machine Interfaces 30 Control ECU 41 Control section 2 Server

Claims

1. An information processing apparatus including a first system and a second system, The first system comprises: A first acquisition unit that acquires, as sensor information, information about an environment in which a predetermined moving object is located, the information being acquired by a sensor installed in the moving object; a first output unit that outputs auxiliary information for assisting in determining control information for controlling a behavior of the moving object based on the sensor information; Equipped with The second system is A second acquisition unit that acquires some sensor information from some of the sensors installed in the moving object; a third acquisition unit that acquires the auxiliary information output by the first output unit; a second output unit that outputs the control information by using at least a part of the sensor information or the auxiliary information; Equipped with The first output unit outputs the auxiliary information including an instruction regarding an action of the moving object and a reason for determining to give the instruction. Information processing device.

2. An information processing device including a first system and a second system, The first system comprises: A first acquisition unit that acquires, as sensor information, information about an environment in which a predetermined moving object is located, the information being acquired by a sensor installed in the moving object; a first output unit that outputs auxiliary information for assisting in determining control information for controlling a behavior of the moving object based on the sensor information; Equipped with The second system is A second acquisition unit that acquires some sensor information from some of the sensors installed in the moving object; a third acquisition unit that acquires the auxiliary information output by the first output unit; a second output unit that outputs the control information by using at least a part of the sensor information or the auxiliary information; Equipped with When the third acquisition unit is unable to acquire the auxiliary information, the moving body imposes stricter restrictions on the autonomous behavior of the moving body than when the third acquisition unit is able to acquire the auxiliary information. Information processing device.

3. The output period of the first output section is longer than the output period of the second output section. The information processing device according to claim 1 .

4. The first acquisition unit acquires the sensor information including information acquired from a camera provided so as to be able to observe directions including front, rear, left and right of the moving body, and a sound collecting device installed outside the moving body. The information processing device according to claim 1 .

5. The sound collecting device installed outside the moving body is provided on the left and right sides of the moving body. The information processing device according to claim 4.

6. The second acquisition unit acquires, as the partial sensor information, information acquired by an imaging device capable of capturing an image of a forward area in a traveling direction of the moving body. The information processing device according to claim 1 .

7. When an action of moving in a direction different from a direction in which the moving object is currently traveling is planned, the second acquisition unit acquires information regarding the different direction as the partial sensor information. The information processing device according to claim 1 .

8. The first acquisition unit acquires the sensor information including information acquired from a sensor related to millimeter wave or sonar information. The information processing device according to claim 4.

9. The second output unit outputs the control information including information on a predicted path of the moving object and a special action in controlling the moving object. The information processing device according to claim 1 .

10. The first output unit further includes a trained model management unit, The learned model management unit executes a process for outputting the auxiliary information by utilizing a large-scale language model. The information processing device according to claim 1 .

11. the auxiliary information includes information regarding a behavior of the moving object at a time after a control time of the moving object based on the control information output by the second output unit; The information processing device according to claim 1 .

12. the first system is connected to a navigation system mounted on the moving object, and the second system is not connected to the navigation system mounted on the moving object; The information processing device according to claim 1 .

13. A program executed by an information processing device including a first system and a second system, The first system comprises: A first acquisition step of acquiring information on an environment in which a predetermined moving object is located, the information being acquired by a sensor installed in the moving object, as sensor information; a first output step of outputting auxiliary information for assisting in determining control information for controlling a behavior of the moving object based on the sensor information; Equipped with The second system is A second acquisition step of acquiring some sensor information from some of the sensors installed in the moving object; a third acquisition step of acquiring the auxiliary information output by the first output step; a second output step of outputting the control information by using at least a part of the part of the sensor information or the auxiliary information; Executing a control process including The first output step outputs the auxiliary information including an instruction regarding an action of the moving object and a reason for deciding to give the instruction. program.

Images