Information processing device, vehicle, information processing method, and program

Abstract

An information processing device according to the present disclosure includes: an information acquisition unit capable of acquiring a plurality of pieces of information associated with a vehicle; an estimation unit that calculates an index value on the basis of the plurality of pieces of information acquired by the information acquisition unit, and estimates a plurality of control variables on the basis of the index value using deep learning; and a driving control unit that executes driving control of the vehicle on the basis of the plurality of control variables.

Description

Technical Field

[0001] The present disclosure relates to an information processing apparatus, a vehicle, an information processing method, and a program that include multivariate analysis using deep learning. Background Art

[0002] A vehicle having an autonomous driving function is described in Japanese Unexamined Patent Application Publication No. 2022-035198. Summary of the Invention

[0003] Problems to be Solved by the Invention

[0004] In existing autonomous driving vehicles, it is difficult to analyze the effects of factors such as air resistance and friction generated during driving on the road while driving.

[0005] Means for Solving the Problems

[0006] According to one embodiment of the present disclosure, there is provided an information processing apparatus including: an information acquisition unit that can acquire a plurality of pieces of information associated with a vehicle; an inference unit that calculates an index value based on the plurality of pieces of information acquired by the information acquisition unit and uses deep learning to infer a plurality of control variables based on the index value; and a driving control unit that executes driving control of the vehicle based on the plurality of control variables.

[0007] According to one embodiment of the present disclosure, in the above information processing apparatus, the inference unit infers the plurality of control variables based on the plurality of pieces of information through multivariate analysis based on an integration method using the deep learning.

[0008] According to one embodiment of the present disclosure, in the above information processing apparatus, the information acquisition unit acquires the plurality of pieces of information in units of the specified cycle, and the inference unit and the driving control unit use the plurality of pieces of information acquired in units of the specified cycle to perform inference of the plurality of control variables and driving control of the vehicle in units of the specified cycle.

[0009] According to one embodiment of the present disclosure, in the above information processing apparatus, a strategy setting unit is further included, the strategy setting unit sets a driving strategy until the vehicle reaches a destination, the driving strategy includes at least one theoretical value of an optimal route to the destination, a driving speed, an inclination, and braking, and the driving control unit includes a strategy update unit that updates the driving strategy based on a difference between the plurality of control variables and the theoretical value.

[0010] According to an embodiment of the present disclosure, in the above information processing device, the information acquisition unit includes a sensor disposed below the vehicle, which is capable of detecting the temperature, material, and inclination of the ground for driving.

[0011] According to an embodiment of the present disclosure, there is provided an information processing device. The information processing device includes: an acquisition unit that acquires a plurality of pieces of information related to the vehicle from a detection unit including a sensor as a cycle for detecting the surrounding conditions of the vehicle, and the sensor detects the surrounding conditions of the vehicle at a second cycle shorter than a first cycle for photographing the surrounding of the vehicle; a calculation unit that calculates an index value related to the surrounding conditions of the vehicle based on the acquired plurality of pieces of information, and calculates a control variable for controlling the behavior of the vehicle based on the calculated index value; and a control unit that controls the behavior of the vehicle based on the calculated control variable.

[0012] According to an embodiment of the present disclosure, in the above information processing device, the calculation unit calculates the control variable based on the index value through multivariate analysis using an integration method based on deep learning.

[0013] According to an embodiment of the present disclosure, in the above information processing device, the acquisition unit acquires the plurality of pieces of information in units of one billionth of a second, and the calculation unit uses the plurality of pieces of information acquired in units of one billionth of a second to perform the calculation of the index value and the control variable in units of one billionth of a second.

[0014] According to an embodiment of the present disclosure, in the above information processing device, the calculation unit predicts a collision of an object with the vehicle based on the acquired plurality of pieces of information, and in a case where the predicted result indicates an inevitable collision, calculates a control variable corresponding to damage in which the damage generated by the vehicle in the inevitable collision is below a predetermined threshold as the control variable.

[0015] According to an embodiment of the present disclosure, in the above information processing device, the damage generated by the vehicle is at least one of a deformation position and a deformation amount of the vehicle.

[0016] According to an embodiment of the present disclosure, in the above information processing device, the control variable corresponding to the damage generated by the vehicle is at least one of a collision angle and a vehicle speed of the vehicle.

[0017] According to an embodiment of the present disclosure, there is provided a vehicle including the above information processing device.

[0018] According to an embodiment of the present disclosure, a program is provided that causes a computer to function as the information processing device.

[0019] According to an embodiment of the present disclosure, an information processing device is provided, characterized by including a processor that acquires detection information obtained by detecting the surrounding conditions of a moving body, point information obtained by capturing an object in the captured image as points, and identification information obtained by identifying the object, calculates a control variable based on the correlation between a variable related to the surrounding conditions of the moving body calculated from the detection information and a control variable inferred using the variable, and controls the autonomous driving of the moving body based on the control variable, the point information, and the identification information.

[0020] In the information processing device, it may be that a coefficient of the variable with respect to the control variable is determined in advance, and the processor uses the coefficient to calculate the control variable.

[0021] In any of the information processing devices, it may be that the processor performs the inference process of the control variable at a second period longer than a first period for acquiring the detection information.

[0022] In any of the information processing devices, it may be that the processor performs the calculation process of the control variable at a third period shorter than the second period.

[0023] In any of the information processing devices, it may be that the processor infers the control variable through multivariate analysis based on an integration method using deep learning.

[0024] In any of the information processing devices, it may be that the processor acquires the point information and the identification information from different other processors respectively.

[0025] According to an embodiment of the present disclosure, an information processing method is provided, in which a computer acquires detection information obtained by detecting the surrounding conditions of a moving body, point information obtained by capturing an object in the captured image as points, and identification information obtained by identifying the object, calculates a control variable based on the correlation between a variable related to the surrounding conditions of the moving body calculated from the detection information and a control variable inferred using the variable, and controls the autonomous driving of the moving body based on the control variable, the point information, and the identification information.

[0026] According to an embodiment of the present disclosure, there is provided a program that causes a computer to acquire detection information obtained by detecting the surrounding conditions of a moving body, point information obtained by capturing an object as points in a captured image, and identification information obtained by identifying the object, calculate a control variable based on the correlation between a variable related to the surrounding conditions of the moving body calculated from the detection information and a control variable inferred using the variable, and control the autonomous driving of the moving body based on the control variable, the point information, and the identification information.

[0027] According to an embodiment of the present disclosure, there is provided an information processing apparatus including a processor that acquires detection information from a detection unit that detects the surrounding conditions of the moving body at a second period shorter than a first period for capturing the surrounding of the moving body, divides a period in which an obstacle collision prediction for the moving body can be performed based on the detection information into a plurality of periods, and calculates, in each of the plurality of periods in time series of the plurality of periods, a control variable for controlling the autonomous driving of the moving body using a plurality of input parameters based on the detection information.

[0028] In the above information processing apparatus, when calculating the control variable in the second and subsequent periods of the plurality of periods, the processor may calculate the control variable by reducing the input parameters compared to the previous period.

[0029] According to an embodiment of the present disclosure, there is provided an information processing method in which a computer performs the following processing: acquiring detection information from a detection unit that detects the surrounding conditions of the moving body at a second period shorter than a first period for capturing the surrounding of the moving body, dividing a period in which an obstacle collision prediction for the moving body can be performed based on the detection information into a plurality of periods, and calculating, in each of the plurality of periods in time series of the plurality of periods, a control variable for controlling the autonomous driving of the moving body using a plurality of input parameters based on the detection information.

[0030] According to an embodiment of the present disclosure, there is provided a program that causes a computer to perform the following processing: acquiring detection information from a detection unit that detects the surrounding conditions of the moving body at a second period shorter than a first period for capturing the surrounding of the moving body; dividing a period in which an obstacle collision prediction for the moving body can be performed based on the detection information into a plurality of periods; and calculating, in each of the plurality of periods in time series of the plurality of periods, a control variable for controlling the autonomous driving of the moving body using a plurality of input parameters based on the detection information.

[0031] According to an embodiment of the present disclosure, there is provided an information processing apparatus including a processor that acquires detection information from a detection unit. The detection unit detects the situation around the moving body at a second period shorter than a first period for photographing the surroundings of the moving body. Based on the detection information, the distance between the moving body and an obstacle is calculated, and based on the distance and a predetermined margin, the approach of the obstacle is predicted.

[0032] In the above information processing apparatus, it may be that the margin is predetermined for the occupants of the moving body.

[0033] In any of the above information processing apparatuses, it may be that the margin is predetermined according to the age of the occupant.

[0034] In any of the information processing apparatuses, it may be that the processor changes the margin according to the moving speed of the moving body or the obstacle.

[0035] In any of the information processing apparatuses, it may be that the processor changes the margin according to the braking distance of the moving body or the obstacle.

[0036] In any of the information processing apparatuses, it may be that the processor controls the autonomous driving of the moving body based on the predicted result.

[0037] According to an embodiment of the present disclosure, there is provided an information processing method in which a computer performs the following processing: acquiring detection information from a detection unit, the detection unit detecting the situation around the moving body at a second period shorter than a first period for photographing the surroundings of the moving body, calculating the distance between the moving body and an obstacle based on the detection information, and predicting the approach of the obstacle based on the distance and a predetermined margin.

[0038] According to an embodiment of the present disclosure, there is provided a program that causes a computer to perform the following processing: acquiring detection information from a detection unit, the detection unit detecting the situation around the moving body at a second period shorter than a first period for photographing the surroundings of the moving body, calculating the distance between the moving body and an obstacle based on the detection information, and predicting the approach of the obstacle based on the distance and a predetermined margin.

[0039] According to an embodiment of the present disclosure, there is provided an information processing apparatus including a processor that obtains detection information from a detection unit. The detection unit detects the situation around the moving body at a second period shorter than a first period for photographing the surroundings of the moving body. Based on the detection information, a collision of an obstacle with the moving body is predicted. In a case where the predicted result indicates an inevitable collision, a control variable is calculated, and the control variable controls the autonomous driving of the moving body so that the obstacle collides with a predetermined part of the moving body.

[0040] In the above information processing apparatus, it may be that the processor sets, as the predetermined part, a part where damage generated when the obstacle collides with the moving body satisfies a predetermined criterion.

[0041] In any of the above information processing apparatuses, it may be that the part where the damage satisfies the predetermined criterion is a part of the moving body having a rigidity equal to or greater than a predetermined threshold.

[0042] In any of the above information processing apparatuses, it may be that the part where the damage satisfies the predetermined criterion is a part at a distance equal to or greater than a predetermined threshold from an engine, a motor, or a battery mounted on the moving body.

[0043] In any of the above information processing apparatuses, it may be that the part where the damage satisfies the predetermined criterion is a part at a distance equal to or greater than a predetermined threshold from an occupant in the moving body.

[0044] In any of the above information processing apparatuses, it may be that the processor receives an input of a design document related to the moving body and determines, based on the design document, the part where the damage satisfies the predetermined criterion.

[0045] According to an embodiment of the present disclosure, there is provided an information processing method in which a computer performs the following processing: obtaining detection information from a detection unit, the detection unit detecting the situation around the moving body at a second period shorter than a first period for photographing the surroundings of the moving body; predicting, based on the detection information, a collision of an obstacle with the moving body; and calculating, in a case where the predicted result indicates an inevitable collision, a control variable that controls the autonomous driving of the moving body so that the obstacle collides with a predetermined part of the moving body.

[0046] According to an embodiment of the present disclosure, a program is provided that causes a computer to perform the following processing: obtain detection information from a detection unit that detects the state of the surroundings of a moving body at a second period shorter than a first period for photographing the surroundings of the moving body; predict a collision of an obstacle with the moving body based on the detection information; and calculate a control variable that controls the autonomous driving of the moving body so that the obstacle collides with a predetermined part of the moving body when the predicted result indicates an inevitable collision.

[0047] The information processing apparatus of the present disclosure includes a processor that obtains detection information from a detection unit that detects the state of the surroundings of a moving body at a second period shorter than a first period for photographing the surroundings of the moving body, calculates a control variable for controlling the autonomous driving of the moving body based on the detection information, and notifies a set contact method when calculating a control variable that reduces the damage caused to the moving body in an inevitable collision.

[0048] The information processing method of the present disclosure causes a computer to perform the following processing: obtain detection information from a detection unit that detects the state of the surroundings of a moving body at a second period shorter than a first period for photographing the surroundings of the moving body; calculate a control variable for controlling the autonomous driving of the moving body based on the detection information; and notify a set contact method when calculating a control variable that reduces the damage caused to the moving body in an inevitable collision.

[0049] The program of the present disclosure causes a computer to perform the following processing: obtain detection information from a detection unit that detects the state of the surroundings of a moving body at a second period shorter than a first period for photographing the surroundings of the moving body, calculate a control variable for controlling the autonomous driving of the moving body based on the detection information, and notify a set contact method when calculating a control variable that reduces the damage caused to the moving body in an inevitable collision.

[0050] According to an embodiment of the present disclosure, there is provided an information processing apparatus including: an acquisition unit that acquires a plurality of pieces of information related to the vehicle from a detection unit of a sensor that detects the surrounding conditions of the vehicle; a calculation unit that calculates an index value related to the surrounding conditions of the vehicle based on the acquired plurality of pieces of information, and calculates a control variable for controlling the behavior of the vehicle based on the calculated index value; and a control unit that controls the behavior of the vehicle based on the calculated control variable. The calculation unit predicts a collision of an object with the vehicle based on the acquired plurality of pieces of information, and calculates a control variable in a case where the predicted result indicates an inevitable collision, the control variable being such that the damage caused by the vehicle in the inevitable collision and a re-collision with another object after the collision is equal to or less than a predetermined threshold value.

[0051] In any of the above information processing apparatuses, it may be that the acquisition unit acquires information related to the surrounding conditions of the vehicle from another vehicle that is a collision object.

[0052] In any of the above information processing apparatuses, it may be that the acquisition unit acquires information related to the surrounding conditions of the vehicle from an external device that sets a travel path for the vehicle to travel.

[0053] In any of the above information processing apparatuses, it may be that the calculation unit calculates the control variable based on the index value through multivariate analysis using an integration method based on deep learning.

[0054] In any of the above information processing apparatuses, it may be that the damage caused by the vehicle is at least one of a deformation position and a deformation amount of the vehicle.

[0055] In any of the above information processing apparatuses, it may be that the control variable corresponding to the damage caused by the vehicle is at least one of a collision angle and a vehicle speed of the vehicle.

[0056] According to an embodiment of the present disclosure, there is provided a vehicle including: the above information processing apparatus; a sensor that is connected to the information processing apparatus and detects the surrounding conditions of the vehicle; and a reception unit that receives the plurality of pieces of information from the external device.

[0057] It should be noted that the above summary of the disclosure does not list all the necessary features of the present disclosure. In addition, sub-combinations of these feature groups may also constitute the object of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1It is a schematic diagram showing an example of a vehicle equipped with a Central Brain.

[0059] Figure 2 It is a block diagram showing an example of an information processing device according to a first embodiment of the present disclosure.

[0060] Figure 3 It is a block diagram showing an example of an information processing device according to a second embodiment of the present disclosure.

[0061] Figure 4 It is a schematic diagram showing an example of the structure of an inference unit included in an information processing device according to a third embodiment of the present disclosure.

[0062] Figure 5 It is a schematic diagram showing an example of the processing contents of an information acquisition unit, an inference unit, and a driving control unit included in an information processing device according to a third embodiment of the present disclosure.

[0063] Figure 6 It is a flowchart showing an example of the process of a traveling speed control process according to a third embodiment of the present disclosure.

[0064] Figure 7 It is a schematic diagram showing an example of the structure of an inference unit included in an information processing device according to a fourth embodiment of the present disclosure.

[0065] Figure 8 It is a schematic diagram showing an example of the processing contents of an information acquisition unit, an inference unit, and a driving control unit included in an information processing device according to a fourth embodiment of the present disclosure.

[0066] Figure 9 It is a flowchart showing an example of the process of a traveling speed control process strategy update process according to a fourth embodiment of the present disclosure.

[0067] Figure 10 It is a schematic diagram showing a modified example of the structure of an information processing device according to a fourth embodiment of the present disclosure.

[0068] Figure 11 It is a diagram schematically showing the danger prediction ability of an AI for ultra-high-performance autonomous driving according to a fifth embodiment of the present disclosure.

[0069] Figure 12 It is a diagram schematically showing an example of a network structure inside a vehicle according to a fifth embodiment of the present disclosure.

[0070] Figure 13 It is a flowchart executed by a Central Brain according to a fifth embodiment of the present disclosure.

[0071] Figure 14 It is the first explanatory diagram showing a control example of central-brain-based autonomous driving according to the fifth embodiment of the present disclosure.

[0072] Figure 15 It is the second explanatory diagram showing a control example of central-brain-based autonomous driving according to the fifth embodiment of the present disclosure.

[0073] Figure 16 It is the third explanatory diagram showing a control example of central-brain-based autonomous driving according to the fifth embodiment of the present disclosure.

[0074] Figure 17 It is the fourth explanatory diagram showing a control example of central-brain-based autonomous driving according to the fifth embodiment of the present disclosure.

[0075] Figure 18 It is the fifth explanatory diagram showing a control example of central-brain-based autonomous driving according to the fifth embodiment of the present disclosure.

[0076] Figure 19 As the sixth explanatory diagram showing a control example of central-brain-based autonomous driving according to the fifth embodiment of the present disclosure, it is a schematic diagram showing a state in which other vehicles are driving around the vehicle.

[0077] Figure 20 It is a block diagram showing an example of the structure of an information processing device including a central brain according to the fifth embodiment of the present disclosure.

[0078] Figure 21 It is a flowchart executed by the central brain according to the fifth embodiment of the present disclosure.

[0079] Figure 22 It is a schematic diagram showing an example of a vehicle equipped with a central brain according to the sixth embodiment of the present disclosure.

[0080] Figure 23 It is the first block diagram showing an example of the structure of an information processing device according to the sixth embodiment of the present disclosure.

[0081] Figure 24 It is the second block diagram showing an example of the structure of an information processing device according to the seventh embodiment of the present disclosure.

[0082] Figure 25 It is an explanatory diagram showing an example of point information output by MoPU.

[0083] Figure 26 It is the third block diagram showing an example of the structure of an information processing device according to the tenth embodiment of the present disclosure.

[0084] Figure 27 The fourth block diagram showing an example of the structure of the information processing apparatus according to the eleventh embodiment of the present disclosure.

[0085] Figure 28 An explanatory diagram showing an example of the association between point information and label information.

[0086] Figure 29 An explanatory diagram showing a schematic structure of a vehicle.

[0087] Figure 30 A block diagram showing an example of the functional structure of the cooling execution apparatus.

[0088] Figure 31 The fifth block diagram showing an example of the structure of the information processing apparatus according to the fourteenth embodiment of the present disclosure.

[0089] Figure 32 The sixth block diagram showing an example of the structure of the information processing apparatus according to the fourteenth embodiment of the present disclosure.

[0090] Figure 33 A diagram schematically showing coordinate detection of an object over time.

[0091] Figure 34 The seventh block diagram showing an example of the structure of the information processing apparatus according to the fifteenth embodiment of the present disclosure.

[0092] Figure 35A An explanatory diagram for explaining an image of an object captured by an event camera.

[0093] Figure 35B An explanatory diagram for explaining an image of an object captured by an event camera.

[0094] Figure 35C An explanatory diagram for explaining an image of an object captured by an event camera.

[0095] Figure 36 A schematic diagram showing a state in which other vehicles are traveling around a vehicle.

[0096] Figure 37 A flowchart executed by a central brain according to the twentieth embodiment of the present disclosure.

[0097] Figure 38 A flowchart executed by a central brain according to the twenty - first embodiment of the present disclosure.

[0098] Figure 39 A structural diagram of a vehicle system according to the twenty - first embodiment of the present disclosure.

[0099] Figure 40 is a flowchart executed by a central brain according to the twenty-first embodiment of the present disclosure.

[0100] Figure 41 is a block diagram showing an example of an information processing apparatus according to the twenty-second embodiment of the present disclosure.

[0101] Figure 42 is a block diagram showing an example of an information processing apparatus according to the twenty-third embodiment of the present disclosure.

[0102] Figure 43 is a diagram for explaining a case where the speed (acceleration / deceleration) of a vehicle is controlled according to the relationship of y = ax n

[0103] Figure 44 is a block diagram schematically showing an example of a hardware configuration of a computer that functions as an information processing apparatus. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0104] The present disclosure will be more fully understood from the following detailed description. The further scope of application of the present disclosure will become apparent from the following detailed description. However, the detailed description and specific examples are preferred embodiments of the present disclosure and are described only for the purpose of illustration. It will be apparent to those skilled in the art that various changes and modifications made in accordance with the detailed description are covered by the spirit and scope of the present disclosure.

[0105] The applicant has no intention of dedicating any of the described embodiments to the public, and in the case of disclosed modifications and alternatives, even if some may not be included in the literal expression of the claims, they are part of the invention under the doctrine of equivalents.

[0106] Hereinafter, the present disclosure will be described by way of embodiments, but the following embodiments are not intended to limit the subject matter recited in the claims. In addition, all combinations of features described in the embodiments are not necessarily essential for solving the problems of the present disclosure.

[0107] (First Embodiment)

[0108] The information processing apparatus of the present disclosure can accurately obtain an index value required for driving control based on a large amount of information related to the control of a vehicle. Therefore, the information processing apparatus of the present disclosure can be at least partially mounted on a vehicle to achieve the control of the vehicle.

[0109] ​In addition, the information processing apparatus of the present disclosure can provide a driving system that can achieve autonomous driving in real time based on data obtained from inputs of multiple sensors at the edge using AI (Artificial Intelligence) / multivariate analysis / goal seek / policy formulation / optimal probability solution / optimal speed solution / optimal route management at usage level L6 (Level6), and perform adjustments based on the Delta optimal solution.

[0110] Figure 1 FIG. is a schematic diagram showing an example of a vehicle equipped with a Central Brain. The Central Brain may be an example of the information processing apparatus according to the present embodiment. As Figure 1 shown, the Central Brain can be communicatively connected to a plurality of gateways (Gate Way). The Central Brain according to the present embodiment can achieve Level 6 autonomous driving based on a plurality of information obtained via the gateways.

[0111] "Level L6" represents a level of autonomous driving, corresponding to a level higher than Level L5 which represents fully autonomous driving. Although Level 5 (Level5) represents fully autonomous driving, it is a level comparable to the level of human driving and there is still a probability of accidents occurring. Level L6 represents a level higher than Level 5, corresponding to a level with a lower probability of accidents than Level 5.

[0112] The computing power of Level L6 (i.e., the computing power for achieving Level L6) is about 1000 times that of Level L5 (i.e., the computing power for achieving Level L5). Therefore, high-performance driving control that cannot be achieved by Level 5 can be realized.

[0113] Figure 2 FIG. is a block diagram showing an example of the information processing apparatus according to the first embodiment of the present disclosure. The information processing apparatus 1 according to the present embodiment at least includes: an information acquisition unit 10 capable of acquiring a plurality of information related to the vehicle, an inference unit 20 for inferring a plurality of index values based on the plurality of information acquired by the information acquisition unit 10, and a driving control unit 30 for performing driving control of the vehicle based on the plurality of index values.

[0114] The information acquisition unit 10 can acquire various information related to the vehicle. As the information acquisition unit 10, for example, it may include sensors installed at various parts of the vehicle, and a communication unit for acquiring information that can be obtained via a network from a server (not shown) or the like. As an example of the sensors included in the information acquisition unit 10, the following can be cited: radar, lidar (LiDAR), high-pixel / long-focal-length / ultra-wide-angle / 360-degree / high-performance cameras, visual recognition, weak sound sensors, ultrasonic sensors, vibration sensors, infrared sensors, ultraviolet sensors, electromagnetic wave sensors, temperature sensors, humidity sensors, spot AI weather forecasts, material sensors, tilt sensors, high-precision multi-channel global positioning system (GPS), and / or low-altitude satellite information, etc. Alternatively, long-tail incident AI data, etc. can be cited. The long-tail incident AI data is the trip data of a vehicle with level L5 installed (i.e., a vehicle installed with a device capable of achieving the computing power of level L5 (here, as an example, it is the information processing device 1)).

[0115] The information that various sensors can acquire can include: the temperature or material of the ground (e.g., the road), the temperature of the outside air, the tilt of the ground, the frozen state or moisture content of the road, the material or wear condition of each tire, or the air pressure, the road width, the presence or absence of overtaking bans, the presence or absence of oncoming vehicles, the vehicle type information of the vehicles in front and behind, the cruising state of these vehicles, and / or the surrounding conditions (birds, animals, footballs, accident vehicles, earthquakes, fires, winds, typhoons, heavy rain, light rain, snowstorms, and / or fog, etc.). In the present embodiment, by using the computing power of level L6, these detections can be performed every one billionth of a second (nanosecond).

[0116] It should be particularly noted that the above-mentioned information acquisition unit 10 includes a vehicle lower sensor provided at the lower part of the vehicle and capable of detecting the temperature, material, and tilt of the ground for driving. By using this vehicle lower sensor, the Independent Smart Tilt function can be executed.

[0117] Alternatively, the inference unit 20 can use machine learning, and more specifically, deep learning, to infer the indexed value related to vehicle control based on the multiple information acquired by the information acquisition unit 10. In other words, the inference unit 20 can be composed of AI (Artificial Intelligence).

[0118] The inference unit 20 uses the computing power used when implementing level L6 (hereinafter also referred to as "the computing power of level L6") to perform multivariate analysis based on the integration method as shown in the following formula (1) (for example, refer to formula (2)) on the data per nanosecond or long-tail event AI data collected by the information acquisition unit 10 through a plurality of sensor groups, etc., whereby an accurate index value can be obtained. More specifically, while obtaining the integrated value of various ultra-high-resolution Delta values at the level of L6 computing power, the indexed value of each variable can be obtained in real time at the edge level, and the result that will occur in the next nanosecond can be obtained with the highest probability value (that is, the value obtained by indexing each variable, that is, the index value). To achieve this, for example, the integrated value obtained by time-integrating the Delta value (for example, the change value in a short time) of a function that can determine each variable such as air resistance, road resistance, road elements (such as garbage), and slip coefficient (in other words, a function representing the change of each variable) is input to the deep learning model of the inference unit 20 (for example, a learned model obtained by performing deep learning on a neural network). The deep learning model of the inference unit 20 outputs an index value corresponding to the input integrated value (for example, the index value with the highest confidence level (that is, evaluation value)). The output of the index value is in units of nanoseconds.

[0119] [Calculation formula 1]

[0120]

[0121] [Calculation formula 2]

[0122] V n = DL(f(A,B,C,D,…,N)(dA n / dt)) (2)

[0123] It should be noted that, as an example, in Formula (1), "f(A)" is, for example, a formula obtained by simplifying the expression of a function representing the changes of various variables such as air resistance, road resistance, road elements (such as garbage), and slip coefficient. Additionally, as an example, Formula (1) is a formula representing the time integral v of "f(A)" from time a to time b. In the formula, DL represents deep learning (for example, a deep learning model optimized by performing deep learning on a neural network), dAn / dt represents the Delta value of f(A, B, C, D,..., N), A, B, C, D,..., N represent air resistance, road resistance, road elements (such as garbage), and slip coefficient, etc., f(A, B, C, D,..., N) represents a function showing the changes of A, B, C, D,..., N, and Vn represents the value output from the deep learning model optimized by performing deep learning on a neural network (i.e., the index value).

[0124] It should be noted that an example form is listed here in which the integral value obtained by performing a time integral on the Delta value of a function is input into the deep learning model of the inference unit 20, but this is merely an example. For example, it is also possible to infer the integral value (for example, the result occurring in the next nanosecond) obtained by performing a time integral on the Delta value of a function representing the changes of various variables such as air resistance, road resistance, road elements, and slip coefficient through the deep learning model of the inference unit 20, and as the inference result, the integral value with the highest confidence level (i.e., the evaluation value) is obtained by the inference unit 20 per nanosecond.

[0125] Furthermore, an example form is listed here in which an integral value is input into the deep learning model or an integral value is output from the deep learning model, but this is merely an example, and the technology of the present disclosure is also applicable even without using the integral value. For example, it is possible to infer at least one index value through a deep learning model optimized by performing deep learning on a neural network using supervised data, where in the supervised data, the values corresponding to A, B, C, D,..., N are used as example data, and the values corresponding to at least one index value (for example, the result occurring in the next nanosecond) are used as correct answer data.

[0126] The value obtained by indexing each variable obtained by the inference unit 20 (i.e., the index value) can be further refined by increasing the number of times of deep learning. For example, it is possible to calculate a more accurate index value by using a large amount of data such as the rotation of tires or motors, steering angles, or the material of the road, weather, garbage, or the influence during quadratic curve deceleration, slip, loss of balance, or steering or speed control methods for re - restoring balance, or long - tail event AI data.

[0127] The driving control unit 30 can perform the driving control of the vehicle based on a plurality of index values determined by the inference unit 20. Also, the driving control unit 30 can implement the autonomous driving control of the vehicle. Specifically, it is possible to obtain the result that will occur in the next nanosecond with the highest probability value from the plurality of index values, and implement the driving control of the vehicle considering this probability value. That is, the driving control unit 30 can be configured to obtain the index value with the highest confidence level (i.e., evaluation value) from the plurality of index values as the result that will occur in the next nanosecond, and perform the driving control of the vehicle according to the obtained index value.

[0128] According to the information processing apparatus 1 having the above structure, since the analysis or inference of information can be implemented using the computing power of level L6, which is much greater than that of level L5, it is possible to perform a delicate analysis that is incomparable with the past. As a result, vehicle control for safe autonomous driving becomes possible. In addition, through the above-described multivariate analysis using AI, a value difference of 1000 times can be generated compared to the world of level L5.

[0129] (Second Embodiment)

[0130] Figure 3 FIG. is a block diagram showing an example of an information processing apparatus according to a second embodiment of the present disclosure. The information processing apparatus 1A according to the present embodiment is different from the information processing apparatus 1 in that, in addition to the information processing apparatus 1 according to the above-described first embodiment, it further includes a strategy setting unit 40 that sets a driving strategy until the vehicle reaches the destination.

[0131] The strategy setting unit 40 can set a driving strategy from the current position to the destination based on information on the destination input by the occupants of the vehicle or the like, or traffic information between the current position and the destination. At this time, the information used for calculating the strategy setting, that is, the data currently acquired by the information acquisition unit 10, can be considered. This is not only for simply calculating the route to the destination, but also for calculating a more realistic theoretical value by considering the surrounding conditions at the current moment. The driving strategy can be configured to include at least one theoretical value of the optimal route (also referred to as the strategy route), driving speed, tilt, and braking until the destination. Preferably, the driving strategy can be composed of all the theoretical values of the above optimal route, driving speed, tilt, and braking.

[0132] A plurality of theoretical values constituting the driving strategy set by the strategy setting unit 40 can be used for the automatic driving control of the driving control unit 30. In addition, preferably, the driving control unit 30 includes a strategy update unit 31 that can update the driving strategy based on the differences between the plurality of index values inferred by the inference unit 20 (for example, an index value representing the driving speed, an index value representing tilt, and an index value representing the braking control value) and the respective theoretical values set by the strategy setting unit 40 (for example, a theoretical value of the driving speed, a theoretical value of tilt, and a theoretical value of the braking control value).

[0133] The index values inferred by the inference unit 20 are information obtained during vehicle driving, specifically detected during actual driving. For example, they are values inferred based on the friction coefficient. Therefore, in the strategy update unit 31, by considering this index value, it is possible to cope with the changes at every moment when passing on the strategy route. Specifically, in the strategy update unit 31, the difference (Delta value) is calculated based on the theoretical value and the index value included in the driving strategy, so that the optimal solution can be derived again and the strategy route can be re-formulated. As a first example of the optimal solution, an index value used instead of the theoretical value can be cited. As a second example of the optimal solution, an adjusted index value used instead of the theoretical value can be cited. As a third example of the optimal solution, a solution obtained by performing regression analysis using at least one theoretical value and at least one index value can be cited. As a fourth example of the optimal solution, a statistical value (for example, median and / or average value, etc.) obtained based on the theoretical value and the index value can be cited. Which of the first to fourth examples the strategy update unit 31 derives the optimal solution from can be determined, for example, according to the magnitude of the difference. It should be noted that it is not limited to the first to fourth examples, and the optimal solution can also be obtained by other methods.

[0134] In this way, by the strategy update unit 31 deriving the optimal solution again, for example, it is also possible to achieve the automatic driving control in a non-slip critical state. That is, in the following cases, that is, if the automatic driving control is performed only based on the theoretical value, the automatic driving control in which the vehicle slips will be implemented; if the automatic driving control is performed only based on the index value, the automatic driving control with a safety margin and without slipping is implemented, the index value after subtracting the safety margin and adjusting is used as the theoretical value by the driving control unit 30, thus achieving the automatic driving control in a non-slip critical state of the vehicle. In addition, when performing such an update process, since the computing power of the above-mentioned level L6 can be used, correction and fine-tuning can be performed in units of one billionth of a second, and more detailed driving control can be achieved.

[0135] In addition, when the information acquisition unit 10 has the above-described vehicle lower sensor, since the vehicle lower sensor also detects the temperature or material of the ground, etc., it is possible to cope with the changes at every moment when passing on the strategic route. When calculating the driving course included in the driving strategy, independent intelligent tilting can also be performed. Furthermore, even when other information (for example, flying tires, debris, animals, etc.) is detected, by coping with the changes at every moment when passing on the strategic route, it is possible to recalculate the optimal driving course instant by instant and implement optimal route management.

[0136] (Third Embodiment)

[0137] In the third embodiment described below, an information processing device and the like that can analyze the influence of factors such as air resistance and / or friction generated when an autonomous vehicle travels on a road while traveling will be mainly described.

[0138] Figure 4 It is a schematic diagram showing an example of the structure of the inference unit 20 included in the information processing device 1B according to the third embodiment of the present disclosure.

[0139] In Figure 4 In the example shown, the information processing device 1B is mounted on the vehicle 48. The inference unit 20 included in the information processing device 1B has a deep learning model 20A. The deep learning model 20A is a learned model optimized by performing deep learning on a neural network using a plurality of supervised data 50. The inference unit 20 uses the deep learning model 20A to infer a plurality of index values based on the environmental information 56 (refer to Figure 5 ).

[0140] The supervised data 50 is a data set having example data 52 and correct answer data 54. The example data 52 has example environmental information 52A. The example environmental information 52A is information assuming the environmental information 56 (refer to Figure 5 ) acquired by the information acquisition unit 10. For example, the example environmental information 52A includes information such as the distance between the vehicle and the obstacle 52A1, the traveling speed 52A2, the air resistance 52A3, the road resistance 52A4, the road element 52A5, and the ground tilt 52A6. Here, the air resistance 52A3 corresponds to the air resistance exemplified in the first embodiment above, the road resistance 52A4 corresponds to the road resistance exemplified in the first embodiment above, the road element 52A5 corresponds to the road element exemplified in the first embodiment above, and the ground tilt 52A6 corresponds to the tilt exemplified in the first embodiment above. In addition, the distance 52A1 between the vehicle and the obstacle is the distance between the vehicle 48 and the obstacle in the traveling direction of the vehicle 48 (in Figure 4In the example shown, it is the distance between a vehicle in front of vehicle 48, namely vehicle 53). The traveling speed 52A2 is the traveling speed of vehicle 48.

[0141] The correct solution data 54 is the correct solution data (i.e., annotation) for the example data 52. The correct solution data 54 has a correct solution distance 54A. The correct solution distance 54A is an example of the "first distance" related to the technology of the present disclosure.

[0142] The correct solution distance 54A is the distance obtained by adding the necessary stop distance 54A1 and the margin distance 54A2. The necessary stop distance 54A1 is the distance required for vehicle 48 to stop without colliding with vehicle 53 between vehicle 48 and vehicle 53. Vehicle 53 is an obstacle in the traveling direction of vehicle 48 and travels in front of vehicle 48. In Figure 4 In the example shown, vehicle 53 traveling in front of vehicle 48 is exemplified, but the technology of the present disclosure is not limited thereto, and it may also be vehicle 53 stopped in front of vehicle 48, or a pedestrian or the like existing in front of vehicle 48, as long as it is an obstacle in the traveling direction of vehicle 48.

[0143] The margin distance 54A2 is an additional determined distance (for example, a distance of about several meters) to improve the certainty of preventing a collision between vehicle 48 and vehicle 53.

[0144] Figure 5 It is a schematic diagram showing an example of the processing contents of the information acquisition unit 10, the inference unit 20, and the driving control unit 30 included in the information processing apparatus 1B according to the third embodiment.

[0145] The information acquisition unit 10 acquires environmental information 56 in the same manner as in the first embodiment described above. The environmental information 56 includes information such as the distance between vehicle obstacles 56A, traveling speed 56B, air resistance 56C, road resistance 56D, road elements 56E, and ground inclination 56F.

[0146] The inference unit 20 inputs the environmental information 56 acquired by the information acquisition unit 10 into the deep learning model 20A. As a result, the deep learning model 20A outputs an index value 58 corresponding to the environmental information 56 (i.e., an index value related to the correct solution distance 54A corresponding to the environmental information 56). The inference unit 20 acquires the index value 58 output from the deep learning model 20A. The index value 58 is an index value inferred as a non-collision distance by the deep learning model 20A. The non-collision distance is, for example, the distance between vehicle 48 and an obstacle in the traveling direction of vehicle 48 (in Figure 4In the example shown, it is the distance at which the vehicle 53) will not collide. The index value 58 can also be an index value obtained by performing the multivariate analysis based on the integration method described in the first embodiment above. It should be noted that the index value 58 is an example of the "first index value" related to the technology of the present disclosure.

[0147] Based on the index value 58 obtained by the inference unit 20, the driving control unit 30 uses the traveling speed calculation formula 60 to calculate the maximum speed 62 at which the vehicle 48 will not collide with an obstacle in the traveling direction (for example, a vehicle or a pedestrian existing in front of the vehicle 48). The traveling speed calculation formula 60 is a formula that takes the index value 58 as the independent variable and the maximum speed 62 as the dependent variable. The maximum speed 62 refers to the maximum speed within the range below the legal speed and below the speed of the vehicle in front. The speed of the vehicle in front is the traveling speed of another vehicle traveling in front of the vehicle 48 (for example, the vehicle immediately in front of the vehicle 48 in the traveling direction). For example, the speed of the vehicle in front is obtained by the information acquisition unit 10 or the like. The driving control unit 30 refers to the speed of the vehicle in front obtained by the information acquisition unit 10 or the like and calculates the maximum speed 62.

[0148] The driving control unit 30 performs driving control of the vehicle based on a plurality of index values inferred by the inference unit 20. The driving control includes traveling speed control. The traveling speed control is the control of the traveling speed of the vehicle 48 and is performed based on the index value 58. The traveling speed control includes the control of causing the vehicle 48 to travel at the maximum speed 62. That is, the driving control unit 30 controls the drive system of the vehicle 48 (for example, the power source that transmits power to the wheels) so that the vehicle 48 travels at the maximum speed 62 calculated based on the index value 58.

[0149] Figure 6 It is a flowchart showing an example of the flow of the traveling speed control process executed by the information processing device 1B.

[0150] In Figure 6 In the traveling speed control process shown, first, in step ST10, the information acquisition unit 10 determines whether the timing specified in nanoseconds has arrived (for example, whether one billionth of a second has passed). In step ST10, if the timing specified in nanoseconds has not arrived, the determination result is negative, and the traveling speed control process transfers to step ST22. In step ST10, if the timing specified in nanoseconds has arrived, the determination result is positive, and the traveling speed control process transfers to step ST12.

[0151] In step ST12, the information acquisition unit 10 acquires the environmental information 56. After the processing of step ST12 is executed, the driving speed control process transfers to step ST14.

[0152] In step ST14, the inference unit 20 inputs the environmental information 56 acquired by the information acquisition unit 10 in step ST12 into the deep learning model 20A. Accordingly, the deep learning model 20A outputs an index value 58 corresponding to the input environmental information 56. After the processing of step ST14 is executed, the driving speed control process transfers to step ST16.

[0153] In step ST16, the inference unit 20 acquires the index value 58 output from the deep learning model 20A. After the processing of step ST16 is executed, the driving speed control process transfers to step ST18.

[0154] In step ST18, the driving control unit 30 calculates the maximum speed 62 using the driving speed arithmetic expression 60 based on the index value 58 acquired by the inference unit 20 in step ST16. After the processing of step ST18 is executed, the driving speed control process transfers to step ST20.

[0155] In step ST20, the driving control unit 30 controls the drive system of the vehicle 48 so that the vehicle 48 travels at the maximum speed 62 calculated in step ST18. After the processing of step ST20 is executed, the driving speed control process transfers to step ST22.

[0156] In step ST22, the driving control unit 30 determines whether the end condition of the driving speed control process is satisfied. As an example of the end condition of the driving speed control process, a condition such as an instruction to end the driving speed control process being given to the information processing device 1B can be cited. In step ST22, if the end condition of the driving speed control process is not satisfied, the determination result is negative, and the driving speed control process transfers to step ST10. In step ST22, if the end condition of the driving speed control process is satisfied, the determination result is positive, and the driving speed control process ends.

[0157] As described above, in the present third embodiment, the inference unit 20 infers the index value 58 related to the correct distance 54A, and the correct distance 54A includes the necessary stopping distance 54A1 required for the vehicle 48 to stop without colliding with an obstacle in the traveling direction of the vehicle 48. And the driving control unit 30 controls the traveling speed of the vehicle 48 based on the index value 58. Therefore, the vehicle 48 can travel safely and in a shorter time to the destination.

[0158] In addition, in this third embodiment, the index value 58 is inferred in nanoseconds, and the maximum speed 62 is calculated based on the index value 58. The maximum speed 62 is the maximum speed at which the vehicle 48 will not collide with an obstacle, and the drive system of the vehicle 48 is controlled by the driving control unit 30 so that the vehicle 48 travels at the maximum speed 62. Therefore, the vehicle 48 can be safely and driven to the destination in the shortest time.

[0159] In addition, in this third embodiment, the maximum speed 62 is calculated within the range below the legal speed and below the speed of the vehicle ahead. Therefore, the vehicle 48 can be safely and driven to the destination in the shortest time while complying with the legal speed.

[0160] In addition, in this third embodiment, as the correct distance 54A, the distance obtained by adding the necessary stopping distance 54A1 and the margin distance 54A2 is used. Therefore, compared with the case where only the necessary stopping distance 54A1 is used as the correct distance 54A, the possibility of the vehicle 48 colliding with the vehicle 53 can be reduced.

[0161] It should be noted that in the above third embodiment, as an example of the index value 58, the index value inferred by the deep learning model 20A as the non-collision distance is exemplified, but the technology of the present disclosure is not limited thereto. For example, as the index value 58, the index value representing the maximum speed 62 can be inferred by the deep learning model 20A. In this case, for example, the drive system of the vehicle 48 is controlled by the driving control unit 30 so that the vehicle 48 travels at the maximum speed 62 represented by the index value 58.

[0162] In addition, in the above third embodiment, the case of calculating the maximum speed 62 is exemplified, but this is only an example, and a speed higher than the specified speed and lower than the maximum speed 62 can also be calculated. In addition, instead of the speed, the number of rotations of the wheel can be calculated as long as information related to the speed (that is, a parameter for controlling the speed of the vehicle 48) is calculated.

[0163] In addition, in the above third embodiment, as the correct distance 54A, the distance obtained by adding the necessary stopping distance 54A1 and the margin distance 54A2 is listed, but this is only an example, and only the necessary stopping distance 54A1 can also be used as the correct distance 54A.

[0164] (Fourth Embodiment)

[0165] In the fourth embodiment described below, an information processing device and the like that can achieve high-precision autonomous driving considering the actual environment during the vehicle's travel until it reaches the destination will be mainly described.

[0166] Figure 7It is a schematic diagram showing an example of the structure of the information processing apparatus 1C according to the fourth embodiment of the present disclosure.

[0167] The information processing apparatus 1C is mounted on multiple vehicles 54 (also refer to Figure 8 ). The information processing apparatus 1C includes an information acquisition unit 10, an inference unit 20, a driving control unit 30, and a strategy setting unit 40 in the same manner as in the first embodiment. The driving control unit 30 has a strategy update unit 31. It should be noted that, in Figure 7 the example shown, the strategy update unit 31 is shown as a part of the driving control unit 30, but this is merely an example, and the strategy update unit 31 may also be provided outside the driving control unit 30.

[0168] The information acquisition unit 10 acquires environmental information 56 in the same manner as in the first embodiment. The environmental information 56 includes a driving route 56A (for example, the route on which the vehicle 54 travels and is a route defined by coordinates) as the route on which the vehicle 54 travels, a driving speed 56B as the speed at which the vehicle 54 travels, a ground inclination 56C equivalent to the tilt listed in the first embodiment, a braking control value 56D (in other words, a parameter for controlling the brake), an air resistance 56E, a road resistance 56F, and road elements 56G, etc.

[0169] The inference unit 20 has a deep learning model 20A. The deep learning model 20A is a learned model optimized by performing deep learning on a neural network using a plurality of supervised data. The inference unit 20 uses the deep learning model 20A to infer a plurality of index values based on the environmental information 56.

[0170] The supervised data used in the deep learning of the deep learning model 20A is a data set having example data and correct answer data (i.e., annotations) for the example data. The example data is data assuming the environmental information 56, and the correct answer data is data assuming a plurality of index values 58.

[0171] The inference unit 20 inputs the environmental information 56 acquired by the information acquisition unit 10 into the deep learning model 20A. As a result, the deep learning model 20A outputs a plurality of index values 58 (i.e., a plurality of control values used in the autonomous driving control of the vehicle 54) corresponding to the environmental information 56.

[0172] The strategy update unit 31 acquires the plurality of index values 58 output from the deep learning model 20A. The strategy setting unit 40 sets a driving strategy 66 until the vehicle 54 reaches the destination. Here, the setting of the driving strategy 66, for example, means the process of giving the driving strategy 66 to the strategy update unit 31.

[0173] The driving strategy 66 includes a plurality of theoretical values 68. The plurality of theoretical values 68 include: the theoretical value of the route for the vehicle 54 to reach the destination (for example, the route assumed as the route traveled by the vehicle 54 and defined by coordinates), the theoretical value of the driving speed for the vehicle 54 to reach the destination, the theoretical value of the ground inclination for the vehicle 54 to reach the destination, and the theoretical value of the braking control value for the vehicle 54 to reach the destination.

[0174] The strategy update unit 31 updates the driving strategy 66 based on the driving strategy 66 set by the strategy setting unit 40 and the plurality of index values 58 output from the inference unit 20. For example, the strategy update unit 31 updates the plurality of theoretical values 68 based on the difference 70 between the plurality of theoretical values 68 included in the driving strategy 66 and the plurality of index values 58, thereby updating the driving strategy 66.

[0175] The update of the theoretical value 68, in other words, represents the derivation of the optimal solution. As a first example of the optimal solution, the index value 58 used in place of the theoretical value 68 can be cited. As a second example of the optimal solution, the adjusted index value 58 used in place of the theoretical value 68 can be cited. As a third example of the optimal solution, the solution obtained by performing regression analysis using at least one theoretical value 68 and at least one index value 58 can be cited. As a fourth example of the optimal solution, the statistical value (for example, median and / or average value, etc.) obtained from the theoretical value 68 and the index value 58 can be cited. In which case among the first to fourth examples the strategy update unit 31 derives the optimal solution can be determined, for example, according to the magnitude of the difference 70. It should be noted that, not limited to the first to fourth examples, the optimal solution can also be obtained by other methods.

[0176] Here, the difference 70 between the plurality of theoretical values 68 and the plurality of index values 58 refers to the difference between the theoretical value 68 and the index value 58 for each of the plurality of items (for example, the route for the vehicle 54 to reach the destination, the driving speed for the vehicle 54 to reach the destination, the ground inclination for the vehicle 54 to reach the destination, the braking control value for the vehicle 54 to reach the destination, etc.).

[0177] It should be noted that the difference 70 is an example of the "result obtained by comparing a plurality of theoretical values and a plurality of index values" involved in the technology of the present disclosure. Here, although the difference 70 is listed, this is only an example. For example, it can also be the ratio of one of the theoretical value 68 and the index value 58 to the other, as long as it is a numerical value that can determine the degree of difference between the plurality of theoretical values and the plurality of index values.

[0178] The driving control unit 30 performs driving control of the vehicle 54 according to the driving strategy 66 updated by the strategy update unit 31 (for example, the updated plurality of theoretical values 68).

[0179] Figure 8 This is a schematic diagram showing an example of a manner in which the driving strategy 66 of the first vehicle 54A among multiple vehicles 54 equipped with the information processing device 1C according to the fourth embodiment of the present disclosure is updated based on a plurality of index values 58 inferred by the second vehicle 54B.

[0180] The first vehicle 54A and the second vehicle 54B travel toward the same destination, and the second vehicle 54B travels in front of the first vehicle 54A with respect to the destination. That is, the second vehicle 54B arrives at the destination earlier than the first vehicle 54A.

[0181] Data on the multiple vehicles 54 is collected and managed by the data management device 72. The data management device 72 is wirelessly communicably connected to a plurality of information processing devices 1C mounted on the multiple vehicles 54. As an example of the data management device 72, for example, a server is cited.

[0182] The data management device 72 constructs a database 74 based on various information obtained from the information processing devices 1C of the respective vehicles 54. The database 74 has a vehicle identifier 76 and a plurality of index values 58. The vehicle identifier 76 is an identifier capable of identifying the vehicle 54. In the database 74, for each vehicle 54, the vehicle identifier 76 is associated with a plurality of index values 58 (that is, a plurality of index values 58 inferred by the inference unit 20 included in the information processing device 1C of the vehicle 54 determined by the corresponding vehicle identifier 76).

[0183] Before the first vehicle 54A reaches the destination and during the travel of the first vehicle 54A, a strategy update unit 31 (hereinafter referred to as "the strategy update unit 31 of the first vehicle 54A") included in the information processing device 1C of the first vehicle 54A acquires a plurality of index values 58 inferred by an inference unit 20 (hereinafter referred to as "the inference unit 20 of the second vehicle 54B") included in the information processing device 1C of the second vehicle 54B from the data management device 72.

[0184] For example, the data management device 72, according to a request from the strategy update unit 31 of the first vehicle 54A, acquires a plurality of index values 58 corresponding to the second vehicle 54B from the database 74 and transmits the plurality of index values 58 acquired from the database 74 to the strategy update unit 31 of the first vehicle 54A. The strategy update unit 31 of the first vehicle 54A receives the plurality of index values 58 transmitted from the data management device 72 and updates the driving strategy 66 based on the received plurality of index values 58.

[0185] For example, the update of the driving strategy 66 is achieved by updating a plurality of theoretical values 68 based on the result obtained by comparing a plurality of theoretical values 68 included in the driving strategy 66 with a plurality of received index values 58. Here, the update of the theoretical value 68 can also be performed, for example, in the same manner as in the first to fourth examples described above.

[0186] Figure 9 It is a flowchart showing an example of the process flow of the driving strategy update process executed by the information processing device 1C of the first vehicle 54A.

[0187] In Figure 9 In the driving strategy update process shown, in step ST30, the strategy setting unit 40 sets the driving strategy 66 for the strategy update unit 31. After the process of step ST30 is executed, the driving strategy update process transfers to step ST32.

[0188] In step ST32, the strategy update unit 31 determines whether the timing specified in nanoseconds has arrived (for example, whether one billionth of a second has passed). In step ST32, if the timing specified in nanoseconds has not arrived, the determination result is negative, and the driving strategy update process transfers to step ST32. In step ST32, if the timing specified in nanoseconds has arrived, the determination result is positive, and the driving strategy update process transfers to step ST34.

[0189] In step ST34, the strategy update unit 31 determines whether the second vehicle information acquisition timing has arrived. The second vehicle information acquisition timing refers to the timing when the strategy update unit 31 acquires a plurality of index values 58 inferred by the inference unit 20 of the second vehicle 54B. As a first example of the second vehicle information acquisition timing, a timing that satisfies the condition that the index value 58 related to the second vehicle 54B stored in the database 74 has been updated can be cited. In addition, as a second example of the second vehicle information acquisition timing, a timing that satisfies the condition that a predetermined time (for example, several seconds) has elapsed since the process of step ST32 was executed can be cited.

[0190] In step ST34, if the second vehicle information acquisition timing has not arrived, the determination result is negative, and the driving strategy update process transfers to step ST40. In step ST34, if the second vehicle information acquisition timing has arrived, the determination result is positive, and the driving strategy update process transfers to step ST36.

[0191] In step ST36, the policy update unit 31 acquires a plurality of index values 58 related to the second vehicle 54B (i.e., a plurality of index values 58 associated with the vehicle identifier 76 capable of identifying the second vehicle 54B) from the data management device 72. After the processing of step ST36 is executed, the driving policy update process proceeds to step ST38.

[0192] In step ST38, the policy update unit 31 updates the driving policy 66 based on the plurality of index values 58 acquired from the data management device 72 in step ST36. That is, the plurality of theoretical values 68 included in the driving policy 66 are updated based on the plurality of index values 58 acquired from the data management device 72. After the processing of step ST38 is executed, the driving policy update process proceeds to step ST40.

[0193] In step ST40, the information acquisition unit 10 acquires the environmental information 56. After the processing of step ST32 is executed, the driving policy update process proceeds to step ST42.

[0194] In step ST42, the inference unit 20 inputs the environmental information 56 acquired by the information acquisition unit 10 in step ST40 into the deep learning model 20A. Accordingly, the deep learning model 20A outputs index values 58 corresponding to the input environmental information 56. After the processing of step ST42 is executed, the driving policy update process proceeds to step ST44.

[0195] In step ST44, the policy update unit 31 acquires the plurality of index values 58 output from the deep learning model 20A. After the processing of step ST44 is executed, the driving policy update process proceeds to step ST46.

[0196] In step ST46, the policy update unit 31 calculates the difference 70 between the plurality of index values 58 and the plurality of theoretical values 68 included in the driving policy 66. After the processing of step ST46 is executed, the driving policy update process proceeds to step ST48.

[0197] In step ST48, the policy update unit 31 updates the plurality of theoretical values 68 included in the driving policy 66 based on the difference 70 calculated in step ST46, thereby updating the driving policy 66. After the processing of step ST48 is executed, the driving policy update process proceeds to step ST50.

[0198] In step ST50, the driving control unit 30 performs driving control of the vehicle 54 according to the driving policy 66 updated in step ST48. After the processing of step ST40 is executed, the driving policy update process proceeds to step ST42.

[0199] In step ST52, the policy update unit 31 determines whether the end condition for the driving policy update process is satisfied. As an example of the end condition for the driving policy update process, a condition such as an instruction to end the driving policy update process being given to the information processing device 1C can be cited. In step ST52, if the end condition for the driving policy update process is not satisfied, the determination result is negative, and the driving policy update process transfers to step ST32. In step ST52, if the end condition for the driving policy update process is satisfied, the determination result is positive, and the driving policy update process ends.

[0200] As described above, in this fourth embodiment, the information processing device 1C is mounted on multiple vehicles 54 including the first vehicle 54A and the second vehicle 54B, and the information processing device 1C has: the information acquisition unit 10, the inference unit 20, the driving control unit 30, the policy update unit 31, and the policy setting unit 40 described in the above first and second embodiments. Before the first vehicle 54A reaches the destination, the policy update unit 31 updates the driving policy 66 set by the policy setting unit 40 based on a plurality of index values 58 inferred by the inference unit 20 of the second vehicle 54B that is traveling ahead of the first vehicle 54A with respect to the destination. Thereby, the information processing device 1C of the first vehicle 54A can obtain a driving policy 66 that conforms to the actual environment until reaching the destination. As a result, highly accurate autonomous driving that takes into account the actual environment in which the first vehicle 54A travels until reaching the destination can be achieved.

[0201] In addition, in this fourth embodiment, before the first vehicle 54A reaches the destination and during the travel of the first vehicle 54A, the policy update unit 31 of the first vehicle 54A acquires a plurality of index values 58 for updating the driving policy 66 (that is, a plurality of index values 58 inferred by the inference unit 20 of the second vehicle 54B). Therefore, highly accurate autonomous driving that takes into account the actual environment in which the first vehicle 54A travels until reaching the destination can be achieved before the first vehicle 54A reaches the destination and during the travel of the first vehicle 54A.

[0202] In addition, in this fourth embodiment, before the second vehicle 54B reaches the destination, the policy update unit 31 of the first vehicle 54A acquires a plurality of index values 58 for updating the driving policy 66 (that is, a plurality of index values 58 inferred by the inference unit 20 of the second vehicle 54B). Therefore, compared with the case where the policy update unit 31 of the first vehicle 54A acquires a plurality of index values 58 inferred by the inference unit 20 of the second vehicle 54B after the second vehicle 54B reaches the destination, the policy update unit 31 of the first vehicle 54A can update the driving policy 66 earlier.

[0203] In addition, in the present fourth embodiment, before the first vehicle 54A reaches the destination, the policy update unit 31 updates a plurality of theoretical values 68 included in the driving policy 66 set by the policy setting unit 40 based on a plurality of index values 58 inferred by the inference unit 20 of the second vehicle 54B that travels ahead of the first vehicle 54A with respect to the destination. Then, the driving control unit 30 of the first vehicle 54A performs driving control of the first vehicle 54A according to the plurality of theoretical values 68 updated by the policy update unit 31. Thus, compared with the case where the plurality of theoretical values 68 included in the driving policy 66 are always fixed, highly accurate autonomous driving can be achieved considering the actual environment in which the first vehicle 54A travels until it reaches the destination.

[0204] In addition, in the present fourth embodiment, the data management device 72 collects and manages a plurality of index values 58 from multiple vehicles 54. And the policy update unit 31 of the first vehicle 54A updates the driving policy 66 based on the plurality of index values 58 obtained from the data management device 72. Therefore, the driving control unit 30 can perform autonomous driving of the first vehicle 54A according to the driving policy 66 updated based on the plurality of index values 58 obtained from the designated vehicle 54 (for example, the second vehicle 54B) among the multiple vehicles 54.

[0205] It should be noted that, in the above fourth embodiment, an example of the method in which the difference value 70 is used for updating the driving policy 66 is given, but the use of the difference value 70 is not limited to this. For example, as Figure 10 shown, it may also be configured such that the information processing device 1C further includes a notification unit 78, and the notification unit 78 performs a notification corresponding to the difference value 70. For example, the notification unit 78 determines whether the difference value 70 exceeds a threshold value, and in the case where the difference value 70 exceeds the threshold value, notifies the situation that the difference value 70 has exceeded the threshold value. The threshold value may be a fixed value (for example, a default value), or may also be a variable value that is changed according to an instruction provided to the information processing device 1C by a user or the like. The notification is realized, for example, by visual display using a display and / or audio output using an audio playback device.

[0206] In addition, instead of performing a notification every time the difference value 70 is calculated, a notification may be performed in the case where the number of times the difference value 70 exceeds the threshold value (for example, the number of times the difference value 70 continuously exceeds the threshold value) exceeds a predetermined number of times (for example, the number of times determined according to an instruction provided to the information processing device 1C by a user or the like), or a notification may be performed in the case where, within a predetermined period (for example, the period determined according to an instruction provided to the information processing device 1C by a user or the like), the number of times the difference value 70 exceeds the threshold value exceeds a predetermined number of times.

[0207] In addition, the difference 70 used for comparison with the threshold value may be the difference between the theoretical value 68 and the index value 58 with respect to one item among a plurality of items (for example, the route of the vehicle 54 until it reaches the destination, the traveling speed of the vehicle 54 until it reaches the destination, the inclination of the ground until the vehicle 54 reaches the destination, the braking control value of the vehicle 54 until it reaches the destination, etc.), or may also be the difference between the theoretical value 68 and the index value 58 with respect to each item among the plurality of items.

[0208] In each of the above embodiments, the difference between the theoretical value 68 and the index value 58 (for example, the difference 70) is exemplified, but this is merely an example, and any value that can determine the result of comparing the theoretical value 68 and the index value 58 is acceptable. As a value that can determine the result of comparing the theoretical value 68 and the index value 58, in addition to the difference, examples also include: the ratio of one of the theoretical value 68 and the index value 58 to the other (in other words, the rate).

[0209] In the above fourth embodiment, although an example of the manner in which the strategy update unit 31 of the first vehicle 54A acquires a plurality of index values 58 corresponding to the second vehicle 54B before the second vehicle 54B reaches the destination is exemplified, this is merely an example. For example, the strategy update unit 31 of the first vehicle 54A may also acquire a plurality of index values 58 corresponding to the second vehicle 54B after the second vehicle 54B reaches the destination.

[0210] In the above fourth embodiment, although an example of the manner in which the strategy update unit 31 of the first vehicle 54A acquires a plurality of index values 58 corresponding to the second vehicle 54B from the data management device 72 is exemplified, the technology of the present disclosure is not limited thereto. For example, the strategy update unit 31 of the first vehicle 54A may also directly communicate between the information processing device 1C of the first vehicle 54A and the information processing device 1C of the second vehicle 54B, and acquire a plurality of index values 58 inferred by the inference unit 20 from the information processing device 1C of the second vehicle 54B. Additionally, for example, whenever the inference unit 20 of the second vehicle 54B infers an index value 58 (for example, whenever the index value 58 corresponding to the second vehicle 54B is updated in the data management device 72), the index value 58 inferred by the inference unit 20 of the second vehicle 54B may be acquired by the strategy update unit 31 of the first vehicle 54A and used for updating the driving strategy 66.

[0211] In the above-described fourth embodiment, an example is given in which a plurality of theoretical values 68 include the theoretical value of the route until the vehicle 54 reaches the destination, the theoretical value of the traveling speed until the vehicle 54 reaches the destination, the theoretical value of the ground inclination until the vehicle 54 reaches the destination, and the theoretical value of the braking control value until the vehicle 54 reaches the destination. However, this is only an example. For example, the plurality of theoretical values 68 may include at least one of the theoretical value of the route until the vehicle 54 reaches the destination, the theoretical value of the traveling speed until the vehicle 54 reaches the destination, the theoretical value of the ground inclination until the vehicle 54 reaches the destination, and the theoretical value of the braking control value until the vehicle 54 reaches the destination.

[0212] In the above-described fourth embodiment, an example is given in which, during the travel of the first vehicle 54A, a plurality of index values 58 for updating the travel strategy 66 are acquired by the strategy update unit 31 of the first vehicle 54A. However, the technology of the present disclosure is not limited to this. For example, before the first vehicle 54A reaches the destination and during the stop of the first vehicle 54A, a plurality of index values 58 for updating the travel strategy 66 may be acquired by the strategy update unit 31 of the first vehicle 54A. Thereby, even if the first vehicle 54A stops before reaching the destination, highly accurate autonomous driving considering the actual environment during the travel of the first vehicle 54A until it reaches the destination can be achieved.

[0213] In the above-described fourth embodiment, an example is given in which the index value 58 inferred by the inference unit 20 of the second vehicle 54B is directly used to update the travel strategy 66 of the first vehicle 54A. However, this is only an example. For example, an adjustment value (e.g., weight) determined according to the confidence level of the index value 58 inferred by the inference unit 20 of the second vehicle 54B may be used to adjust the index value 58 (i.e., the index value 58 inferred by the inference unit 20 of the second vehicle 54B), and based on the adjusted index value 58, the theoretical value 68 may be updated in the same manner as in the above-described fourth embodiment.

[0214] For example, regarding the confidence level of the index value 58 inferred by the inference unit 20 of the second vehicle 54B, when the first vehicle 54A and the second vehicle 54B pass through the same location (hereinafter referred to as the "first location"), it varies depending on the time interval between the time when the index value 58 is inferred by the inference unit 20 of the second vehicle 54B while traveling at the first location and the time when the first vehicle 54A travels at the first location. For example, when the time interval between the time when the index value 58 is inferred by the inference unit 20 of the second vehicle 54B while traveling at the first location and the time when the first vehicle 54A travels at the first location is longer, the confidence level regarding the index value 58 inferred by the inference unit 20 of the second vehicle 54B is lower. On the contrary, when traveling at the first location, the shorter the time interval between the time when the index value 58 is inferred by the inference unit 20 of the second vehicle 54B and the time when the first vehicle 54A travels, the higher the confidence level regarding the index value 58 inferred by the inference unit 20 of the second vehicle 54B.

[0215] Therefore, the longer the above-mentioned time interval is, when the strategy update unit 31 of the first vehicle 54A updates the theoretical value 68, it uses an adjustment value that reduces the influence of the index value 58 (for example, the index value 58 inferred by the inference unit 20 of the second vehicle 54B while traveling at the first location) on the theoretical value 68 to adjust the index value 58, and updates the theoretical value 68 based on the adjusted index value 58. In addition, the shorter the above-mentioned time interval is, when the strategy update unit 31 of the first vehicle 54A updates the theoretical value 68, it uses an adjustment value that increases the influence of the index value 58 (for example, the index value 58 inferred by the inference unit 20 of the second vehicle 54B while traveling at the first location) on the theoretical value 68 to adjust the index value 58, and updates the theoretical value 68 based on the adjusted index value 58.

[0216] In addition, here, the time interval between the time when the index value 58 is inferred by the inference unit 20 of the second vehicle 54B while traveling at the first location and the time when the first vehicle 54A travels is illustrated, but this is only an example. For example, the index value 58 (that is, the index value 58 inferred by the inference unit 20 of the second vehicle 54B) can be adjusted according to an adjustment value (for example, a weight) determined based on the difference between the external environment of the second vehicle 54B while traveling at the first location and the external environment of the first vehicle 54A while traveling at the first location, and the theoretical value 68 can be updated based on the adjusted index value 58 in the same manner as in the fourth embodiment above.

[0217] In this case, for example, when the external environment when the second vehicle 54B travels at the first location is exactly the same as the external environment when the first vehicle 54A travels at the first location, no adjustment value is required (that is, no adjustment of the index value 58 is required). The greater the degree of difference between the external environment when the second vehicle 54B travels at the first location and the external environment when the first vehicle 54A travels at the first location, the greater the adjustment value is used to adjust the index value 58 (that is, the index value 58 inferred by the inference unit 20 of the second vehicle 54B). In the same manner as in the above-described fourth embodiment, the theoretical value 68 can be updated based on the adjusted index value 58.

[0218] It should be noted that the external environment refers to the environment outside the vehicle 54. As an example of the external environment, meteorological conditions can be cited. The so-called meteorological conditions, for example, refer to the degree of rainfall, the degree of snowfall, the temperature of the outside air, the humidity of the outside air, the wind speed, and / or the wind direction, etc. The degree of difference between the external environment when the second vehicle 54B travels at the first location and the external environment when the first vehicle 54A travels at the first location can be determined, for example, based on the degree of difference (such as the difference value or ratio, etc.) between the meteorological condition-related index value (that is, the index value 58 related to the meteorological conditions) inferred by the inference unit 20 of the first vehicle 54A and the meteorological condition-related index value inferred by the inference unit 20 of the second vehicle 54B.

[0219] (Fifth Embodiment)

[0220] Figure 11 Schematically shows the ability of the AI for high-performance autonomous driving involved in the fifth embodiment of the present disclosure to predict risks. In this embodiment, multiple sensor information based on multiple sensors as detection units is digitalized by AI and stored in the cloud. The AI predicts and judges the optimal mixture of conditions every nanosecond (one billionth of a second) to optimize the operation of the vehicle 2012.

[0221] Figure 12 Is a schematic diagram showing an example of a vehicle 2012 equipped with a central brain 120. The central brain 2120 can be an example of the information processing device 1D involved in this embodiment. As Figure 12 shown, the central brain 2120 can be communicably connected to multiple gateways. The central brain 2120 involved in this embodiment can achieve level 6 autonomous driving based on multiple information obtained via the gateways. The central brain 2120 is an example of an information processing device.

[0222] As Figure 12As shown, the central brain 2120 is communicably connected to a plurality of gateways. The central brain 2120 is connected to an external cloud via the gateways. The central brain 2120 is configured to be able to access the external cloud via the gateways. On the other hand, due to the existence of the gateways, it is configured that the central brain 2120 cannot be directly accessed from the outside.

[0223] The central brain 2120 outputs a request signal to the server every predetermined time. Specifically, the central brain 2120 outputs a request signal indicating an inquiry to the server every one billionth of a second.

[0224] Examples of the sensors included in the vehicle 2012 used in the present embodiment may include radar, lidar (LiDAR), high-pixel / long-focus / ultra-wide-angle / 360-degree / high-performance cameras, visual recognition, faint sound, ultrasonic waves, vibration, infrared rays, ultraviolet rays, electromagnetic waves, temperature, humidity, spot AI weather forecast, high-precision multi-channel global positioning system (GPS), low-altitude satellite information, long-tail event AI data, etc. The long-tail event AI data refers to the trip data of a vehicle equipped with functions capable of achieving Level 5 autonomous driving.

[0225] The above sensors include sensors that detect the conditions around the vehicle. As the period for detecting the conditions around the vehicle, the sensors that detect the conditions around the vehicle detect the conditions around the vehicle at a second period shorter than the first period for photographing the surroundings of the vehicle with a camera or the like.

[0226] Examples of the sensor information obtained from various sensors may include the movement of the center of gravity of the weight, the detection of the road material, the detection of the external air temperature, the detection of the external air humidity, the detection of the inclination angles in the up / down, lateral, and diagonal directions of the slope, the freezing mode of the road, the detection of the moisture content, the material, wear condition, and air pressure of each tire, the road width, the presence or absence of overtaking prohibition, oncoming vehicles, the model information of the front and rear vehicles, the cruising states of these vehicles, the surrounding conditions (e.g., birds, animals, footballs, accident vehicles, earthquakes, fires, winds, typhoons, heavy rains, light rains, snowstorms, or fog), etc. In the present embodiment, these detections are performed every one billionth of a second.

[0227] The central brain 2120 that functions as an example of the information processing device according to the present embodiment at least includes the respective functions of an acquisition unit, a calculation unit, and a control unit, where the acquisition unit can acquire a plurality of information related to the vehicle, the calculation unit calculates a control variable based on the plurality of information acquired by the acquisition unit, and the control unit performs driving control of the vehicle based on the control variable.

[0228] For example, the central brain 2120 performs the following functions: using one or more pieces of sensor information detected by the above sensors to calculate control variables for the wheel speed, tilt, and for each supporting wheel, the wheel speed, tilt, and suspension of the four wheels of the vehicle. It should be noted that the tilt of the wheel includes both the tilt of the wheel relative to the axis horizontal to the road and the tilt of the wheel relative to the axis perpendicular to the road.

[0229] Here, one or more pieces of sensor information can apply the sensor information from sensors that detect the conditions around the vehicle. In addition, when using multiple pieces of sensor information as one or more pieces of sensor information, a predetermined number of sensor information can be applied. The predetermined number is, for example, 3. Based on the 3 pieces of sensor information, index values for controlling the wheel speed, tilt, and suspension are calculated. The number of index values calculated according to the combination of the 3 pieces of sensor information is, for example, 3. Among the index values for controlling the wheel speed, tilt, and suspension, for example, there are index values calculated according to the information related to air resistance in the sensor information, index values calculated according to the information related to road resistance in the sensor information, index values calculated according to the information related to the slip coefficient in the sensor information, and so on.

[0230] In addition, the index values calculated for each combination of different multiple pieces of sensor information for one piece of sensor information or a combination of sensor information are aggregated to calculate the control variables for controlling the wheel speed, tilt, and suspension. For example, index values are calculated according to the sensor information of sensors that detect the conditions around the vehicle, and the control variables are calculated. In addition, when using multiple pieces of sensor information, for example, multiple index values are calculated through the combination of sensor 1, sensor 2, and sensor 3, multiple index values are calculated through the combination of sensor 4, sensor 5, and sensor 6, multiple index values are calculated through the combination of sensor 1, sensor 3, and sensor 7, and the index values are aggregated to calculate the control variables. In this way, while changing the combination of sensor information, a predetermined number, for example, 300 index values are calculated, and the control variables are calculated. Specifically, the calculation unit can use machine learning, and more specifically, can use deep learning, to calculate the control variables according to the sensor information. In other words, the calculation unit for calculating the index values and the control variables can be composed of AI (Artificial Intelligence).

[0231] The calculation unit uses the computing power at level 6 to perform multivariate analysis based on the integration method shown in the following formula (1) on the data collected every nanosecond by a large number of sensor groups, etc. (for example, refer to formula (2)), so as to be able to obtain accurate control variables. More specifically, while obtaining the integral value of the Delta value of various ultra-high resolutions at level 6 of the computing power, the indexed values of each variable are obtained at the edge level and in real time, and the result that will occur in the next nanosecond can be obtained with the highest probability value.

[0232] [Calculation formula 1]

[0233]

[0234] [Calculation formula 2]

[0235] V n =DL(f(A, B, C,..., N)(dA n / dt)) (2)

[0236] It should be noted that DL in the formula represents deep learning, and A, B, C,..., N are index values calculated based on sensor information, such as index values calculated based on air resistance, index values calculated based on road resistance, index values calculated based on road elements, and index values calculated based on slip coefficient, etc. When the index values calculated while changing the combination of a pre-determined number of sensor information are 300, the index values of A to N in the formula also become 300, and the 300 index values are summarized.

[0237] In addition, in the above formula (2), the wheel speed (V) is calculated, but the control variables for controlling the tilt (steering angle, camber) and suspension are also calculated similarly.

[0238] If a detailed explanation is given, regarding the tilt (steering angle R), accurate control variables can be obtained by performing multivariate analysis based on the integration method shown in the following formula (3) (for example, refer to formula (4)).

[0239] [Calculation formula 3]

[0240]

[0241] [Calculation formula 4]

[0242] R n =DL(f(A, B, C,..., N)(dA n / dt)) (4)

[0243] In addition, regarding the inclination (camber angle C), by performing multivariate analysis based on the integration method shown in the following formula (5) (for example, refer to formula (6)), accurate control variables can be obtained.

[0244] [Calculation formula 5]

[0245]

[0246] [Calculation formula 6]

[0247] C n = DL(f(A, B, C,..., N)(dA n / dt)) (6)

[0248] Furthermore, regarding the suspension S, by performing multivariate analysis based on the integration method shown in the following formula (7) (for example, refer to formula (8)), accurate control variables can be obtained.

[0249] [Calculation formula 7]

[0250]

[0251] [Calculation formula 8]

[0252] S n = DL(f(A, B, C,..., N)(dA n / dt)) (8)

[0253] Specifically, the central brain 2120 calculates a total of 16 control variables, which are used to control the wheel speeds of each of the 4 wheels, the inclination of each of the 4 wheels relative to the axis horizontal to the road (steering angle, camber angle), the inclination of each of the 4 wheels relative to the axis perpendicular to the road (steering angle, camber angle), and the suspensions that support each of the 4 wheels. In this embodiment, the calculation of the above 16 control variables is performed every one billionth of a second. It should be noted that the wheel speeds of each of the 4 wheels can also be referred to as "the rotation speeds of the in-wheel motors mounted on each of the 4 wheels", and the inclination of each of the 4 wheels relative to the axis horizontal to the road (steering angle) can also be referred to as "the horizontal angles of each of the 4 wheels".

[0254] The inclination of each of the 4 wheels relative to the axis perpendicular to the above road (camber angle) can be referred to as "the vertical angles of each of the 4 wheels". In addition, the suspensions (coil springs, shock absorbers) that determine the positions of the wheels relative to the above road can be referred to as "the attenuation amounts that absorb the impacts from the road received by each of the 4 wheels".

[0255] Also, for example, when the vehicle is traveling on a mountain road, the above control variable is a value for performing optimal steering that matches the mountain road, and when the vehicle is parked in a parking lot, the above control variable is a value for traveling at an optimal angle that matches the parking lot.

[0256] In addition, in the present embodiment, the central brain 2120 calculates a total of 16 control variables for controlling the wheel speed of each of the 4 wheels, the inclination (steering angle) of each of the 4 wheels with respect to an axis horizontal to the road, the inclination (camber angle) of each of the 4 wheels with respect to an axis perpendicular to the road, and the suspension that supports each of the 4 wheels. However, this calculation does not have to be performed by the central brain 2120, and a dedicated anchor chip for calculating the above control variables may be provided separately. In this case, DL in formula (2) also represents deep learning, and A, B, C,..., N also represent index values calculated based on sensor information. When the number of aggregated indexes is 300 as described above, the number of indexes in this formula also becomes 300.

[0257] In addition, in the present embodiment, the central brain 2120 functions as a control unit that controls autonomous driving in units of one billionth of a second based on the calculated control variables. Specifically, the central brain 2120 controls the in-wheel motors mounted on the 4 wheels respectively based on the above 16 control variables, thereby controlling the wheel speed, inclination (steering angle, camber angle), and the suspension that supports each of the 4 wheels of the vehicle 2012, and performing autonomous driving.

[0258] It should be noted that the index value calculated based on sensor information can be calculated based on the sensor information of one sensor. For example, when it is a sensor for detecting the surrounding conditions of the vehicle, the index value is calculated based on the sensor information of the sensor for detecting the surrounding conditions of the vehicle, and the control variable is calculated.

[0259] The central brain 120 repeatedly executes Figure 13 the flowchart shown.

[0260] In step S10, the central brain 120 acquires sensor information including road information detected by the sensor. Then, the central brain 120 proceeds to step S11. The process of step S10 is an example of the function of the acquisition unit.

[0261] In step S11, the central brain 2120 calculates the above 16 control variables based on the sensor information acquired in step S10. Then, the central brain 120 proceeds to step S12. The process of step S11 is an example of the function of the calculation unit.

[0262] In step S12, the central brain 120 controls autonomous driving based on the control variables calculated in step S11. Then, the central brain 2120 ends the processing of this flowchart. The processing of step S12 is an example of the function of the control unit.

[0263] Figures 14 to 18 It is an explanatory diagram showing an example of the control of autonomous driving based on the central brain 120. It should be noted that Figures 14 to 16 is an explanatory diagram from the perspective of observing the vehicle 12 from the front, Figure 17 and Figure 18 is an explanatory diagram from the perspective of observing the vehicle 2012 from below.

[0264] Figure 14 It shows the situation where the vehicle 2012 is traveling on the flat road R1. The central brain 2120 controls the in-wheel motors 2031 respectively mounted on the four wheels 2030 based on the above 16 control variables calculated according to the road R1, so as to control the wheel speed, inclination (steering angle, camber angle) of each of the four wheels 2030 and the suspensions 2032 respectively supporting the four wheels 2030, and perform autonomous driving.

[0265] Figure 15 It shows the situation where the vehicle 2012 is traveling on the mountain road R2. The central brain 2120 controls the in-wheel motors 2031 respectively mounted on the four wheels 2030 based on the above 16 control variables calculated according to the mountain road R2, so as to control the wheel speed, inclination (steering angle, camber angle) of each of the four wheels 2030 and the suspensions 2032 respectively supporting the four wheels 2030, and perform autonomous driving.

[0266] Figure 16 It shows the situation where the vehicle 2012 is traveling in the waterlogged section R3. The central brain 2120 controls the in-wheel motors 2031 respectively mounted on the four wheels 2030 based on the above 16 control variables calculated according to the waterlogged section R3, so as to control the wheel speed, inclination (steering angle, camber angle) of each of the four wheels 2030 and the suspensions 2032 respectively supporting the four wheels 2030, and perform autonomous driving.

[0267] Figure 17 It shows the situation where the vehicle 2012 turns in the direction shown by the arrow A1. The central brain 120 controls the in-wheel motors 2031 respectively mounted on the four wheels 2030 based on the above 16 control variables calculated according to the incoming curve, so as to control the wheel speed, inclination (steering angle, camber angle) of each of the four wheels 2030 and the suspensions 2032 (not shown) respectively supporting the four wheels 2030, and perform autonomous driving.

[0268] Figure 18The case where the vehicle 2012 moves parallel in the direction shown by the arrow A2 is shown. The central brain 2120 controls the in-wheel motors 2031 respectively mounted on the four wheels 2030 based on the above 16 control variables calculated according to the parallel movement in the direction shown by the arrow A2, thereby controlling the wheel speed, inclination (steering angle, camber angle) of each of the four wheels 2030 and the suspension 2032 (not shown) that supports the four wheels 2030 respectively, and performs autonomous driving.

[0269] It should be noted that Figures 14 to 18 The states (inclinations) of the wheels 2030 and the suspension 2032 shown are just examples, and it goes without saying that the states of the wheels 2030 and the suspension 2032 different from the states shown in each figure may occur.

[0270] Here, although the existing in-wheel motors mounted on vehicles can independently control their respective drive wheels, in this vehicle, it is impossible to analyze road conditions and the like to control the in-wheel motors. Therefore, in this vehicle, for example, when driving on a mountain road or a waterlogged section, etc., appropriate autonomous driving based on road conditions and the like cannot be performed.

[0271] However, through the vehicle 2012 according to the present embodiment, based on the structure described above, autonomous driving that can control speed, steering, etc. according to the environment such as road conditions can be performed.

[0272] However, during the driving of the vehicle, sometimes an obstacle approaches the vehicle. In this case, preferably, the vehicle changes its behavior and drives to avoid contact or collision with the obstacle. As an example of the obstacle, other vehicles, walls, guardrails, curbstones, and other installations other than the vehicle itself during driving can be listed. In the following description, the case where another vehicle approaching the vehicle 2012 is taken as an example of the obstacle is described. It should be noted that the obstacle is an example of the object.

[0273] Figure 19 It is a diagram schematically showing the state where the vehicle 2012 traveling on a two-way two-lane road is taken as the vehicle 2012A and other vehicles 2012B, 2012C, and 2012D are traveling around the vehicle 2012A. In the example of this figure, it is the following state: Other vehicle 2012B is traveling behind the vehicle 2012A, other vehicle 2012D is traveling in front on the oncoming lane, and other vehicle 2012C is traveling behind.

[0274] The central brain 2120 of the vehicle 2012A controls the in-wheel motors 2031 respectively mounted on the four wheels 2030 based on the control variables that change moment by moment and are calculated according to the driving state on the driving path, so as to control the wheel speeds, tilts (steering angles, camber angles) of the four wheels 2030 respectively, and the suspensions 2032 that support the four wheels 2030 respectively, and perform autonomous driving. In addition, the central brain 2120 detects the behaviors of other vehicles 2012B, 2012C, 2012D around the vehicle 2012A through sensors and obtains them as sensor information.

[0275] The central brain 2120 at least has the following functions: predicting the collision of an obstacle as an object with the vehicle including contact based on the acquired sensor information, and driving to avoid the obstacle. As Figure 19 shown, when the central brain 2120 predicts that another vehicle 2012D enters in front of the vehicle 2012A based on the sensor information, it calculates the control variables representing the behavior of the vehicle 2012A that can at least avoid contact with the other vehicle 2012D.

[0276] For example, in Figure 19 the example shown, the vehicle 2012A collides with another vehicle 2012D on the path 2012Ax1 where it is driving in the current driving state. Therefore, the central brain 2120 calculates paths such as 2012Ax2 and 2012Ax3 that avoid the collision with the other vehicle 2012D, selects any one of them, and calculates the control variables. Regarding the selection of the path, it is only necessary to select the path of the control variable in the vehicle 2012A where the load caused by the control variable is lower than a predetermined specified value (for example, the minimum value).

[0277] The central brain 2120 that can achieve the above autonomous driving will be further described. The central brain 2120 is configured as Figure 20 the information processing device 1D shown. It should be noted that the above central brain 2120 is a general processing device that functions as an information processing device including a gateway, and the central brain 2125 described later is a narrow processing device when the functions are classified by the processor.

[0278] Figure 20FIG. 0 is a block diagram showing an example of the configuration of an information processing apparatus 1D including a central brain according to the fifth embodiment. The information processing apparatus 1D includes an IPU (Image Processing Unit) 2121, a MoPU (Motion Processing Unit) 2122, a central brain 2125, and a memory 2126. The central brain 2125 includes a GNPU (Graphics Neural network Processing Unit) 2123 and a CPU (Central Processing Unit) 2124.

[0279] The IPU 2121 may be built into an ultra-high resolution camera (not shown) provided in a vehicle. The IPU 2121 performs predetermined image processing such as Bayer conversion, demosaicing, denoising, and sharpening on an image of an object existing around the vehicle, and outputs the processed image of the object at a frame rate of, for example, 10 frames per second and a resolution of 12 million pixels. The image output from the IPU 2121 is provided to the central brain 2125 and the memory 2126.

[0280] The MoPU 2122 may be built into a low-resolution camera different from the ultra-high resolution camera provided in the vehicle. The MoPU 2122 outputs motion information indicating the motion of the photographed object at a frame rate of, for example, 1920 frames per second. That is, the frame rate of the output of the MoPU 2122 is one hundred times the frame rate of the output of the IPU 2121. The MoPU 2122 outputs vector information indicating the motion of a point indicating the existence position of the object along a predetermined coordinate axis as the motion information. That is, the motion information output from the MoPU 2122 does not include information required to identify what the photographed object is (for example, a person or an obstacle), and includes only information indicating the motion (moving direction and moving speed) of the center point (or center of gravity point) of the object on the coordinate axes (x-axis, y-axis, z-axis). The image output from the MoPU 2122 is provided to the central brain 2125 and the memory 2126. Since the motion information does not include image information, the amount of information transmitted to the central brain 2125 and the memory 2126 can be suppressed.

[0281] This embodiment includes: a first processor that outputs an image of an object captured at a first frame rate; and a second processor that outputs motion information representing the motion of the object captured at a second frame rate higher than the first frame rate. That is, the detection unit that captures the surroundings of the vehicle with the first period as the period for detecting the surroundings of the vehicle in this embodiment is an example of the above-mentioned "first processor", and the IPU 2121 is an example of the "first processor". In addition, the detection unit including a sensor that detects the surroundings of the vehicle with a second period shorter than the first period in this embodiment is an example of the above-mentioned "second processor", and the MoPU 2122 is an example of the "second processor".

[0282] The central brain 2125 performs driving control of the vehicle based on the image output from the IPU 2121 and the motion information output from the MoPU 2122. For example, the central brain 2125 identifies objects (people, animals, roads, traffic lights, signs, crosswalks, obstacles, buildings, etc.) existing around the vehicle based on the image output from the IPU 2121. In addition, the central brain 2125 identifies the motion of the objects whose identities have been recognized existing around the vehicle based on the motion information output from the MoPU 2122. Based on the recognized information, the central brain 2125 performs, for example, control (speed control) of the motor driving the wheels, braking control, and steering wheel control. In the central brain 2125, the GNPU 2123 can undertake the processing related to image recognition, and the CPU 2124 can undertake the processing related to vehicle control.

[0283] Generally, an ultra-high-resolution camera is used for image recognition in autonomous driving. It is possible to identify what objects are included in the image captured by the high-resolution camera. However, in the autonomous driving of level 6 era, this is not enough. In the level 6 era, it is also necessary to identify the motion of the objects. By identifying the motion of the objects, for example, a vehicle traveling through autonomous driving can perform an avoidance action of avoiding obstacles with higher accuracy. However, in a high-resolution camera, only about 10 frames of images can be acquired per second, and it is difficult to analyze the motion of the objects. On the other hand, although the camera equipped with the MoPU 2122 has a low resolution, it can output at a high frame rate of, for example, 1920 frames per second.

[0284] Therefore, in the technology of this embodiment, two independent processors, namely IPU 2121 and MoPU 2122, are used. The high-resolution camera (IPU 2121) is given the role of acquiring the image information required to identify what the captured object is, and MoPU 2122 is given the role of detecting the movement of the object. MoPU 2122 captures the object as a point and analyzes in which directions on the x-axis, y-axis, and z-axis and at what speed the coordinates of this point move. Since the overall contour of the object and the detection of what the object is can be carried out through the image from the high-resolution camera, as long as it is known how the center point of the object moves through MoPU 2122, it is known how the whole object behaves.

[0285] According to the method of only analyzing the movement and speed of the center point of the object, compared with the method of judging how the whole image of the object moves, the amount of information transmitted to the central brain 2125 can be greatly suppressed, and the amount of calculation in the central brain 2125 can be greatly reduced. For example, when sending an image of 1000 pixels × 1000 pixels to the central brain 2125 at a frame rate of 1920 frames per second, if color information is included, 4 billion bits per second of data will be sent to the central brain 2125. By only sending the motion information indicating the movement of the center point of the object, MoPU 2122 can compress the amount of data transmitted to the central brain 2125 to 20,000 bits per second. That is, the amount of data transmitted to the central brain 2125 is compressed to 1 / 200,000.

[0286] In this way, by combining and using the low frame rate and high-resolution image output from IPU 2121 and the high frame rate and lightweight motion information output from MoPU 2122, object recognition including the movement of the object can be achieved with a smaller amount of data.

[0287] It should be noted that when one MoPU 2122 is used, vector information indicating the movement of the point representing the existence position of the object along each of the two coordinate axes (x-axis and y-axis) in the three-dimensional orthogonal coordinate system can be obtained. The principle of a stereo camera can be used, and two MoPU 2122 can be used to output vector information indicating the movement of the point representing the existence position of the object along each of the three coordinate axes (x-axis, y-axis, and z-axis) in the three-dimensional orthogonal coordinate system. The z-axis is the axis along the depth direction (the driving of the vehicle).

[0288] In the present embodiment, as sensor information, as a cycle for detecting the surrounding conditions of the vehicle 2012, the surrounding conditions of the vehicle can be detected at a second cycle shorter than the first cycle of photographing the surrounding of the vehicle using a camera or the like. That is, in the above-mentioned level 5, the surrounding conditions can be detected twice every 0.3 seconds using a camera or the like, but in the present embodiment, the surrounding conditions can be detected 576 times using level 6. Further, an index value and a control variable can be calculated every time the surrounding conditions are detected 576 times, the behavior of the vehicle can be controlled, and a faster and safer driving can be performed compared to the autonomous driving performed at level 5.

[0289] In the above, the case of avoiding a collision by selecting a path has been described, but it is not limited thereto. For example, in a case where it is difficult to avoid a collision between the own vehicle 2012A and another vehicle, a control variable for reducing the damage caused to the vehicle in an inevitable collision can be calculated. That is, the central brain 2120 may calculate a control variable corresponding to damage such that the damage caused to the vehicle in an inevitable collision is equal to or less than a predetermined threshold value as the control variable. Therefore, including the above calculation unit, based on the acquired plurality of sensor information, predicting a collision of an object with the vehicle 2012, and in a case where the predicted prediction result indicates an inevitable collision, calculating a control variable corresponding to damage such that the damage caused to the vehicle in an inevitable collision is equal to or less than a predetermined threshold value as the control variable. The damage caused to the vehicle can apply at least one of the deformation position and the deformation amount of the above vehicle. Further, the control variable corresponding to the damage caused to the vehicle can apply at least one of the collision angle and the vehicle speed of the vehicle.

[0290] Figure 21 is a flowchart showing an example of a process in the central brain 2120 that can reduce the damage caused to the vehicle 2012 in an inevitable collision. In Figure 21 in, the Figure 13 processing of step S11 shown is replaced with steps S11A to S11D and executed. The central brain 2120 can repeatedly execute the Figure 21 processing shown to replace the Figure 13 processing shown.

[0291] In step S11A, the central brain 2120 calculates the relationship between the own vehicle and another vehicle. Specifically, in step S11A, based on the sensor information, predicting a collision of an object (for example, another vehicle) with the vehicle 2012, and determining whether the prediction result is an inevitable collision. And, in step S11B, it is determined whether the determination result based on the prediction is an inevitable collision. In a case where it is determined to be affirmative in step S11B, the process transfers to step S11D, and in a case where it is determined to be negative, it is the same as the above Figure 13Similar to step S11, a control variable is calculated in step S11C. In step S11D, as described above, a control variable that reduces the damage caused by the vehicle in an inevitable collision is calculated, and then step S12 is entered.

[0292] The processing of steps S11A, S11B, S11C, and S11D is not limited to the above order. For example, after the processing of step S11C, the processing of steps S11B and S11D can be executed. Specifically, first, in step S11C, a collision of another vehicle with vehicle 2012 is predicted, and a control variable representing the relationship with the lowest risk is calculated for the mutual relationship between the own vehicle and the other vehicle. The relationship with the lowest risk can include, for example, a relationship including contact or collision and a relationship with a lower possibility of including contact or collision in the relationship of avoiding this relationship. That is, a control variable with the lowest collision risk is calculated. Next, in step S11B, based on the calculated control variable, the behavior of vehicle 2012 is predicted, and a collision of another vehicle with vehicle 2012 is predicted, and it is judged whether the collision is inevitable based on the prediction result. In this case, if the judgment in step S11B is negative, step S12 can be directly entered. On the other hand, if the judgment is positive, in step S11D, based on the control variable calculated in step S11C, a control variable that can further reduce the collision risk is calculated. Specifically, at least one of the collision angle and the vehicle speed of the vehicle that reduces the damage caused by the contact or collision between the own vehicle and the other vehicle is calculated.

[0293] It should be noted that for the control variable related to the speed of vehicle 2012, the acceleration side can be set to 10 types each of large, medium, and small (L, M, S), a total of 30 modes are applied, and the deceleration side can be set to 10 types each of large, medium, and small (L, M, S), a total of 30 modes are applied for selection. In this case, since the relationship between the own vehicle and the other vehicle changes every moment, in the state where the own vehicle and the other vehicle are approaching, the distance changes in the approaching direction, so the selection options for the mode become smaller every moment, the processing time for selection can be reduced, and further, the adjustment of the difference of Delta becomes smaller.

[0294] In addition, in the above, the case of applying other vehicle 2012D to another vehicle with respect to own vehicle 2012A is described, but at least one and all of the other vehicles 2012B, 2012C, and 2012D around own vehicle 2012A can also be used as the object, and a control variable that reduces the collision risk for the other vehicle as the object is calculated.

[0295] Therefore, based on the control variables calculated according to the prediction of collisions including contacts, the central brain 2120 controls, for example, the in-wheel motors 2031 respectively mounted on the four wheels 2030, thereby controlling the wheel speeds, tilts (steering angles, camber angles) of the four wheels 2030 respectively, and the suspensions 2032 that support the four wheels 2030 respectively, so that autonomous driving can be performed while avoiding collisions or reducing the damage caused to the vehicle during collisions.

[0296] (Sixth Embodiment)

[0297] In addition, when providing an autonomous driving function, it is necessary to process the information of various sensors to derive the control variables for controlling autonomous driving. However, in existing autonomous driving, there is a problem that the amount of computation or the computation time required to derive the control variables increases. Therefore, an information processing device and the like that can derive the control variables without performing the inference process of the control variables every time will be described below.

[0298] Figure 22 FIG. is a schematic diagram showing an example of a vehicle 3100 equipped with a central brain 3015. A plurality of gateways are communicably connected to the central brain 3015. The central brain 3015 is connected to an external cloud server via the gateway. The central brain 3015 is configured to be able to access the external cloud server via the gateway. On the other hand, due to the presence of the gateway, it is configured that the central brain 3015 cannot be directly accessed from the outside.

[0299] Every time a predetermined time elapses, the central brain 3015 outputs a request signal to the cloud server. Specifically, the central brain 3015 outputs a request signal indicating an inquiry to the cloud server every one billionth of a second. As an example, the central brain 3015 controls the autonomous driving of level 6 based on a plurality of information obtained via the gateway.

[0300] Figure 23 FIG. is a first block diagram showing an example of the structure of the information processing device 1E. The information processing device 1E includes an IPU (Image Processing Unit) 3011, a MoPU (Motion Processing Unit) 3012, a central brain 3015, and a memory 3016. The central brain 3015 is configured to include a GNPU (Graphics Neural network Processing Unit) 3013 and a CPU (Central Processing Unit) 3014.

[0301] The IPU 3011 is built into a super high-resolution camera (not shown) provided in the vehicle 3100. The IPU 3011 performs predetermined image processing such as Bayer transformation, demosaicing, denoising, and sharpening on the image of an object existing around the vehicle 3100 captured by the super high-resolution camera, and outputs the processed image of the object at a frame rate of, for example, 10 frames per second and a resolution of 12 million pixels. In addition, the IPU 3011 outputs identification information for identifying the object captured based on the image of the object captured by the super high-resolution camera. The identification information is information required to identify what the captured object is (for example, a person or an obstacle). In the present embodiment, the IPU 3011 outputs label information indicating the category of the captured object (for example, information indicating whether the captured object is a dog, a cat, or a bear) as the identification information. Further, the IPU 3011 outputs position information indicating the position of the captured object in the camera coordinate system of the super high-resolution camera. The image, label information, and position information output from the IPU 3011 are provided to the central brain 3015 and the memory 3016. The IPU 3011 is an example of a "second processor", and the super high-resolution camera is an example of a "second camera".

[0302] The MoPU 3012 is built into another camera (not shown) different from the super high-resolution camera provided in the vehicle 3100. The MoPU 3012 outputs point information obtained by capturing the captured object as a point at a frame rate of, for example, 100 frames per second or more based on the image of the object captured by the other camera facing the direction corresponding to the super high-resolution camera at a frame rate of 100 frames per second or more. The point information output from the MoPU 3012 is provided to the central brain 3015 and the memory 3016. Thus, the image used by the MoPU 3012 to output the point information and the image used by the IPU 3011 to output the identification information refer to the images captured by the other camera and the super high-resolution camera facing the corresponding direction. Here, the "corresponding direction" means the direction in which the imaging range of the other camera overlaps with the imaging range of the super high-resolution camera. In the above case, the other camera captures an object facing the direction overlapping with the imaging range of the super high-resolution camera. It should be noted that the super high-resolution camera and the other camera capture an object facing the corresponding direction, for example, by previously obtaining the correspondence relationship of the camera coordinate systems between the super high-resolution camera and the other camera.

[0303] For example, as point information, MoPU 3012 outputs the coordinate values of at least two coordinate axes in a three-dimensional orthogonal coordinate system representing the position where an object exists. As an example, this coordinate value represents the center point (or centroid) of the object. In addition, as the coordinate values of two coordinate axes, MoPU 3012 outputs the coordinate value of the axis (x-axis) in the width direction in this three-dimensional orthogonal coordinate system (hereinafter referred to as "x coordinate value") and the coordinate value of the axis (y-axis) in the height direction (hereinafter referred to as "y coordinate value"). It should be noted that the x-axis is the axis along the vehicle width direction of the vehicle 3100, and the y-axis is the axis along the vehicle height direction of the vehicle 3100.

[0304] According to the above structure, since the point information output by MoPU 3012 per second contains more than 100 frames of x coordinate values and y coordinate values, based on this point information, the movement (moving direction and moving speed) of the object on the x-axis and y-axis in the above three-dimensional orthogonal coordinate system can be grasped. That is to say, the point information output by MoPU 3012 contains position information representing the position of the object in the above three-dimensional orthogonal coordinate system and movement information representing the movement of the object.

[0305] As described above, in the point information output from MoPU 3012, the information required to identify what the captured object is (for example, a person or an obstacle) is not included, and only the information representing the movement (moving direction and moving speed) of the center point (or centroid) of the object on the x-axis and y-axis is included. And since the point information output from MoPU 3012 does not include image information, the data volume output to the central brain 3015 and the memory 3016 can be significantly reduced. MoPU 3012 is an example of a "first processor", and other cameras are examples of "first cameras".

[0306] As described above, in this embodiment, the frame rate of other cameras with MoPU 3012 built in is higher than the frame rate of the ultra-high resolution camera with IPU 3011 built in. Specifically, the frame rate of other cameras is 100 frames per second or more, and the frame rate of the ultra-high resolution camera is 10 frames per second. That is to say, the frame rate of other cameras is more than 10 times the frame rate of the ultra-high resolution camera.

[0307] The central brain 3015 associates the point information output from MoPU 3012 with the label information output from IPU 3011. For example, due to the frame rate difference between the above other cameras and the ultra-high resolution camera, the central brain 3015 is in a state where it has obtained the point information related to the object but has not obtained the label information. In this state, the central brain 3015 identifies the x coordinate value and y coordinate value of the object based on the point information, but does not identify what the object is.

[0308] After that, in the case where the tag information related to the above object is obtained, the central brain 3015 exports the category of the tag information (for example: PERSON (human)). And, the central brain 3015 associates the tag information with the point information obtained above. Thus, the central brain 3015 identifies the x coordinate value and y coordinate value of the object based on the point information, and identifies what the object is. The central brain 3015 is an example of the "third processor".

[0309] Here, when there are multiple objects captured by the ultra-high resolution camera and other cameras, such as object A and object B, the central brain 3015 associates the point information and tag information related to each object in the following manner. Due to the frame rate difference between the above-mentioned other camera and the ultra-high resolution camera, the central brain 3015 is in a state where it has obtained the point information related to object A and object B (hereinafter referred to as "point information A" and "point information B"), but has not obtained the tag information. In this state, the central brain 3015 identifies the x coordinate value and y coordinate value of object A based on point information A, and identifies the x coordinate value and y coordinate value of object B based on point information B, but does not identify what these objects are.

[0310] After that, in the case where one piece of tag information is obtained, the central brain 3015 exports the category of the one piece of tag information (for example: PERSON). And, the central brain 3015 determines the point information associated with the one piece of tag information based on the position information output from the IPU 3011 together with the one piece of tag information and the position information included in the obtained point information A and point information B. For example, the central brain 3015 determines the point information including the position information that represents the position closest to the position of the object represented by the position information output from the IPU 3011, and associates the point information with the one piece of tag information. In the case where the point information determined above is point information A, the central brain 3015 associates the one piece of tag information with point information A, identifies the x coordinate value and y coordinate value of object A based on point information A, and identifies what object A is.

[0311] As described above, when there are multiple objects captured by the ultra-high resolution camera and other cameras, the central brain 3015 associates the point information with the tag information based on the position information output from the IPU 3011 and the position information included in the point information output from the MoPU 3012.

[0312] In addition, the central brain 3015 identifies objects (persons, animals, roads, signals, signs, crosswalks, obstacles, buildings, etc.) existing around the vehicle 3100 based on the images and label information output from the IPU 3011. In addition, the central brain 3015 identifies the positions and movements of the objects, whose identities have been recognized, existing around the vehicle 3100 based on the point information output from the MoPU 3012. Based on the recognized information, the central brain 3015 performs controls such as control (speed control) of the electric motor for driving the wheels, braking control, and steering wheel control, to control the autonomous driving of the vehicle 3100. For example, the central brain 3015 controls the autonomous driving of the vehicle 3100 according to the position information and movement information included in the point information output from the MoPU 3012, so as to avoid collisions with objects. In the central brain 3015, the GNPU 13 can undertake the processing related to image recognition, and the CPU 3014 can undertake the processing related to the control of the vehicle 3100.

[0313] Generally, an ultra-high-resolution camera is used for image recognition in autonomous driving. Here, based on the image captured by the ultra-high-resolution camera, it is possible to identify what the objects included in the image are. However, in the autonomous driving in the level 6 era, this alone is not sufficient. In the level 6 era, it is also necessary to identify the movement of objects with higher precision. The MoPU 3012 is used to identify the movement of objects with higher precision, so that it is possible to perform avoidance actions such as the vehicle 3100 avoiding obstacles during autonomous driving with higher precision. However, through the ultra-high-resolution camera, only about 10 frames of images can be obtained per second, and the precision of analyzing the movement of objects is lower than that of the camera equipped with the MoPU 3012. On the other hand, through the camera equipped with the MoPU 3012, for example, it is possible to output at a high frame rate of 100 frames per second.

[0314] Therefore, the information processing device 1E according to the sixth embodiment includes two independent processors, namely the IPU 3011 and the MoPU 3012. The information processing device 1E assigns the role of obtaining the information required to identify what the captured objects are to the IPU 3011 built into the ultra-high-resolution camera, and assigns the role of detecting the positions and movements of objects to the MoPU 3012 built into other cameras. The MoPU 3012 captures the photographed objects as points, and analyzes in which directions on at least the x-axis and y-axis in the above-mentioned three-dimensional orthogonal coordinate system the coordinates of the points move at what speed. Since the overall contour of the object and the detection of what the object is can be performed through the images from the ultra-high-resolution camera, as long as it is known how, for example, the center point of the object moves through the MoPU 3012, it is known what behavior the entire object makes.

[0315] According to the method of only analyzing the movement and speed of the center point of an object, compared with the case of judging how the entire image of the object moves, the amount of data output to the central brain 3015 can be greatly suppressed, and the amount of computation in the central brain 3015 can be greatly reduced. For example, in the case of outputting an image of 1000 pixels × 1000 pixels to the central brain 3015 at a frame rate of 1000 frames per second, if color information is included, 4 billion bits per second of data will be output to the central brain 3015. By only outputting the point information representing the movement of the center point of the object, MoPU 3012 can compress the amount of data output to the central brain 3015 to 20,000 bits per second. That is, the amount of data output to the central brain 3015 is compressed to 1 / 200,000.

[0316] In this way, by combining and using the low frame rate and high-resolution image and label information output from the IPU 3011 with the high frame rate and lightweight point information output from the MoPU 3012, object recognition including the movement of the object can be achieved with a small amount of data.

[0317] In addition, in the information processing device 1E, by associating the point information output from the MoPU 3012 with the label information output from the IPU 3011 by the central brain 3015, information related to what kind of object is performing what kind of movement can be grasped.

[0318] (Seventh Embodiment)

[0319] Next, the seventh embodiment related to this embodiment will be described by omitting or simplifying the parts that overlap with the above sixth embodiment.

[0320] Figure 24 It is a second block diagram showing an example of the structure of the information processing device 1E related to the seventh embodiment. As Figure 24 shown, the information processing device 1E mounted on the vehicle 3100 includes MoPU 3012L equivalent to the left eye, MoPU 3012R equivalent to the right eye, IPU 3011, and central brain 3015.

[0321] MoPU 3012L is equipped with a camera 3030L, a radar 3032L, an infrared camera 3034L, and a core 3017L. In addition, MoPU 3012R is equipped with a camera 3030R, a radar 3032R, an infrared camera 3034R, and a core 3017R. It should be noted that hereinafter, when not distinguishing between MoPU 3012L and MoPU 3012R, it is denoted as "MoPU 3012", when not distinguishing between camera 3030L and camera 3030R, it is denoted as "camera 3030", when not distinguishing between radar 3032L and radar 3032R, it is denoted as "radar 3032", when not distinguishing between infrared camera 3034L and infrared camera 3034R, it is denoted as "infrared camera 3034", and when not distinguishing between core 3017L and core 3017R, it is denoted as "core 3017".

[0322] The camera 3030 equipped in MoPU 3012 captures an object at a frame rate greater than that of the ultra-high-resolution camera (e.g., 10 frames per second) equipped in IPU 3011 (120, 240, 480, 960, or 1920 frames per second). The frame rate of the camera 3030 is variable. The camera 3030 is an example of the "first camera".

[0323] The radar 3032 equipped in MoPU 3012 acquires a radar signal, which is a signal of a reflected wave based on an electromagnetic wave irradiated on an object and reflected from the object. The infrared camera 3034 equipped in MoPU 3012 is a camera that captures an infrared image.

[0324] The core 3017 (e.g., composed of one or more CPUs) equipped in MoPU 3012 extracts feature points for each frame image captured by the camera 3030, and outputs the x coordinate value and y coordinate value of the object in the above three-dimensional orthogonal coordinate system as point information. The core 3017 uses, for example, the center point (center of gravity point) of the object extracted from the image as a feature point. It should be noted that the point information output by the core 3017 is similar to that in the above embodiment and includes position information and motion information.

[0325] IPU 3011 is equipped with an ultra-high-resolution camera (not shown), and outputs an image of an object captured by the ultra-high-resolution camera, label information indicating the category of the object, and position information indicating the position of the object in the camera coordinate system of the ultra-high-resolution camera.

[0326] The central brain 3015 acquires the point information output from the MoPU 3012, and the image, label information, and position information output from the IPU 3011. Further, the central brain 3015 associates the label information related to the object existing at the position corresponding to the position information included in the point information output from the MoPU 3012 and the position information output from the IPU 3011 with the point information. Thereby, in the information processing apparatus 1E, it is possible to associate the information on what the object represented by the label information is with the position and movement of the object represented by the point information.

[0327] Here, the MoPU 3012 changes the frame rate of the camera 3030 according to a specified factor. In the present embodiment, as an example of the specified factor, the MoPU 3012 changes the frame rate of the camera 3030 according to the score related to the external environment. In this case, the MoPU 3012 calculates the score related to the external environment for the vehicle 3100, and changes the frame rate of the camera 3030 according to the calculated score. Further, the MoPU 3012 outputs a control signal for the camera 3030 to capture an image at the changed frame rate. Thereby, the camera 3030 captures an image at the frame rate indicated by the control signal. With this configuration, according to the information processing apparatus 1E, it is possible to capture an image of an object at a frame rate suitable for the external environment.

[0328] Note that the information processing apparatus 1E mounted on the vehicle 3100 includes a variety of sensors (not shown). The MoPU 3012 calculates the degree of risk related to the movement of the vehicle 3100 as the score related to the external environment for the vehicle 3100 based on the sensor information (for example, the movement of the center of gravity of the weight, the detection of the road material, the detection of the external air temperature, the detection of the external air humidity, the detection of the up / down / left / right inclination angle of the slope, the freezing mode of the road, the detection of the moisture content, the material, wear condition, and air pressure of each tire, the road width, the presence or absence of overtaking prohibition, the oncoming vehicle, the vehicle types of the front and rear vehicles, the cruising states of these vehicles, or the surrounding conditions (birds, animals, footballs, accident vehicles, earthquakes, fires, winds, typhoons, heavy rain, light rain, snowstorms, fog, etc.)) acquired from the variety of sensors and the point information. The degree of risk indicates the degree to which the vehicle 3100 will travel in a dangerous place in the future. In this case, the MoPU 3012 changes the frame rate of the camera 3030 according to the calculated degree of risk. The vehicle 3100 is an example of a "moving body". With this configuration, according to the information processing apparatus 1E, it is possible to change the frame rate of the camera 3030 according to the degree of risk related to the movement of the vehicle 3100. The sensor is an example of a "detection unit", and the sensor information is an example of "detection information".

[0329] For example, the higher the calculated risk level is, the higher the frame rate of the camera 3030 is set by the MoPU 3012. When the calculated risk level is less than the first threshold, the MoPU 3012 changes the frame rate of the camera 3030 to 120 frames per second. In addition, when the calculated risk level is greater than or equal to the first threshold and less than the second threshold, the MoPU 3012 changes the frame rate of the camera 3030 to any one of 240, 480, and 960 frames per second. Further, when the calculated risk level is greater than or equal to the second threshold, the MoPU 3012 changes the frame rate of the camera 3030 to 1920 frames per second. It should be noted that in any of the above cases of the risk level, in addition to causing the camera 3030 to capture images at the selected frame rate, the MoPU 3012 can also output control signals to the radar 3032 and the infrared camera 3034 so that radar signals are acquired and infrared images are captured at values corresponding to the frame rate.

[0330] For example, the lower the calculated risk level is, the lower the frame rate of the camera 3030 is set by the MoPU 3012. When the frame rate of the camera 3030 is set to 1920 frames per second, and the calculated risk level is greater than or equal to the first threshold and less than the second threshold, the MoPU 3012 changes the frame rate of the camera 3030 to any one of 240, 480, and 960 frames per second. In addition, when the frame rate of the camera 3030 is set to 1920 frames per second, and the calculated risk level is less than the first threshold, the MoPU 3012 changes the frame rate of the camera 3030 to 120 frames per second. Further, when the frame rate of the camera 3030 is set to any one of 240, 480, and 960 frames per second, and the calculated risk level is less than the first threshold, the MoPU 3012 changes the frame rate of the camera 3030 to 120 frames per second. It should be noted that in this case, similar to the above, control signals can be output to the radar 3032 and the infrared camera 3034 so that radar signals are acquired and infrared images are captured at values corresponding to the changed frame rate of the camera 3030.

[0331] In addition, the MoPU 3012 can use big data related to driving, such as long-tail event AI (Artificial Intelligence) DATA (e.g., Trip data of a vehicle equipped with a level 5 autonomous driving control mode) or map information, which is known before the vehicle 3100 starts driving, as information for predicting the risk level to calculate the risk level.

[0332] In the above, a risk level is calculated as a score related to the external environment, but the index as the score related to the external environment is not limited to the risk level. For example, the MoPU 3012 may calculate a score related to the external environment different from the risk level based on the moving direction or speed of an object captured by the camera 3030, and change the frame rate of the camera 3030 according to the score. Hereinafter, the following case will be described: the MoPU 3012 calculates a score related to the speed of an object captured by the camera 3030, that is, a speed score, and changes the frame rate of the camera 3030 according to the speed score. As an example, the speed score is set such that the faster the speed of the object, the higher the speed score, and the slower the speed of the object, the lower the speed score. Also, the higher the calculated speed score, the higher the frame rate of the camera 3030 that the MoPU 3012 sets, and the lower the calculated speed score, the lower the frame rate of the camera 3030 that the MoPU 3012 sets. Therefore, when the calculated speed score becomes greater than or equal to the threshold due to the high speed of the object, the MoPU 3012 changes the frame rate of the camera 3030 to 1920 frames / second. In addition, when the calculated speed score becomes less than the threshold due to the low speed of the object, the MoPU 3012 changes the frame rate of the camera 3030 to 120 frames / second. It should be noted that in this case as well, similar to the above, a control signal can be output to the radar 3032 and the infrared camera 3034 so that radar signals and infrared images are acquired at values corresponding to the changed frame rate of the camera 3030.

[0333] Next, the following situation will be described: MoPU 3012 calculates a score related to the moving direction of an object captured by the camera 3030, that is, a direction score, and changes the frame rate of the camera 3030 according to the direction score. As an example, the direction score is set such that the direction score becomes higher when the moving direction of the object is the direction approaching the road, and the direction score becomes lower when the moving direction of the object is the direction moving away from the road. Moreover, the higher the calculated direction score, the higher the frame rate of the camera 3030 that MoPU 3012 makes, and the lower the calculated direction score, the lower the frame rate of the camera 3030 that MoPU 3012 makes. Specifically, MoPU 3012 determines the moving direction of the object by using AI or the like, and calculates the direction score based on the determined moving direction. And, when the direction score calculated due to the moving direction of the object being the direction approaching the road becomes greater than or equal to the threshold value, MoPU 3012 changes the frame rate of the camera 3030 to 1920 frames per second. In addition, when the direction score calculated due to the moving direction of the object being the direction moving away from the road becomes less than the threshold value, MoPU 3012 changes the frame rate of the camera 3030 to 120 frames per second. It should be noted that in this case, similar to the above, a control signal can also be output to the radar 3032 and the infrared camera 3034 so that the radar signal and the infrared image are captured with values corresponding to the changed frame rate of the camera 3030.

[0334] In addition, MoPU 3012 can output point information only for objects whose calculated score related to the external environment is greater than or equal to a specified threshold value. In this case, for example, MoPU 3012 can determine whether to output point information related to the object according to the moving direction of the object captured by the camera 3030. For example, MoPU 3012 may not output point information related to an object that has little influence on the driving of the vehicle 3100. Specifically, MoPU 3012 calculates the moving direction of the object captured by the camera 3030, and does not output point information related to an object such as a pedestrian gradually moving away from the road. On the other hand, MoPU 3012 outputs point information related to an object approaching the road (for example, an object such as a pedestrian about to rush onto the road). With this structure, according to the information processing device 1E, point information related to an object that has little influence on the driving of the vehicle 3100 can be not output.

[0335] In addition, in the above description, the case where the MoPU 3012 calculates the risk level is illustrated, but the disclosed technology is not limited to this method. For example, the central brain 3015 can calculate the risk level instead of the MoPU 3012. In this case, the central brain 3015 calculates the risk level related to the movement of the vehicle 3100 based on the sensor information obtained from various sensors and the point information output from the MoPU 3012, as a score related to the external environment for the vehicle 3100. And, the central brain 3015 outputs an instruction to change the frame rate of the camera 3030 to the MoPU 3012 according to the calculated risk level.

[0336] In addition, in the above description, the case where the MoPU 3012 outputs point information based on the image captured by the camera is illustrated, but the disclosed technology is not limited to this method. For example, the MoPU 3012 can output point information based on radar signals and infrared images instead of based on the image captured by the camera 3030. Similar to the image captured by the camera 3030, the MoPU 3012 can derive the x - coordinate value and y - coordinate value of an object from the infrared image of the object captured by the infrared camera 3034. The radar 3032 can obtain the three - dimensional point cloud data of an object based on radar signals. That is, the radar 3032 can detect the coordinate of the z - axis in the above - mentioned three - dimensional orthogonal coordinate system. Here, the z - axis is the axis along the depth direction of the object and the traveling direction of the vehicle 3100. Hereinafter, the coordinate value of the z - axis will be recorded as "z - coordinate value". In this case, the MoPU 3012 uses the principle of a stereo camera and combines the x - coordinate value and y - coordinate value of the object captured by the infrared camera 3034 with the z - coordinate value of the object represented by the three - dimensional point cloud data at the same timing as when the radar 3032 obtains the three - dimensional point cloud data of the object, and derives the coordinate values of the three axes (x - axis, y - axis, and z - axis) of the object as point information. And, the MoPU 3012 outputs the derived point information to the central brain 3015.

[0337] In addition, in the above description, the case where the MoPU 3012 derives point information is illustrated, but the disclosed technology is not limited to this method. For example, the central brain 3015 can derive point information instead of the MoPU 3012. The central brain 3015 derives point information by combining the information detected by, for example, the camera 3030L, the camera 3030R, the radar 3032, and the infrared camera 3034. As a specific example, the central brain 3015 performs three - point measurement based on the x - coordinate value and y - coordinate value of the object captured by the camera 3030L and the x - coordinate value and y - coordinate value of the object captured by the camera 3030R, and thus derives the coordinate values of the three axes (x - axis, y - axis, and z - axis) of the object as point information.

[0338] In addition, in the above description, an example is given where the central brain 3015 controls the autonomous driving of the vehicle 3100 based on the image and label information output from the IPU 3011 and the point information output from the MoPU 3012. However, the disclosed technology is not limited to this method. For example, the central brain 3015 can perform motion control of a robot based on the above information output from the IPU 3011 and the MoPU 3012. The robot can be a humanoid intelligent robot that performs tasks instead of humans. In this case, the central brain 3015 performs motion control of the robot's arms, palms, fingers, feet, etc. based on the above information output from the IPU 3011 and the MoPU 3012, so that it can perform actions such as grasping, holding, hugging, carrying on the back, moving, transporting, throwing, kicking, and avoiding objects. When the central brain 3015 performs motion control of the robot, the IPU 3011 and the MoPU 3012 can be mounted at the positions of the right and left eyes of the robot. That is, the IPU 3011 and the MoPU 3012 for the right eye can be mounted on the right eye, and the IPU 3011 and the MoPU 3012 for the left eye can be mounted on the left eye.

[0339] (Eighth Embodiment)

[0340] Next, the eighth embodiment of the present disclosure will be described by omitting or simplifying the repeated parts of the above sixth and seventh embodiments.

[0341] As an example, the information processing device 1E according to the eighth embodiment has a structure similar to that of the sixth embodiment Figure 23 shown.

[0342] The MoPU 3012 according to the eighth embodiment outputs the coordinate values of at least two diagonal points among the vertices of a polygon that encloses the contour of an object recognized from an image captured by another camera as point information. Similar to the sixth embodiment, the coordinate values are the x coordinate value and the y coordinate value of the object in the above three-dimensional orthogonal coordinate system.

[0343] Figure 25 is an explanatory diagram showing an example of the point information output by the MoPU 3012. In Figure 25 , for each of the 4 objects included in the image captured by another camera, the MoPU 3012 shows the bounding boxes 3021, 3022, 3023, and 2024 formed by enclosing the contour of the object with a quadrilateral. And, Figure 25Illustrates the manner in which MoPU 3012 outputs, as point information, the coordinate values of two diagonal points among the vertices of the quadrilateral bounding boxes 3021, 3022, 3023, and 2024 that enclose the contour of an object. In this way, MoPU 3012 can capture the object not as a point but as an object with a certain size.

[0344] In addition, when capturing the object as an object with a certain size, MoPU 3012 may not output, as point information, the coordinate values of two diagonal points among the vertices of the polygon that encloses the contour of the object identified from the image captured by another camera, but may output the coordinate values of multiple vertices of the polygon that encloses the contour of the object as point information. For example, if Figure 25 is taken as an example, then MoPU 3012 outputs, as point information, the coordinate values of all 4 vertices of the bounding boxes 3021, 3022, 3023, and 2024 that enclose the contour of the object with a quadrilateral.

[0345] (Ninth Embodiment)

[0346] Next, the ninth embodiment of the present disclosure will be described by omitting or simplifying the repetitive parts of the above sixth to eighth embodiments.

[0347] As an example, the information processing apparatus 1E according to the ninth embodiment has a structure similar to that of the sixth embodiment Figure 23 shown.

[0348] The vehicle 3100 equipped with the information processing apparatus 1E according to the ninth embodiment includes a sensor composed of at least one of a radar, a LiDAR, a high-pixel / long-focus / ultra-wide-angle / 360-degree / high-performance camera, a vision sensor, a sound sensor, an ultrasonic sensor, a vibration sensor, an infrared sensor, a ultraviolet sensor, a radio wave sensor, a temperature sensor, and a humidity sensor. Examples of the sensor information obtained by the information processing apparatus 1E from the sensor include the movement of the center of gravity of the body weight, the detection of the road material, the detection of the external air temperature, the detection of the external air humidity, the detection of the up / down / horizontal / oblique inclination angles of the slope, the freezing mode of the road, the detection of the moisture content, the material, wear condition, and air pressure of each tire, the road width, the presence or absence of overtaking prohibition, the vehicle types of oncoming vehicles and vehicles in front and behind, the cruising states of these vehicles, or the surrounding conditions (birds, animals, footballs, accident vehicles, earthquakes, fires, winds, typhoons, heavy rains, light rains, snowstorms, fogs, etc.). The sensor is an example of the "detection unit", and the sensor information is an example of the "detection information".

[0349] The central brain 3015 according to the ninth embodiment calculates control variables for controlling the autonomous driving of the vehicle 3100 based on sensor information detected by sensors. The central brain 3015 acquires sensor information every one billionth of a second. Specifically, the central brain 3015 calculates control variables for the wheel speed, tilt, and suspension supporting the wheels of each of the four wheels of the vehicle 3100. It should be noted that the tilt of the wheel includes both the tilt of the wheel with respect to the axis horizontal to the road and the tilt of the wheel with respect to the axis perpendicular to the road. In this case, the central brain 3015 calculates a total of 16 control variables, and the total 16 control variables are used to control the wheel speed of each of the four wheels, the tilt of each of the four wheels with respect to the axis horizontal to the road, the tilt of each of the four wheels with respect to the axis perpendicular to the road, and the suspensions respectively supporting the four wheels.

[0350] Moreover, the central brain 3015 controls the autonomous driving of the vehicle 3100 based on the control variables calculated above, the point information output from the MoPU 3012, and the label information output from the IPU 3011. Specifically, the central brain 3015 controls the in-wheel motors respectively mounted on the four wheels based on the above 16 control variables, thereby controlling the wheel speed, tilt, and suspensions respectively supporting the four wheels of the vehicle 3100 to perform autonomous driving. In addition, the central brain 3015 identifies the positions and movements of the objects that are recognized as what around the vehicle 3100 based on the point information and the label information, and controls the autonomous driving of the vehicle 3100 based on the recognized information to avoid, for example, colliding with the objects. In this way, the central brain 3015 controls the autonomous driving of the vehicle 3100, so that, for example, when the vehicle 3100 is driving on a mountain road, it can perform the optimal steering matching the mountain road, and when the vehicle 3100 is parked in a parking lot, it can drive at the optimal angle matching the parking lot.

[0351] Here, the central brain 3015 can be a unit capable of using machine learning, and more specifically, using deep learning, to infer control variables based on the above sensor information and information that can be obtained via a network from a server (not shown) and the like. In other words, the central brain 3015 can be composed of AI.

[0352] The central brain 3015 is the computing power for the above-mentioned sensor information and long-tail event AI data per billionth of a second. Using the computing power used at implementation level 6 (hereinafter also referred to as "the computing power of level 6"), by performing multivariate analysis based on the integral method shown in the following formula (1) (for example, refer to formula (2)), the control variables can be obtained. More specifically, while obtaining the integral value of the Delta value of various ultra-high resolutions at the computing power of level 6, the control variables can be obtained at the edge level and in real time, and the results (i.e., the control variables) that will occur in the next billionth of a second can be obtained with the highest probability value. To achieve this, the integral value obtained by time-integrating the Delta value (for example, the change value in a tiny time) of a function that can determine various variables such as air resistance, road resistance, road elements (such as garbage), and slip coefficient (for example, the above-mentioned sensor information and information that can be obtained via the network) (in other words, a function representing the changes of various variables) is input to the deep learning model of the central brain 3015 (for example, a learned model obtained by performing deep learning on a neural network). The deep learning model of the central brain 3015 outputs the control variables corresponding to the input integral value (for example, the control variable with the highest confidence level (i.e., evaluation value)). The output of the control variables is carried out in units of billionths of a second.

[0353] [Calculation formula 1]

[0354]

[0355] [Calculation formula 2]

[0356] V n =DL(f(A,B,C,D,…,N)(dA n / dt)) (2)

[0357] It should be noted that, as an example, in formula (1), "f(A)" is a formula that simplifies and expresses a function representing the changes of various variables such as air resistance, road resistance, road elements (such as garbage), and slip coefficient. In addition, as an example, formula (1) is a formula representing the time integral v of "f(A)" from time a to time b. DL in formula (2) represents deep learning (for example, a deep learning model optimized by performing deep learning on a neural network), dAn / dt represents the Delta value of f(A, B, C,..., N), A, B, C,..., N represent various variables such as air resistance, road resistance, road elements (such as garbage), and slip coefficient, f(A, B, C,..., N) represents a function used to represent the changes of A, B, C,..., N, and Vn represents the value output from the deep learning model optimized by performing deep learning on a neural network (control variable).

[0358] It should be noted that an example of the method of inputting the integral value obtained by time-integrating the Delta value of the function into the deep learning model of the central brain 3015 is listed here, but this is just one example. It can also be set that, for example, the deep learning model of the central brain 3015 infers the integral value obtained by time-integrating the Delta value of the function representing the changes of various variables such as air resistance, road resistance, road elements, and slip coefficient (for example, the result occurring in the next billionth of a second), and the central brain 3015 obtains the integral value with the highest confidence level (i.e., evaluation value) per billionth of a second as the inference result.

[0359] In addition, an example of the method of inputting the integral value into the deep learning model or outputting the integral value from the deep learning model is listed here, but this is just one example. Even without using the integral value, the technology of the present disclosure is still valid. For example, it can also be that at least one control variable is inferred by using the deep learning model optimized by performing deep learning on the neural network using the following supervised data. The supervised data takes the values equivalent to A, B, C,..., N as example data and the values equivalent to at least one control variable (for example, the result occurring in the next billionth of a second) as correct answer data.

[0360] The control variable obtained by the central brain 3015 can be further refined by increasing the number of times of deep learning. For example, a large amount of data such as the rotation of tires or electric motors, steering angles, road materials, weather, garbage, or the influence during quadratic curve deceleration, slip, and methods of steering or speed control for losing balance or regaining balance, or long-tail event AI data can be used to calculate a more accurate control variable.

[0361] (Tenth Embodiment)

[0362] Next, the tenth embodiment of the present disclosure will be described by omitting or simplifying the repeated parts of the above sixth to ninth embodiments.

[0363] Figure 26 It is a third block diagram showing an example of the structure of the information processing apparatus 1E according to the tenth embodiment. It should be noted that Figure 26 Only a part of the structure of the information processing apparatus 1E is shown.

[0364] As Figure 26As shown, through the MoPU 3012, the visible light image and the infrared image of the object captured by the camera 3030 are respectively input to the core 3017 at a frame rate of 100 frames per second or more. The camera 3030 is configured to include a visible light camera 3030A capable of capturing a visible light image of the object and an infrared camera 3030B capable of capturing an infrared image of the object. And, the core 3017 outputs point information to the central brain 3015 based on at least one of the input visible light image and infrared image.

[0365] Here, when the object can be recognized from the visible light image of the object captured by the visible light camera 3030A, the core 3017 outputs point information based on the visible light image. On the other hand, when the object cannot be captured from the visible light image due to a specified factor, the core 3017 outputs point information based on the infrared image of the object captured by the infrared camera 3030B. For example, consider a case where the core 3017 cannot capture the object from the visible light image due to the influence of darkness as a specified factor. In this case, the core 3017 uses the infrared camera 3030B to detect the heat of the object, and outputs the point information of the object based on the infrared image as the detection result. It should be noted that, without limitation, the core 3017 may also output point information based on the visible light image and the infrared image.

[0366] In addition, the MoPU 3012 synchronizes the timing of capturing the visible light image by the visible light camera 3030A with the timing of capturing the infrared image by the infrared camera 3030B. Specifically, the MoPU 3012 outputs a control signal to the camera 3030 so that the visible light image and the infrared image are captured at the same timing. Thereby, the number of images per second captured by the visible light camera 3030A is synchronized with the number of images per second captured by the infrared camera 3030B (for example, 1920 frames per second).

[0367] (Eleventh Embodiment)

[0368] Next, the eleventh embodiment of the present disclosure will be described by omitting or simplifying the repeated parts of the above sixth to tenth embodiments.

[0369] Figure 27 It is a fourth block diagram showing an example of the structure of the information processing apparatus 1E according to the eleventh embodiment. It should be noted that Figure 27 only shows a part of the structure of the information processing apparatus 1E.

[0370] As Figure 27As shown, through the MoPU 3012, the image of the object captured by the camera 3030 and the radar signal based on the reflected wave of the electromagnetic wave irradiated onto the object by the radar 3032 and reflected from the object are respectively input to the core 3017 at a frame rate of 100 frames per second or more. And, based on the input image of the object and the radar signal, the core 3017 outputs point information to the central brain 3015. The core 3017 can derive the x coordinate value and y coordinate value of the object from the input image of the object. As described above, the radar 3032 can obtain the three-dimensional point cloud data of the object based on the radar signal and detect the coordinate of the z axis in the above-mentioned three-dimensional orthogonal coordinate system. In this case, the core 3017 uses the principle of a stereo camera to combine the x coordinate value and y coordinate value of the object captured by the camera 3030 at the same timing as the timing when the radar 3032 obtains the three-dimensional point cloud data of the object with the z coordinate value of the object represented by the three-dimensional point cloud data, and derives the coordinate values of the three axes (x axis, y axis, and z axis) of the object as point information. It should be noted that the image of the object input to the core 3017 may also include at least one of a visible light image and an infrared image.

[0371] In addition, the MoPU 3012 synchronizes the timing of capturing an image by the camera 3030 with the timing of the radar 3032 obtaining the three-dimensional point cloud data of the object based on the radar signal. Specifically, the MoPU 3012 captures images at the same timing and outputs control signals to the camera 3030 and the radar 3032 to obtain the three-dimensional point cloud data of the object. Thereby, the number of images per second captured by the camera 3030 is synchronized with the number of three-dimensional point cloud data per second obtained by the radar 3032 (for example, 1920 frames per second). In this way, the number of images per second captured by the camera 3030 and the number of three-dimensional point cloud data per second obtained by the radar 3032 are greater than the frame rate of the ultra-high resolution camera provided in the IPU 3011, that is, the number of images per second captured by the ultra-high resolution camera.

[0372] (The twelfth embodiment)

[0373] Next, the twelfth embodiment of the present disclosure will be described by omitting or simplifying the repeated parts of the above sixth to eleventh embodiments.

[0374] As an example, the information processing device 1E according to the twelfth embodiment has a structure similar to that of the sixth embodiment Figure 23 shown.

[0375] The central brain 3015 according to the twelfth embodiment associates the point information output from the MoPU 3012 at the same timing as the timing at which the IPU 3011 outputs the tag information with the tag information. In addition, when the central brain 3015 outputs new point information from the MoPU 3012 after associating the point information with the tag information, the central brain 3015 also associates the new point information with the tag information. The new point information is the point information of an object that is the same object as the object represented by the point information after being associated with the tag information, and is one or more pieces of point information during the period from when this association is made until the next tag information is output. In the twelfth embodiment, similar to the above-described embodiment, the frame rate of other cameras equipped with the MoPU 3012 is 100 frames per second or more (for example, 1920 frames per second), and the frame rate of the ultra-high-resolution camera equipped with the IPU 3011 is 10 frames per second.

[0376] Figure 28 It is an explanatory diagram showing an example of the association between point information and tag information. In the following description, the number of pieces of point information output per second from the MoPU 3012 is referred to as the "output rate of point information", and the number of pieces of tag information output per second from the IPU 3011 is referred to as the "output rate of tag information".

[0377] Figure 28 It shows the time series of the output rate of the point information P4 of the object B14. The output rate of the point information P4 related to the object B14 is 1920 frames per second. In addition, the point information P4 moves from right to left in the figure. The output rate of the tag information related to the object B14 is 10 frames per second, which is lower than the output rate of the point information P4.

[0378] First, at the time point of time t0, the tag information related to the object B14 has not been output from the IPU 3011. Therefore, at the time point of time t0, the central brain 3015 identifies the coordinate value (position information) of the object B14 based on the point information P4, but does not identify what the object B14 is.

[0379] Next, at the time point of time t1, the tag information related to the object B14 is output from the IPU 3011. Therefore, the central brain 3015 derives the tag information "PERSON" for the object B14 based on this tag information. And the central brain 3015 associates the tag information "PERSON" derived at time t1 with the coordinate value (position information) of the point information P4 output from the MoPU 3012 at time t1. Thereby, at the time point of time t1, the central brain 3015 identifies the coordinate value (position information) of the object B14 based on the point information P4, and also identifies what the object B14 is.

[0380] At Figure 28In this case, the timing at which the next label information related to the object B14 is output from the IPU 3011 is defined as time t2. Therefore, at the time point of time t2, the central brain 3015 derives the label information "PERSON" for the object B14 based on this label information output from the IPU 3011. And the central brain 3015 associates the label information "PERSON" derived at time t2 with the coordinate value (position information) of the point information P4 output from the MoPU 3012 at time t2.

[0381] Here, due to the frame rate difference between the other camera equipped with the MoPU 3012 and the ultra-high resolution camera equipped with the IPU 3011, during the period from time t1 to time t2, the central brain 3015 acquires the point information P4 related to the object B14, while not acquiring the label information. In this case, for the point information P4 acquired during the period from time t1 to time t2, the central brain 3015 associates the point information P4 with the label information "PERSON" associated with the upcoming time t1. Here, the point information P4 acquired by the central brain 3015 during the period from time t1 to time t2 is an example of "new point information". In Figure 28 the example shown, since a plurality of point information P4 is output from the MoPU 3012 during the period from time t1 to time t2, the central brain 3015 acquires a plurality of point information P4. Therefore, in Figure 28 the example shown, for any one of the plurality of point information P4 acquired during the period from time t1 to time t2, the central brain 3015 associates it with the label information "PERSON" associated with the upcoming time t1. It should be noted that, different from Figure 28 the example shown, in the case where one point information P4 is output from the MoPU 3012 during the period from time t1 to time t2, for this one point information P4, the central brain 3015 associates it with the label information "PERSON" associated with the upcoming time t1.

[0382] Here, with respect to the central brain 3015, since even during the period when the type of the object whose movement is being tracked is uncertain, the point information of this object is continuously output at a high frame rate, the risk of losing the coordinate value (position information) of this object is low. Therefore, in the case where the association between the point information and the label information is performed once, for the point information acquired during the period until the next label information is acquired, the central brain 3015 can presumptively assign the previous label information.

[0383] (The thirteenth embodiment)

[0384] Next, the redundant parts with the sixth to twelfth embodiments described above are omitted or simplified to describe the thirteenth embodiment of the present disclosure.

[0385] When the information processing device 1E that controls the autonomous driving of the vehicle 3100 performs high-level arithmetic processing, heat generation becomes an issue. Therefore, the thirteenth embodiment provides a vehicle 3100 having a cooling function for the information processing device 1E.

[0386] Figure 29 It is an explanatory diagram showing the schematic structure of the vehicle 3100. As Figure 29 shown, an information processing device 1E, a cooling execution device 3110, and a cooling unit 3120 are mounted on the vehicle 3100.

[0387] The information processing device 1E according to the thirteenth embodiment is a device that controls the autonomous driving of the vehicle 3100. As an example, the information processing device 1E has a structure similar to that of the sixth embodiment Figure 23 shown. The cooling execution device 3110 obtains the detection result of an object based on the information processing device 1E, and based on this detection result, causes the cooling unit 3120 to execute cooling of the information processing device 1E. The cooling unit 3120 cools the information processing device 1E using at least one cooling mechanism such as an air cooling mechanism, a water cooling mechanism, and a liquid nitrogen cooling mechanism. Hereinafter, the object to be cooled in the information processing device 1E is set as the central brain 3015 that controls the autonomous driving of the vehicle 3100 (specifically, the CPU 3014 that constitutes the central brain 3015) for description, but it is not limited thereto.

[0388] The information processing device 1E and the cooling execution device 3110 are communicably connected via a network (not shown). This network can be any one of a vehicle network, the Internet, a Local Area Network (LAN), and a mobile communication network. The mobile communication network can be based on any one of communication methods such as 5G (5th Generation), LTE (Long Term Evolution), 3G (3rd Generation), and 6G (6th Generation) and subsequent communication methods

[0389] Figure 30 It is a block diagram showing an example of the functional structure of the cooling execution device 3110. As Figure 30 shown, the cooling execution device 3110 has an acquisition unit 3112, an execution unit 3114, and a prediction unit 3116 as functional structures.

[0390] The acquisition unit 3112 acquires the detection result of the object based on the information processing device 1E. For example, as this detection result, the acquisition unit 3112 acquires the point information of the object output from the MoPU 3012.

[0391] The execution unit 3114 cools the central brain 3015 based on the detection result of the object acquired by the acquisition unit 3112. For example, when it is recognized based on the point information of the object output from the MoPU 3012 that the object is moving, the execution unit 3114 causes the cooling unit 3120 to start cooling the central brain 3015.

[0392] It should be noted that the execution unit 3114 is not limited to cooling the central brain 3015 based on the detection result of the object, and can also cool the central brain 3015 based on the prediction result of the working condition of the information processing device 1E.

[0393] Here, the prediction unit 3116 predicts the working condition of the information processing device 1E based on the detection result of the object acquired by the acquisition unit 3112. Specifically, it predicts the working condition of the central brain 3015. For example, the prediction unit 3116 acquires a learning model stored in a specified storage area. And the prediction unit 3116 predicts the working condition of the central brain 3015 by inputting the point information of the object output from the MoPU 3012 acquired by the acquisition unit 3112 into the learning model. Here, the learning model outputs the state and change amount of the computing power of the central brain 3015 as this working condition. In addition, the prediction unit 3116 can predict and output the temperature change of the information processing device 1E together with this working condition. Specifically, it is the temperature change of the central brain 3015. For example, the prediction unit 3116 predicts the temperature change of the central brain 3015 based on the number of point information of the object output from the MoPU 3012 acquired by the acquisition unit 3112. In this case, the prediction unit 3116 predicts that the greater the number of point information, the greater the temperature change, and the smaller the number of point information, the smaller the temperature change.

[0394] In the above case, the execution unit 3114 causes the cooling unit 3120 to start cooling the central brain 3015 based on the prediction result of the working condition of the central brain 3015 by the prediction unit 3116. For example, when the state and change amount of the computing power of the central brain 3015 predicted as this working condition exceed a predetermined threshold, the execution unit 3114 causes the cooling based on the cooling unit 3120 to start. In addition, when the temperature based on the temperature change of the central brain 3015 predicted as this working condition exceeds a specified threshold, the execution unit 3114 causes the cooling based on the cooling unit 3120 to start.

[0395] In addition, the execution unit 3114 can also use a cooling mechanism corresponding to the prediction result of the prediction unit 3116 regarding the temperature change of the central brain 3015 to perform cooling on the central brain 3015. For example, the execution unit 3114 can use more cooling mechanisms when the predicted temperature of the central brain 3015 is higher to cause the cooling unit 3120 to perform cooling. As a specific example, when it is predicted that the temperature of the central brain 3015 exceeds the first threshold, the execution unit 3114 uses one cooling mechanism to cause the cooling unit 3120 to perform cooling. On the other hand, when it is predicted that the temperature of the central brain 3015 exceeds the second threshold which is higher than the first threshold, the execution unit 3114 uses multiple cooling mechanisms to cause the cooling unit 3120 to perform cooling.

[0396] In addition, the execution unit 3114 can use a more powerful cooling mechanism when the predicted temperature of the central brain 3015 is higher to perform cooling on the central brain 3015. For example, when it is predicted that the temperature of the central brain 3015 exceeds the first threshold, the execution unit 3114 uses an air cooling mechanism to cause the cooling unit 3120 to perform cooling. In addition, when it is predicted that the temperature of the central brain 3015 exceeds the second threshold which is higher than the first threshold, the execution unit 3114 uses a water cooling mechanism to cause the cooling unit 3120 to perform cooling. Further, when it is predicted that the temperature of the central brain 3015 exceeds the third threshold which is higher than the second threshold, the execution unit 3114 uses a liquid nitrogen cooling mechanism to cause the cooling unit 3120 to perform cooling.

[0397] Further, the execution unit 3114 can determine the cooling mechanism used for cooling based on the number of point information of the object output from the MoPU 3012 acquired by the acquisition unit 3112. In this case, the execution unit 3114 can use a more powerful cooling mechanism when the number of point information is larger to perform cooling on the central brain 3015. For example, when the number of point information exceeds the first threshold, the execution unit 3114 uses an air cooling mechanism to cause the cooling unit 3120 to perform cooling. In addition, when the number of point information exceeds the second threshold which is higher than the first threshold, the execution unit 3114 uses a water cooling mechanism to cause the cooling unit 3120 to perform cooling. Further, when the number of point information exceeds the third threshold which is higher than the second threshold, the execution unit 3114 uses a liquid nitrogen cooling mechanism to cause the cooling unit 3120 to perform cooling.

[0398] As an opportunity for the central brain 3015 to operate, moving objects present on the lane are sometimes detected. For example, when the vehicle 3100 is in autonomous driving and a moving object on the lane is detected, the central brain 3015 sometimes performs arithmetic processing for controlling the vehicle 3100 on that object. As described above, heat generation during the high-level arithmetic processing performed by the central brain 3015 that controls the autonomous driving of the vehicle 3100 becomes an issue. Therefore, the cooling execution device 3110 according to the thirteenth embodiment predicts the heat dissipation of the central brain 3015 based on the detection result of the object by the information processing device 1E, and executes cooling of the central brain 3015 before or simultaneously with the start of heat dissipation. Thereby, it is possible to suppress the central brain 3015 from becoming high-temperature during the autonomous driving of the vehicle 3100, and high-level arithmetic processing by the central brain 3015 during that autonomous driving becomes possible.

[0399] (Fourteenth Embodiment)

[0400] Next, the fourteenth embodiment of the present disclosure will be described by omitting or simplifying the overlapping parts with the sixth to thirteenth embodiments described above.

[0401] The MoPU 3012 included in the information processing device 1E according to the fourteenth embodiment derives the z coordinate value of an object as point information from an image of the object captured by the camera 3030. Hereinafter, each mode of the information processing device 1E according to the fourteenth embodiment will be described in sequence.

[0402] The information processing device 1E according to the first mode has a structure similar to that of the seventh embodiment Figure 24 as shown.

[0403] In the above first mode, the MoPU 3012 derives the z coordinate value of an object as point information from images of the object captured by a plurality of cameras 3030, specifically, from the images of the object captured by the camera 3030L and the camera 3030R. As described above, when one MoPU 3012 is used, the x coordinate value and the y coordinate value of the object can be derived as point information. Here, when two MoPU 3012s are used, based on the principle of a stereo camera, the z coordinate value of the object can be derived as point information from the images of the object captured by the two cameras 3030. Therefore, this first mode uses the principle of a stereo camera and derives the z coordinate value of the object as point information from the images of the object captured by the camera 3030L of the MoPU 3012L and the camera 3030R of the MoPU 3012R, respectively.

[0404] The information processing device 1E according to the second mode has a structure similar to that of the seventh embodiment Figure 24 as shown.

[0405] In the second method described above, the MoPU 3012 derives the x - coordinate value, y - coordinate value, and z - coordinate value of an object as point information based on the image of the object captured by the camera 3030 and the radar signal of the reflected wave reflected from the object based on the electromagnetic wave irradiated onto the object by the radar 3032. As described above, the radar 3032 can obtain the three - dimensional point cloud data of the object based on the radar signal. That is, the radar 3032 can detect the coordinate of the z - axis in the above - mentioned three - dimensional orthogonal coordinate system. In this case, the MoPU 3012 uses the principle of a stereo camera to combine the x - coordinate value and y - coordinate value of the object captured by the camera 3030 at the same timing as the timing when the three - dimensional point cloud data of the object is obtained by the radar 3032, and the z - coordinate value of the object represented by the three - dimensional point cloud data, and derives the coordinate values of the three axes of the object as point information.

[0406] The information processing device 1E involved in the third method includes Figure 31 the structure shown. Figure 31 It is the fifth block diagram showing an example of the structure of the information processing device 1E. It should be noted that Figure 31 only a part of the structure of the information processing device 1E is shown.

[0407] In the third method described above, the MoPU 3012 derives the z - coordinate value of an object as point information based on the image of the object captured by the camera 3030 and the result of photographing the structured light irradiated onto the object by the irradiation device 130.

[0408] As Figure 31 shown, through the MoPU 3012, the image of the object captured by the camera 3030 and the distortion information indicating the distortion of the pattern of the structured light, which is the result of photographing the structured light irradiated onto the object by the camera 140, are respectively input to the core 3017 at a frame rate of 100 frames per second or more. And the core 3017 outputs the point information to the central brain 3015 based on the input object image and distortion information.

[0409] Here, as one of the methods for identifying the three - dimensional position or shape of an object, there is the structured light method. The structured light method irradiates structured light patterned into dots onto an object and obtains depth information based on the distortion of the pattern. The structured light method is disclosed, for example, in the reference (http: / / ex - press.jp / wp - content / uploads / 2018 / 10 / 018_teledyne_3rd.pdf).

[0410] Figure 31The irradiation device 130 shown irradiates structured light onto an object. In addition, the camera 140 captures the structured light irradiated onto the object by the irradiation device 130. Further, the camera 140 outputs distortion information based on the distortion of the pattern of the captured structured light to the core 3017.

[0411] Here, the MoPU 3012 synchronizes the timing of capturing an image by the camera 3030 with the timing of capturing the structured light by the camera 140. Specifically, the MoPU 3012 outputs a control signal to the camera 3030 and the camera 140 so that the camera 3030 and the camera 140 capture images at the same timing. Thereby, the number of images per second captured by the camera 3030 is synchronized with the number of images per second captured by the camera 140 (for example, 1920 frames / second). In this way, the number of images per second captured by the camera 3030 and the number of images per second captured by the camera 140 are more than the frame rate of the ultra-high-resolution camera included in the IPU 3011, that is, the number of images per second captured by the ultra-high-resolution camera.

[0412] Further, the core 3017 combines the x coordinate value and the y coordinate value of the object captured by the camera 3030 at the same timing as the timing of capturing the structured light by the camera 140 with the distortion information based on the distortion of the pattern of the structured light, and derives the z coordinate value of the object as point information.

[0413] The information processing device 1E according to the fourth mode includes Figure 32 the structure shown. Figure 32 This is the sixth block diagram showing an example of the structure of the information processing device 1E. Note that Figure 32 only a part of the structure of the information processing device 1E is shown.

[0414] Figure 32 The block diagram shown is Figure 23 a block diagram obtained by adding the Lidar sensor 3018 to the structure of the block diagram shown. The Lidar sensor 3018 is a sensor that acquires point cloud data including an object existing in a three-dimensional space and the road surface on which the vehicle 3100 travels. The information processing device 1E can derive the position information in the depth direction of the object, that is, the z coordinate value of the object, by using the point cloud data acquired by the Lidar sensor 3018. Note that it is assumed that the point cloud data obtained by the Lidar sensor 3018 is obtained at intervals longer than the x coordinate value and the y coordinate value of the object output from the MoPU 3012. In addition, the MoPU 3012 includes the camera 3030 similarly to the above-described mode of the fourteenth embodiment.

[0415] In the fourth method, the MoPU 3012 uses the principle of a stereo camera to combine the x - coordinate value and y - coordinate value of an object captured by the camera 3030 at the same timing as the timing of the point cloud data of the object obtained by the Lidar sensor 3018, and the z - coordinate value of the object represented by the point cloud data, and derives the coordinate values of the three axes of the object as point information.

[0416] Here, in the above - mentioned fourth method, the MoPU 3012 derives the z - coordinate value of the object at time t + 1 as point information based on the x - coordinate value, y - coordinate value, and z - coordinate value of the object at time t, and the x - coordinate value and y - coordinate value of the object at the next time point (for example, time t + 1) after time t. Time t is an example of the "first time point", and time t + 1 is an example of the "second time point". In the fourth method, the z - coordinate value of the object at time t + 1 is derived using shape information, that is, geometric shape. Hereinafter, the details will be described.

[0417] Figure 33 is a diagram schematically showing the coordinate detection of an object in a time series. In Figure 33 it, J represents the position of an object represented by a rectangle, and the position of the object moves in time series from J1 to J2. In Figure 33 the coordinate values of the object at time t when the object is at J1 are (x1, y1, z1), and the coordinate values of the object at time t + 1 when the object is at J2 are (x2, y2, z2).

[0418] First, time t will be described.

[0419] The MoPU 3012 derives the x - coordinate value and y - coordinate value of the object from the image of the object captured by the camera 3030. Then, the MoPU 3012 integrates the z - coordinate value of the object represented by the point cloud data obtained from the Lidar sensor 3018 and the above - mentioned x - coordinate value and y - coordinate value, and derives the three - dimensional coordinate values (x1, y1, z1) of the object at time t.

[0420] Next, time t + 1 will be described.

[0421] The MoPU 3012 derives the z - coordinate value of the object at time t + 1 based on the geometric shape of the space and the changes in the x - coordinate value and y - coordinate value of the object from time t to time t + 1. The geometric shape of the space includes the shape of the road surface and the shape of the vehicle 3100 obtained from the image captured by the ultra - high - resolution camera provided in the IPU 3011 and the point cloud data of the Lidar sensor 3018.

[0422] The geometry representing the shape of the road surface is pre-generated at the time point of time t. MoPU 3012 uses the geometry representing the shape of the vehicle 3100 in combination with the geometry representing the shape of the road surface, so that it can simulate the situation of the vehicle 3100 driving on the road surface and can estimate the movement amounts of the respective axes of the x-axis, y-axis, and z-axis.

[0423] Therefore, MoPU 3012 derives the x-coordinate value and y-coordinate value of the object at time t+1 from the image of the object captured by the camera 3030. MoPU 3012 simulates and calculates the movement amount of the z-axis when the object changes from the x-coordinate value and y-coordinate value (x1, y1) at time t to the x-coordinate value and y-coordinate value (x2, y2) at time t+1, so that it can derive the z-coordinate value of the object at time t+1. And, MoPU 3012 integrates the above x-coordinate value, y-coordinate value, and z-coordinate value to derive the three-dimensional coordinate value (x2, y2, z2) of the object at time t+1.

[0424] As Figure 33 shown, since the object moves in the depth direction together with the movement of the plane coordinates (i.e., the x-axis and y-axis), in order to accurately control the autonomous driving of the vehicle 3100, it is also necessary to detect the movement in the z-axis direction. Here, MoPU 3012 sometimes cannot obtain the z-coordinate value of the object as fast as the x-coordinate value and y-coordinate value of the object that can be derived from the point cloud data of the Lidar sensor 3018. Therefore, in the above fourth method, it is assumed that MoPU 3012 derives the z-coordinate value of the object at time t+1 based on the x-coordinate value, y-coordinate value, and z-coordinate value of the object at time t and the x-coordinate value and y-coordinate value of the object at time t+1. Thus, according to the information processing device 1E related to the above fourth method, through MoPU 3012, two-dimensional motion detection and three-dimensional motion detection based on high-speed frame shooting can be achieved with high performance and low-usage data.

[0425] In addition, in the above description, an example is given where the MoPU 3012 derives the z - coordinate value of an object as point information from the image of the object captured by the camera 3030. However, the disclosed technology is not limited to this method. For example, the central brain 3015 can derive the z - coordinate value of the object as point information instead of the MoPU 3012. In this case, the central brain 3015 derives the z - coordinate value of the object as point information by performing the processing that was executed by the MoPU 3012 in the above description on the image of the object captured by the camera 3030. As an example, the central brain 3015 derives the z - coordinate value of the object as point information from the images of the object captured by multiple cameras 3030, specifically, the camera 3030L and the camera 3030R. In this case, the central brain 3015 uses the principle of a stereo camera and derives the z - coordinate value of the object as point information based on the images of the object captured by the camera 3030L of the MoPU 3012L and the camera 3030R of the MoPU 3012R respectively.

[0426] (The Fifteenth Embodiment)

[0427] Next, the fifteenth embodiment of the present disclosure will be described by omitting or simplifying the parts that overlap with the sixth to fourteenth embodiments described above.

[0428] Figure 34 FIG. 7 is a seventh block diagram showing an example of the structure of the information processing apparatus 1E according to the fifteenth embodiment. It should be noted that Figure 34 only a part of the structure of the information processing apparatus 1E is shown.

[0429] As Figure 34 shown, through the MoPU 3012, the image of the object captured by the event camera 3030C (hereinafter sometimes also referred to as an "event image") is input to the core 3017. Moreover, the core 3017 outputs point information to the central brain 3015 based on the input event image. It should be noted that event cameras are disclosed, for example, in the reference (https: / / dendenblog.xyz / event - based - camera / ).

[0430] FIG. 35 is an explanatory diagram for explaining the image (event image) of the object captured by the event camera 3030C. Figure 35A FIG. 36 is a diagram showing the object that is the subject of photography by the event camera 3030C. Figure 35B FIG. 37 is a diagram showing an example of the event image. Figure 35CThis is a diagram showing an example in which the center of gravity of the difference portion between the image captured at the current moment and the image captured at the previous moment, which is represented by an event image, is calculated as point information. In the event image, the difference portion between the image captured at the current moment and the image captured at the previous moment is extracted as points. Therefore, in the case of using the event camera 3030C, for example, as Figure 35B shown, Figure 35A the points of the moving parts in the person area as shown are extracted.

[0431] In contrast, as Figure 35C shown, after the core 3017 extracts the person as an object, the coordinates of the feature points representing the person area (for example, only one point) are extracted. Thereby, the amount of data transmitted to the central brain 3015 and the memory 3016 can be suppressed. The event image can extract the person as an object at an arbitrary frame rate. Therefore, in the case of the event camera 3030C, in the above-described embodiment, it is also possible to extract at a frame rate higher than the maximum frame rate (for example: 1920 frames / second) of the camera 3030 mounted on the MoPU 3012, and the point information of the object can be captured with high precision.

[0432] It should be noted that, similar to the above-described embodiment, for the information processing apparatus 1E according to the fifteenth embodiment, in addition to the event camera 3030C, the MoPU 3012 may further include a visible light camera 3030A. In this case, in the MoPU 3012, the visible light image and the event image of the object captured by the visible light camera 3030A are respectively input to the core 3017. And the core 3017 outputs the point information to the central brain 3015 based on at least one of the input visible light image and event image.

[0433] For example, when the core 3017 is able to identify an object from the visible light image of the object captured by the visible light camera 3030A, it outputs point information based on the visible light image. On the other hand, when the object cannot be captured from the visible light image due to a specified factor, the core 3017 outputs point information based on the event image. The specified factors include at least one of the cases where the moving speed of the object is equal to or higher than a specified value and the change in the amount of ambient light per unit time is equal to or higher than a specified value. For example, when the object moves at a high speed and cannot be captured from the visible light image, the core 3017 identifies the object based on the event image and outputs the x coordinate value and y coordinate value of the object as point information. In addition, when the object cannot be captured from the visible light image due to a sharp change in the amount of ambient light such as backlighting, the core 3017 identifies the object based on the event image and outputs the x coordinate value and y coordinate value of the object as point information. With this structure, according to the information processing device 1E, the camera 3030 for capturing an object can be used in different cases according to specified factors.

[0434] (Sixteenth Embodiment)

[0435] Next, the sixteenth embodiment of the present disclosure will be described by omitting or simplifying the overlapping parts with the sixth to fifteenth embodiments described above.

[0436] As an example, the information processing device 1E according to the sixteenth embodiment has a structure similar to that of the ninth embodiment Figure 23 shown.

[0437] In the ninth embodiment, the following case was described: whenever the central brain 3015 acquires sensor information, multivariate analysis (for example, refer to Equation (2)) based on the integration method shown in Equation (1) is performed to infer the control variable. That is, if the central brain 3015 acquires sensor information every one billionth of a second, it performs multivariate analysis using the sensor information acquired each time, thereby inferring the control variable in units of one tenth of a second. In contrast, in the sixteenth embodiment, the central brain 3015 calculates the control variable based on the correlation between the variable and the control variable. A detailed description thereof will be given below.

[0438] The central brain 3015 is similar to the ninth embodiment and acquires the sensor information detected by the sensors. In addition, the central brain 3015 is similar to the ninth embodiment and acquires the point information output from the MoPU 3012 and the label information output from the IPU 3011. Here, the "central brain 3015" is an example of a "processor", the sensor information is an example of "detection information", the MoPU 3012 and the IPU 3011 are examples of "other different processors", and the label information is an example of "identification information". That is, the processor functions as an acquisition unit that acquires the detection information obtained by detecting the conditions around the moving body, the point information obtained by capturing the photographed object as points, and the identification information obtained by identifying the object. At this time, the processor can acquire the point information and the identification information from other different processors respectively.

[0439] Next, the central brain 3015 is similar to the ninth embodiment and calculates various variables (such as A, B, C,..., N in Equation (2)), such as air resistance, road resistance, road elements (such as garbage), and slip coefficient, based on the acquired sensor information. That is, the processor calculates the variables related to the conditions around the moving body based on the acquired detection information. And the central brain 3015 infers the control variable Vn through, for example, Equation (2). That is, the processor can infer the control variable through multivariate analysis based on the integration method using deep learning.

[0440] So far, it is similar to the ninth embodiment, but in the sixteenth embodiment, the central brain 3015 is configured to be able to calculate the control variable Vn' based on the correlation between the variable and the control variable Vn. It should be noted that "Vn'" represents the control variable calculated based on the correlation, but when there is no need to specifically distinguish it from the inferred control variable "Vn", it is only represented as the control variable Vn. For example, when the coefficient of the variable with respect to the control variable Vn is predetermined, the central brain 3015 can use this variable to calculate the control variable Vn'. That is, the processor can use the predetermined coefficient of the variable with respect to the control variable Vn to calculate the control variable Vn'.

[0441] As an example, as coefficients of variables A, B, C, ..., N with respect to the control variable Vn, regression coefficients WA, WB, WC, WD, ..., WN are respectively determined in advance. In this case, if the differences of variable A, variable B, variable C, variable D, ...... variable N are respectively set as ΔA, ΔB, ΔC, ΔD, …, ΔN, the central brain 3015 can calculate the control variable Vn' through the following formula. That is, the central brain 3015 can calculate the control variable Vn' by performing regression analysis based on the inferred control variable Vn. In this way, the processor functions as a calculation unit that calculates the control variable Vn' based on the correlation between the variables related to the surrounding conditions of the moving body calculated from the detection information and the control variable Vn inferred using the variables.

[0442] Vn’ = Vn + WA×ΔA + WB×ΔB + WC×ΔC + WD×ΔD + … + WN×ΔN

[0443] It should be noted that in the above description, as an example, the case where the regression coefficients have been determined in advance is shown as the coefficients of the variables with respect to the control variable Vn, but it is not limited to this. For example, the regression coefficient WA can be obtained through the following formula using the correlation coefficient rA of variable A with respect to the control variable Vn. The same applies to other variables B, C...., N. Therefore, such correlation coefficients can be determined in advance as the coefficients of the variables with respect to the control variable Vn. In addition, the determination coefficient can be obtained by squaring the correlation coefficient. Therefore, such determination coefficients (also called contribution rates) can also be determined in advance as the coefficients of the variables with respect to the control variable Vn.

[0444] WA = rA×(standard deviation of Vn / standard deviation of A)

[0445] In this way, as the coefficients of the variables with respect to the control variable Vn, any coefficients that can calculate the control variable Vn' based on the differences of the variables can be determined in advance. It should be noted that such coefficients can be manually set via user input, or can be automatically set according to simulation results or learning results after machine learning of the relationship between the control variable Vn and the variables.

[0446] Thus, even if the central brain 3015 does not perform the inference process of the control variable Vn every time sensor information is acquired, it can maintain the derivation frequency of the control variable Vn by performing the calculation process of the control variable Vn' based on the correlation. That is, when the central brain 3015 acquires sensor information every one billionth of a second, for example, the inference process of the control variable Vn is performed in units of one hundred billionths of a second, and for the period when the inference process is not performed, it can be supplemented by the calculation process of the control variable Vn' based on the correlation.

[0447] For example, the central brain 3015 may perform the inference process of the control variable Vn in a second cycle that is longer than the first cycle for acquiring sensor information. In this case, preferably, the second cycle is shorter than the imaging cycle of the ultra-high-resolution camera included in the IPU 3011, and more preferably, the second cycle is shorter than the imaging cycle of other cameras included in the MPU. In addition, the central brain 3015 may perform the calculation process of the correlation-based control variable Vn' in a third cycle that is shorter than the second cycle for performing the inference process of the control variable Vn. In this case, if the third cycle is the same as the first cycle, it is preferable because the control variable Vn can be derived at the same frequency as the acquisition frequency of the sensor information.

[0448] And, the central brain 3015 controls the autonomous driving of the vehicle 3100, for example, based on the control variable Vn, the point information, and the label information thus derived. In this way, the processor functions as a control unit that controls the autonomous driving of the moving body based on the control variable Vn, the point information, and the identification information.

[0449] As described above, when the information processing device 1E according to the sixteenth embodiment controls the autonomous driving of the moving body based on the control variable, the point information, and the identification information, it calculates the control variable according to the correlation between the variable and the control variable. Thus, according to the information processing device 1E according to the sixteenth embodiment, even if the inference process of the control variable is not performed every time, the control variable can be calculated according to the correlation between the variable and the control variable, so that the execution frequency of the inference process of the control variable can be reduced, and thus the amount of computation and the computation time required for the control of the autonomous driving can be reduced.

[0450] (Seventeenth Embodiment)

[0451] In addition, in the case of an autonomous driving vehicle, multiple images of the surroundings of the vehicle captured by a camera are used to control the autonomous driving. Therefore, in the existing autonomous driving, there are the following problems: the amount of data acquired by the processor that controls the autonomous driving increases, and the amount of computation required for the control of the autonomous driving increases. Therefore, hereinafter, as the seventeenth embodiment of the present disclosure, an information processing device and the like that can reduce the amount of computation required for the control of the autonomous driving will be described.

[0452] The overlapping parts with the sixth to sixteenth embodiments described above are omitted or simplified to describe the seventeenth embodiment of the present disclosure.

[0453] As an example, the information processing device 1E according to the seventeenth embodiment is configured to include a structure similar to that of the ninth embodiment Figure 23 as shown.

[0454] During the driving of vehicle 3100, sometimes obstacles approach vehicle 3100. In such a case, preferably, vehicle 3100 changes its behavior and drives to avoid contact or collision with the obstacles. As an example of an obstacle, other vehicles other than the own vehicle in motion, walls, guardrails, curbstones, and other installations can be listed. In the following description, a case where another vehicle approaching vehicle 3100 is taken as an example of an obstacle will be described.

[0455] Similar to the ninth embodiment, the central brain 3015 controls the in-wheel motors respectively mounted on the four wheels based on the above control variables that change every moment and are calculated according to the driving state on the driving path, thereby controlling the wheel speeds, tilts of the four wheels respectively, and the suspensions supporting the four wheels, and performs autonomous driving. In addition, similar to the ninth embodiment, the central brain 3015 obtains sensor information from sensors that detect the conditions around vehicle 3100 (for example, the behavior of other vehicles).

[0456] The central brain 3015 at least has the following function: Based on the acquired sensor information, predict the collision of the obstacle with vehicle 3100 including contact, and drive to avoid the obstacle.

[0457] The central brain 3015 can obtain sensor information from sensors that detect the conditions around vehicle 3100 every one-billionth of a second. That is, the processor can obtain detection information from a detection unit that detects the conditions around the moving body at a second period shorter than the first period for photographing the surroundings of the moving body.

[0458] The central brain 3015 according to the present embodiment divides the period during which it is possible to predict the collision of the obstacle with vehicle 3100 based on the sensor information into multiple periods. The period during which collision prediction can be performed is preset to, for example, a value such as 0.3 seconds. The period during which collision prediction can be performed can be dynamically set according to the vehicle speed and the like. For example, the central brain 3015 divides the period during which it is possible to predict the collision of the obstacle with vehicle 3100 into three periods: an initial period, a middle period, and a late period. It should be noted that the central brain 3015 can divide the period during which it is possible to predict the collision of the obstacle with vehicle 3100 into two periods, or can divide it into four or more periods.

[0459] Moreover, the central brain 3015 of the present embodiment calculates control variables for controlling the autonomous driving of vehicle 3100 using a plurality of input parameters (for example, the inputs in the above formulas (1) and (2)) based on the sensor information in each of the multiple periods in the time series of the multiple periods. When the central brain 3015 calculates the control variables in the second and subsequent periods among the multiple periods, it can calculate the control variables with fewer input parameters compared to the previous period.

[0460] For example, at the beginning of the three periods, the central brain 3015 of the present embodiment predicts the collision direction by performing comprehensive collision prediction based on all sensor information. Next, in the middle of the three periods, the central brain 3015 predicts the collision location between the vehicle 3100 and the obstacle. And in the later stage of the three periods, the central brain 3015 determines the collision location and determines the avoidance countermeasure.

[0461] As a specific example, at the beginning, the central brain 3015 predicts a collision with an obstacle from the left front of the vehicle 3100 based on all sensor information. Next, based on the sensor information detecting the situation on the left front of the vehicle 3100, the central brain 3015 predicts the left fender as the collision location between the vehicle 3100 and the obstacle. In this way, in the middle period, the central brain 3015 can reduce the input parameters used when calculating the control variable based on the prediction result in the initial stage.

[0462] And the central brain 3015 determines the collision location and determines the avoidance countermeasure based on the sensor information detecting the situation around the left fender. The avoidance countermeasure includes avoiding the collision. In addition, in the case where the collision is inevitable, the avoidance countermeasure includes colliding in a way that reduces the degree of damage to the vehicle 3100. In this way, in the later stage, the central brain 3015 can reduce the input parameters used when calculating the control variable based on the prediction results in the initial and middle stages.

[0463] As described above, according to the seventeenth embodiment, the computational amount required for the control of autonomous driving can be reduced.

[0464] (Eighteenth Embodiment)

[0465] In addition, when providing autonomous driving, in order to avoid collisions with obstacles, a function for predicting collisions is required. However, it is desirable to expand the prediction target from collisions with obstacles to the approach of obstacles. Therefore, hereinafter, as the eighteenth embodiment of the present disclosure, an information processing device and the like that can predict the approach of an obstacle will be described.

[0466] Next, the overlapping parts with the sixth to seventeenth embodiments described above will be omitted or simplified, and the eighteenth embodiment related to the present embodiment will be described.

[0467] As an example, the information processing device 1E according to the eighteenth embodiment has a structure similar to that of the Figure 23 shown in the ninth embodiment.

[0468] During the driving of a vehicle, sometimes an obstacle approaches the vehicle. In such a case, preferably, the vehicle travels by changing its behavior to avoid contact or collision with the obstacle. As an example of the obstacle, other vehicles, walls, guardrails, curbstones, and other installations other than the own vehicle during driving can be listed. In the following description, a case will be described in which another vehicle approaching the vehicle 3100 is taken as an example of the obstacle. It should be noted that the obstacle is an example of the object.

[0469] Figure 36 FIG. schematically shows a state in which the vehicle 3100A traveling on a two-lane road in both directions is taken as the own vehicle 3100A and other vehicles 3100B, 3100C, and 3100D are traveling around the own vehicle 3100A. In the example of this figure, it is the following state: The other vehicle 3100B is traveling behind the own vehicle 3100A, the other vehicle 3100D is traveling ahead on the oncoming lane, and the other vehicle 3100C is traveling behind.

[0470] The central brain 3015 of the own vehicle 3100A controls the in-wheel motors respectively mounted on the four wheels based on the above control variables that change every moment and are calculated according to the traveling state on the traveling path, thereby controlling the wheel speeds, tilts of the four wheels respectively, and the suspensions supporting the four wheels, and performs autonomous driving. In addition, the central brain 15 obtains sensor information from sensors that detect the conditions around the own vehicle 3100A (for example, the behaviors of other vehicles 3100B, 3100C, and 3100D). Here, the "central brain 3015" is an example of the "processor", the "sensor" is an example of the "detection unit", and the "sensor information" is an example of the "detection information".

[0471] The central brain 3015 at least has the following functions: According to the acquired sensor information, it predicts the collision of the obstacle as the object with the own vehicle including contact, so as to travel while avoiding the obstacle. As Figure 36 shown, when the central brain 3015 predicts that the other vehicle 3100D intrudes in front of the own vehicle 3100A according to the sensor information, it calculates the control variable representing the behavior of the own vehicle 3100A that can at least avoid contact with the other vehicle 3100D.

[0472] For example, in Figure 36In the example shown, the vehicle 3100A collides with another vehicle 3100D on the path Ax1 while traveling in the current driving state. Therefore, the central brain 3015 calculates paths such as Ax2 and Ax3 to avoid a collision with the other vehicle 3100D, selects any one of them, and calculates the control variables. Regarding the selection of the path, as long as a path with a load caused by the control variable in the vehicle 3100A being lower than a predetermined specified value (for example, the minimum value) is selected.

[0473] So far, the function of the central brain 3015 to predict collisions with obstacles has been described, but the central brain 3015 also has the function of predicting the approach of obstacles. The central brain 3015 can obtain sensor information from sensors that detect the situation around the vehicle 3100A every one-billionth of a second. That is, the processor can obtain detection information from a detection unit that detects the situation around the moving body at a second period shorter than the first period of photographing the surroundings of the moving body.

[0474] Moreover, the central brain 3015 can calculate the distance between the vehicle 3100A and another vehicle 3100C based on the detection information, and predict the approach of the other vehicle 3100C based on the calculated distance and a predetermined margin.

[0475] For example, as a path to avoid a collision with the other vehicle 3100D, the path Ax2 is selected. When the vehicle 3100A travels along the path Ax2, the vehicle 3100A may approach another vehicle 3100C traveling behind the other vehicle 3100D. Therefore, the central brain 3015 can calculate the distance between the vehicle 3100A and the other vehicle 3100C based on the sensor information, and compare the calculated distance with the margin to predict the approach of the other vehicle 3100C.

[0476] It should be noted that the margin can be predetermined, or a value obtained in advance through experiments, or a value obtained in advance through user input. In addition, the sense of fear of the approach of an obstacle is not constant but varies from person to person. Therefore, the margin can also be determined in advance for each occupant of the vehicle 3100A. In particular, children and the elderly may have a strong sense of fear of the approach of an obstacle. Therefore, the margin can also be determined in advance according to the age of the occupants of the vehicle 3100A.

[0477] When setting the margin, for example, facial authentication can be performed on the camera image captured inside the vehicle 3100A to determine the occupants of the vehicle 3100A, and the margin determined for each occupant can be set. Alternatively, the camera image can be analyzed to estimate the age of the occupants of the vehicle 3100A, and the pre-determined margin can be set according to the age. It should be noted that when multiple passengers are in the vehicle 3100A, the central brain 3015 sets the margin that is determined to be the maximum value among the multiple margins, that is, the margin can be set according to the passenger with the strongest sense of fear.

[0478] In addition, the sense of fear of approaching becomes stronger according to the moving speed. Therefore, the central brain 3015 can change the set margin according to the moving speed of the vehicle 3100A or other vehicle 3100C. Preferably, it can be changed according to the sum of the moving speeds of the vehicle 3100A and other vehicle 3100C. In particular, the braking distance traveled from the start of brake operation until the vehicle 3100 stops is usually proportional to the square of the speed of the vehicle 3100. Therefore, the central brain 3015 can also change the set margin according to the braking distance of the vehicle 3100A or other vehicle 3100C. Preferably, it can be changed according to the sum of the braking distances of the vehicle 3100A and other vehicle 3100C. It should be noted that such a braking distance not only varies depending on the speed of the vehicle 3100, but also varies depending on the state of the road surface, the weight of the vehicle 3100, the number of passengers, the cargo, the conditions of the tires, etc. Therefore, when the central brain 3015 can obtain this information, it can also consider this information to change the margin.

[0479] Here, a margin M is set, and this margin M is changed to a margin M'. In this case, the central brain 3015 can compare the distance Dac between the vehicle 3100A and other vehicle 3100C with the margin M'. And when Dac > M', the central brain 3015 can predict that the other vehicle 3100C is not approaching. On the other hand, when Dac ≤ M', the central brain 3015 can predict that the other vehicle 3100C is approaching.

[0480] Moreover, the central brain 3015 can also control the autonomous driving of the vehicle 3100A based on the predicted result. For example, when it is predicted that the other vehicle 3100C is approaching, the central brain 3015 can also correct the path Ax2 to reduce the overrun into the oncoming lane. In addition, when it is predicted that the other vehicle 3100C is approaching, the central brain 3015 can also increase the vehicle speed of the vehicle 3100A so that it can return from the oncoming lane in advance.

[0481] As described above, the information processing device 1E according to the eighteenth embodiment acquires detection information from a detection unit that detects the surrounding conditions of the moving body at a second cycle shorter than the first cycle of photographing the surroundings of the moving body, calculates the distance between the moving body and the obstacle based on the detection information, and predicts the approach of the obstacle based on the calculated distance and the predetermined margin. Thus, according to the information processing device 1E according to the eighteenth embodiment, a function of predicting the approach of an obstacle can be provided. At this time, the margin can also be predetermined according to the occupant or age of the moving body. Thus, according to the information processing device 1E according to the eighteenth embodiment, the sense of security of each occupant can be improved. In addition, the information processing device 1E according to the eighteenth embodiment can also change the margin according to the moving speed or braking distance of the moving body or the obstacle. Thus, according to the information processing device 1E according to the eighteenth embodiment, the approach of the obstacle can be predicted in conjunction with the fear of the occupant. Furthermore, according to the information processing device 1E according to the eighteenth embodiment, the automatic driving of the moving body can be controlled based on the predicted result. Thus, according to the information processing device 1E according to the eighteenth embodiment, when the approach of the obstacle is predicted, the moving body can be automatically driven more safely.

[0482] (Nineteenth Embodiment)

[0483] In addition, when providing automatic driving, an unavoidable collision with an obstacle may occur. In such a case, it is desirable to control the portion that causes the obstacle to collide. Therefore, as the eighteenth embodiment of the present disclosure, an information processing device or the like that can control the portion that causes the obstacle to collide is described below.

[0484] The nineteenth embodiment of the present disclosure will be described by omitting or simplifying the overlapping parts with the sixth to eighteenth embodiments described above.

[0485] As an example, the information processing device 1E according to the nineteenth embodiment includes the same Figure 23 The structure shown.

[0486] Figure 37 3100 is a flowchart showing an example of the process flow in the central brain 3015 that can control the part that causes the obstacle to collide. Here, as an example of an obstacle, other vehicles other than the driving vehicle, walls, guardrails, curbs, and other installations can be listed. In the following description, a case where other vehicles approaching the vehicle 3100 are used as an example of an obstacle is described. During the driving of the vehicle 3100, other vehicles may approach the vehicle 3100. In this case, preferably, the vehicle 3100 changes the behavior of the vehicle 3100 and drives to avoid contact or collision with other vehicles.

[0487] However, in a situation where a collision with other vehicles is unavoidable, an inevitable collision with an obstacle may occur. To handle such a situation, the central brain 3015 executes the process shown in this figure.

[0488] In step S20, as described above, the central brain 3015 can obtain sensor information from sensors that detect the surrounding conditions (including the behavior of other vehicles) of the detection vehicle 3100 every one billionth of a second. Here, the "central brain 3015" is an example of a "processor", the "vehicle" is an example of a "mobile body", the "sensor" is an example of a "detection unit", and the "sensor information" is an example of "detection information". That is, the processor can function as an acquisition unit that acquires detection information from the detection unit, and the detection unit detects the surrounding conditions of the mobile body with a second period shorter than the first period of photographing the surrounding of the mobile body.

[0489] In step S21A, the central brain 3015 can calculate the relationship between the own vehicle and other vehicles. For example, the central brain 3015 can calculate the position, distance, relative speed, etc. between the own vehicle and other vehicles based on the sensor information obtained in step S20.

[0490] In step S21B, the central brain 3015 can predict a collision with other vehicles and determine whether the collision is inevitable. For example, the central brain 3015 can use the relationship calculated in step S21A and a pre-determined threshold for collision determination to determine whether the collision is inevitable. That is, the processor can function as a prediction unit that predicts a collision of an obstacle with the mobile body based on the detection information.

[0491] When it is determined in step S21B that the collision is not inevitable ("No"), the central brain 3015 advances the process to step S21C. In step S21C, the central brain 3015 calculates a control variable. For example, the central brain 3015 calculates the control variable by a method similar to that of the ninth embodiment based on the sensor information obtained in step S20. On the other hand, when it is determined in step S21C that the collision is inevitable ("Yes"), the central brain 3015 advances the process to step S21D.

[0492] In step S21D, the central brain 3015 calculates a control variable for controlling the collision part. For example, the central brain 3015 can calculate at least one of the collision angle and vehicle speed of the vehicle 3100 that satisfies the pre-determined partial collision limitation conditions between the other vehicle and the vehicle 3100. That is, the processor can function as a calculation unit that calculates a control variable for controlling the autonomous driving of the mobile body so that in the case where the predicted result indicates an inevitable collision, a pre-determined partial collision between the obstacle and the mobile body occurs.

[0493] Here, the central brain 3015 can preset the part where the damage caused by the collision is below the threshold, preferably the minimum, as the collision part. That is, the processor can function as a setting unit that sets the part where the damage generated when the moving body collides with an obstacle meets a predetermined criterion as a predetermined part. Such a part can be, for example, a part in the vehicle 3100 where the rigidity is above a predetermined threshold. In addition, such a part can also be a part where the distance to the engine, motor, or battery, which are the main components mounted on the vehicle 3100, is above a predetermined threshold. In addition, such a part can also be a part in the vehicle 3100 where the distance to the occupant is above a predetermined threshold. It should be noted that in the above description, only the case of considering the damage on the vehicle 3100 side is shown as an example, but in addition to this, other damages on the vehicle side can also be considered. That is, the central brain 3015 can calculate a control variable so that the vehicle 3100 collides with a predetermined part of another vehicle.

[0494] In addition, when determining the part where the damage meets the predetermined criterion, the central brain 3015 can also accept user input and manually determine such a part by the user. As an alternative, the central brain 3015 can accept the input of a design document related to the vehicle (at least any one of a design drawing, a design specification, or a specification) and automatically determine such a part based on the input design document. At this time, the central brain 3015 can, for example, based on the design document, score the rigidity, the distance to the main component, and the distance to the occupant for each part respectively, and determine the part with the maximum addition operation and (or weighted addition operation sum) of the scores. In this way, the processor can also function as an input unit that accepts the input of a design document related to the moving body and a specific unit that determines the part where the damage meets the predetermined criterion based on the design document.

[0495] In step S22, the central brain 3015 controls the autonomous driving. For example, the central brain 3015 controls the autonomous driving based on the control variable calculated in step S21C or step S21D. After that, the central brain 3015 ends the processing of this flowchart.

[0496] As described above, the information processing apparatus 1E according to the nineteenth embodiment acquires detection information from a detection unit that detects the state of the surroundings of the moving body at a second period shorter than the first period for photographing the surroundings of the moving body, and predicts a collision of an obstacle with the moving body based on the detection information. When the predicted result indicates an inevitable collision, a control variable for controlling the autonomous driving of the moving body is calculated so that the obstacle collides with a predetermined part of the moving body. Thus, according to the information processing apparatus 1E according to the nineteenth embodiment, even when an inevitable collision with an obstacle occurs, it is possible to control the part where the obstacle collides. At this time, the information processing apparatus 1E according to the nineteenth embodiment may set a part where the damage generated when the obstacle collides with the moving object satisfies a predetermined criterion as the predetermined part. Thus, with the information processing apparatus 1E according to the nineteenth embodiment, it is possible to reduce, preferably minimize, the damage caused by the collision. More specifically, by setting at least any one of a part in the moving body having a rigidity equal to or higher than a predetermined threshold, a part having a distance equal to or higher than a predetermined threshold from an engine, a motor, or a battery mounted on the moving body, or a part having a distance equal to or higher than a predetermined threshold from an occupant in the moving body as such a part, with the information processing apparatus 1E according to the nineteenth embodiment, it is possible to reduce, preferably minimize, dents in the moving body, damage to main components, or injuries to occupants. In addition, the information processing apparatus 1E according to the nineteenth embodiment may accept an input of a design document related to the moving body and determine a part where the damage satisfies a predetermined criterion based on the design document. Thus, with the information processing apparatus 1E according to the nineteenth embodiment, it is possible to automatically determine a part for reducing damage without manual operation.

[0497] (Twenty - thirtieth Embodiment)

[0498] In addition, preferably, in the case of an autonomous driving vehicle, when the vehicle collides with an obstacle, a notification is made promptly. Therefore, as the twentieth embodiment of the present disclosure, an information processing apparatus and the like that can promptly notify when the vehicle collides with an obstacle will be described below.

[0499] Next, the twentieth embodiment of the present disclosure will be described by omitting or simplifying the overlapping parts with the sixth to nineteenth embodiments described above.

[0500] As an example, the information processing apparatus 1E according to the twentieth embodiment has a structure similar to that of the Figure 23 shown in the ninth embodiment.

[0501] During the driving of vehicle 3100, sometimes an obstacle approaches vehicle 3100. In this case, preferably, vehicle 3100 changes its behavior and drives to avoid contact or collision with the obstacle. As an example of the obstacle, other vehicles other than the own vehicle in motion, walls, guardrails, curbs, and other installations can be listed. In the following description, a case where another vehicle approaching vehicle 3100 is taken as an example of the obstacle will be described.

[0502] Similar to the ninth embodiment, the central brain 3015 controls the in-wheel motors respectively mounted on the four wheels based on the above control variables that change every moment and are calculated according to the driving state on the driving path, thereby controlling the wheel speeds, tilts of the four wheels respectively, and the suspensions supporting the four wheels, and performs autonomous driving. In addition, similar to the ninth embodiment, the central brain 3015 obtains sensor information from sensors that detect the conditions around vehicle 3100 (such as the behavior of other vehicles).

[0503] The central brain 3015 has at least the following functions: predicting a collision of the obstacle with vehicle 3100 including contact based on the acquired sensor information, and driving to avoid the obstacle.

[0504] The central brain 3015 can obtain sensor information from sensors that detect the conditions around vehicle 3100 every one-billionth of a second. That is, the processor can obtain detection information from a detection unit that detects the conditions around the moving body at a second period shorter than the first period of photographing the surroundings of the moving body.

[0505] Figure 38 It is a flowchart showing an example of the process executed by the central brain 3015 as a process capable of reducing the damage caused to the vehicle in an inevitable collision.

[0506] In step S30, as described above, the central brain 3015 obtains sensor information from sensors that detect the conditions around vehicle 3100. In step S32, the central brain 3015 calculates the relationship between the own vehicle and other vehicles. For example, the central brain 3015 can calculate the position, distance, and relative speed, etc. between the own vehicle and other vehicles based on the sensor information obtained in step S30.

[0507] In step S34, the central brain 3015 predicts a collision with another vehicle and determines whether the collision is unavoidable. For example, the central brain 3015 can use the relationship calculated in step S32 and a predetermined threshold value for collision determination to determine whether the collision is unavoidable. That is, the processor can also function as a prediction unit that predicts a collision of an obstacle with a moving body based on the detection information. In the case where the determination in step S34 is positive, that is, in the case where it is determined that the collision is unavoidable, the process is transferred to step S36.

[0508] In step S36, the central brain 3015 calculates control variables that reduce the damage caused by the vehicle in an unavoidable collision. For example, the central brain 3015 can calculate control variables corresponding to the damage caused by the vehicle 3100 in an unavoidable collision being below a predetermined threshold. The damage caused by the vehicle 3100 can apply at least one of the deformation position and deformation amount of the vehicle 3100. In addition, the control variable corresponding to the damage caused by the vehicle 3100 can apply at least one of the collision angle and vehicle speed of the vehicle 3100.

[0509] In step S38, the central brain 3015 notifies the preset contact via the communication line. Examples of the contact in this case include the emergency notification method of the police station and the emergency notification method of the fire department. In addition, examples of the contact in this case include the owner of the vehicle 3100 and the family of the owner of the vehicle 3100. When the processing of step S38 is completed, the processing is transferred to step S32.

[0510] On the other hand, if the determination in step S34 is negative, that is, if it is determined that the collision is unavoidable, the process transfers to step S40. In step S40, the central brain 3015 calculates the control variable. For example, the central brain 3015 calculates the control variable based on the sensor information acquired in step S30 by a method similar to the ninth embodiment. If the process of step S40 is completed, the process transfers to step S42.

[0511] In step S42, the central brain 3015 controls the automatic driving. For example, the central brain 3015 controls the automatic driving based on the control variables calculated in step S36 or step S40. If the processing of step S42 ends, the processing of the flowchart ends.

[0512] As described above, according to the twentieth embodiment, when a vehicle collides with an obstacle, notification can be promptly made.

[0513] In some of the above embodiments, the processing set to be executed by each processor (e.g., IPU 3011, MoPU 3012, and central brain 3015) is merely exemplary, and the processor that executes each processing is not limited. For example, the processing set to be executed by MoPU 3012 in the above embodiments can be executed by central brain 3015 instead of MoPU 3012, or can be executed by other processors other than IPU 3011, MoPU 3012, and central brain 3015.

[0514] (Twenty-first embodiment)

[0515] Hereinafter, as a twenty-first embodiment of the present disclosure, an information processing device and the like for protecting passengers in a vehicle having an automatic driving function during a collision with another vehicle or an object and a secondary collision will be described.

[0516] In the fifth embodiment described above, if Figure 19 As shown, an example of a situation in which a collision between the vehicle 2012A and other surrounding vehicles (for example, other vehicles 2012B, 2012C, and 2012D) is inevitable in the process for reducing the above-mentioned damage is described. In contrast, in the twenty-first embodiment, not only the situation of collision with other vehicles, but also the process for reducing the damage in the case of collision with other objects again after collision with other vehicles is performed. Here, the second collision includes secondary collision, tertiary collision, etc. The following describes the differences from the fifth embodiment. It should be noted that the same figure marks are marked on the same structures as those described in the fifth embodiment, and the detailed description is omitted.

[0517] Figure 39 FIG. 4 shows a structural diagram of a vehicle system 4001 according to the present embodiment. Figure 39 As shown, the vehicle system 4001 of this embodiment is configured to include a plurality of vehicles 4012, an information providing device 4014, and a server 4016. The vehicles 4012 include a vehicle 4012A and other vehicles 4012B, 4012C, and 4012D traveling around the vehicle 4012A. In this embodiment, in addition to the vehicle 4012A, each of the other vehicles 4012B, 4012C, and 4012D is also equipped with a central brain 4120 as an information processing device, but the present invention is not limited thereto. The information providing device 4014 and the server 4016 are examples of external devices.

[0518] The information providing device 4014 is a monitoring unit having a camera or a radar, etc. The information providing device 4014 is installed around the travel path of the vehicle 4012, for example, on the top of a traffic light, a pillar of a street light, a door supporting a sign, etc.

[0519] As an example, server 4016 includes map information server 4016A and weather information server 4016B. Map information server 4016A has a map information database and can provide detailed map information as map information to vehicle 4012A. A detailed map refers to a map that contains information about the terrain or structures around the driving path. Weather information server 4016B can provide vehicle 4012A with current and near future (for example, one hour later) weather information. Map information server 4016A and weather information server 4016B are connected to vehicle 4012A via a public network N. It should be noted that in Figure 39 In the figure, the map information server 4016A and the weather information server 4016B are connected only to the vehicle 4012A, but are not limited thereto and may be connected to other vehicles 4012B, 4012C, and 4012D.

[0520] Figure 40 This is a flowchart showing an example of the process flow in the central brain 4120 that can reduce the damage caused by the vehicle 4012A in an unavoidable collision. It should be noted that the processing of step S50 and step S51 is an example of the function of the acquisition unit. In addition, the processing of step S52 to step S57 is an example of the function of the calculation unit. Furthermore, the processing of step S58 is an example of the function of the control unit.

[0521] exist Figure 40 In step S50, the central brain 4120 obtains sensor information including road information detected by the sensor.

[0522] In step S51, the central brain 4120 obtains external information. Specifically, the central brain 4120 obtains the captured images of the surroundings of the vehicle 4012A, the detection data of vehicles and objects, and the information of traffic lights from the information providing device 4014. In addition, the central brain 4120 obtains map information from the map information server 4016A and obtains weather information from the weather information server 4016B.

[0523] In step S52, the central brain 4120 calculates the relationship with other vehicles and objects with respect to the vehicle 4012A. Specifically, the central brain 4120 calculates the position and speed of the types of other vehicles with respect to the vehicle 4012A (vehicle sizes such as two-wheeled vehicles and large vehicles or the types of communicable vehicles (other vehicles 4012B, 4012C, 4012D)). In addition, the central brain 4120 calculates the types (people, animals, dropped objects, etc.), positions, and speeds of objects on the driving path of the vehicle 4012A.

[0524] In step S53, the central brain 4120 determines whether the vehicle 4012A has contacted or collided with other vehicles or objects. When the central brain 4120 determines that the vehicle 4012A has contacted or collided with other vehicles or objects (the case of "Yes (Y)" in step S53), the central brain 4120 proceeds to step S57. On the other hand, when the central brain 4120 determines that the vehicle 4012A has not yet contacted or collided with other vehicles or objects, that is, when there is neither contact nor collision (the case of "No (N)" in step S53), the central brain 4120 proceeds to step S54.

[0525] In step S54, the central brain 4120 determines whether the collision between the vehicle 4012A and other vehicles or objects is inevitable. If the central brain 4120 determines that the collision between the vehicle 4012A and other vehicles or objects is inevitable (if the answer is "yes" in step S54), the central brain 4120 proceeds to step S56. On the other hand, if the central brain 4120 determines that the collision between the vehicle 4012A and other vehicles or objects is not inevitable (if the answer is "no" in step S54), the central brain 4120 proceeds to step S55.

[0526] In step S55, the central brain 4120 calculates the above 16 control variables (Vn, Rn, Cn, Sn) based on the sensor information acquired in step S50 and the external information acquired in step S51. Then, it proceeds to step S58.

[0527] In step S56, as a damage reduction mode, the central brain 4120 calculates control variables (Vn, Rn, Cn, Sn) that reduce the damage caused to the vehicle 4012A in an unavoidable collision based on the sensor information obtained in step S50 and the external information obtained in step S51. Then, the process proceeds to step S58.

[0528] In the above step S53, if it is determined that the vehicle 4012A has already made contact or collided, step S57 is executed. In step S57, as a re-collision mode, the central brain 4120 calculates the control variables (Vn, Rn, Cn, Sn) for reducing the damage caused to the vehicle 4012A in the re-collision based on the sensor information obtained in step S50 and the external information obtained in step S51. Then, step S58 is entered.

[0529] In step S58, the central brain 4120 controls the automatic driving of the vehicle 4012A based on the calculated control variables, and then returns to step S50.

[0530] According to the present embodiment, similar to the fifth embodiment, in the case where the collision of the present vehicle 4012A is inevitable, the control variables representing the relationship with the lowest risk are calculated based on the mutual relationship between the present vehicle 4012A and other vehicles or objects. In addition, in the case where the present vehicle 4012A has already contacted or collided with other vehicles or objects, the control variables representing the relationship with the lowest risk are also calculated based on the mutual relationship with other vehicles or objects. For example, in the case where the present vehicle 4012A exceeds the opposite lane due to the reaction when colliding with other vehicles, the control variables are calculated so that the damage to the present vehicle 4012A when it collides with other vehicles again is minimized. In addition, for example, in the case where the outside of the driving path is a cliff, the control variables are calculated so that the present vehicle 4012A stays on the driving path so that the present vehicle 4012A does not fall into the cliff due to the reaction of the collision with other vehicles.

[0531] As described above, according to the information processing device involved in this embodiment (more specifically, the central brain 4120), when the vehicle 4012A collides, it is possible to accurately calculate the incident angle to the object, the reflection angle after the collision, the rebound speed, the moving distance or the moving direction, and prevent secondary damage caused by secondary collisions, tertiary collisions, etc. after the collision. It should be noted that it is also possible to determine whether to stop at a single collision, stop at a secondary collision, or allow more than three collisions based on the object of the collision, the time until the collision, the surrounding environment, etc.

[0532] In addition, according to the present embodiment, the vehicle 4012A and other vehicles 4012 (e.g., rear vehicles) or other objects (e.g., pedestrians, animals, etc.) around the object of collision are also considered to collide with the object again. Thus, it is possible to take surrounding vehicles into consideration.

[0533] Furthermore, according to the present embodiment, the surrounding conditions of the vehicle 4012A (e.g., the presence of cliffs or guardrails, etc.) can be considered to prevent a re-collision. For example, the central brain 4120 considers the surrounding conditions at the time of the collision (whether it is a cliff, a guardrail, or a sidewalk), and also calculates the position to which the vehicle 4012A and the object will rebound and move after the collision. In particular, in the case where there are cliffs or sidewalks with people around, the method of calculating the re-collision of the vehicle 4012A or the object without moving in that direction after the collision is calculated, and in the case where there are no people on the sidewalk, roadside trees, or guardrails around, the calculation is performed to determine how much the vehicle 4012A or the object can collide again in that direction. After this calculation, the central brain 4120 calculates the control variables (Vn, Rn, Cn, Sn).

[0534] (Twenty-second embodiment)

[0535] In the above-mentioned embodiments, for example, in the first embodiment, a method of controlling the vehicle by obtaining an index value is illustrated, but as an alternative, the vehicle can also be controlled by obtaining a control variable for driving control. Therefore, the following is a description of an information processing device using a control variable as the twenty-second embodiment of the present disclosure.

[0536] The information processing device of this embodiment can obtain the control variables required for driving control with high accuracy based on multiple information related to vehicle control. It should be noted that the information processing device described below is similar to the first embodiment except that the control variables are used. Therefore, in order to avoid repeated description, the description of the structure similar to the first embodiment is omitted.

[0537] The information processing device according to this embodiment can be implemented by, for example, Figure 1 The central brain is realized by the central brain shown. The central brain is connected to multiple gateways in a communicative manner. The central brain is connected to the external cloud via the gateway. The central brain is configured to be able to access the external cloud via the gateway. On the other hand, due to the existence of the gateway, the central brain is configured to be unable to be directly accessed from the outside.

[0538] The central brain outputs a request signal to the server every time a predetermined time has passed. Specifically, the central brain outputs a request signal indicating an inquiry to the server in a predetermined cycle, for example, every nanosecond.

[0539] Figure 41 1F is a block diagram showing an example of an information processing device involved in the twenty-second embodiment of the present disclosure. The information processing device 1F involved in this embodiment includes at least: an information acquisition unit 5010, which can acquire multiple information associated with the vehicle; an inference unit 5020, which obtains an index value based on the multiple information acquired by the information acquisition unit 5010 and infers multiple control variables using the index value; and a driving control unit 5030, which performs driving control of the vehicle based on the multiple control variables.

[0540] The information acquisition unit 5010 can acquire various information associated with the vehicle. As the information acquisition unit 5010, for example, it can include sensors installed in various parts of the vehicle or a communication unit for acquiring information that can be obtained via a network from a server not shown in the figure. As sensors included in the information acquisition unit 5010, radar, LiDAR (laser radar), high-pixel / telephoto / ultra-wide-angle / 360-degree / high-performance cameras, visual recognition, micro-sound sensors, ultrasonic sensors, vibration sensors, infrared sensors, ultraviolet sensors, electromagnetic wave sensors, temperature sensors, humidity sensors, fixed-point (Spot) AI weather forecasts, high-precision multi-channel GPS, low-altitude satellite information, etc. can be listed. Alternatively, long-tail event AI data, etc. can be listed. Long-tail event AI data refers to the trip data of a car equipped with level 5.

[0541] Among the sensor information that can be obtained by various sensors, there can be listed the temperature or material of the ground (such as the road), detection of the outside air temperature, detection of the outside air humidity, detection of the lateral and oblique inclination angles of the ramp in the up and down directions, the freezing method of the road, detection of the moisture content, the material of each tire, the wear condition, detection of the air pressure, the width of the road, whether there is a prohibition on overtaking, information on the types of oncoming vehicles, front and rear vehicles, the cruising status of these vehicles, the surrounding conditions (birds, animals, football, accident vehicles, earthquakes, fires, wind, typhoons, heavy rain, light rain, blizzards, fog, etc.), etc. In the present embodiment, these detections are performed in units of specified periods, for example, every billionth of a second.

[0542] It is particularly important to note that the information acquisition unit 5010 includes a vehicle bottom sensor, which is disposed at the bottom of the vehicle and can detect the temperature, material and inclination of the ground for driving. The vehicle bottom sensor can be used to perform independent intelligent tilting.

[0543] Here, one or more sensor information can apply sensor information from a sensor that detects the condition around the vehicle. In addition, in the case of using a plurality of sensor information as one or more sensor information, a predetermined number of sensor information can be applied. The predetermined number is, for example, 3. Based on the 3 sensor information, index values ​​for controlling wheel speed, tilt, and suspension are calculated. The number of index values ​​calculated based on the combination of the 3 sensor information is, for example, 3. The index values ​​for controlling wheel speed, tilt, and suspension include, for example, index values ​​calculated based on information related to air resistance in the sensor information, index values ​​calculated based on information related to road resistance in the sensor information, index values ​​calculated based on information related to slip coefficient in the sensor information, and the like.

[0544] In addition, the inference unit 5020 aggregates the index values ​​calculated for each combination of one sensor information or a plurality of sensor information having different combinations of sensor information, and calculates the control variables for controlling the wheel speed, tilt, and suspension. For example, the inference unit 5020 calculates the index values ​​based on the sensor information of the sensor that detects the surrounding conditions of the vehicle, and calculates the control variables. In addition, when using a plurality of sensor information, the inference unit 5020 calculates a plurality of index values, for example, by a combination of sensor 1, sensor 2, and sensor 3, calculates a plurality of index values ​​by a combination of sensor 4, sensor 5, and sensor 6, and calculates a plurality of index values ​​by a combination of sensor 1, sensor 3, and sensor 7, and aggregates the index values ​​to calculate the control variables.

[0545] For example, the inference unit 5020 uses information on the material, weather, wind, and inclination of the road to calculate the index value based on the information related to air resistance. Similarly, the inference unit 5020 uses information on the material, wind, and inclination of the road to calculate the index value based on the information related to road resistance. Similarly, the inference unit 5020 uses information on the material of the road to calculate the index value based on the information related to road elements.

[0546] In this way, the inference unit 5020 calculates a predetermined number of index values, such as 300, while changing the combination of sensor information, and calculates the control variable based on the calculated index value. Specifically, the calculation unit can be a unit that can use machine learning, more specifically, deep learning, to calculate the control variable based on the sensor information. In other words, the calculation unit that calculates the index value and the control variable can be composed of AI.

[0547] The inference unit 5020 uses the computing power of level 6 to perform multivariate analysis (e.g., refer to equation (2)) based on the integration method shown in the following equation (1) on the data of each nanosecond or the long-tail event AI data collected by a large number of sensor groups in the information acquisition unit 5010, thereby being able to obtain accurate control variables. In more detail, the inference unit 5020 obtains the indexed value of each variable at the edge level and in real time while obtaining the integral value of various ultra-high resolution Delta values ​​with the computing power of level 6, and can obtain the result that occurs in the next nanosecond with the highest probability value. In order to achieve this, the integral value obtained by time-integrating the Delta value (e.g., the change value of a small time) of the function (in other words, the function representing the change of each variable) that can determine each variable such as air resistance, road resistance, road element (e.g., garbage) and slip coefficient (e.g., a plurality of information obtained by the information acquisition unit 5010) is input to the deep learning model of the inference unit 5020. The deep learning model of the inference unit 5020 outputs a control variable corresponding to the input integral value (for example, a control variable with the highest confidence (ie, evaluation value)). The output of the control variable is performed in nanoseconds.

[0548] [Calculation formula 1]

[0549]

[0550] [Calculation formula 2]

[0551] V n =DL(f(A,B,C,D,…,N)(dA n / dt)) (2)

[0552] It should be noted that, as an example, in formula (1), "f(A)" is a formula that simplifies the function that represents the change of each variable such as air resistance, road resistance, road elements (such as garbage) and sliding coefficient. In addition, as an example, formula (1) is a formula that represents the time integral v of "f(A)" from time a to time b. DL in formula (2) represents deep learning (for example, a deep learning model optimized by deep learning of a neural network), dAn / dt represents the Delta value of f(A, B, C, D, ..., N), and A, B, C, ..., N are index values ​​calculated based on sensor information, such as index values ​​calculated based on air resistance, index values ​​calculated based on road resistance, index values ​​calculated based on road elements, and index values ​​calculated based on sliding coefficient. f(A, B, C, ..., N) represents a function used to represent the change of A, B, C, ..., N, and Vn represents the value output from the deep learning model optimized by deep learning of a neural network (that is, a control variable).

[0553] It should be noted that the example of inputting the integral value obtained by time-integrating the Delta value of the function to the deep learning model of the inference unit 5020 is listed here, but this is only an example. It can also be set, for example, that the deep learning model of the inference unit 5020 infers the integral value (for example, the result occurring in the next nanosecond) obtained by time-integrating the Delta value of the function representing the change of each variable such as air resistance, road resistance, road element and slip coefficient, and the inference unit 5020 obtains the integral value with the highest confidence (i.e., evaluation value) as the inference result per nanosecond.

[0554] In addition, examples of inputting integral values ​​to or outputting integral values ​​from a deep learning model are listed here, but this is only an example, and the technology disclosed herein also works even if integral values ​​are not used. For example, at least one control variable may be inferred by a deep learning model optimized by deep learning a neural network using the following supervisory data, where the supervisory data uses values ​​equivalent to A, B, C, ..., N as example data and uses values ​​equivalent to at least one control variable (e.g., the result occurring in the next nanosecond) as correct answer data.

[0555] Each control variable obtained by the inference unit 5020 can be further refined by increasing the number of deep learning. For example, a more accurate control variable can be calculated using a large amount of data or long-tail event AI data such as tire or motor rotation, steering angle or road material, weather, garbage or the impact of quadratic deceleration, slip, steering or speed control methods for losing balance or regaining balance.

[0556] The driving control unit 5030 may perform driving control of the vehicle based on the multiple control variables determined by the inference unit 5020. Alternatively, the driving control unit 5030 may implement automatic driving control of the vehicle. Specifically, the highest probability value of the result occurring in the next nanosecond may be obtained based on the multiple control variables, and driving control of the vehicle taking the probability value into consideration may be implemented.

[0557] According to the information processing device 1F having the above structure, similar to the information processing device of the first embodiment, since the level 6 computing power, which is much greater than the level 5 computing power, can be used to implement information analysis or inference, it is possible to perform a more detailed analysis than before. As a result, vehicle control for safe autonomous driving can be performed. Furthermore, through the above-mentioned multivariate analysis based on AI, a value difference of 1000 times can be generated compared to the level 5 world.

[0558] (Twenty-third embodiment)

[0559] Figure 421 is a block diagram showing an example of an information processing device according to the twenty-third embodiment of the present disclosure. The information processing device 1G according to this embodiment is different from the information processing device 1F according to the twenty-second embodiment in that, in addition to the information processing device 1F, it further includes a strategy setting unit 5040 for setting a driving strategy until the vehicle reaches the destination.

[0560] The strategy setting unit 5040 can set a driving strategy from the current location to the destination based on the information of the destination input by the occupants of the vehicle or the traffic information between the current location and the destination. At this time, the information at the time point when the strategy is set can be considered, that is, the data currently acquired by the information acquisition unit 5010. It is not only a simple route calculation to reach the destination, but also a theoretical value that is more in line with reality by considering the surrounding conditions at that moment. The driving strategy can be composed of the theoretical value of at least one of the optimal route (strategic route) to reach the destination, driving speed, inclination, and braking. Preferably, the driving strategy can be composed of all the theoretical values ​​of the above-mentioned optimal route, driving speed, inclination, and braking.

[0561] The plurality of theoretical values ​​constituting the driving strategy set by the strategy setting unit 5040 may be used for the automatic driving control in the driving control unit 5030. In addition, preferably, the driving control unit 5030 includes a strategy updating unit 5031, which can update the driving strategy based on the difference between the plurality of control variables inferred by the inference unit 5020 and each theoretical value set by the strategy setting unit 5040.

[0562] The control variables inferred by the inference unit 5020 are based on information obtained during vehicle driving, specifically, based on information such as the friction coefficient detected during actual driving. Therefore, in the strategy update unit 5031, by considering the control variables, it is possible to cope with changes that change every moment when the strategy route passes. Specifically, in the strategy update unit 5031, by calculating the difference (Delta value) between the theoretical value and the control variable contained in the driving strategy, the optimal solution can be derived again and the strategy route can be re-formulated. In this way, it is also possible to achieve automatic driving control without slipping. In addition, during such an update process, since the computing power of the above-mentioned level 6 can be used, corrections and fine-tuning can be performed in units of specified periods, such as units of billionths of a second, so that more precise driving control can be achieved.

[0563] In addition, when the information acquisition unit 5010 has the above-mentioned lower vehicle sensor, since the lower vehicle sensor also detects the temperature or material of the ground, etc., it can cope with the changes that change every moment when the strategic route passes. When calculating the driving distance included in the driving strategy, independent intelligent tilt can also be implemented. Furthermore, even if other information is detected (for example, flying tires, debris, animals, etc.), by coping with the changes that change every moment when the strategic route passes, the optimal driving distance can be recalculated in an instant and the optimal route management can be implemented.

[0564] In the twenty-second and twenty-third embodiments, the central brain functions as a control unit that calculates a control variable for controlling the speed of the vehicle and controls the automatic driving of the vehicle in units of a predetermined period, for example, in units of nanoseconds, based on the control variable. n The relationship between the two controls the speed of the vehicle (acceleration / deceleration). And, it enables frictionless and perfect turns.

[0565] Figure 43 It means that according to y=ax n Here, x is time, y is speed, and a is a proportional constant, which is an example of a control variable. In addition, the value of n is 2, for example, but it can also be other values. That is, a perfect turn without friction means a beautiful, smooth, perfect acceleration. n The acceleration is calculated by the relationship between y and ax to control the automatic driving at the start of acceleration. That is, the acceleration can change every moment. And if it is close to the target speed, y = 1-ax n Such an n-order function decelerates. 1 is the result after the target speed is normalized.

[0566] In addition, when decelerating, y=1-ax can be used at the beginning of deceleration. n represents the acceleration. 1 is the result of normalizing the speed during deceleration. n The acceleration is calculated by the relationship between y and ax to control the automatic driving at the beginning of deceleration. And, if it approaches the target speed, y = ax n Such n-th order function slows down.

[0567] In order to satisfy the above y=ax when the vehicle accelerates and decelerates n , y=1-ax n The relationship between the control variables used to control the driving of the vehicle is the control variable inferred by the inference unit 5020.

[0568] The proportional constant can be smoothly changed in any number of stages at a specified period, such as nanoseconds. The central brain calculates based on sensor information such as driving time, battery consumption, avoidance of unexpected accidents, material conditions such as tires, and variables such as wind speed, and compares it with data stored in the cloud, fine-tunes the difference, and correctly transmits the value of the proportional constant inferred by the goal seek to the wheel hub motors and four spin angles installed on the four wheels, thereby achieving perfect speed control and perfect steering control for optimal acceleration and deceleration. This is an object-oriented goal seek driving system. And achieving this type of driving is a level 6 function.

[0569] (Replenish)

[0570] Figure 44 An example of the hardware configuration of the computer 1200 that functions as the above-mentioned information processing device (e.g., central brain) or cooling execution device is schematically shown. The program installed in the computer 1200 can make the computer 1200 function as one or more "parts" of the device involved in the above-mentioned embodiment, or make the computer 1200 perform the operation associated with the device involved in the above-mentioned embodiment or the one or more "parts", and / or can make the computer 1200 perform the process involved in the above-mentioned embodiment or the stage of the process. Such a program can be executed by the CPU 1212 to make the computer 1200 perform specific operations associated with some or all of the blocks in the flowcharts and block diagrams recorded in this specification.

[0571] The computer 1200 according to various embodiments may include a CPU 1212, a RAM 1214, and a graphics controller 1216 that may be connected to each other through a host controller 1210. The computer 1200 may further include a communication interface 1222, a storage device 1224, an input / output unit such as a DVD drive and an IC card drive, which may be connected to the host controller 1210 via the input / output controller 1220. The DVD drive may be a DVD-ROM drive and a DVD-RAM drive, etc. The storage device 1224 may be a hard disk drive and a solid state drive, etc. The computer 1200 may further include a ROM 1230 and a conventional input / output unit such as a keyboard, which may be connected to the input / output controller 1220 via an input / output chip 1240.

[0572] The CPU 1212 can control each unit by operating according to the program stored in the ROM 1230 and the RAM 1214. The graphic controller 1216 can obtain image data generated by the CPU 1212 from a frame buffer or the like provided in the RAM 1214 or in itself, and display the image data on the display device 1218.

[0573] The communication interface 1222 can communicate with other electronic devices via a network. The storage device 1224 can store programs and data used by the CPU 1212 in the computer 1200. The DVD drive can read programs or data from a DVD-ROM or the like and provide them to the storage device 1224. The IC card drive can read programs and data from an IC card and / or write programs and data to an IC card.

[0574] The ROM 1230 can store therein a boot program or the like executed by the computer 1200 at startup, and / or a program depending on the hardware of the computer 1200. The input / output chip 1240 can also connect various input / output units to the input / output controller 1220 via a USB port, a parallel port, a serial port, a keyboard port, a mouse port, and the like.

[0575] The program can be provided by a computer-readable storage medium such as a DVD-ROM or an IC card. The program can be read from the computer-readable storage medium, installed in the storage device 1224, RAM 1214, or ROM 1230, which is also an example of a computer-readable storage medium, and executed by the CPU 1212. The information processing described in these programs can be read by the computer 1200, and the program and the aforementioned various types of hardware resources can be coordinated. The device or method can be configured by realizing the operation or processing of information according to the use of the computer 1200.

[0576] For example, when communication is performed between the computer 1200 and an external device, the CPU 1212 can execute a communication program loaded into the RAM 1214, and instruct the communication interface 1222 to perform communication processing based on the processing described in the communication program. Under the control of the CPU 1212, the communication interface 1222 can read the transmission data stored in the transmission buffer provided in the RAM 1214, the storage device 1224, a recording medium such as a DVD-ROM or an IC card, and transmit the read transmission data to the network, or write the reception data received from the network to the reception buffer provided on the recording medium, etc.

[0577] In addition, the CPU 1212 can cause all or a necessary part of a file or a database stored in an external recording medium such as the storage device 1224, a DVD drive (DVD-ROM), an IC card, etc. to be read into the RAM 1214, and perform various types of processing on the data on the RAM 1214. Next, the CPU 1212 can write the processed data back to the external recording medium.

[0578] Various types of information such as various types of programs, data, tables, and databases can be stored in the recording medium to receive information processing. The CPU 1212 can perform various types of processing on the data read from the RAM 1214, and write the results back to the RAM 1214. The various types of processing include various types of operations, information processing, conditional judgments, conditional branches, unconditional branches, information retrieval / replacement, etc. recorded in various places of the present disclosure and specified by the instruction sequence of the program. In addition, the CPU 1212 can retrieve information in files, databases, etc. in the recording medium. For example, in the case where multiple entries each having an attribute value of a first attribute associated with an attribute value of a second attribute are stored in the recording medium, the CPU 1212 can retrieve an entry that is consistent with the condition specifying the attribute value of the first attribute from the multiple entries, and read the attribute value of the second attribute stored in the entry, thereby obtaining the attribute value of the second attribute associated with the first attribute that meets the predetermined condition.

[0579] The program or software module described above may be stored in a computer-readable storage medium on or near the computer 1200. In addition, a recording medium such as a hard disk or RAM provided in a server system connected to a dedicated communication network or the Internet may be used as a computer-readable storage medium, thereby enabling the program to be provided to the computer 1200 via the network.

[0580] The flowcharts and boxes in the block diagrams in this embodiment may represent the stages of the process of performing an operation or the "parts" of the device having the function of performing an operation. Specific stages and "parts" may be implemented by dedicated circuits, programmable circuits supplied together with computer-readable instructions stored on a computer-readable storage medium, and / or processors supplied together with computer-readable instructions stored on a computer-readable storage medium. Dedicated circuits may include digital and / or analog hardware circuits, and may also include integrated circuits (ICs) and / or discrete circuits. Programmable circuits may include reconfigurable hardware circuits such as field programmable gate arrays (FPGAs) and programmable logic arrays (PLAs), which include logical AND, logical OR, logical XOR, logical NAND, logical NOR, and other logical operations, triggers, registers, and storage elements.

[0581] Computer-readable storage media may include any tangible device capable of storing instructions executed by an appropriate device, with the result that a computer-readable storage medium having instructions stored in a tangible device has a product including instructions that can be executed to generate a unit for performing the operations specified in the flowchart or block diagram. Examples of computer-readable storage media include electronic storage media, magnetic storage media, optical storage media, electromagnetic storage media, semiconductor storage media, etc. More specific examples of the computer-readable storage medium may include a floppy disk (registered trademark) disk, a magnetic disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an electrically erasable programmable read-only memory (EEPROM), a static random access memory (SRAM), a compact disc read-only memory (CD-ROM), a digital versatile disc (DVD), a Blu-ray disc (Blu-ray (registered trademark) Disk), a memory stick, an integrated circuit card, and the like.

[0582] Computer readable instructions may include assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state setting data, or any source code or object code described in any combination of one or more programming languages, wherein the one or more programming languages ​​include object-oriented programming languages ​​such as Smalltalk (registered trademark), JAVA (registered trademark), C++, etc. and traditional procedural programming languages ​​such as the "C" programming language or similar programming languages.

[0583] The computer-readable instructions can be provided to a processor or programmable circuit of a general-purpose computer, a special-purpose computer or other programmable data processing device locally or through a local LAN (Local Area Network, LAN), a wide area network (Wide Area Network, WAN) such as the Internet, etc., so that the processor or programmable circuit of the general-purpose computer, special-purpose computer or other programmable data processing device executes the computer-readable instructions to generate a unit for performing the operations specified in the flowchart or block diagram. Examples of processors include computer processors, processing units, microprocessors, digital signal processors, controllers, microcontrollers, etc.

[0584] The above embodiments are used to illustrate the technology of the present disclosure, but the technical scope of the present disclosure is not limited to the scope described in the above embodiments. It should be clear to those skilled in the art that various changes or improvements can be made to the above embodiments. It can be seen from the description in the claims that the embodiments with such changes or improvements can also be included in the technical scope of the present disclosure.

[0585] It should be noted that the execution order of each process such as actions, sequences, steps and stages in the devices, systems, programs and methods shown in the claims, specifications and drawings is not specifically indicated as "before", "earlier than", etc., or as long as the output of the previous process is not used in the subsequent process, it can be implemented in any order. Even if the action flow in the claims, specifications and drawings is described using "first", "next", etc. for convenience, it does not mean that it must be implemented in this order.

[0586] As used in this specification, "α and / or β" has the same meaning as "at least one of α and β". That is, "α and / or β" means that it can be only α, only β, or a combination of α and β. In addition, in this specification, when "and / or" is used to connect and express three or more items, the same understanding as "α and / or β" is also applicable.

[0587] This application is based on Special Application No. 2022-170165 filed on October 24, 2022, No. 2022-172777 filed on October 27, 2022, No. 2022-175679 filed on November 1, 2022, No. 2022-181362 filed on November 11, 2022, No. 2022-182131 filed on November 14, 2022, Special Application No. 2022-186040 filed on November 21, 2022, No. 2022-187648 filed on November 24, 2022, No. 2022-187649 filed on November 24, 2022, and No. 2022-187640 filed on November 28, 2022 No. 2-189546, Special Application No. 2022-210851 filed on December 27, 2022, No. 2023-36967 filed on March 9, 2023, Special Application No. 2023-58312 filed on March 31, 2023, Special Application No. 2023-78139 filed on May 10, 2023, Special Application No. 2023-82445 filed on May 18, 2023, Special Application No. 2023-87263 filed on May 26, 2023, Special Application No. 2023-90418 filed on May 31, 2023, and Special Application No. 2023-93420 filed on June 6, 2023, the contents of which form a part of this application as the contents of this application.

[0588] All documents, including publications, patent applications, and patents, cited in this specification are incorporated into this specification by reference to the same extent as if each document was specifically indicated, incorporated by reference, and set forth in its entirety herein.

[0589] The use of nouns used in association with the description of the present disclosure (especially in association with the following claims) and the use of the same indicators, unless otherwise specified in this specification or clearly contradicted by the context, should be interpreted as covering both the singular and the plural. Unless otherwise specified, the terms "have", "have", "include" and "include" are interpreted as open terms (i.e., "including but not limited to"). Unless otherwise specified in this specification, the description of the numerical range in this specification is intended only as a shorthand expression for referring to each corresponding value within the range separately, and each numerical value should be incorporated into the specification as if it has been listed separately in this specification. Unless otherwise specified in this specification or clearly contradicted by the context, all methods described in this specification can be implemented in any suitable order. Unless otherwise specifically stated, any examples or exemplary expressions (e.g., "etc.") used in this specification are only intended to better illustrate the present disclosure, not to set limitations on the scope of the present disclosure. Any statement in the specification should not be interpreted as representing elements not recorded in the claims as essential elements for the implementation of the present disclosure.

[0590] In this specification, preferred embodiments of the prese...

Claims

1. An information processing device, wherein, the information processing device includes: an information acquisition unit that can acquire a plurality of pieces of information associated with a vehicle; an inference unit that calculates an index value based on the plurality of pieces of information acquired by the information acquisition unit, and uses deep learning to infer a plurality of control variables based on the index value; and a driving control unit that executes driving control of the vehicle based on the plurality of control variables.

2. The information processing device according to claim 1, wherein, the inference unit infers the plurality of control variables based on the plurality of pieces of information through multivariate analysis based on an integration method using the deep learning.

3. The information processing device according to claim 1, wherein, the information acquisition unit acquires the plurality of pieces of information in units of a predetermined period, and the inference unit and the driving control unit use the plurality of pieces of information acquired in units of the predetermined period to perform inference of the plurality of control variables and driving control of the vehicle in units of the predetermined period.

4. The information processing device according to claim 1, wherein, the information processing device further includes a strategy setting unit, the strategy setting unit sets a driving strategy until the vehicle reaches a destination, the driving strategy includes at least one theoretical value of an optimal route to the destination, a driving speed, an inclination, and a braking, the driving control unit includes a strategy update unit that updates the driving strategy based on a difference between the plurality of control variables and the theoretical value.

5. The information processing device according to claim 1, wherein, the information acquisition unit includes a sensor that is provided below the vehicle and can detect the temperature, material, and inclination of the ground for driving.

6. An information processing device, wherein, the information processing device includes: an acquisition unit that acquires a plurality of pieces of information associated with the vehicle from a detection unit including a sensor. As a period for detecting the surrounding conditions of the vehicle, the sensor detects the surrounding conditions of the vehicle at a second period shorter than a first period for photographing the surrounding of the vehicle; a calculation unit that calculates an index value related to the surrounding conditions of the vehicle based on the acquired plurality of pieces of information, and calculates a control variable for controlling the behavior of the vehicle based on the calculated index value; and a control unit that controls the behavior of the vehicle based on the calculated control variable.

7. The information processing device according to claim 6, wherein, the calculation unit calculates the control variable based on the index value through multivariate analysis based on an integration method using deep learning.

8. The information processing device according to claim 6, wherein, the acquisition unit acquires the plurality of pieces of information in units of one billionth of a second, and the calculation unit uses the plurality of pieces of information acquired in units of one billionth of a second to perform calculation of the index value and the control variable in units of one billionth of a second.

9. The information processing device according to claim 6, wherein, The calculation unit predicts a collision of an object against the vehicle based on the acquired multiple pieces of information, and when the predicted result indicates an inevitable collision, calculates a control variable corresponding to damage that is equal to or less than a predetermined threshold value among the damage generated by the vehicle in the inevitable collision as the control variable.

10. The information processing device according to claim 9, wherein the damage generated by the vehicle is at least one of a deformation position and a deformation amount of the vehicle.

11. The information processing device according to claim 9, wherein the control variable corresponding to the damage generated by the vehicle is at least one of a collision angle and a vehicle speed of the vehicle.

12. A vehicle, wherein the vehicle includes the information processing device according to any one of claims 1 to 11.

13. A program, wherein the program causes a computer to function as the information processing device according to any one of claims 1 to 11.

14. An information processing device, wherein the information processing device includes a processor, the processor acquires detection information obtained by detecting the state of the surroundings of the moving body, point information obtained by capturing an object in the captured image as points, and identification information for identifying the object, calculates the control variable based on the correlation between a variable related to the state of the surroundings of the moving body calculated from the detection information and the control variable inferred using the variable, controls the autonomous driving of the moving body based on the control variable, the point information, and the identification information.

15. The information processing device according to claim 14, wherein a coefficient of the variable with respect to the control variable is predetermined, the processor calculates the control variable using the coefficient.

16. The information processing device according to claim 14, wherein the processor executes the inference process of the control variable at a second period that is longer than a first period for acquiring the detection information.

17. The information processing device according to claim 16, wherein the processor executes the calculation process of the control variable at a third period that is shorter than the second period.

18. The information processing device according to any one of claims 14 to 17, wherein the processor infers the control variable through multivariate analysis based on an integration method using deep learning.

19. The information processing device according to any one of claims 14 to 17, wherein the processor acquires the point information and the identification information from different other processors respectively.

20. An information processing method, wherein the information processing method executed by a computer includes the following processes: acquiring detection information obtained by detecting the state of the surroundings of the moving body, point information obtained by capturing an object in the captured image as points, and identification information for identifying the object, calculating the control variable based on the correlation between a variable related to the state of the surroundings of the moving body calculated from the detection information and the control variable inferred using the variable, Control the autonomous driving of the moving body based on the control variable, the point information, and the recognition information.

21. A program, wherein, the program causes a computer to perform the following processing: Obtain detection information obtained by detecting the situation around the moving body, point information obtained by capturing an object as points in a captured image, and recognition information obtained by recognizing the object, Calculate the control variable based on the correlation between a variable related to the situation around the moving body calculated from the detection information and a control variable inferred using the variable, Control the autonomous driving of the moving body based on the control variable, the point information, and the recognition information.

22. An information processing device, wherein, the information processing device includes a processor, the processor Obtains detection information from a detection unit that detects the situation around the moving body at a second period shorter than a first period for capturing the surroundings of the moving body, Divides a period during which an obstacle collision prediction for the moving body can be made based on the detection information into a plurality of periods, In each of the plurality of periods in time series of the plurality of periods, calculates a control variable for controlling the autonomous driving of the moving body using a plurality of input parameters based on the detection information.

23. The information processing device according to claim 22, wherein, When calculating the control variable in the second and subsequent periods among the plurality of periods, the processor calculates the control variable with fewer input parameters than in the previous period.

24. An information processing method, wherein, the information processing method causes a computer to perform the following processing: Obtain detection information from a detection unit that detects the situation around the moving body at a second period shorter than a first period for capturing the surroundings of the moving body, Divides a period during which an obstacle collision prediction for the moving body can be made based on the detection information into a plurality of periods, In each of the plurality of periods in time series of the plurality of periods, calculates a control variable for controlling the autonomous driving of the moving body using a plurality of input parameters based on the detection information.

25. A program, wherein, the program causes a computer to perform the following processing: Obtain detection information from a detection unit that detects the situation around the moving body at a second period shorter than a first period for capturing the surroundings of the moving body, Divides a period during which an obstacle collision prediction for the moving body can be made based on the detection information into a plurality of periods, In each of the plurality of periods in time series of the plurality of periods, calculates a control variable for controlling the autonomous driving of the moving body using a plurality of input parameters based on the detection information.

26. An information processing device, wherein, the information processing device includes a processor, the processor Obtains detection information from a detection unit that detects the situation around the moving body at a second period shorter than a first period for capturing the surroundings of the moving body, Calculates the distance between the moving body and an obstacle based on the detection information, Predict the approach of the obstacle based on the distance and a predetermined margin.

27. The information processing device according to claim 26, wherein the margin is predetermined by an occupant of the moving body.

28. The information processing device according to claim 27, wherein the margin is predetermined according to the age of the occupant.

29. The information processing device according to claim 26, wherein the processor changes the margin according to the moving speed of the moving body or the obstacle.

30. The information processing device according to claim 29, wherein the processor changes the margin according to the braking distance of the moving body or the obstacle.

31. The information processing device according to any one of claims 26 to 30, wherein the processor controls the autonomous driving of the moving body based on the predicted result.

32. An information processing method, wherein the information processing method is executed by a computer to perform the following processing: Obtain detection information from a detection unit that detects the state of the surroundings of the moving body at a second period shorter than a first period for photographing the surroundings of the moving body, Calculate the distance between the moving body and an obstacle based on the detection information, Predict the approach of the obstacle based on the distance and a predetermined margin.

33. A program, wherein the program causes a computer to perform the following processing: Obtain detection information from a detection unit that detects the state of the surroundings of the moving body at a second period shorter than a first period for photographing the surroundings of the moving body, Calculate the distance between the moving body and an obstacle based on the detection information, Predict the approach of the obstacle based on the distance and a predetermined margin.

34. An information processing device, wherein the information processing device includes a processor, the processor obtains detection information from a detection unit that detects the state of the surroundings of the moving body at a second period shorter than a first period for photographing the surroundings of the moving body, predicts a collision of an obstacle with the moving body based on the detection information, calculates a control variable in a case where the predicted result indicates an inevitable collision, the control variable controlling the autonomous driving of the moving body so that the obstacle collides with a predetermined part of the moving body.

35. The information processing device according to claim 34, wherein the processor sets as the predetermined part a part where the damage generated when the obstacle collides with the moving body satisfies a predetermined criterion.

36. The information processing device according to claim 35, wherein the part where the damage satisfies a predetermined criterion is a part of the moving body having a rigidity equal to or higher than a predetermined threshold.

37. The information processing device according to claim 35, wherein the part where the damage satisfies a predetermined criterion is a part at a distance equal to or higher than a predetermined threshold from an engine, a motor, or a battery mounted on the moving body.

38. The information processing device according to claim 35, wherein The part of the damage that meets a predetermined criterion is the part where the distance to the occupant in the moving body is greater than a predetermined threshold value.

39. The information processing device according to any one of claims 36 to 38, wherein, the processor accepts an input of a design document related to the moving body, and based on the design document, determines the part of the damage that meets a predetermined criterion.

40. An information processing method, wherein, the information processing method is executed by a computer to perform the following processing: acquire detection information from a detection unit that detects the state of the surroundings of the moving body at a second period shorter than a first period for photographing the surroundings of the moving body, predict a collision of an obstacle with the moving body based on the detection information, and when the predicted result indicates an inevitable collision, calculate a control variable that controls the autonomous driving of the moving body so that the obstacle collides with a predetermined part in the moving body.

41. A program, wherein, the program causes a computer to perform the following processing: acquire detection information from a detection unit that detects the state of the surroundings of the moving body at a second period shorter than a first period for photographing the surroundings of the moving body, predict a collision of an obstacle with the moving body based on the detection information, and when the predicted result indicates an inevitable collision, calculate a control variable that controls the autonomous driving of the moving body so that the obstacle collides with a predetermined part in the moving body.

42. An information processing device, wherein, the information processing device includes a processor, the processor acquires detection information from a detection unit that detects the state of the surroundings of the moving body at a second period shorter than a first period for photographing the surroundings of the moving body, calculates a control variable for controlling the autonomous driving of the moving body based on the detection information, and when calculating a control variable that reduces the damage caused to the moving body in an inevitable collision, makes a notification to a set contact method.

43. An information processing method, wherein, the information processing method is executed by a computer to perform the following processing: acquire detection information from a detection unit that detects the state of the surroundings of the moving body at a second period shorter than a first period for photographing the surroundings of the moving body, calculate a control variable for controlling the autonomous driving of the moving body based on the detection information, and when calculating a control variable that reduces the damage caused to the vehicle in an inevitable collision, makes a notification to a set contact method.

44. A program, wherein, the program causes a computer to perform the following processing: acquire detection information from a detection unit that detects the state of the surroundings of the moving body at a second period shorter than a first period for photographing the surroundings of the moving body, calculate a control variable for controlling the autonomous driving of the moving body based on the detection information, and when calculating a control variable that reduces the damage caused to the vehicle in an inevitable collision, makes a notification to a set contact method.

45. An information processing device, wherein, the information processing device includes: An acquisition unit that acquires a plurality of pieces of information related to the vehicle from a detection unit including sensors that detect the surrounding conditions of the vehicle; A calculation unit that calculates an index value related to the surrounding conditions of the vehicle based on the acquired plurality of pieces of information, and calculates a control variable for controlling the behavior of the vehicle based on the calculated index value; And A control unit that controls the behavior of the vehicle based on the calculated control variable, The calculation unit predicts a collision of an object with the vehicle based on the acquired plurality of pieces of information, and when the predicted result indicates an inevitable collision, calculates a control variable such that the damage caused by the vehicle in the inevitable collision and a re-collision with another object after the collision is below a predetermined threshold.

46. The information processing device according to claim 45, wherein, The acquisition unit acquires information related to the surrounding conditions of the vehicle from another vehicle that is a collision object.

47. The information processing device according to claim 45, wherein, The acquisition unit acquires information related to the surrounding conditions of the vehicle from an external device that sets a driving path for the vehicle to travel.

48. The information processing device according to claim 45, wherein, The calculation unit calculates the control variable based on the index value through multivariate analysis based on an integration method using deep learning.

49. The information processing device according to claim 45, wherein, The damage caused by the vehicle is at least one of the deformation position and deformation amount of the vehicle.

50. The information processing device according to claim 45, wherein, The control variable corresponding to the damage caused by the vehicle is at least one of the collision angle and vehicle speed of the vehicle.

51. A vehicle, wherein, The vehicle includes: The information processing device according to claim 47; The sensor, which is connected to the information processing device and detects the surrounding conditions of the vehicle; And A receiving unit that receives the plurality of pieces of information from the external device.

Images