Information processor, information processing method, and information processing program

Abstract

To provide an information processor that can reduce an amount of calculation when controlling the driving of a vehicle by considering each pattern of movement of surrounding objects.SOLUTION: The information processor includes: a first processor that outputs point information that identifies a photographed object as a point from an image of the object photographed by a first camera mounted on a vehicle; a second processor that outputs identification information identifying the photographed object from an image of the object photographed by a second camera mounted on the vehicle and facing in a direction corresponding to that of the first camera; and a third processor that associates the point information output from the first processor and the identification information output from the second processor. The third processor further controls the driving of the vehicle based on results of simulating the movement of the object on a digital map on which the point information and the identification information that are associated with each other are plotted.SELECTED DRAWING: Figure 2A

Description

[Technical field]

[0001] The present disclosure relates to an information processing device, an information processing method, and an information processing program. [Background technology]

[0002] Patent Document 1 describes a vehicle with an automatic driving function. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2022-035198 A Summary of the Invention [Problem to be solved by the invention]

[0004] Incidentally, when a vehicle is driven autonomously as in Patent Document 1, the autonomous driving is controlled using multiple images of the surroundings of the vehicle captured by a camera. In this case, if every possible pattern of the movement of surrounding objects is taken into account, there is a problem that the amount of calculation required for controlling the autonomous driving becomes large.

[0005] Therefore, an object of the present disclosure is to provide an information processing device, an information processing method, and an information processing program that can reduce the amount of calculations when controlling the driving of a vehicle by taking into account each pattern of the movement of surrounding objects. [Means for solving the problem]

[0006] The information processing device disclosed herein includes a first processor that outputs point information that captures an object as a point from an image of the object photographed by a first camera mounted on a vehicle, a second processor that outputs identification information that identifies the object photographed from an image of the object photographed by a second camera mounted on the vehicle and facing in a direction corresponding to the first camera, and a third processor that matches the point information output from the first processor and the identification information output from the second processor, and the third processor further controls driving of the vehicle based on a result of a simulation of the movement of the object on a digital map on which the matched point information and identification information are plotted.

[0007] In addition, the first processor of the information processing device disclosed herein outputs the point information including movement information indicating the movement of the object, and in the simulation, a collision risk between the vehicle and the object is predicted using each movement pattern of the object determined according to the movement information included in the point information and the identification information, and the third processor performs driving control of the vehicle based on the collision risk predicted in the simulation.

[0008] In addition, the third processor of the information processing device disclosed herein controls the driving of the vehicle so as to avoid a collision in which the collision risk predicted in the simulation is equal to or greater than a threshold, and when all of the collision risk predicted in the simulation are less than a threshold, controls the driving of the vehicle so as to prevent following vehicles from getting stuck in a traffic jam.

[0009] Also, in the information processing device of the present disclosure, in the simulation, a collision risk between the vehicle and the object is predicted using a point representing the object or a polygon surrounding a contour of the object.

[0010] The information processing method disclosed herein includes a computer that outputs point information that captures an object as a point from an image of the object captured by a first camera mounted on a vehicle, outputs identification information that identifies the captured object from an image of the object captured by a second camera mounted on the vehicle and facing in a direction corresponding to the first camera, matches the output point information with the output identification information, and executes a process of controlling the driving of the vehicle based on the results of a simulation of the movement of the object on a digital map on which the matched point information and identification information are plotted.

[0011] The information processing program disclosed herein causes a computer to execute a process of outputting point information that captures an object as a point from an image of the object photographed by a first camera mounted on a vehicle, outputting identification information that identifies the photographed object from an image of the object photographed by a second camera mounted on the vehicle and facing in a direction corresponding to the first camera, matching the output point information with the output identification information, and performing a process of controlling the driving of the vehicle based on the results of a simulation of the movement of the object on a digital map on which the matched point information and identification information are plotted.

[0012] The above summary of the invention does not list all of the necessary features of the present invention. Also, subcombinations of these features may also be inventions. [Brief description of the drawings]

[0013] [Figure 1A] FIG. 1 is a schematic diagram illustrating an example of an information processing system. [Figure 1B] FIG. 1 is a schematic diagram showing an example of a vehicle equipped with a Central Brain. [Figure 2A] 1 is a first block diagram illustrating an example of a configuration of an information processing device. [Figure 2B] 11 is a flowchart showing an example of a processing flow of a Central Brain of the information processing device. [Figure 2C]13 is a flowchart illustrating an example of a server flow. [Diagram 3] FIG. 2 is a second block diagram showing an example of the configuration of the information processing device. [Figure 4] FIG. 11 is an explanatory diagram showing an example of point information output by the MoPU. [Diagram 5] FIG. 13 is a third block diagram showing an example of the configuration of the information processing device. [Figure 6] FIG. 4 is a fourth block diagram showing an example of the configuration of an information processing device. [Figure 7] FIG. 11 is an explanatory diagram showing an example of association between point information and label information. [Figure 8] FIG. 1 is an explanatory diagram showing a schematic configuration of a vehicle; [Figure 9] 2 is a block diagram showing an example of a functional configuration of a cooling execution device. FIG. [Figure 10] FIG. 5 is a fifth block diagram showing an example of the configuration of an information processing device. [Figure 11] FIG. 6 is a sixth block diagram showing an example of the configuration of an information processing device. [Figure 12] FIG. 1 is a diagram illustrating detection of coordinates of an object in a time series; [Figure 13] FIG. 7 is a seventh block diagram showing an example of the configuration of an information processing device. [Figure 14] FIG. 2 is an explanatory diagram for explaining an image of an object captured by an event camera. [Figure 15] FIG. 2 is an explanatory diagram illustrating an example of a hardware configuration of a computer that functions as an information processing device or a cooling execution device. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0014] Hereinafter, embodiments of the present disclosure will be described, but the following embodiments do not limit the invention according to the claims. Furthermore, not all of the combinations of features described in the embodiments are necessarily essential to the solution of the invention.

[0015] (First embodiment) First, a first embodiment according to the present disclosure will be described. As an example, at least a part of the information processing device according to the present disclosure is mounted on a vehicle 100 to perform autonomous driving control of the vehicle 100. In addition, the information processing device can provide a driving system that can realize autonomous driving in real time based on data obtained by Level 6 AI / multivariate analysis / goal seeking / strategy planning / optimal probability solution / optimal speed solution / optimal course management / multiple sensor inputs at the edge, and is adjusted based on a delta optimal solution. The vehicle 100 is an example of an "object."

[0016] Here, "Level 6" refers to a level of autonomous driving, which is a level higher than Level 5, which represents fully autonomous driving. Although Level 5 represents fully autonomous driving, it is at the same level as a human driving, and there is still a possibility of accidents occurring. Level 6 represents a level higher than Level 5, which means that the probability of accidents occurring is lower than that of Level 5.

[0017] The computing power at Level 6 is about 1000 times that of Level 5. Therefore, it is possible to realize high-performance operation control that was not possible at Level 5.

[0018] 1A is a schematic diagram showing an example of an information processing system 1000. The information processing system 1000 includes a vehicle 100 equipped with a Central Brain 15, and a server 200, and the Central Brain 15 and the server 200 are connected via a network N.

[0019] FIG. 1B is a schematic diagram showing an example of a vehicle 100 equipped with a Central Brain 15. A plurality of Gateways are communicatively connected to the Central Brain 15. The Central Brain 15 is connected to an external cloud server via the Gateways. The Central Brain 15 is configured to be able to access the external cloud server via the Gateways. On the other hand, due to the presence of the Gateways, the Central Brain 15 is configured not to be directly accessible from the outside.

[0020] The Central Brain 15 outputs a request signal to the cloud server every time a predetermined time has elapsed. Specifically, the Central Brain 15 outputs a request signal representing an inquiry to the cloud server every billionth of a second. As an example, the Central Brain 15 controls Level 6 autonomous driving based on multiple pieces of information acquired via the Gateway.

[0021] 2A is a first block diagram showing an example of the configuration of an information processing device 10. The information processing device 10 includes an IPU (Image Processing Unit) 11, a MoPU (Motion Processing Unit) 12, a Central Brain 15, and a memory 16. The Central Brain 15 includes a GNPU (Graphics Neural network Processing Unit) 13 and a CPU (Central Processing Unit) 14.

[0022] The IPU 11 is built in an ultra-high resolution camera (not shown) installed in the vehicle 100. The IPU 11 performs predetermined image processing such as Bayer transformation, demosaic, noise removal, and sharpening on an image of an object existing around the vehicle 100 captured by the ultra-high resolution camera, and outputs the processed image of the object at a frame rate of 10 frames / second and a resolution of 12 million pixels, for example. The IPU 11 also outputs identification information that identifies the captured object from the image of the object captured by the ultra-high resolution camera. The identification information is information required to identify what the captured object is (for example, whether it is a person or an obstacle). In this embodiment, the IPU 11 outputs label information indicating the type of the captured object (for example, information indicating whether the captured object is a dog, a cat, or a bear) as the identification information. Furthermore, the IPU 11 outputs position information indicating the position of the captured object in the camera coordinate system of the ultra-high resolution camera. The image, label information, and position information output from the IPU 11 are supplied to the Central Brain 15 and the memory 16. The IPU 11 is an example of a "second processor," and the ultra-high resolution camera is an example of a "second camera."

[0023] The MoPU 12 is built in a separate camera (not shown) other than the super-high resolution camera installed in the vehicle 100. The MoPU 12 outputs point information capturing the captured object as a point from an image of the object captured at a frame rate of 100 frames / second or more by a separate camera facing a direction corresponding to the super-high resolution camera, for example, at a frame rate of 100 frames / second or more. The point information output from the MoPU 12 is supplied to the Central Brain 15 and the memory 16. In this way, the image used by the MoPU 12 to output the point information and the image used by the IPU 11 to output the identification information are images captured by the separate camera and the super-high resolution camera facing in corresponding directions. Here, the "corresponding direction" is the direction in which the shooting range of the separate camera and the shooting range of the super-high resolution camera overlap. In the above case, the separate camera photographs the object facing in a direction overlapping with the shooting range of the super-high resolution camera. Note that the super-high resolution camera and the separate camera photograph the object facing in corresponding directions, for example, by previously determining the correspondence relationship of the camera coordinate system between the super-high resolution camera and the separate camera.

[0024] For example, the MoPU 12 outputs, as point information, coordinate values ​​of at least two coordinate axes in a three-dimensional orthogonal coordinate system of a point indicating the location of an object. As an example, the coordinate values ​​indicate the center point (or center of gravity) of the object. Furthermore, the MoPU 12 outputs, as the coordinate values ​​of the two coordinate axes, a coordinate value of an axis (x-axis) along the width direction in the three-dimensional orthogonal coordinate system (hereinafter referred to as the "x-coordinate value") and a coordinate value of an axis (y-axis) along the height direction in the three-dimensional orthogonal coordinate system (hereinafter referred to as the "y-coordinate value"). Note that the x-axis is an axis along the width direction of the vehicle 100, and the y-axis is an axis along the height direction of the vehicle 100.

[0025] With the above configuration, the point information for one second output by the MoPU 12 includes x-coordinate values ​​and y-coordinate values ​​for 100 frames or more, so that the movement (movement direction and movement speed) of an object on the x-axis and y-axis in the above-mentioned three-dimensional orthogonal coordinate system can be grasped based on the point information. In other words, the point information output by the MoPU 12 includes position information indicating the position of the object in the above-mentioned three-dimensional orthogonal coordinate system and movement information indicating the movement of the object.

[0026] As described above, the point information output from MoPU 12 does not include information necessary to identify what the photographed object is (for example, whether it is a person or an obstacle), but only includes information indicating the movement (direction and speed of movement) of the center point (or center of gravity) of the object on the x-axis and y-axis. Furthermore, because the point information output from MoPU 12 does not include image information, the amount of data output to Central Brain 15 and memory 16 can be dramatically reduced. MoPU 12 is an example of a "first processor," and the separate camera is an example of a "first camera."

[0027] As described above, in this embodiment, the frame rate of the separate camera incorporating the MoPU 12 is higher than the frame rate of the ultra-high resolution camera incorporating the IPU 11. Specifically, the frame rate of the separate camera is 100 frames / second or higher, and the frame rate of the ultra-high resolution camera is 10 frames / second. In other words, the frame rate of the separate camera is 10 times or higher than the frame rate of the ultra-high resolution camera.

[0028] The Central Brain 15 associates the point information output from the MoPU 12 with the label information output from the IPU 11. For example, due to the difference in frame rate between the separate camera and the super-high resolution camera, there is a state in which the Central Brain 15 acquires point information about an object but does not acquire label information. In this state, the Central Brain 15 recognizes the x and y coordinate values ​​of the object based on the point information, but does not recognize what the object is.

[0029] Thereafter, when label information about the object is acquired, the Central Brain 15 derives the type of the label information (e.g., PERSON). The Central Brain 15 then associates the label information with the point information acquired above. As a result, the Central Brain 15 recognizes the x-coordinate value and the y-coordinate value of the object based on the point information, and recognizes what the object is. The Central Brain 15 is an example of a "third processor."

[0030] Here, when there are a plurality of objects, such as object A and object B, photographed by the super-high resolution camera and another camera, the Central Brain 15 associates point information and label information for each object as follows. Due to the difference in frame rate between the other camera and the super-high resolution camera, the Central Brain 15 acquires point information for object A and object B (hereinafter referred to as "point information A" and "point information B"), but there is a state in which it does not acquire label information. In this state, the Central Brain 15 recognizes the x-coordinate value and y-coordinate value of object A based on point information A, and recognizes the x-coordinate value and y-coordinate value of object B based on point information B, but does not recognize what those objects are.

[0031] Thereafter, when a piece of label information is acquired, the Central Brain 15 derives the type (e.g., PERSON) of the piece of label information. Then, the Central Brain 15 specifies point information to be associated with the piece of label information based on the position information output from the IPU 11 together with the piece of label information and the position information included in the acquired point information A and point information B. For example, the Central Brain 15 specifies point information including position information indicating a position closest to the position of the object indicated by the position information output from the IPU 11, and associates the point information with the piece of label information. When the point information specified above is point information A, the Central Brain 15 associates the piece of label information with the point information A, recognizes the x-coordinate value and the y-coordinate value of the object A based on the point information A, and recognizes what the object A is.

[0032] As described above, when there are multiple objects photographed by the ultra-high resolution camera and another camera, the Central Brain 15 matches the point information with the label information based on the position information output from the IPU 11 and the position information contained in the point information output from the MoPU 12.

[0033] Furthermore, the Central Brain 15 recognizes objects (people, animals, roads, traffic lights, signs, crosswalks, obstacles, buildings, etc.) present around the vehicle 100 based on the image and label information output from the IPU 11. Furthermore, the Central Brain 15 recognizes the position and movement of recognized objects present around the vehicle 100 based on the point information output from the MoPU 12. Based on the recognized information, the Central Brain 15 performs, for example, control of the motor driving the wheels (speed control), brake control, and steering control to control the automatic driving of the vehicle 100. For example, the Central Brain 15 controls the automatic driving of the vehicle 100 so as to avoid collision with objects based on the position information and movement information included in the point information output from the MoPU 12. In the Central Brain 15, the GNPU 13 may be responsible for processing related to image recognition, and the CPU 14 may be responsible for processing related to vehicle control.

[0034] In general, ultra-high resolution cameras are used to perform image recognition in autonomous driving. Here, it is possible to recognize what objects are included in an image captured by an ultra-high resolution camera. However, this alone is not sufficient for autonomous driving in the Level 6 era. In the Level 6 era, it is also necessary to recognize the movement of objects with higher accuracy. By recognizing the movement of objects with higher accuracy using the MoPU 12, for example, the vehicle 100 traveling by autonomous driving can perform an avoidance operation to avoid an obstacle with higher accuracy. However, an ultra-high resolution camera can only acquire images at about 10 frames per second, and the accuracy of analyzing the movement of an object is lower than that of a camera equipped with the MoPU 12. On the other hand, a camera equipped with the MoPU 12 can output at a high frame rate of, for example, 100 frames / second.

[0035] Therefore, the information processing device 10 according to the first embodiment includes two independent processors, an IPU 11 and a MoPU 12. In the information processing device 10, the IPU 11 built in the super-high resolution camera is given the role of acquiring information necessary to identify what the photographed object is, and the MoPU 12 built in another camera is given the role of detecting the position and movement of the object. The MoPU 12 captures the photographed object as a point, and analyzes in which direction and at what speed the coordinates of the point move on at least the x-axis and y-axis in the above-mentioned three-dimensional orthogonal coordinate system. Since the entire contour of the object and what the object is can be detected by the image from the super-high resolution camera, for example, if the MoPU 12 knows how the center point of the object moves, it is possible to know how the entire object behaves.

[0036] According to a method of analyzing only the movement and speed of the center point of an object, it is possible to significantly suppress the amount of data output to the Central Brain 15 and significantly reduce the amount of calculation in the Central Brain 15, compared to determining how the entire image of the object moves. For example, when an image of 1000 pixels x 1000 pixels is output to the Central Brain 15 at a frame rate of 1000 frames / second, including color information, 4 billion bits / second of data will be output to the Central Brain 15. By the MoPU 12 outputting only point information indicating the movement of the center point of the object, the amount of data output to the Central Brain 15 can be compressed to 20,000 bits / second. In other words, the amount of data output to the Central Brain 15 is compressed to 1 / 200,000.

[0037] In this way, by combining the low frame rate, high resolution image and label information output from the IPU 11 with the high frame rate, lightweight point information output from the MoPU 12, it is possible to achieve object recognition, including object movement, with a small amount of data.

[0038] In addition, in the information processing device 10, the Central Brain 15 associates the point information output from the MoPU 12 with the label information output from the IPU 11, thereby making it possible to grasp information regarding what kind of object is moving in what way.

[0039] <Driving control for autonomous driving> In this embodiment, a high performance digital map is used in the simulation for driving control.

[0040] The MoPU 12 detects objects on the front, rear, left and right sides of the actual vehicle 2000 times per second, and extracts object movement information. The MoPU 12 also obtains label information indicating the results of identifying the types of objects, animals, pedestrians, etc. on the front, rear, left and right sides of the vehicle 100 from the IPU 11. The server 200 plots the results of associating the obtained object movement information with the label information on a digital map.

[0041] The server 200 grasps the characteristics of the object from the label information, and simulates how the object will move next based on the object's movement information, taking into account the characteristics of the object. Here, the characteristics of the object are, for example, characteristics of a cat, a human, a bird, or the characteristics of the other vehicle, and it is possible to infer to some extent how the other object will move from the characteristics of the other object. In other words, if the other object is a cat, it can be inferred that it moves at an extremely fast speed, but its speed is at most 48 lm / h, and its range of movement is also limited, and if the other object is a bird, it can be assumed that the direction of movement is directly above.

[0042] Specifically, it vectorizes the movement information, understands the characteristics of the object, and grasps all the possibilities of what the object will do next. For example, it infers all the possibilities, such as moving 30 cm to the right at 2 m / s, jumping, crouching, moving 50 cm diagonally forward to the left at 1 m / s, etc.

[0043] Then, from the inference results of the movements of the vehicle 100 and all objects in front, behind, left and right of the vehicle 100, all events that indicate what will happen next are inferred. For example, multiple events are predicted, such as the vehicle 100 colliding with another vehicle, the vehicle 100 hitting a roadside tree, and the vehicle 100 avoiding contact with an oncoming bicycle by just 1 mm. Damage at that time is also predicted. This inference may be made in hundreds or even trillions of ways.

[0044] The driving control of the vehicle 100 is performed so as to avoid a serious collision event among the inferred events. Alternatively, if a serious collision event does not occur, the driving control is performed so as to prevent the following vehicles from being congested. In this way, optimal driving control is performed according to the surrounding situation.

[0045] Furthermore, in the inference of object movement and event inference, the amount of calculation increases because all possibilities are inferred. Therefore, in this embodiment, the extracted movement information is used as the movement of the object. In other words, instead of using all the image processing information, the amount of data used for inference is minimized by using polygons (wire frames) that include the points and contours detected by the MoPU 12.

[0046] As described above, the Central Brain 15 performs driving control for the automatic driving of the vehicle 100. At this time, the Central Brain 15 transmits the associated point information and label information, and a driving control sequence for a predetermined time ahead to the server 200. The driving control sequence is time-series data of a combination of steering amount, access operation amount, and brake operation amount.

[0047] The server 200 simulates the movement of the object on a digital map on which the associated point information and label information are plotted. In the simulation, the server 200 predicts the collision risk between the vehicle 100 and the object by using each movement pattern of the object determined according to the movement information and label information included in the point information.

[0048] In the simulation, the server 200 predicts the collision risk between the vehicle and the object using points representing the object or a polygon surrounding the outline of the object. The collision risk indicates the possibility of a collision and the severity of the collision.

[0049] The Central Brain 15 performs driving control of the vehicle 100 based on the results of simulating the movement of objects on a digital map on which the associated point information and label information are plotted.

[0050] Specifically, the Central Brain 15 controls the driving of the vehicle 100 based on the collision risk predicted by the simulation.

[0051] More specifically, the Central Brain 15 controls the driving of the vehicle 100 so as to avoid a collision in which the collision risk predicted by the simulation is equal to or greater than a threshold value. When all of the collision risk predicted by the simulation is less than a threshold value, the Central Brain 15 controls the driving of the vehicle 100 so as to prevent a traffic jam for the following vehicles.

[0052] Next, the flow of processing in the information processing system 1000 in automatic driving control will be described. First, the flow of processing in the Central Brain 15 will be described with reference to Fig. 2B. This processing is repeatedly executed at predetermined intervals.

[0053] First, in step S100, the Central Brain 15 acquires point information including movement information from the MoPU 12.

[0054] In step S102, the Central Brain 15 acquires, from the IPU 11, label information that is identification information of an object.

[0055] In step S103, the Central Brain 15 associates the point information acquired in step S100 with the label information acquired in step S102.

[0056] In step S104, the Central Brain 15 generates a driving control sequence for driving toward the destination up to a predetermined time ahead.

[0057] In step S106, the Central Brain 15 transmits the result of the association in step S103, the driving control sequence acquired in step S104, and the position information of the vehicle 100 to the server 200. The position information of the vehicle 100 may be acquired by a GPS sensor (not shown) mounted on the vehicle 100.

[0058] Here, when the server 200 receives the result of associating the point information and the label information, the driving control sequence, and the position information of the vehicle 100 from the Central Brain 15, the server 200 executes a processing routine shown in FIG. 2C.

[0059] In step S120, the server 200 uses the position information of the vehicle 100 to plot the associated point information and label information on a digital map.

[0060] In step S122, the server 200 simulates the movement of the object. At this time, in the simulation, the server 200 calculates the movement of the object using each movement pattern of the object determined according to the movement information and label information included in the point information.

[0061] In step S124, the server 200 predicts the risk of collision between the vehicle 100 and the object for each movement pattern based on the result of simulating the movement of the object.

[0062] In step S126, the server 200 determines whether the collision risk predicted for at least one motion pattern is equal to or greater than a threshold value. If the collision risk predicted for at least one motion pattern is equal to or greater than the threshold value, the process proceeds to step S128. On the other hand, if the collision risk predicted for all motion patterns is less than the threshold value, the process proceeds to step S130.

[0063] In step S128, the server 200 corrects the driving control sequence so that the vehicle 100 does not collide with an object in a movement pattern in which the collision risk is equal to or greater than a threshold. The correction of the driving control sequence may be performed using a learned model. The model may be learned using teacher data that is a combination of a movement pattern in which the collision risk is equal to or greater than a threshold and a correct driving control sequence (driving control sequence for manual driving) for preventing the vehicle 100 from colliding with an object at that time. The model is also learned using teacher data at level 2, which represents automatic driving. Specifically, the model is learned so as to be updated using the difference between the driving control sequence obtained by the model and the driving control sequence for manual driving. The model is also learned using teacher data at level 3, which represents automatic driving, and teacher data at level 4, which represents automatic driving, in that order.

[0064] In step S130, the server 200 corrects the driving control sequence so as to avoid congestion in the following vehicle. This correction of the driving control sequence may be performed using a trained model. This model may be trained using teacher data that is a combination of point information of the following vehicle and a correct driving control sequence (a driving control sequence for manual driving) for preventing congestion in the following vehicle at this time.

[0065] In step S132, the driving control sequence corrected in step S128 or S130 is transmitted to the Central Brain 15.

[0066] Then, in step S108 of FIG. 2B, the Central Brain 15 acquires the driving control sequence transmitted from the server 200.

[0067] In step S110, the Central Brain 15 performs driving control of the vehicle 100 using the acquired driving control sequence.

[0068] Although the simulation is performed by the server 200 in the above example, the simulation may be performed by the Central Brain 15. Also, the generation of a driving control sequence for a predetermined time ahead when traveling toward a destination is performed by the Central Brain 15 in the above example, but the generation may be performed by the server 200.

[0069] Second Embodiment Next, a second embodiment of the present invention will be described while omitting or simplifying parts that overlap with the above embodiment.

[0070] 3 is a second block diagram showing an example of the configuration of the information processing device 10. As shown in FIG. 3, the information processing device 10 mounted on the vehicle 100 includes an MoPU 12L corresponding to the left eye, an MoPU 12R corresponding to the right eye, an IPU 11, and a Central Brain 15.

[0071] The MoPU12L includes a camera 30L, a radar 32L, an infrared camera 34L, and a core 17L. The MoPU12R includes a camera 30R, a radar 32R, an infrared camera 34R, and a core 17R. In the following, when there is no distinction between the MoPU12L and the MoPU12R, it is referred to as "MoPU12", when there is no distinction between the camera 30L and the camera 30R, it is referred to as "camera 30", when there is no distinction between the radar 32L and the radar 32R, it is referred to as "radar 32", when there is no distinction between the infrared camera 34L and the infrared camera 34R, it is referred to as "infrared camera 34", and when there is no distinction between the core 17L and the core 17R, it is referred to as "core 17".

[0072] The camera 30 of the MoPU 12 captures an object at a frame rate (120, 240, 480, 960, or 1920 frames / sec) greater than the ultra-high resolution camera (e.g., 10 frames / sec) of the IPU 11. The frame rate of the camera 30 is variable. The camera 30 is an example of a "first camera."

[0073] The radar 32 included in the MoPU 12 acquires a radar signal that is a signal based on electromagnetic waves irradiated onto an object and reflected from the object. The infrared camera 34 included in the MoPU 12 is a camera that captures infrared images.

[0074] Core 17 (e.g., composed of one or more CPUs) included in MoPU 12 extracts feature points for each frame of image captured by camera 30, and outputs the x-coordinate value and the y-coordinate value of the object in the above-mentioned three-dimensional orthogonal coordinate system as point information. Core 17 takes, for example, the center point (center of gravity) of the object extracted from the image as the feature point. Note that the point information output by core 17 includes position information and movement information, similar to the above embodiment.

[0075] IPU11 is equipped with an ultra-high resolution camera (not shown) and outputs an image of an object captured by the ultra-high resolution camera, label information indicating the type of the object, and position information indicating the position of the object in the camera coordinate system of the ultra-high resolution camera.

[0076] The Central Brain 15 acquires the point information output from the MoPU 12, and the image, label information, and position information output from the IPU 11. The Central Brain 15 then associates the point information with label information about an object that exists at a position corresponding to the position information included in the point information output from the MoPU 12 and the position information output from the IPU 11. This enables the information processing device 10 to associate information about the object indicated by the label information with the position and movement of the object indicated by the point information.

[0077] Here, the MoPU 12 changes the frame rate of the camera 30 in response to a predetermined factor. In this embodiment, the MoPU 12 changes the frame rate of the camera 30 in response to a score related to the external environment, which is an example of the predetermined factor. In this case, the MoPU 12 calculates a score related to the external environment for the vehicle 100, and changes the frame rate of the camera 30 in response to the calculated score. Then, the MoPU 12 outputs a control signal to the camera 30 to cause the camera 30 to capture images at the changed frame rate. As a result, the camera 30 captures images at the frame rate indicated by the control signal. With this configuration, the information processing device 10 can capture images of an object at a frame rate suitable for the external environment.

[0078] The information processing device 10 mounted on the vehicle 100 includes a plurality of types of sensors (not shown). The MoPU 12 calculates a degree of danger regarding the movement of the vehicle 100 as a score regarding the external environment for the vehicle 100 based on the sensor information (e.g., shift of the center of gravity of the body weight, detection of the road material, detection of the outside air temperature, detection of the outside air humidity, detection of the up, down, side, diagonal inclination angle of the slope, detection of the road freezing condition, detection of the amount of moisture, detection of the material of each tire, wear condition, detection of the air pressure, road width, presence or absence of overtaking prohibition, oncoming vehicles, vehicle type information of the front and rear vehicles, cruising state of those vehicles, or surrounding conditions (birds, animals, soccer balls, wrecked vehicles, earthquakes, housework, wind, typhoon, heavy rain, light rain, snowstorm, fog, etc.)) and point information acquired from the plurality of types of sensors. The degree of danger indicates the degree of danger in a place where the vehicle 100 will be traveling in the future. In this case, the MoPU 12 changes the frame rate of the camera 30 according to the calculated degree of danger. The vehicle 100 is an example of a "moving body". With this configuration, the information processing device 10 can change the frame rate of the camera 30 depending on the degree of danger regarding the movement of the vehicle 100. The sensor is an example of a "detection unit", and the sensor information is an example of "detection information".

[0079] For example, the MoPU 12 increases the frame rate of the camera 30 as the calculated risk level increases. When the calculated risk level is less than a first threshold, the MoPU 12 changes the frame rate of the camera 30 to 120 frames / second. When the calculated risk level is equal to or greater than the first threshold and less than a second threshold, the MoPU 12 changes the frame rate of the camera 30 to one of 240, 480, or 960 frames / second. When the calculated risk level is equal to or greater than the second threshold, the MoPU 12 changes the frame rate of the camera 30 to 1920 frames / second. When the calculated risk level is equal to or greater than the second threshold, the MoPU 12 may output a control signal to the radar 32 and the infrared camera 34 so as to obtain a radar signal and capture an infrared image at a numerical value corresponding to the frame rate, in addition to causing the camera 30 to capture an image at the selected frame rate when the risk level is one of the above levels.

[0080] For example, the MoPU 12 decreases the frame rate of the camera 30 as the calculated risk level decreases. When the frame rate of the camera 30 is set to 1920 frames / sec and the calculated risk level is equal to or greater than the first threshold and less than the second threshold, the MoPU 12 changes the frame rate of the camera 30 to one of 240, 480, and 960 frames / sec. When the frame rate of the camera 30 is set to 1920 frames / sec and the calculated risk level is less than the first threshold, the MoPU 12 changes the frame rate of the camera 30 to 120 frames / sec. When the frame rate of the camera 30 is set to 240, 480, and 960 frames / sec and the calculated risk level is less than the first threshold, the MoPU 12 changes the frame rate of the camera 30 to 120 frames / sec. When the frame rate of the camera 30 is set to 240, 480, and 960 frames / sec and the calculated risk level is less than the first threshold, the MoPU 12 changes the frame rate of the camera 30 to 120 frames / sec. In this case, similarly to the above, a control signal may be output to the radar 32 and the infrared camera 34 so as to acquire a radar signal and capture an infrared image with values ​​according to the changed frame rate of the camera 30.

[0081] In addition, MoPU12 may calculate the degree of danger by using big data regarding driving that is known before vehicle 100 starts driving, such as long-tail incident AI (Artificial Intelligence) DATA (e.g., Trip data from a vehicle equipped with a level 5 autonomous driving control system) or map information as information to predict the degree of danger.

[0082] In the above, the degree of danger is calculated as a score related to the external environment, but the index that becomes the score related to the external environment is not limited to the degree of danger. For example, the MoPU 12 may calculate a score related to the external environment other than the degree of danger based on the moving direction or speed of an object captured by the camera 30, and change the frame rate of the camera 30 according to the score. Hereinafter, a case will be described in which the MoPU 12 calculates a speed score that is a score related to the speed of an object captured by the camera 30, and changes the frame rate of the camera 30 according to the speed score. As an example, the speed score is set to be higher as the speed of the object is faster and lower as the speed of the object is slower. The MoPU 12 increases the frame rate of the camera 30 as the calculated speed score is higher and decreases the frame rate of the camera 30 as the calculated speed score is lower. For this reason, when the calculated speed score is equal to or higher than a threshold value because the speed of the object is fast, the MoPU 12 changes the frame rate of the camera 30 to 1920 frames / second. Furthermore, if the speed score calculated is less than the threshold because the object is moving slowly, the MoPU 12 changes the frame rate of the camera 30 to 120 frames / second. Note that, in this case, similarly to the above, a control signal may be output to the radar 32 and the infrared camera 34 so that the radar signal is acquired and the infrared image is captured at a value corresponding to the changed frame rate of the camera 30.

[0083] Next, a case will be described in which the MoPU 12 calculates a direction score, which is a score related to the moving direction of an object captured by the camera 30, and changes the frame rate of the camera 30 according to the direction score. As an example, the direction score is set so that it is higher when the moving direction of the object is a direction approaching the road and lower when the moving direction is a direction away from the road. The MoPU 12 increases the frame rate of the camera 30 as the calculated direction score increases, and decreases the frame rate of the camera 30 as the calculated direction score decreases. Specifically, the MoPU 12 identifies the moving direction of the object by using AI or the like, and calculates the direction score based on the identified moving direction. The MoPU 12 changes the frame rate of the camera 30 to 1920 frames / second when the calculated direction score is equal to or greater than a threshold because the moving direction of the object is a direction approaching the road. The MoPU 12 changes the frame rate of the camera 30 to 120 frames / second when the calculated direction score is less than a threshold because the moving direction of the object is a direction away from the road. In this case, similarly to the above, a control signal may be output to the radar 32 and the infrared camera 34 so as to acquire a radar signal and capture an infrared image with values ​​according to the changed frame rate of the camera 30.

[0084] Furthermore, the MoPU 12 may output point information only for objects whose calculated score related to the external environment is equal to or greater than a predetermined threshold. In this case, for example, the MoPU 12 may determine whether or not to output point information for an object according to the moving direction of the object captured by the camera 30. For example, the MoPU 12 may not output point information for an object that has a low impact on the traveling of the vehicle 100. Specifically, the MoPU 12 calculates the moving direction of an object captured by the camera 30, and does not output point information for an object such as a pedestrian moving away from the road. On the other hand, the MoPU 12 outputs point information for an object approaching the road (for example, an object such as a pedestrian that is about to jump out onto the road). With this configuration, the information processing device 10 does not need to output point information for an object that has a low impact on the traveling of the vehicle 100.

[0085] In addition, in the above description, the case where the MoPU 12 calculates the risk level has been exemplified, but the disclosed technology is not limited to this aspect. For example, the Central Brain 15 may calculate the risk level instead of the MoPU 12. In this case, the Central Brain 15 calculates the risk level regarding the movement of the vehicle 100 as a score regarding the external environment for the vehicle 100 based on the sensor information acquired from multiple types of sensors and the point information output from the MoPU 12. Then, the Central Brain 15 outputs an instruction to the MoPU 12 to change the frame rate of the camera 30 according to the calculated risk level.

[0086] In the above description, the MoPU 12 outputs point information based on an image captured by the camera 30, but the disclosed technology is not limited to this embodiment. For example, the MoPU 12 may output point information based on a radar signal and an infrared image instead of an image captured by the camera 30. The MoPU 12 can derive the x-coordinate value and the y-coordinate value of an object from the infrared image of the object captured by the infrared camera 34 in the same manner as the image captured by the camera 30. The radar 32 can acquire three-dimensional point cloud data of the object based on the radar signal. That is, the radar 32 can detect the coordinate of the z-axis in the above three-dimensional orthogonal coordinate system. Here, the z-axis is an axis along the depth direction of the object and the traveling direction of the vehicle 100, and the coordinate value of the z-axis will be referred to as the "z-coordinate value" below. In this case, the MoPU 12 utilizes the principle of a stereo camera to combine the x- and y-coordinate values ​​of the object photographed by the infrared camera 34 at the same timing as the radar 32 acquires the 3D point cloud data of the object with the z-coordinate value of the object indicated by the 3D point cloud data, and derives the coordinate values ​​of the three coordinate axes (x-axis, y-axis, and z-axis) of the object as point information.The MoPU 12 then outputs the derived point information to the Central Brain 15.

[0087] In addition, in the above description, the case where the MoPU 12 derives the point information has been exemplified, but the disclosed technology is not limited to this embodiment. For example, the Central Brain 15 may derive the point information instead of the MoPU 12. The Central Brain 15 derives the point information by combining information detected by, for example, the camera 30L, the camera 30R, the radar 32, and the infrared camera 34. As a specific example, the Central Brain 15 derives the coordinate values ​​of the three coordinate axes (x-axis, y-axis, and z-axis) of the object as point information by performing triangulation based on the x-coordinate value and the y-coordinate value of the object photographed by the camera 30L and the x-coordinate value and the y-coordinate value of the object photographed by the camera 30R.

[0088] In the above description, the Central Brain 15 controls the automatic driving of the vehicle 100 based on the image and label information output from the IPU 11 and the point information output from the MoPU 12, but the disclosed technology is not limited to this embodiment. For example, the Central Brain 15 may control the operation of a robot based on the above information output from the IPU 11 and the MoPU 12. The robot may be a humanoid smart robot that performs work in place of a human. In this case, the Central Brain 15 controls the operation of the arms, palms, fingers, feet, etc. of the robot based on the above information output from the IPU 11 and the MoPU 12, and causes the robot to perform operations such as grasping, grabbing, holding, carrying, moving, carrying, throwing, kicking, and avoiding an object. When the Central Brain 15 controls the operation of the robot, the IPU 11 and the MoPU 12 may be mounted at the positions of the right and left eyes of the robot. That is, the right eye may be equipped with an IPU 11 and MoPU 12 for the right eye, and the left eye may be equipped with an IPU 11 and MoPU 12 for the left eye.

[0089] (Third embodiment) Next, a third embodiment according to the present invention will be described while omitting or simplifying parts that overlap with the above-described embodiments. As an example, the information processing device 10 according to the third embodiment has the same configuration as that of the first embodiment shown in FIG.

[0090] The MoPU 12 according to the third embodiment outputs, as point information, the coordinate values ​​of at least two diagonal vertices of a polygon surrounding the contour of an object recognized from an image captured by another camera. The coordinate values ​​are the x and y coordinate values ​​of the object in the above-mentioned three-dimensional orthogonal coordinate system, as in the first embodiment.

[0091] Fig. 4 is an explanatory diagram showing an example of point information output by the MoPU 12. In Fig. 4, bounding boxes 21, 22, 23, and 24 that surround the contours of four objects included in an image captured by another camera are shown. Fig. 4 also shows an example in which the MoPU 12 outputs, as point information, the coordinate values ​​of two diagonal points of the vertices of the rectangular bounding boxes 21, 22, 23, and 24 that surround the contours of the objects. In this way, the MoPU 12 may capture objects as objects having a certain size, rather than as points.

[0092] Furthermore, when an object is recognized as an object having a certain size, the MoPU 12 may output, as point information, the coordinate values ​​of multiple vertices of a polygon surrounding the contour of the object, instead of the coordinate values ​​of two diagonal points of the vertices of the polygon surrounding the contour of the object recognized from an image captured by another camera. For example, in the example of Fig. 4, the MoPU 12 may output, as point information, the coordinate values ​​of all four vertices of bounding boxes 21, 22, 23, and 24 that surround the contour of the object in a rectangular shape.

[0093] (Fourth embodiment) Next, a fourth embodiment of the present invention will be described while omitting or simplifying parts that overlap with the above-mentioned embodiments. As an example, the information processing device 10 according to the fourth embodiment has the same configuration as that of the first embodiment shown in FIG.

[0094] The vehicle 100 on which the information processing device 10 according to the fourth embodiment is mounted is equipped with sensors consisting of at least one of radar, LiDAR, high-pixel, telephoto, ultra-wide-angle, 360-degree, high-performance cameras, vision sensors, sound sensors, ultrasonic sensors, vibration sensors, infrared sensors, ultraviolet sensors, radio wave sensors, temperature sensors, and humidity sensors. Examples of sensor information that the information processing device 10 receives from the sensors include shift of the center of gravity of body weight, detection of road material, detection of outside air temperature, detection of outside air humidity, detection of up, down, side, and diagonal inclination angles of a slope, how the road is frozen, detection of moisture content, material of each tire, wear status, detection of air pressure, road width, presence or absence of overtaking prohibition, oncoming vehicles, vehicle type information of front and rear vehicles, cruising status of those vehicles, surrounding conditions (birds, animals, soccer balls, wrecked vehicles, earthquakes, fires, wind, typhoons, heavy rain, light rain, snowstorms, fog, etc.), etc. The sensors are an example of a "detection unit," and the sensor information is an example of "detection information."

[0095] The Central Brain 15 according to the fourth embodiment calculates control variables for controlling the autonomous driving of the vehicle 100 based on sensor information detected by the sensors. The Central Brain 15 acquires sensor information every one billionth of a second. Specifically, the Central Brain 15 calculates control variables for controlling the wheel speed, inclination, and suspension supporting the wheels of each of the four wheels of the vehicle 100. The inclination of the wheels includes both the inclination of the wheels with respect to an axis horizontal to the road and the inclination of the wheels with respect to an axis perpendicular to the road. In this case, the Central Brain 15 calculates a total of 16 control variables for controlling the wheel speed of each of the four wheels, the inclination of each of the four wheels with respect to an axis horizontal to the road, the inclination of each of the four wheels with respect to an axis perpendicular to the road, and the suspension supporting each of the four wheels.

[0096] Then, the Central Brain 15 controls the automatic driving of the vehicle 100 based on the control variables calculated above, the point information output from the MoPU 12, and the label information output from the IPU 11. Specifically, the Central Brain 15 controls the in-wheel motors mounted on the four wheels based on the 16 control variables, thereby controlling the wheel speed and inclination of each of the four wheels of the vehicle 100 and the suspensions supporting each of the four wheels to perform automatic driving. In addition, the Central Brain 15 recognizes the position and movement of an object that is recognized as being present around the vehicle 100 based on the point information and the label information, and controls the automatic driving of the vehicle 100 to avoid, for example, a collision with the object based on the recognized information. By the Central Brain 15 controlling the automatic driving of the vehicle 100 in this way, for example, when the vehicle 100 runs on a mountain road, it can perform optimal steering suited to the mountain road, and when the vehicle 100 is parked in a parking lot, it can run at an optimal angle suited to the parking lot.

[0097] Here, the Central Brain 15 may be capable of inferring control variables from the above sensor information and information obtainable via a network from a server (not shown) or the like, using machine learning, more specifically, deep learning. In other words, the Central Brain 15 may be configured with AI.

[0098] The Central Brain 15 can obtain the control variables by performing a multivariate analysis (see, for example, formula (2)) using the integration method as shown in the following formula (1) using the computational power of the sensor information and long-tail incident AI data for every 1 billionth of a second (hereinafter, also referred to as the "computational power of Level 6") used to realize Level 6. More specifically, while obtaining the integral value of the delta values ​​of various Ultra High Resolutions using the computational power of Level 6, each control variable is obtained at the edge level and in real time, and the result (i.e., each control variable) occurring in the next 1 billionth of a second can be obtained with the highest probability value. To achieve this, for example, an integral value obtained by time integrating the delta value (e.g., a change value over a small time period) of a function that can identify each variable (e.g., the above sensor information and information obtainable via a network) such as air resistance, road resistance, road elements (e.g., garbage), and slip coefficient is input to the deep learning model of the Central Brain 15 (e.g., a learned model obtained by performing deep learning on a neural network). The deep learning model of Central Brain15 outputs the control variable corresponding to the input integral value (for example, the control variable with the highest confidence (i.e., the evaluation value)). The output of the control variable is performed in units of 1 billionth of a second.

[0099]

number

[0100]

number

[0101] As an example, in formula (1), "f(A)" is an equation that expresses a simplified function that indicates the behavior of each variable, such as air resistance, road resistance, road elements (e.g., garbage), and slip coefficient. Also, as an example, formula (1) is an equation that indicates the time integral v of "f(A)" from time a to time b. DL in formula (2) indicates deep learning (e.g., a deep learning model optimized by performing deep learning on a neural network), and dA n / dt indicates the delta value of f(A,B,C,D,...,N), where A,B,C,D,...,N indicate variables such as air resistance, road resistance, road elements (e.g., garbage), and slip coefficient, and f(A,B,C,D,...,N) is a function that indicates the behavior of A,B,C,D,...,N, and V n indicates the value (control variable) output from the deep learning model optimized by performing deep learning on the neural network.

[0102] Note that, although an example is given here in which an integral value obtained by time-integrating the delta value of a function is input to the deep learning model of the Central Brain 15, this is merely one example. For example, an integral value (e.g., a result occurring in the next billionth of a second) obtained by time-integrating the delta value of a function indicating the behavior of each variable such as air resistance, road resistance, road elements, and slip coefficient may be inferred by the deep learning model of the Central Brain 15, and the integral value with the highest degree of certainty (i.e., evaluation value) may be obtained by the Central Brain 15 every billionth of a second as an inference result.

[0103] In addition, although an example of inputting an integral value to a deep learning model and outputting an integral value from a deep learning model is given here, this is merely an example, and the technology of the present disclosure can be realized even without using an integral value. For example, values ​​corresponding to A, B, C, D, ..., N are used as example data, and a value corresponding to at least one control variable (e.g., the result occurring in the next billionth of a second) is used as correct answer data to perform deep learning on a neural network, using teacher data, so that at least one control variable can be inferred by an optimized deep learning model.

[0104] The control variables obtained by Central Brain 15 can be further refined by increasing the number of deep learning rounds. For example, more accurate control variables can be calculated using huge amounts of data such as tires, motor rotation, steering angle, road material, weather, garbage, effects of quadratic deceleration, slips, loss of balance and how to control steering and speed to regain balance, and long-tail incident AI data.

[0105] Fifth embodiment Next, a fifth embodiment according to the present invention will be described while omitting or simplifying parts that overlap with the above-mentioned embodiments. Fig. 5 is a third block diagram showing an example of the configuration of the information processing device 10. Note that Fig. 5 shows only a portion of the configuration of the information processing device 10.

[0106] 5, in the MoPU 12, a visible light image and an infrared image of an object captured by the camera 30 are input to the core 17 at a frame rate of 100 frames / second or more. The camera 30 includes a visible light camera 30A capable of capturing a visible light image of the object and an infrared camera 30B capable of capturing an infrared image of the object. The core 17 then outputs point information to the Central Brain 15 based on at least one of the input visible light image and infrared image.

[0107] Here, when the core 17 can identify an object from a visible light image of the object captured by the visible light camera 30A, it outputs point information based on the visible light image. On the other hand, when the core 17 cannot capture an object from the visible light image due to a predetermined factor, it outputs point information based on an infrared image of the object captured by the infrared camera 30B. For example, it is assumed that the core 17 cannot capture an object from a visible light image due to the influence of darkness as a predetermined factor. In this case, the core 17 detects the heat of the object using the infrared camera 30B, and outputs point information of the object based on the infrared image that is the detection result. However, without being limited to this, the core 17 may output point information based on a visible light image and an infrared image.

[0108] Furthermore, the MoPU 12 synchronizes the timing of capturing visible light images by the visible light camera 30A with the timing of capturing infrared images by the infrared camera 30B. Specifically, the MoPU 12 outputs a control signal to the camera 30 so as to capture visible light images and infrared images at the same timing. This synchronizes the number of images captured per second by the visible light camera 30A with the number of images captured per second by the infrared camera 30B (e.g., 1920 frames / second).

[0109] Sixth embodiment Next, a sixth embodiment according to the present invention will be described while omitting or simplifying parts that overlap with the above-mentioned embodiments. Fig. 6 is a fourth block diagram showing an example of the configuration of the information processing device 10. Note that Fig. 6 shows only a portion of the configuration of the information processing device 10.

[0110] As shown in FIG. 6, in the MoPU 12, an image of an object captured by the camera 30 and a radar signal based on the electromagnetic waves irradiated by the radar 32 and reflected from the object are input to the core 17 at a frame rate of 100 frames / second or more. Then, the core 17 outputs point information to the Central Brain 15 based on the input image of the object and the radar signal. The core 17 can derive the x-coordinate value and the y-coordinate value of the object from the input image of the object. As described above, the radar 32 can acquire three-dimensional point cloud data of the object based on the radar signal and detect the coordinate of the z-axis in the above-mentioned three-dimensional orthogonal coordinate system. In this case, the core 17 utilizes the principle of a stereo camera to combine the x-coordinate value and the y-coordinate value of the object captured by the camera 30 at the same timing as the timing when the radar 32 acquires the three-dimensional point cloud data of the object and the z-coordinate value of the object indicated by the three-dimensional point cloud data, and derive the coordinate values ​​of the three coordinate axes (x-axis, y-axis, and z-axis) of the object as point information. The image of the object input to the core 17 may include at least one of a visible light image and an infrared image.

[0111] Furthermore, the MoPU 12 synchronizes the timing of capturing images by the camera 30 with the timing of acquiring 3D point cloud data of an object based on a radar signal by the radar 32. Specifically, the MoPU 12 outputs control signals to the camera 30 and the radar 32 so that they capture images at the same timing and acquire 3D point cloud data of the object. This synchronizes the number of images captured per second by the camera 30 with the number of pieces of 3D point cloud data acquired per second by the radar 32 (for example, 1920 frames / second). In this way, the number of images captured per second by the camera 30 and the number of pieces of 3D point cloud data acquired per second by the radar 32 are greater than the frame rate of the ultra-high resolution camera included in the IPU 11, i.e., the number of images captured per second by the ultra-high resolution camera.

[0112] Seventh embodiment Next, a seventh embodiment according to the present invention will be described while omitting or simplifying parts that overlap with the above-mentioned embodiments. As an example, the information processing device 10 according to the seventh embodiment has the same configuration as that of the first embodiment shown in FIG.

[0113] The Central Brain 15 according to the seventh embodiment associates the point information output from the MoPU 12 with the label information at the same timing as the IPU 11 outputs the label information. Furthermore, when new point information is output from the MoPU 12 after the point information and the label information are associated, the Central Brain 15 also associates the new point information with the label information. The new point information is point information of the same object as the object indicated by the point information associated with the label information, and is one or more pieces of point information between the time the association is performed and the time the next label information is output. In the seventh embodiment, as in the above embodiment, the frame rate of the separate camera incorporating the MoPU 12 is 100 frames / second or more (for example, 1920 frames / second), and the frame rate of the super-high resolution camera incorporating the IPU 11 is 10 frames / second.

[0114] 7 is an explanatory diagram showing an example of the correspondence between point information and label information. In the following description, the number of pieces of point information output per second from the MoPU 12 is referred to as the "output rate of point information," and the number of pieces of label information output per second from the IPU 11 is referred to as the "output rate of label information."

[0115] 7 shows a time series of the output rate of the point information P4 of the object B14. The output rate of the point information P4 for the object B14 is 1920 frames / second. The point information P4 moves from right to left in the figure. The output rate of the label information for the object B14 is 10 frames / second, which is lower than the output rate of the point information P4.

[0116] First, at time t0, label information about object B14 is not output from the IPU 11. Therefore, at time t0, the Central Brain 15 recognizes the coordinate values ​​(position information) of object B14 based on point information P4, but does not recognize what object B14 is.

[0117] Next, at time t1, label information about object B14 is output from IPU 11. Therefore, the Central Brain 15 derives label information "PERSON" about object B14 based on the label information. Then, the Central Brain 15 associates the label information "PERSON" derived at time t1 with the coordinate values ​​(position information) of point information P4 output from MoPU 12 at time t1. As a result, at time t1, the Central Brain 15 recognizes the coordinate values ​​(position information) of object B14 based on point information P4 and recognizes what object B14 is.

[0118] 7, the timing when the next label information for object B14 is output from the IPU 11 is set to time t2. Therefore, at the time point of time t2, the Central Brain 15 derives label information "PERSON" for object B14 based on the label information output from the IPU 11. Then, the Central Brain 15 associates the label information "PERSON" derived at time t2 with the coordinate values ​​(position information) of the point information P4 output from the MoPU 12 at time t2.

[0119] Here, due to the difference in frame rate between the separate camera in which the MoPU 12 is built and the super-high resolution camera in which the IPU 11 is built, during the period from time t1 to time t2, the Central Brain 15 acquires point information P4 about the object B14, but does not acquire label information. In this case, the Central Brain 15 associates the point information P4 acquired during the period from time t1 to time t2 with the label information "PERSON" associated with the previous time t1 for the point information P4. Here, the point information P4 acquired by the Central Brain 15 during the period from time t1 to time t2 is an example of "new point information". In the example shown in FIG. 7, since multiple pieces of point information P4 are output from the MoPU 12 during the period from time t1 to time t2, the Central Brain 15 acquires multiple pieces of point information P4. Therefore, in the example shown in Fig. 7, the Central Brain 15 associates each of the multiple point information P4 acquired during the period from time t1 to time t2 with the label information "PERSON" associated with the immediately preceding time t1. Note that, unlike the example shown in Fig. 7, when one piece of point information P4 is output from the MoPU 12 during the period from time t1 to time t2, the Central Brain 15 associates the one piece of point information P4 with the label information "PERSON" associated with the immediately preceding time t1.

[0120] Here, even if there is a period in which the type of object being tracked is uncertain, the Central Brain 15 continuously outputs point information of the object at a high frame rate, so there is a low risk of losing track of the coordinate values ​​(position information) of the object. Therefore, once the Central Brain 15 has associated point information with label information, it is possible to presumptively assign the previous label information to point information acquired until the next label information is acquired.

[0121] Eighth embodiment Next, an eighth embodiment according to the present invention will be described while omitting or simplifying parts that overlap with the above-mentioned embodiments. When the information processing device 10 that controls the autonomous driving of the vehicle 100 performs advanced arithmetic processing, heat generation becomes an issue. Therefore, the eighth embodiment provides a vehicle 100 having a cooling function for the information processing device 10.

[0122] Fig. 8 is an explanatory diagram showing a schematic configuration of the vehicle 100. As shown in Fig. 8, the vehicle 100 is equipped with an information processing device 10, a cooling execution device 110, and a cooling section 120.

[0123] The information processing device 10 according to the eighth embodiment is a device that controls the automatic driving of the vehicle 100, and includes, as an example, the configuration shown in FIG. 2 similar to that of the first embodiment. The cooling execution device 110 acquires the object detection result by the information processing device 10, and causes the cooling unit 120 to execute cooling of the information processing device 10 based on the detection result. The cooling unit 120 cools the information processing device 10 using at least one cooling means such as air cooling means, water cooling means, and liquid nitrogen cooling means. In the following, the object to be cooled in the information processing device 10 will be described as the Central Brain 15 (more specifically, the CPU 14 constituting the Central Brain 15) that controls the automatic driving of the vehicle 100, but is not limited thereto.

[0124] The information processing device 10 and the cooling execution device 110 are communicatively connected via a network (not shown). The network may be any of a vehicle network, the Internet, a LAN (Local Area Network), and a mobile communication network. The mobile communication network may be compliant with any of the following communication methods: 5G (5th Generation) communication method, LTE (Long Term Evolution) communication method, 3G (3rd Generation) communication method, and 6G (6th Generation) communication method and later.

[0125] Fig. 9 is a block diagram showing an example of the functional configuration of the cooling execution device 110. As shown in Fig. 9, the cooling execution device 110 has an acquisition unit 112, an execution unit 114, and a prediction unit 116 as its functional configuration.

[0126] The acquisition unit 112 acquires a result of object detection by the information processing device 10. For example, the acquisition unit 112 acquires point information of the object output from the MoPU 12 as the detection result.

[0127] The execution unit 114 executes a central The execution unit 114 causes the cooling unit 120 to start cooling the Central Brain 15. For example, when the execution unit 114 recognizes that an object is moving based on the point information of the object output from the MoPU 12, the execution unit 114 causes the cooling unit 120 to start cooling the Central Brain 15.

[0128] Note that the execution unit 114 is not limited to executing cooling of the Central Brain 15 based on the result of object detection, and may execute cooling of the Central Brain 15 based on the result of prediction of the operating status of the information processing device 10.

[0129] Here, the prediction unit 116 predicts the operating status of the information processing device 10, specifically, the Central Brain 15, based on the detection result of the object acquired by the acquisition unit 112. For example, the prediction unit 116 acquires a learning model stored in a predetermined storage area. Then, the prediction unit 116 predicts the operating status of the Central Brain 15 by inputting the point information of the object output from the MoPU 12 acquired by the acquisition unit 112 into the learning model. Here, the learning model outputs the status and change amount of the computing power of the Central Brain 15 as the operating status. In addition, the prediction unit 116 may predict and output the temperature change of the information processing device 10, specifically, the Central Brain 15, together with the operating status. For example, the prediction unit 116 predicts the temperature change of the Central Brain 15 based on the number of point information of the object output from the MoPU 12 acquired by the acquisition unit 112. In this case, the prediction unit 116 predicts that the temperature change will be larger as the number of point information becomes larger, and predicts that the temperature change will be smaller as the number of point information becomes smaller.

[0130] In the above case, the execution unit 114 starts cooling the Central Brain 15 by the cooling unit 120 based on the prediction result of the operation status of the Central Brain 15 by the prediction unit 116. For example, the execution unit 114 starts cooling by the cooling unit 120 when the state and change amount of the computing power of the Central Brain 15 predicted as the operation status exceeds a predetermined threshold. In addition, the execution unit 114 starts cooling by the cooling unit 120 when the temperature based on the temperature change of the Central Brain 15 predicted as the operation status exceeds a predetermined threshold.

[0131] Furthermore, the execution unit 114 may cause the cooling unit 120 to perform cooling using a cooling means according to the prediction result of the temperature change of the Central Brain 15 by the prediction unit 116. For example, the higher the predicted temperature of the Central Brain 15, the more cooling means the execution unit 114 may cause the cooling unit 120 to perform cooling. As a specific example, when it is predicted that the temperature of the Central Brain 15 will exceed a first threshold, the execution unit 114 causes the cooling unit 120 to perform cooling using one cooling means. On the other hand, when it is predicted that the temperature of the Central Brain 15 will exceed a second threshold higher than the first threshold, the execution unit 114 causes the cooling unit 120 to perform cooling using multiple cooling means.

[0132] Furthermore, the execution unit 114 may use a more powerful cooling means to cool the Central Brain 15 as the predicted temperature of the Central Brain 15 increases. For example, when the execution unit 114 predicts that the temperature of the Central Brain 15 will exceed a first threshold, the execution unit 114 causes the cooling unit 120 to perform cooling using air cooling means. When the execution unit 114 predicts that the temperature of the Central Brain 15 will exceed a second threshold higher than the first threshold, the execution unit 114 causes the cooling unit 120 to perform cooling using water cooling means. When the execution unit 114 predicts that the temperature of the Central Brain 15 will exceed a third threshold higher than the second threshold, the execution unit 114 causes the cooling unit 120 to perform cooling using liquid nitrogen cooling means.

[0133] Furthermore, the execution unit 114 may determine the cooling means to be used for cooling based on the number of point information of the object output from the MoPU 12 acquired by the acquisition unit 112. In this case, the execution unit 114 may use a more powerful cooling means to perform cooling on the Central Brain 15 as the number of point information increases. For example, when the number of point information exceeds a first threshold, the execution unit 114 causes the cooling unit 120 to perform cooling using an air cooling means. Also, when the number of point information exceeds a second threshold higher than the first threshold, the execution unit 114 causes the cooling unit 120 to perform cooling using a water cooling means. Furthermore, when the number of point information exceeds a third threshold higher than the second threshold, the execution unit 114 causes the cooling unit 120 to perform cooling using a liquid nitrogen cooling means.

[0134] Incidentally, the Central Brain 15 may be activated when a moving object on the roadway is detected. For example, when a moving object on the roadway is detected while the vehicle 100 is driving autonomously, the Central Brain 15 may perform arithmetic processing to control the vehicle 100 with respect to the object. As described above, heat generated when the Central Brain 15 that controls the autonomous driving of the vehicle 100 performs advanced arithmetic processing is an issue. In view of this, the cooling execution device 110 according to the eighth embodiment predicts heat dissipation of the Central Brain 15 based on the object detection result by the information processing device 10, and causes the Central Brain 15 to perform cooling before or simultaneously with the start of heat dissipation. This prevents the Central Brain 15 from becoming too hot during the autonomous driving of the vehicle 100, and enables advanced arithmetic processing during the autonomous driving.

[0135] Ninth embodiment Next, a ninth embodiment according to the present invention will be described while omitting or simplifying parts that overlap with the above-mentioned embodiments. The MoPU 12 included in the information processing device 10 according to the ninth embodiment derives a z coordinate value of an object as point information from an image of the object captured by the camera 30. Hereinafter, each aspect of the information processing device 10 according to the ninth embodiment will be described in order.

[0136] The information processing device 10 according to the first aspect has the same configuration as that of the second embodiment shown in FIG.

[0137] In the first aspect, the MoPU 12 derives the z coordinate value of an object as point information from an image of the object captured by a plurality of cameras 30, specifically, the camera 30L and the camera 30R. As described above, when one MoPU 12 is used, it is possible to derive the x coordinate value and the y coordinate value of the object as point information. Here, when two MoPUs 12 are used, it is possible to derive the z coordinate value of the object as point information based on the images of the object captured by the two cameras 30 by utilizing the principle of a stereo camera. Therefore, in the first aspect, the z coordinate value of the object is derived as point information based on the images of the object captured by the camera 30L of the MoPU 12L and the camera 30R of the MoPU 12R by utilizing the principle of a stereo camera.

[0138] The information processing device 10 according to the second aspect has the same configuration as that of the second embodiment shown in FIG.

[0139] In the second embodiment, the MoPU 12 derives the x, y, and z coordinate values ​​of an object as point information from an image of the object captured by the camera 30 and a radar signal based on the electromagnetic waves irradiated to the object by the radar 32 and reflected from the object. As described above, the radar 32 is capable of acquiring three-dimensional point cloud data of the object based on the radar signal. That is, the radar 32 is capable of detecting the coordinate of the z axis in the three-dimensional orthogonal coordinate system. In this case, the MoPU 12 utilizes the principle of a stereo camera to combine the x and y coordinate values ​​of the object captured by the camera 30 at the same timing as the radar 32 acquires the three-dimensional point cloud data of the object with the z coordinate value of the object indicated by the three-dimensional point cloud data, thereby deriving the coordinate values ​​of the three coordinate axes of the object as point information.

[0140] The information processing device 10 according to the third embodiment has the configuration shown in Fig. 10. Fig. 10 is a fifth block diagram showing an example of the configuration of the information processing device 10. Note that Fig. 10 shows only a part of the configuration of the information processing device 10.

[0141] In the third embodiment, the MoPU 12 derives the z coordinate value of the object as point information from the image of the object captured by the camera 30 and the result of capturing the structured light irradiated onto the object by the irradiation device 130.

[0142] 10, in the MoPU 12, an image of an object captured by the camera 30 and distortion information indicating distortion of a pattern of structured light resulting from capturing, by the camera 140, structured light irradiated onto an object by an irradiation device 130 are each input to the core 17 at a frame rate of 100 frames / second or more. Then, the core 17 outputs point information to the Central Brain 15 based on the input image of the object and distortion information.

[0143] Here, one of the methods for identifying the three-dimensional position or shape of an object is the structured light method. The structured light method irradiates an object with structured light patterned in dots, and obtains depth information from the distortion of the pattern. The structured light method is disclosed, for example, in the reference document (http: / / ex-press.jp / wp-content / uploads / 2018 / 10 / 018_teledyne_3rd.pdf).

[0144] 10 irradiates an object with structured light. Furthermore, camera 140 captures an image of the structured light irradiated onto the object by irradiation device 130. Then, camera 140 outputs to core 17 distortion information based on distortion of the captured structured light pattern.

[0145] Here, the MoPU 12 synchronizes the timing of capturing images by the camera 30 with the timing of capturing images of the structured light by the camera 140. Specifically, the MoPU 12 outputs a control signal to the camera 30 and the camera 140 so that the images are captured at the same timing. This synchronizes the number of images captured per second by the camera 30 with the number of images captured per second by the camera 140 (e.g., 1920 frames / second). In this way, the number of images captured per second by the camera 30 and the number of images captured per second by the camera 140 are greater than the frame rate of the ultra-high resolution camera provided in the IPU 11, i.e., the number of images captured per second by the ultra-high resolution camera.

[0146] Then, core 17 combines the x-coordinate value and y-coordinate value of the object photographed by camera 30 at the same time that the structured light was photographed by camera 140 with distortion information based on the distortion of the pattern of the structured light to derive the z-coordinate value of the object as point information.

[0147] The information processing device 10 according to the fourth aspect has the configuration shown in Fig. 11. Fig. 11 is a sixth block diagram showing an example of the configuration of the information processing device 10. Note that Fig. 11 shows only a part of the configuration of the information processing device 10.

[0148] The block diagram shown in FIG. 11 is obtained by adding a Lidar sensor 18 to the configuration of the block diagram shown in FIG. 2. The Lidar sensor 18 is a sensor that acquires point cloud data including objects existing in a three-dimensional space and the road surface on which the vehicle 100 is traveling. The information processing device 10 can derive position information of an object in the depth direction, that is, the z coordinate value of the object, by using the point cloud data acquired by the Lidar sensor 18. It is assumed that the point cloud data acquired by the Lidar sensor 18 is acquired at intervals longer than the x coordinate value and the y coordinate value of the object output from the MoPU 12. The MoPU 12 also includes a camera 30, similar to the above aspect of the ninth embodiment.

[0149] In the fourth aspect, the MoPU 12 utilizes the principle of a stereo camera to combine the x and y coordinate values ​​of the object photographed by the camera 30 at the same time that the Lidar sensor 18 acquires the point cloud data of the object with the z coordinate value of the object indicated by the point cloud data, to derive coordinate values ​​of the three coordinate axes of the object as point information.

[0150] Here, in the above fourth aspect, the MoPU 12 derives the z coordinate value of the object at time t+1 as point information from the x, y, and z coordinate values ​​of the object at time t and the x and y coordinate values ​​of the object at the next time point after time t (e.g., time t+1). Time t is an example of a "first time point," and time t+1 is an example of a "second time point." In the fourth aspect, the z coordinate value of the object at time t+1 is derived using shape information, i.e., geometry. This will be described in detail below.

[0151] Fig. 12 is a diagram showing a schematic diagram of coordinate detection of an object in a time series. In Fig. 12, J indicates the position of an object represented by a rectangle, and the object's position moves in a time series from J1 to J2. In Fig. 12, the coordinate values ​​of the object at time t when the object is located at J1 are (x1, y1, z1), and the coordinate values ​​of the object at time t+1 when the object is located at J2 are (x2, y2, z2).

[0152] First, the time t will be described. The MoPU 12 derives the x-coordinate value and the y-coordinate value of the object from the image of the object captured by the camera 30. Next, the MoPU 12 integrates the z-coordinate value of the object indicated by the point cloud data acquired from the Lidar sensor 18 with the above-mentioned x-coordinate value and y-coordinate value to derive the three-dimensional coordinate value (x1, y1, z1) of the object at time t.

[0153] Next, time t+1 will be described. The MoPU 12 derives the z-coordinate value of the object at time t+1 based on the spatial geometry and the changes in the x-coordinate value and the y-coordinate value of the object from time t to time t+1. The spatial geometry includes the shape of the road surface obtained from the images captured by the super-high resolution camera equipped in the IPU 11 and the point cloud data of the Lidar sensor 18, and the shape of the vehicle 100.

[0154] The geometry representing the shape of the road surface is generated in advance at time t. The MoPU 12 can simulate the vehicle 100 traveling on the road surface by using the geometry representing the shape of the vehicle 100 together with the geometry representing the shape of the road surface, and can estimate the amount of movement on each of the x-axis, y-axis, and z-axis.

[0155] Therefore, the MoPU 12 derives the x and y coordinate values ​​of the object at time t+1 from the image of the object captured by the camera 30. The MoPU 12 can derive the z coordinate value of the object at time t+1 by calculating, from a simulation, the amount of movement along the z axis when the object changes from its x and y coordinate values ​​(x1, y1) at time t to its x and y coordinate values ​​(x2, y2) at time t+1. The MoPU 12 then combines the x and y coordinate values ​​with the z coordinate value to derive the three-dimensional coordinate value (x2, y2, z2) of the object at time t+1.

[0156] As shown in FIG. 12, since the object moves in the depth direction together with the movement of the planar coordinates (i.e., the x-axis and y-axis), it is necessary to detect the movement in the z-axis direction in order to control the autonomous driving of the vehicle 100 with high accuracy. Here, the MoPU 12 may not be able to acquire the z-coordinate value of the object that can be derived from the point cloud data of the Lidar sensor 18 as quickly as the x-coordinate value and the y-coordinate value of the object. Therefore, in the above-mentioned fourth aspect, the MoPU 12 derives the z-coordinate value of the object at time t+1 from the x-coordinate value, the y-coordinate value, and the z-coordinate value of the object at time t and the x-coordinate value and the y-coordinate value of the object at time t+1. As a result, according to the information processing device 10 according to the above-mentioned fourth aspect, the MoPU 12 can realize two-dimensional motion detection and three-dimensional motion detection by high-speed frame shot with high performance and low data volume.

[0157] In the above description, the MoPU 12 derives the z coordinate value of an object as point information from an image of the object captured by the camera 30, but the disclosed technology is not limited to this embodiment. For example, the Central Brain 15 may derive the z coordinate value of an object as point information instead of the MoPU 12. In this case, the Central Brain 15 derives the z coordinate value of an object as point information by performing the processing performed by the MoPU 12 in the above description on the image of the object captured by the camera 30. As an example, the Central Brain 15 derives the z coordinate value of an object as point information from images of the object captured by a plurality of cameras 30, specifically, the camera 30L and the camera 30R. In this case, the Central Brain 15 utilizes the principle of a stereo camera to derive the z coordinate value of an object as point information based on images of the object captured by the camera 30L of the MoPU 12L and the camera 30R of the MoPU 12R, respectively.

[0158] (Tenth embodiment) Next, a tenth embodiment according to the present invention will be described while omitting or simplifying parts that overlap with the above-mentioned embodiments. Fig. 13 is a seventh block diagram showing an example of the configuration of the information processing device 10. Note that Fig. 13 shows only a portion of the configuration of the information processing device 10.

[0159] 13, in the MoPU 12, an image of an object captured by an event camera 30C (hereinafter, also referred to as an "event image") is input to the core 17. Then, the core 17 outputs point information to the Central Brain 15 based on the input event image. Note that the event camera is disclosed in, for example, reference material (https: / / dendenblog.xyz / event-based-camera / ).

[0160] FIG. 14 is an explanatory diagram for explaining an image of an object (event image) captured by the event camera 30C. FIG. 14(A) is a diagram showing an object to be captured by the event camera 30C. FIG. 14(B) is a diagram showing an example of an event image. FIG. 14(C) is a diagram showing an example in which the center of gravity of the difference between the image captured at the current time and the image captured at the previous time, which is represented by the event image, is calculated as point information. In the event image, the difference between the image captured at the current time and the image captured at the previous time is extracted as a point. Therefore, when the event camera 30C is used, for example, as shown in FIG. 14(B), points of each moving part of the person area shown in FIG. 14(A) are extracted.

[0161] In response to this, as shown in Fig. 14(C), the core 17 extracts the person as an object, and then extracts the coordinates of the feature point (for example, only one point) that represents the person area. This makes it possible to reduce the amount of data transferred to the Central Brain 15 and the memory 16. Since the event image allows the person as an object to be extracted at any frame rate, in the case of the event camera 30C, extraction can be performed at a frame rate equal to or higher than the maximum frame rate (for example, 1920 frames / second) of the camera 30 mounted on the MoPU 12 in the above embodiment, and point information of the object can be captured with high accuracy.

[0162] In the information processing device 10 according to the tenth embodiment, the MoPU 12 may include a visible light camera 30A in addition to the event camera 30C, as in the above embodiments. In this case, in the MoPU 12, a visible light image and an event image of an object captured by the visible light camera 30A are input to the core 17. Then, the core 17 outputs point information to the Central Brain 15 based on at least one of the input visible light image and the event image.

[0163] For example, when the core 17 can identify an object from a visible light image of the object captured by the visible light camera 30A, the core 17 outputs point information based on the visible light image. On the other hand, when the core 17 cannot capture an object from the visible light image due to a predetermined factor, the core 17 outputs point information based on an event image. The predetermined factor includes at least one of a case where the moving speed of the object is equal to or greater than a predetermined value and a case where the change in the amount of ambient light per unit time is equal to or greater than a predetermined value. For example, when the object moves at a high speed and the object cannot be captured from the visible light image, the core 17 identifies the object based on the event image and outputs the x-coordinate value and the y-coordinate value of the object as point information. Also, when the object cannot be captured from the visible light image due to a sudden change in the amount of ambient light such as backlighting, the core 17 identifies the object based on the event image and outputs the x-coordinate value and the y-coordinate value of the object as point information. With this configuration, the information processing device 10 can use the camera 30 for capturing an object differently depending on the predetermined factor.

[0164] 15 is a schematic diagram showing an example of a hardware configuration of a computer 1200 functioning as the information processing device 10 or the cooling execution device 110. A program installed in the computer 1200 can cause the computer 1200 to function as one or more "parts" of the device according to the present embodiment, or cause the computer 1200 to execute operations or one or more "parts" associated with the device according to the present embodiment, and / or cause the computer 1200 to execute a process or steps of the process according to the present embodiment. Such a program can be executed by the CPU 1212 to cause the computer 1200 to execute specific operations associated with some or all of the blocks of the flowcharts and block diagrams described herein.

[0165] The computer 1200 according to this embodiment includes a CPU 1212, a RAM 1214, and a graphics controller 1216, which are connected to each other by a host controller 1210. The computer 1200 also includes input / output units such as a communication interface 1222, a storage device 1224, a DVD drive, and an IC card drive, which are connected to the host controller 1210 via an input / output controller 1220. The DVD drive may be a DVD-ROM drive, a DVD-RAM drive, etc. The storage device 1224 may be a hard disk drive, a solid state drive, etc. The computer 1200 also includes a legacy input / output unit such as a ROM 1230 and a keyboard, which are connected to the input / output controller 1220 via an input / output chip 1240.

[0166] The CPU 1212 operates according to a program stored in the ROM 1230 and the RAM 1214, thereby controlling each unit. The graphic controller 1216 acquires image data generated by the CPU 1212 into a frame buffer or the like provided in the RAM 1214 or into itself, and causes the image data to be displayed on the display device 1218.

[0167] The communication interface 1222 communicates with other electronic devices via a network. The storage device 1224 stores programs and data used by the CPU 1212 in the computer 1200. The DVD drive reads programs or data from a DVD-ROM or the like and provides them to the storage device 1224. The IC card drive reads programs and data from an IC card and / or writes programs and data to an IC card.

[0168] The ROM 1230 stores therein a boot program or the like that is executed by the computer 1200 upon activation, and / or a program that depends on the hardware of the computer 1200. The input / output chip 1240 may also connect various input / output units to the input / output controller 1220 via a USB port, a parallel port, a serial port, a keyboard port, a mouse port, and the like.

[0169] The programs are provided by a computer-readable storage medium such as a DVD-ROM or an IC card. The programs are read from the computer-readable storage medium, installed in the storage device 1224, the RAM 1214, or the ROM 1230, which are also examples of computer-readable storage media, and executed by the CPU 1212. Information processing described in these programs is read by the computer 1200, and brings about cooperation between the programs and the various types of hardware resources described above. An apparatus or method may be constructed by implementing operations or processing of information according to the use of the computer 1200.

[0170] For example, when communication is performed between the computer 1200 and an external device, the CPU 1212 may execute a communication program loaded in the RAM 1214 and instruct the communication interface 1222 to perform communication processing based on the processing described in the communication program. Under the control of the CPU 1212, the communication interface 1222 reads transmission data stored in a transmission buffer area provided in the RAM 1214, the storage device 1224, a DVD-ROM, or a recording medium such as an IC card, and transmits the read transmission data to the network, or writes reception data received from the network to a reception buffer area or the like provided on the recording medium.

[0171] Furthermore, the CPU 1212 may cause all or a necessary portion of a file or database stored in an external recording medium such as the storage device 1224, a DVD drive (DVD-ROM), an IC card, etc. to be read into the RAM 1214, and perform various types of processing on the data on the RAM 1214. The CPU 1212 may then write back the processed data to the external recording medium.

[0172] Various types of information, such as various types of programs, data, tables, and databases, may be stored in the recording medium and undergo information processing. The CPU 1212 may perform various types of processing on the data read from the RAM 1214, including various types of operations, information processing, conditional judgment, conditional branching, unconditional branching, information search / replacement, etc., as described throughout this disclosure and specified by the instruction sequence of the program, and write back the results to the RAM 1214. The CPU 1212 may also search for information in a file, database, etc. in the recording medium. For example, when a plurality of entries each having an attribute value of a first attribute associated with an attribute value of a second attribute are stored in the recording medium, the CPU 1212 may search for an entry whose attribute value of the first attribute matches a specified condition from among the plurality of entries, read the attribute value of the second attribute stored in the entry, and thereby obtain the attribute value of the second attribute associated with the first attribute that satisfies a predetermined condition.

[0173] The above-described programs or software modules may be stored in a computer-readable storage medium on the computer 1200 or in the vicinity of the computer 1200. In addition, a recording medium such as a hard disk or a RAM provided in a server system connected to a dedicated communication network or the Internet can be used as a computer-readable storage medium, thereby providing the programs to the computer 1200 via the network.

[0174] The blocks in the flowcharts and block diagrams in the present embodiment may represent stages of a process in which an operation is performed or "parts" of an apparatus responsible for performing the operation. Particular stages and "parts" may be implemented by dedicated circuitry, programmable circuitry provided with computer-readable instructions stored on a computer-readable storage medium, and / or a processor provided with computer-readable instructions stored on a computer-readable storage medium. The dedicated circuitry may include digital and / or analog hardware circuits, and may include integrated circuits (ICs) and / or discrete circuits. The programmable circuitry may include reconfigurable hardware circuits, such as, for example, field programmable gate arrays (FPGAs), programmable logic arrays (PLAs), and the like, including AND, OR, XOR, NAND, NOR, and other logical operations, flip-flops, registers, and memory elements.

[0175] A computer-readable storage medium may include any tangible device capable of storing instructions that are executed by a suitable device, such that a computer-readable storage medium having instructions stored thereon comprises an article of manufacture that includes instructions that can be executed to create means for performing the operations specified in the flowcharts or block diagrams. Examples of computer-readable storage media may include electronic storage media, magnetic storage media, optical storage media, electromagnetic storage media, semiconductor storage media, and the like. More specific examples of computer-readable storage media may include floppy disks, diskettes, hard disks, random access memories (RAMs), read-only memories (ROMs), erasable programmable read-only memories (EPROMs or flash memories), electrically erasable programmable read-only memories (EEPROMs), static random access memories (SRAMs), compact disk read-only memories (CD-ROMs), digital versatile disks (DVDs), Blu-ray disks, memory sticks, integrated circuit cards, and the like.

[0176] The computer readable instructions may include either assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk®, JAVA®, C++, etc., and conventional procedural programming languages ​​such as the “C” programming language or similar programming languages.

[0177] Computer readable instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus, or to a programmable circuit, either locally or over a local area network (LAN), a wide area network (WAN), such as the Internet, etc., to cause the processor of the general purpose computer, special purpose computer, or other programmable data processing apparatus, or to a programmable circuit, to execute the computer readable instructions to generate means for performing the operations specified in the flowcharts or block diagrams. Examples of processors include computer processors, processing units, microprocessors, digital signal processors, controllers, microcontrollers, etc.

[0178] Although the present invention has been described above using the embodiments, the technical scope of the present invention is not limited to the scope described in the above embodiments. It is clear to those skilled in the art that various modifications and improvements can be made to the above embodiments. It is clear from the description of the claims that such modifications and improvements can also be included in the technical scope of the present invention.

[0179] It should be noted that the order of execution of each process, such as operations, procedures, steps, and stages, in the devices, systems, programs, and methods shown in the claims, specifications, and drawings is not specifically stated as "before" or "prior to," and that the process may be performed in any order unless the output of a previous process is used in a later process. Even if the operational flow in the claims, specifications, and drawings is described using "first," "next," etc. for convenience, it does not mean that the process must be performed in this order.

[0180] In the above embodiment, the processes executed by each processor (e.g., IPU 11, MoPU 12, and Central Brain 15) are merely examples, and the processor that executes each process is not limited. For example, the processes executed by the MoPU 12 in the above embodiment may be executed by the Central Brain 15 instead of the MoPU 12, or may be executed by a processor other than the IPU 11, MoPU 12, and Central Brain 15. [Explanation of symbols]

[0181] 10. Information processing device 11 IPU (second processor) 12 MoPU (first processor) 15 Central Brain (third processor) 200 Servers 1000 Information Processing Systems

Claims

1. a first processor that outputs point information that captures an object as a point from an image of the object captured by a first camera mounted on a vehicle; a second processor that is mounted on the vehicle and that outputs identification information that identifies the photographed object from an image of the object photographed by a second camera that is oriented in a direction corresponding to the first camera; a third processor that associates the point information output from the first processor with the identification information output from the second processor; Equipped with The third processor further performs driving control of the vehicle based on a result of simulating the movement of the object on a digital map on which the associated point information and the identification information are plotted. Information processing device.

2. The first processor outputs the point information including motion information indicative of a motion of the object; In the simulation, a collision risk between the vehicle and the object is predicted using each movement pattern of the object determined according to the movement information included in the point information and the identification information; and The information processing apparatus according to claim 1 , wherein the third processor performs driving control of the vehicle based on the collision risk predicted by the simulation.

3. The third processor performs driving control of the vehicle so as to avoid a collision in which the collision risk predicted in the simulation is equal to or greater than a threshold value, 3. The information processing device according to claim 2, wherein, when all of the collision risks predicted in the simulation are less than the threshold value, driving control of the vehicle is performed so as to prevent a traffic jam of following vehicles.

4. 2 . The information processing apparatus according to claim 1 , wherein in the simulation, a collision risk between the vehicle and the object is predicted using a point representing the object or a polygon surrounding a contour of the object.

5. outputting point information that captures an object as a point from an image of the object photographed by a first camera mounted on the vehicle; outputting identification information that identifies the photographed object from an image of the object photographed by a second camera mounted on the vehicle and facing in a direction corresponding to the first camera; Correlating the output point information with the output identification information; The driving control of the vehicle is performed based on a result of simulating the movement of the object on a digital map on which the associated point information and the identification information are plotted. An information processing method in which processing is performed by a computer.

6. On the computer, outputting point information that captures an object as a point from an image of the object photographed by a first camera mounted on the vehicle; outputting identification information that identifies the photographed object from an image of the object photographed by a second camera mounted on the vehicle and facing in a direction corresponding to the first camera; Correlating the output point information with the output identification information; The driving control of the vehicle is performed based on a result of simulating the movement of the object on a digital map on which the associated point information and the identification information are plotted. An information processing program for executing processing.

Images