In-vehicle information processing device, in-vehicle information processing system, and in-vehicle information processing method

Abstract

The present invention realizes an in-vehicle information processing device that makes it possible to suppress recognition delay with respect to an object having high urgency by providing a larger amount of computation resources to a region having a high priority than a region having a low priority. The in-vehicle information processing device is provided with a control unit (110). The control unit (110) has: an external field data acquisition unit (111) that acquires external field data from external field sensors (210), (260), (290), (310)-(360) mounted on a vehicle (10); a point group cluster generation unit (112) that generates a point group cluster by performing clustering with respect to a plurality of points constituting the external field data acquired from all or some of the external field sensors (210), (260), (290), (310)-(360); a priority setting unit (113) that sets a priority in the point group cluster or an external field region in which the point group cluster is present, on the basis of a relative speed of the point group cluster with respect to the vehicle (10), the relative speed having been obtained from the external field data related to the point group cluster; and a computation resource amount determination unit (114) that determines a computation resource amount for external field data recognition processing according to the priority.

Description

[Technical Field]

[0001] The present invention relates to an in-vehicle information processing device, an in-vehicle information processing system, and an in-vehicle information processing method. [Background technology]

[0002] In recent years, the development of technologies necessary for vehicle driving assistance and autonomous driving has been progressing, and one of the elemental technologies for these technologies is the technology to grasp the situation around the vehicle. As driving assistance and autonomous driving become more advanced, it has become important to grasp the situation around the vehicle in more detail, and as a means to achieve this, a large number of external sensors such as cameras, millimeter-wave radar, and LiDAR are being used.

[0003] In driving assistance and autonomous driving, external sensing data acquired by these external sensors is subjected to recognition processing to detect obstacles to be avoided and areas in which the vehicle should be driven, and the recognition results are then comprehensively judged, taking into account the vehicle driver's operations as necessary, to ultimately determine the necessary action.

[0004] Patent document 1 describes a technology that acquires parallax images using a stereo camera, derives the coordinates of an object in a world coordinate system from the parallax images, and determines whether an object located in the vehicle's planned travel area is a person or not, using the object as a priority object. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Patent Publication No. 2020-160914 Summary of the Invention [Problem to be solved by the invention]

[0006] As the number of external sensors increases, the amount of external sensing data to be processed for recognition also increases, increasing the amount of calculation required for recognition processing, and reducing the computational load becomes an issue. In particular, when handling high-resolution data or performing high-precision recognition processing, the amount of calculation required for recognition processing increases even for each piece of sensing data from an individual external sensor, creating a growing demand for reducing the computational load. One method for reducing the amount of calculation is to perform recognition processing by prioritizing the area sensed by the external sensor, thereby improving the efficiency of the recognition processing. A technique described in Patent Document 1 is known as such a method. However, when providing driving assistance or controlling autonomous driving for vehicles traveling on public roads, there are situations where there are many objects in the vicinity, and in such situations, there are many priority objects, making it difficult to reduce the amount of calculation. It is also necessary to respond to fast-moving objects in the vicinity, such as bicycles and cars, and in cases where a surrounding object approaches, recognition of the priority object may be delayed, leading to delays in control. The object of the present invention is to realize an in-vehicle information processing device, an in-vehicle information processing system, and an in-vehicle information processing method that can reduce recognition delays for objects with high urgency by increasing the amount of computing resources for areas with high priority compared to areas with low priority. [Means for solving the problem]

[0007] In order to achieve the above object, the present invention is configured as follows.

[0008] The in-vehicle information processing device includes a control unit, the control unit including: an external data acquisition unit that acquires external data from external sensors mounted on the vehicle; a point cloud cluster generation unit that performs clustering on a plurality of points constituting the external data acquired from all or some of the external sensors to generate a point cloud cluster; a priority setting unit that sets a priority to the point cloud cluster or an external area in which the point cloud cluster exists based on a relative speed of the point cloud cluster with respect to the vehicle calculated from the external data related to the point cloud cluster; and a computational resource amount determination unit that determines a computational resource amount for recognition processing of the external data in accordance with the priority. an image correction processing unit that corrects an image of the outside world data acquired by the outside world data acquisition unit; an image recognition processing unit that recognizes the image corrected by the image correction processing unit based on the amount of computational resources determined by the computational resource amount determination unit; a point cloud cluster recognition processing unit that performs object type detection and noise removal on the point cloud cluster generated by the point cloud cluster generation unit and performs recognition processing on the point cloud cluster; and a recognition result integration processing unit that checks consistency of recognition results between a processing result of the point cloud cluster recognition processing unit and a processing result of the image recognition processing unit, selects and removes recognition results with inconsistencies, and integrates the processing result of the point cloud cluster recognition processing unit and the processing result of the image recognition processing unit to output the result as an outside world recognition result.It has.

[0009] The in-vehicle information processing system also includes a plurality of external sensors that detect objects around the vehicle, and a control unit having a configuration similar to that of the control unit of the in-vehicle information device.

[0010] The in-vehicle information processing method further includes detecting an object around the vehicle using a plurality of external sensors, and acquiring external data from the plurality of external sensors; The control unit of the in-vehicle information processing device Clustering is performed on a plurality of points constituting external world data acquired from all or some of the external world sensors to generate a point cloud cluster, and a priority is set for the point cloud cluster or an external world area in which the point cloud cluster exists based on a relative speed of the point cloud cluster with respect to the vehicle calculated from the external world data related to the point cloud cluster, and an amount of computational resources for recognition processing of the external world data is determined according to the priority. correcting the acquired image of the external world data, recognizing the corrected image based on the amount of computational resources, detecting the type of object and eliminating noise from the point cloud cluster, performing recognition processing on the point cloud cluster, confirming the consistency of the recognition results between the recognition processing result of the point cloud cluster and the recognized corrected image, selecting and excluding recognition results that are inconsistent, and integrating the recognition processing result of the point cloud cluster and the recognized corrected image to output as an external world recognition result. . [Effects of the Invention]

[0011] It is possible to realize an in-vehicle information processing device, an in-vehicle information processing system, and an in-vehicle information processing method that can appropriately determine priority according to the surrounding area being sensed, even in a vehicle traveling on a public road, and allocate more computing resources to areas with higher priority than to areas with lower priority, thereby suppressing delays in recognition of objects with high urgency. Furthermore, by using computing resources to improve recognition accuracy and recognition function, it is possible to suppress the occurrence of erroneous recognition or non-recognition of objects with high urgency, and to obtain detailed information such as the type and behavior of the object. [Brief explanation of the drawings]

[0012] [Figure 1] 1 is a diagram showing an example of a sensor configuration of an external sensing system to which the present invention is applied; [Figure 2] 1A to 1C are diagrams illustrating an example of the operation of the external sensing system according to the present invention. [Figure 3] 1 is a diagram illustrating an example of a connection configuration of an external sensing system according to a first embodiment. [Figure 4] FIG. 2 is a functional block diagram of a control unit according to the first embodiment. [Figure 5] FIG. 2 is a diagram showing a processing flow of the external sensing system according to the first embodiment. [Figure 6] FIG. 10 is a diagram illustrating an example of region division of an image captured by a camera. [Figure 7] FIG. 10 is a diagram illustrating an example of the order of recognition processing of a camera-acquired image. [Figure 8] FIG. 10 is a diagram illustrating an example of a sensor configuration of an external sensing system according to a second embodiment. [Figure 9A] FIG. 10 is a diagram illustrating an example of priority weighting of recognition processing depending on the state of a vehicle. [Figure 9B] FIG. 10 is a diagram illustrating an example of priority weighting of recognition processing depending on the state of a vehicle. [Figure 9C] FIG. 10 is a diagram illustrating an example of priority weighting of recognition processing depending on the state of a vehicle. [Figure 9D] FIG. 10 is a diagram illustrating an example of priority weighting of recognition processing depending on the state of a vehicle. [Figure 9E] FIG. 10 is a diagram illustrating an example of priority weighting of recognition processing depending on the state of a vehicle. [Figure 10] FIG. 10 is a diagram illustrating an example of a sensor configuration of an external sensing system according to a third embodiment. [Figure 11] FIG. 1 is a diagram illustrating an example of a processing flow of an external sensing system using multiple ECUs. [Figure 12] FIG. 10 is a diagram illustrating an example of image data transfer settings via an inter-ECU image transfer compatible communication path. [Figure 13] FIG. 10 is a diagram illustrating the usage settings of a communication path compatible with image transfer between ECUs. [Figure 14] FIG. 10 is a diagram illustrating an example of a sensor configuration of an external sensing system according to a fourth embodiment. [Figure 15] FIG. 11 is a diagram illustrating an example of dividing the angle of view of a camera in an example of a connection configuration according to a third embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0013] Hereinafter, embodiments of the present invention will be described with reference to the drawings. [Example]

[0014] Example 1 An example of the sensor configuration of an external sensing system to which the present invention is applied will be described with reference to FIG.

[0015] 1, the host vehicle 10 has six cameras and a LiDAR A290 capable of sensing the entire periphery as external sensors that detect objects around the host vehicle 10. The six cameras are a camera F310 for monitoring the front, a camera FL320 for monitoring the left front, a camera FR330 for monitoring the right front, a camera RL340 for monitoring the left rear, a camera RR350 for monitoring the right rear, and a camera R360 for monitoring the rear.

[0016] The horizontal angle of view of camera F310 is shown as camera angle of view F410, the horizontal angle of view of camera FL320 as camera angle of view FL420, the horizontal angle of view of camera FR330 as camera angle of view FR430, the horizontal angle of view of camera RL340 as camera angle of view RL440, the angle of view of camera RR350 as camera angle of view RR450, and the horizontal angle of view of camera R360 as camera angle of view R460.

[0017] An example of the operation of the external sensing system according to the present invention will be described using Fig. 2. In Fig. 2, there are three surrounding vehicles: vehicle A20, vehicle B30, and vehicle C40. Vehicle A is approaching the host vehicle 10 at a speed Va, vehicle B is moving away at a speed Vb, and vehicle C is approaching at a speed Vc. In this situation, the host vehicle 10 uses LiDAR A290 to acquire a point cloud including distance information reflecting the surrounding vehicles.

[0018] Then, a point cloud cluster, which is a collection of spatially nearby points, is created, and a representative relative distance from the vehicle 10 is obtained for each point cloud cluster. Furthermore, the change in relative distance over time is found for each point cloud cluster, and the relative speed with respect to the vehicle 10 is calculated.

[0019] If the LiDAR A290 can directly measure the relative velocity by sensing, the relative velocity obtained by sensing may be used instead of the relative velocity calculated from the change in relative distance over time. Also, any external sensor other than LiDAR may be used as long as it can obtain the information necessary to create a point cloud cluster.

[0020] A point cloud cluster may be a collection of pixels, which are the points that make up an image. In other words, a point cloud cluster may be created based on image data acquired by a camera. However, since distance information is required to determine whether pixels are spatially close, distance information must be provided to each pixel using techniques such as stereo vision using multiple cameras before use. In this case, pixels for which distance information cannot be obtained using stereo vision or other methods must be excluded or estimated from distance information of neighboring pixels.

[0021] Based on the direction and relative speed at which the object was detected by LiDAR A290, the recognition process prioritizes image data acquired by camera F310, camera FL320, and camera R360, which are cameras corresponding to camera view angles F410, FL420, and R460 that overlap greatly with the direction in which the object approaching the vehicle 10 is located.

[0022] Judging from the relative speed, the priority of the recognition process is not increased for image data acquired by camera FR330, which is a camera corresponding to camera angle of view F430 that has a large overlap with only the direction of objects moving away from the host vehicle 10. By increasing the priority of the recognition process only for images corresponding to vehicles A20 and C40 without increasing the priority of the recognition process for images that only contain objects moving away from the host vehicle 10, it becomes possible to allocate more computing resources to the recognition process for vehicles A20 and C40, thereby improving the recognition accuracy for objects that may have an impact on the host vehicle 10 and shortening the time required for recognition.

[0023] The priority of the recognition process may be increased for smaller units than the image data units acquired by each camera. For example, the unit may be a unit obtained by dividing the image data acquired by each camera into multiple units, or it may be only image data in an area on the image corresponding to the part where the target point cloud cluster exists. When increasing the priority of the area on the image corresponding to the part where the point cloud cluster exists, it may be possible to set the area of ​​sensing data where the recognition process is performed slightly larger than the area where the target point cloud cluster exists, taking into account the positional deviation between the external sensor that obtains the sensing data used to create the point cloud cluster and the external sensor that obtains the sensing data used to perform the recognition process.

[0024] By using the relative speed (positive for the direction away from the vehicle 10 and negative for the direction approaching) in combination with the relative distance, for example, by dividing the relative distance by the relative speed and inverting the sign, the time until collision (hereinafter referred to as the collision grace period) assuming that the state at the time of sensing continues can be calculated, and it is also possible to set the priority of the recognition process in stages.

[0025] For example, it is considered that the smaller the positive value of the collision grace period for a point cloud cluster, the shorter the time grace period in case of a collision. Therefore, an object corresponding to such a point cloud cluster can be determined to be a recognition target with a high degree of urgency, and a higher priority can be given to the recognition process for image data acquired by a camera corresponding to an angle of view that covers the direction in which the point cloud cluster exists.

[0026] If the relative velocity of a point cloud cluster is positive, i.e., if the collision grace period is negative, it indicates that the object corresponding to the point cloud cluster is moving away, so the recognition processing priority is not increased for image data acquired by a camera corresponding to an angle of view that covers the direction in which the point cloud cluster exists.

[0027] When there are multiple point cloud clusters in the corresponding angle of view of a certain camera, the priority of the recognition process for the image acquired by that camera is set to the point cloud cluster with the smallest positive collision grace period among those multiple point cloud clusters, i.e., the point cloud cluster with the shortest time grace period in the event of a collision.

[0028] When calculating the collision grace period by dividing the relative distance by the relative speed, if the relative speed is close to 0, the collision grace period becomes very large, and if the relative speed is 0, it becomes infinite. In such cases, clipping is performed to set the collision grace period to a large fixed value that will not be treated as a target for high priority in the priority adjustment of the recognition process, thereby preventing the collision grace period from becoming infinite.

[0029] In the situation shown in FIG. 2, if the priority of the recognition process is determined using the collision margin time, the result will be, for example, as follows.

[0030] Since vehicle B30 is moving away from the host vehicle 10, although it is the closest to the host vehicle 10 among the surrounding vehicles, this does not become a factor for increasing the priority of the recognition process. Between vehicle A20 and vehicle C40, vehicle C40 is closer to the host vehicle 10, but the relative speed Va of vehicle A20 is greater than the relative speed Vc of vehicle C40. In the situation shown in FIG. 2, a relationship between the relative speeds Va and Vc is assumed such that the collision grace period for vehicle A20 is shorter than the collision grace period for vehicle C40. In this situation, it is determined that the recognition process of the direction in which there is a point cloud cluster corresponding to vehicle A20, which is farther away than vehicle C40 but has a shorter collision grace period, has a higher priority.

[0031] As a result, the recognition process priority is given highest to image data acquired at camera view angles F410 and FL420, which overlap closely with the orientation of vehicle A20, and the recognition process priority is given next to image data acquired by camera R360, which corresponds to camera view angle R460, which overlaps closely with the orientation of vehicle C40. In other words, compared to when the recognition process priority is simply determined based on the relative distance to the object, it is possible to determine the recognition process priority in accordance with the level of urgency.

[0032] Although a simple method using relative distance and relative speed has been shown for calculating the time to wait for collision, it is also possible to estimate the time to wait for collision using more advanced methods, such as taking into account changes over time in the relative speed and relative direction with respect to the object, and in such cases it is also possible to lower the level of urgency for objects that have become less likely to collide.

[0033] Fig. 3 shows an example of the connection configuration of an external sensing system that realizes the example operation described using Fig. 2. As shown in Fig. 3, external sensors LiDAR A290, camera F310, camera FL320, camera FR330, camera RL340, camera RR350, and camera R360 are directly connected to ECU Type A110 (control unit) that has a signal processing function including external world recognition processing.

[0034] It is necessary to transmit point cloud data from LiDAR A290 and image data from Camera F310, Camera FL320, Camera FR330, Camera RL340, Camera RR350, and Camera R360 to ECU Type A110 at a frame rate of several tens of frames per second. For this reason, a direct connection is used to ensure the necessary sensing data communication bandwidth and to reduce communication delays. For communication between the external sensors and ECU Type A110, high-speed serial communication of several to several tens of Gbps is used.

[0035] The ECU Type A110 also communicates with other ECUs installed in the vehicle 10 via an interface such as CAN. By connecting with other ECUs, the ECU Type A110 can transmit the results of its recognition of the external world to ECUs that make driving decisions and control the driving, and can obtain the driving status of the vehicle. Depending on the processing power of the ECU, the ECU Type A110 may also perform processes other than those related to external world sensing.

[0036] FIG. 4 is a schematic internal functional block diagram of the ECU Type A110.

[0037] 4, the ECU Type A110 includes an external data acquisition unit 111 that acquires external data from external sensors 290, 310 to 360 mounted on the vehicle 10, a point cloud cluster generation unit 112 that performs clustering on a plurality of points constituting the external data, of which three-dimensional position information is available, among the external data acquired by the external sensor acquisition unit 111 from the external sensors 290, 310 to 360, to generate a point cloud cluster, an image correction processing unit 116 that performs corrections such as image distortion of the external data, image brightness correction, color reproduction and correction, and image misalignment, and a priority setting unit 113 that sets a priority to a point cloud cluster or an external area in which the point cloud cluster is located, based on the relative speed of the point cloud cluster with respect to the vehicle 10 calculated from the external data related to the point cloud cluster generated by the point cloud cluster generation unit 112. Furthermore, the ECU Type A110 includes a computational resource amount determination unit 114 that determines the amount of computational resources for recognition processing of external world data in accordance with the priority set by the priority setting unit 113, an image recognition processing unit 117 that performs recognition processing on images, a point cloud cluster recognition processing unit 118 that performs recognition processing on point cloud clusters generated by the point cloud cluster generation unit 112 based on point cloud data acquired from the LiDAR A290, and a recognition result integration processing unit 115 that integrates the processing result of the point cloud cluster recognition processing unit 118 and the processing result of the image recognition processing unit 117 and outputs the result as an external world recognition result.

[0038] An example of the processing performed by ECU Type A110 is shown in Figure 5. Figure 5 shows the processing for one frame, separated into LiDAR point cloud processing and camera image processing. In other words, the processing shown in Figure 5 is repeated for each frame. LiDAR point cloud frame processing processes external sensing data acquired by LiDAR A290, while camera image frame processing processes image data acquired by camera F310, camera FL320, camera FR330, camera RL340, camera RR350, and camera R360.

[0039] The recognition results obtained by the LiDAR point cloud processing and camera image processing are sent to the recognition result integration processing unit 115, where fusion processing is performed to obtain the final recognition results of the surrounding conditions of the vehicle 10.

[0040] When LiDAR point cloud processing, camera image processing, and recognition result integration processing are executed within the ECU, they may be executed as distributed processing using multiple processor cores, or as time-sharing processing, or a combination of distributed processing and time-sharing processing.

[0041] Although time-sharing or distributed processing may be used to process the image data obtained from multiple cameras, the amount of computing resources allocated to the recognition processing of each image, i.e., the number of available processor cores, the available processing time, and the allocation of various dedicated processing circuits, is adjusted based on the recognition priority.

[0042] In the LiDAR point cloud frame processing, external sensing data acquired by the LiDAR A290 in LiDAR point cloud data acquisition step S710 is imported into the external data acquisition unit 111, and the imported data is subjected to point cloud acquisition signal processing step S720 by the point cloud cluster generation unit 112 to perform error correction associated with individual differences between sensors and errors in mounting the sensor on the vehicle. Additionally, noise contained in the sensing signal is removed, such as by filtering input signals that do not meet a certain level.

[0043] Next, in point cloud clustering processing step S730, the point cloud cluster generation unit 112 clusters the point cloud. Point cloud clustering is performed by, for example, detecting surrounding points within a certain distance in space from a given point, and if there are a certain number or more of such surrounding points, designating the point as a valid point and then repeatedly classifying the point as belonging to the same cluster as the detected surrounding points. However, since it is necessary to separate objects from the road surface, the road surface is estimated in advance taking into account the relationship between the sensor position and the horizontal plane, and points located below the road surface in space are excluded. Furthermore, if the number of points constituting a cluster as a result of clustering is below a certain number, the point cloud cluster is excluded as it is considered to be noise.

[0044] In the point cloud clustering process S730, for each extracted point cloud cluster, several to several tens of points constituting the cluster are selected in order of the shortest relative distance from the host vehicle 10, and the relative distances of these points are averaged to determine the relative distance from the host vehicle 10 to the point cloud cluster. If even a part of an object corresponding to a point cloud cluster comes into contact with the host vehicle 10, it indicates a collision, so the value of the smallest relative distance among the points constituting the point cloud cluster is used, but since the influence of noise is large at a single point, the relative distance to the point cloud cluster is determined using the average of multiple points.

[0045] The point cloud clusters extracted in point cloud clustering processing step S730 are associated with point cloud clusters in the previous frame in point cloud cluster relative velocity detection step S740, and relative velocity is calculated from the difference in elapsed time and relative distance from the previous frame. Relative velocity is considered to be a change in relative distance over time, and the direction of approach is taken as a negative value. Association with point cloud clusters in the previous frame can be performed, for example, by associating the point cloud cluster with the most overlapping points, or by estimating the position of the point cloud cluster in the previous frame from the elapsed time while taking into account the movement tendency of the point cloud cluster up to the previous frame, and then associating the point cloud cluster with the most overlap with the estimated position.

[0046] In the point cloud cluster relative velocity detection step S740, if a millimeter wave radar is used instead of LiDAR, or if the sensing method is such that the relative velocity can be obtained even with LiDAR, the relative velocity obtained as a sensing result may be used.

[0047] After obtaining the relative speed of the peripheral object with respect to the host vehicle 10 in point cloud cluster relative speed detection step S740, the priority setting unit 113 calculates the collision grace time for each point cloud cluster by dividing the relative distance by the relative speed and inverting the sign for each point cloud cluster in collision grace time calculation step S750. That is, the priority setting unit 113 uses the relative speed to calculate the collision grace time that indicates the time required for the point cloud cluster to come into contact with the host vehicle 10, and sets a priority for the point cloud cluster or the external area in the direction in which the point cloud cluster is located based on the collision grace time.

[0048] In addition, in preparation for the case where the absolute value of the relative velocity is very small and the collision grace period is very large, for example, if the collision grace period exceeds 30 seconds, processing is performed such as clipping at 30 seconds. Also, even if the calculated collision grace period is negative, that is, if the relative velocity is positive, the collision grace period is set to the maximum possible value of 30 seconds, and the point cloud cluster corresponding to the moving away object is treated as a point cloud cluster with a long collision grace period, thereby preventing the recognition process from being given a high priority.

[0049] In the collision grace period notification step S760, the range of the direction in which each point cloud cluster exists is calculated for each point cloud cluster, and the calculated range of the direction is notified together with the collision grace period of the point cloud cluster to the recognition processing priority setting step S735, and the priority is set by the priority setting unit 113. The notification is performed by a method suitable for the implementation form, such as a variable, a shared memory area, or inter-process communication.

[0050] Regarding the orientation, in the first embodiment, image data is acquired by multiple cameras, with the horizontal direction being divided, so when performing recognition processing in units of image data acquired by each camera, only the horizontal direction is required. However, when a configuration using a separate camera in the vertical direction is included, the vertical direction orientation is also required. Furthermore, when images captured by individual cameras are divided horizontally and recognition processing is performed separately for the top and bottom, the vertical direction orientation is also required to determine the priority of recognition processing for each divided image area. When determining the priority of recognition processing for each point cloud cluster, horizontal and vertical direction information is also required to identify the range in which the recognition processing target exists.

[0051] In point cloud cluster recognition processing step S770, the point cloud cluster recognition processing unit 118 performs recognition processing for each point cloud cluster by referring to the point cloud data that constitutes the point cloud cluster. Based on the spatial positional relationship of each point data that constitutes the point cloud cluster and the distribution of signal intensity of the reflected laser light, the type of object is detected and point cloud clusters that appear to be noise are excluded.

[0052] The recognition priority may also be taken into consideration when performing the point cloud cluster recognition process. That is, the recognition process may be performed in order of the point cloud clusters with the shortest collision time calculated in the collision time calculation step S750, or a relatively light recognition process may be performed on point cloud clusters with a long collision time by only performing noise removal processing to detect the presence of objects, and a relatively heavy recognition process may be performed on point cloud clusters with a collision time of a certain period or less by detecting the type of vehicle (four-wheeled or larger vehicle, two-wheeled vehicle, bicycle, pedestrian, etc.).

[0053] By performing the recognition process in order from the point cloud cluster with the shortest collision grace period, it is possible to obtain the advantage of being able to quickly obtain recognition results for objects with high urgency. Also, by performing the recognition process to identify the object type after the collision grace period has fallen below a certain time, it is possible to perform high-load recognition processing only for objects near the vehicle 10 that have a large impact on driving route determination and collision avoidance operations, which has the advantage of being able to reduce the overall recognition processing load.

[0054] The recognition results of the point cloud cluster recognition processing step S770 are sent to a recognition result integration process that integrates the recognition results of the sensing data obtained from multiple external sensors, and the final external recognition result is obtained after synthesizing the recognition results of various directions of the vehicle 10, checking the consistency of the recognition results from the multiple external sensors, and selecting or excluding recognition results that have caused inconsistencies.

[0055] In camera image processing, first, in camera image data acquisition step S715, the external world data acquisition unit 111 acquires image data from each camera. In image correction processing step S725, the image correction processing unit 116 corrects the image data acquired from each camera for image distortion, image brightness correction, color reproduction and correction, image misalignment, etc. At this time, corrections due to individual differences between each camera and errors in installing the camera on the vehicle are also performed. Regarding color, when using image data acquired by an image sensor that uses color filters with different transmission wavelength characteristics such as red, green, and blue that vary depending on the pixel, color restoration is performed using multiple surrounding pixels.

[0056] In the image correction processing step S725, image data acquired from a certain camera may be divided into multiple image regions and processed as necessary. Also, different region divisions may be performed for each camera. For example, if a certain camera acquires an image using a fisheye lens with a wide angle of view, the image data acquired by that camera will have significantly different distortions between the center and periphery of the image. Therefore, it is conceivable to divide the image data for that camera into a center region and multiple periphery regions, cut out each region as a separate image, and perform appropriate distortion correction for each region.

[0057] In the recognition processing priority setting step S735, the priority setting unit 113 determines the priority of the recognition processing for each piece of image data output from the image correction processing step S725 based on the range of directions of each point cloud cluster and the collision allowance time obtained in the collision allowance time notification step S760. However, each piece of image data output from the image correction processing step S725 may be divided into multiple regions on the image, and the priority may be determined for each divided unit.

[0058] The recognition priority is determined by the unit of image data to be processed for recognition, and the point cloud cluster that overlaps with the direction included in the image in a certain direction or more is set as the point cloud cluster to be considered, and the collision grace period of the point cloud cluster is used to determine the priority. The unit of image data to be processed for recognition may be a range corresponding to a point cloud cluster, in which case the priority is determined by the collision grace period of the point cloud cluster.

[0059] The criteria for the degree of overlap when determining whether a point cloud cluster should be considered are determined, for example, taking into account the specifications of the recognition process. That is, the minimum proportion of the object contained in the image required for object recognition in the recognition process is taken into account. If the object portion is not included in a proportion that is recognizable in the recognition process, the object cannot be recognized in the recognition process, so information about point cloud clusters corresponding to such objects is excluded and handled.

[0060] A simple method for determining the recognition priority is to treat it as having a high priority when the collision time is below a certain level. When dividing the recognition priority into multiple stages, the collision time value is divided into stages and a tiered priority is assigned to each level. For example, it is possible to divide the priority into five stages such as urgent, highest, high, medium, and low, and then divide the collision time range into ranges and assign it in order of shortest time.

[0061] In the recognition processing priority setting step S735, when determining the priorities, the priorities may be weighted based on the driving state and operation state of the host vehicle 10. For example, during normal driving, it is conceivable to give a higher priority to images acquired by camera F310 corresponding to the front, and during reversing, give a higher priority to images acquired by cameras R360, RL340, and RR350 corresponding to the rear and diagonally rear. Because the host vehicle 10 often makes large turns when reversing, such as when parking in a garage, the priority of diagonally rearward images is also increased. Because the angle of view of camera RL340 and camera RR350 includes part of the lateral front, the images acquired by these cameras may be divided vertically, and only the image of the area corresponding to the rearward area may be given a higher priority.

[0062] When performing recognition processing on image data in units of point cloud clusters, it is also possible to perform recognition processing preferentially on the area corresponding to the point cloud cluster on the image acquired by the camera with the highest priority weighting among cameras that take images including the orientation in which the point cloud cluster exists.

[0063] In the recognition computation resource amount determination step S745, the computation resource amount determination unit 114 determines the computation resources to be allocated to the recognition processing for each image data unit of the recognition processing performed in the image recognition processing step S755 based on the recognition processing priority determined in the recognition processing priority setting step S735. The computation resources include the processing time, the number of processors used for the processing, the dedicated circuits used for the recognition processing, the memory access bandwidth and memory capacity used for the operation of the processors and dedicated circuits, etc. The order in which the recognition processing is performed is also determined. The computation resource amount determination unit 114 determines the computation resource amount for the recognition processing for each of the multiple image regions (or each image) captured by the cameras F310, FL320, FR330, RL340, RR350, and R360 according to the priority.

[0064] In order to allocate the amount of computational resources according to priority, for low-priority recognition processes, the resolution of the image data used for the recognition process may be lowered, or the frequency of actual recognition processing may be reduced by thinning out frames to be subjected to recognition processing (omitting recognition processing for frames at the timing of thinning out), or recognition processes with logics that differ in recognition performance (recognition rate) and recognition functionality (number of types of objects that can be distinguished) may be prepared, and logic with lower performance and functionality but lighter computational processing may be used. It is possible to accommodate multiple priorities by varying the degree to which the resolution is lowered, the frequency with which frames are thinned out, and switching to recognition logic with different computational processing loads.

[0065] When the recognition frequency is reduced, in order to prevent the recognition process from concentrating on processing of a specific frame or from not performing recognition processing of images in a specific direction relative to the vehicle 10 for a long period of time, the process is carried out by taking into consideration multiple frames and planning when and in which direction to perform recognition processing of image data.

[0066] The order of recognition processing is normally such that the highest priority is performed first, but if it is detected that recognition processing of images in a specific direction has been continuously thinned out for a long period of time, processing of image data in that direction will be performed first except for image data with a particularly high priority, such as an urgent priority. By performing recognition processing in this manner, it is possible to avoid a situation where recognition processing of image data in a certain direction is not performed for a long period of time, even if there is an error in allocating the amount of computing resources and the recognition processing is terminated midway due to a lack of processing time.

[0067] In image recognition processing step S755, the image recognition processing unit 117 performs recognition processing on each image data based on the content determined in recognition computation resource amount determination step S745. Recognition processing is omitted for image data for which it was determined in recognition computation resource amount determination step S745 that recognition processing should be omitted. When performing recognition processing on multiple image data, each image data may be processed sequentially, or parallel processing may be performed using multiple processor cores. Parallel processing and sequential processing may also be combined, such as performing parallel processing on two image data at a time and processing these parallel processes sequentially. If the recognition processing time becomes longer than expected and it is likely to cause problems with processing the next frame, the processing is aborted.

[0068] The recognition result obtained in the image recognition processing step S755 is sent to the recognition result integration processing unit 115, which integrates the recognition results for the sensing data obtained from the multiple external sensors 290, 310 to 360, and after checking the consistency of the recognition results and selecting or excluding inconsistent recognition results, the final external recognition result is obtained.

[0069] 6 and 7, an example is shown in which image data acquired by camera F310, camera FL320, and camera FR330 is divided into multiple image regions and handled, and recognition processing with different priorities for each region is performed by changing the recognition frequency.

[0070] 6, an image acquired by camera FL320 is divided into areas D514 and F516 for handling. An image acquired by camera F310 is divided into areas A511, B512, and C513 for handling. An image acquired by camera FR330 is assumed to be divided into areas E515 and G517 for handling.

[0071] Fig. 7 shows an example of the order in which recognition processing is performed for each area shown in Fig. 6. The letters A to G in Fig. 7 correspond to areas A511 to G517 shown in Fig. 6, respectively, and indicate that recognition processing is performed on the unit enclosed in square brackets. For example, [B, A, C] 612 indicates that recognition processing is performed on area B512, area A511, and area C513 collectively. In other words, this indicates that recognition processing is performed on the entire image acquired by camera F310. [A] 611 indicates that recognition processing is performed on only area A511.

[0072] By grouping the areas of images acquired by the same camera together as enclosed in square brackets and performing recognition processing all at once, even if the image of a certain object is only partially contained in each image area due to image area division, it is possible to perform recognition processing without being affected by area division when processing the images together, thereby improving the recognition rate for images.

[0073] In Figure 7, it is assumed that the vehicle 10 is moving straight ahead at high speed, and the order in which the recognition process is performed is such that area A511 has a high priority, area B512, area C513, area D514 and area E515 have a medium priority, and area F516 and area G517 have a low priority.

[0074] According to this order, in the eight consecutive recognition processes shown in FIG. 7, the recognition process can be performed four times on area A511, two times on area B512, area C513, area D514, and area E515, and once on area F516 and area G517.

[0075] As shown in Figure 7, by appropriately adjusting the order of recognition processes according to their priority, it is possible to increase the recognition frequency of high-priority areas while suppressing the effects of area segmentation of images acquired with the same camera.

[0076] In the first embodiment, the in-vehicle information processing device includes an ECU Type A110 (control unit).

[0077] According to the first embodiment of the present invention, it is possible to realize an in-vehicle information processing device, an in-vehicle information processing system, and an in-vehicle information processing method that can reduce recognition delays for objects with high urgency by increasing the amount of computational resources for areas with high priority compared to areas with low priority.

[0078] In addition, in Example 1, the adjustment of the priority for the recognition of images acquired by a camera was shown, but even when performing recognition processing on sensing data acquired by other external sensors such as LiDAR or millimeter wave radar to detect object types, etc., the generation of a point cloud cluster is performed before the recognition processing, so it is possible to adjust the priority of the recognition processing using the relative speed with respect to the point cloud cluster and the collision grace time.

[0079] Example 2 Next, a second embodiment of the present invention will be described.

[0080] A second embodiment will be described in which the LiDARs are changed from the configuration using the LiDAR A290 shown in the first embodiment to a LiDAR F210 for forward monitoring and a LiDAR R260 for rearward monitoring.

[0081] FIG. 8 is a diagram illustrating an example of a system connection configuration according to the second embodiment.

[0082] In the second embodiment, the vehicle 10 has six cameras as external sensors, a LiDAR F210 for forward monitoring, and a LiDAR R260 for rearward monitoring. The angle of view of each camera is the same as in the first embodiment. Unlike the first embodiment, the LiDAR is for forward monitoring and rearward monitoring, and there is no LiDAR for left and right monitoring. Due to the change in the external sensor configuration, the ECU having a signal processing function including external recognition processing has been changed from ECU Type A110 to ECU Type B120 (control unit).

[0083] The ECU Type B120 also communicates with other ECUs installed in the vehicle 10 via an interface such as CAN. By connecting with other ECUs, it is possible to transmit the results of recognizing the outside world to ECUs that make driving decisions and control driving, and to obtain the driving status of the vehicle 10.

[0084] In Example 2, the method for determining the priority of the recognition process in the range that can be sensed by the LiDAR F210 and the LiDAR R260 is the same as the method shown in Example 1. That is, point cloud clusters are generated from point cloud data obtained by the LiDAR F210 or the LiDAR R260, the collision grace period for each point cloud cluster is calculated, and the priority of the recognition process is increased for the camera image corresponding to the direction of the point cloud cluster that shows an object with a short collision grace period.

[0085] The left and right sides that are not covered by LiDAR are addressed by weighting the priority of the recognition process according to the vehicle behavior and state. An example of changing the weighting according to the vehicle behavior and state will be described with reference to Figures 9A to 9E.

[0086] 9A to 9E, it is assumed that five modes are used as a method for weighting the priority of the recognition process, and the angle of view of the area with higher priority for each mode is shown.

[0087] Mode DFF, shown in Figure 9A, is a mode for high-speed cruising, and is primarily used when cruising at high speeds on expressways and other roads for expressways. When cruising at high speeds, curves are gentle, so the area where priority is increased can be narrowed to the front. However, since distant objects are likely to approach in a short period of time, recognition processing is required for high-resolution images in order to recognize distant objects, and it is desirable to increase the frequency of recognition processing. Therefore, the area where priority is increased for recognition processing is limited to the camera angle of view F410.

[0088] Mode DF shown in Figure 9B is a forward mode, and is mainly used when cruising on public roads. When driving on public roads, there is less need to recognize distant objects than when cruising at high speeds, but it is still necessary to deal with objects suddenly appearing from the left and right and sharp curves, so in addition to the camera angle of view F410, the recognition priority for the camera angle of view FL420 and the camera angle of view FR430 is also increased.

[0089] Mode DR shown in Fig. 9C is the mode for reversing, and is the mode when the reverse gear is engaged. When reversing, the driving speed is slow, but the steering angle is often large when parking in a garage, etc. Therefore, in addition to camera view angle R460, which is the view angle for rear monitoring, the recognition priority is also increased for the camera view angle RL440 and camera view angle RR450, which are mainly used for diagonal rear monitoring.

[0090] Mode DFL shown in Fig. 9D is a mode when the steering angle is turned to the left by a certain amount or more, other than when reversing, and when the left turn signal is on. When the steering angle is turned to the left by a certain amount or more, or when the left turn signal is on, it is expected that the vehicle will turn left or change lanes to the left lane, so in addition to camera angle of view F410 that covers the front, the recognition priority is also increased for camera angle of view FL420 that covers the left front that is expected to be the destination, and camera angle of view RL440 for detecting objects approaching from the rear left or objects that may be hit by a left turn.

[0091] Mode DFR shown in FIG. 9E is a mode when the steering angle is turned to the right by a certain amount or more in a state other than when reversing, and when the right turn indicator is on, and is a mode opposite to Mode DFL in terms of left and right.

[0092] When increasing the priority of the recognition process, if the angle of view for which the priority is to be increased matches the angle of view of each camera as shown in Figures 9A to 9E, the priority of the recognition process can be increased in units of image data acquired by each camera.

[0093] However, if the ECU has sufficient processing performance, for example, in Mode DF shown in Fig. 9B, it is also possible to increase the priority of recognition processing for areas in the images acquired by camera RL340 and camera RR350 that overlap with the camera angles of view FL420 and FR430. Even if an object is in the same direction as seen from vehicle 10, by performing recognition processing on multiple image data acquired by multiple cameras, it is possible to recognize an object that is in a blind spot for one of the cameras and to improve recognition accuracy by comparing the recognition results for each image.

[0094] Strictly speaking, the recognition process for image data acquired by the cameras is a recognition process for image data acquired by each camera that has been subjected to image correction processing. The same applies below.

[0095] When weighting the recognition processing priority determined according to the vehicle behavior and state is taken into account in the recognition processing priority determined from the collision grace time of each point cloud cluster based on the sensing results of the LiDAR F210 and LiDAR R260, for example, the priority weighting result can be treated as the priority itself, and compared with the priority determined from the collision grace time, with the higher priority treated as the priority for that direction. In this case, the weighting of the recognition processing priority determined according to the vehicle behavior and state is also divided into three levels: high, medium, and low, and the recognition priority is set to high, medium, or low corresponding to the weighting.

[0096] When weighting the priority of recognition processing is set to three levels, for example, in Mode DF, in the medium speed range, the camera angle of view F410 is weighted high, and the camera angle of view FL420 and the camera angle of view FR430 are weighted medium, and in the low speed range, the camera angle of view F410, the camera angle of view FL420, and the camera angle of view FR430 are all weighted high. In other words, in the medium speed range, since the vehicle is traveling on a main road where travel is possible at a certain speed, more emphasis is placed on forward monitoring, and in the low speed range, taking into account travel on residential roads, etc., recognition processing can be performed by uniformly increasing the priority for a wide angle of view.

[0097] The priority of the recognition process determined based on the vehicle's behavior and condition is weighted at three levels: high, medium, and low, and the priority of the recognition process using the collision grace period of the point cloud cluster is set to emergency, highest, high, medium, and low. If the higher priority is set as the priority for the direction, the priority from high to low is selected based on the vehicle's behavior and condition as well as the collision grace period of the detected point cloud cluster, and the emergency and highest priorities are determined only by the collision grace period of the detected point cloud cluster.

[0098] That is, in the configuration of Example 2, prioritization of recognition processing using the collision grace time of point cloud clusters is performed only in the forward / backward direction, which can be sensed by LiDAR F210 and LiDAR R260, and therefore, urgent and highest priority are also generated only in the forward / backward direction. Although the host vehicle 10 moves in the forward / backward direction, including at high speeds, in the lateral direction, movement is limited to steering in normal situations where there is no slippage, etc., and since the movement speed is limited, there is little need for remote monitoring.

[0099] Therefore, the configuration of the second embodiment may be considered, taking into consideration the balance between the cost of the sensor and the processing load of the ECU.

[0100] Furthermore, if it is desired to more strictly determine the priority in the horizontal direction, it is possible to perform stereoscopic vision of the area where the angles of view overlap by combining cameras FL320 and RL340 in the left direction and cameras FR330 and RR350 in the right direction, and calculate the spatial position of each pixel from the distance information of each pixel obtained by stereoscopic vision and the position information of that pixel on the image, thereby treating each pixel as a point having spatial position information, and treating the set of these points as a point cloud. Once a point cloud is obtained, point cloud clusters can be obtained in the horizontal direction as well, and the collision grace time for each point cloud cluster can also be obtained, and the priority of the recognition process can be determined based on this information.

[0101] Furthermore, it is also possible to omit only the LiDAR R260 or both the LiDAR F210 and the LiDAR R260, and widen the angle of view of the necessary cameras so that the area around the vehicle 10 that cannot be sensed by the LiDAR can be photographed by two or more cameras, extract a point cloud cluster from the distance information of each pixel obtained by stereoscopic vision using images acquired by multiple cameras, and determine the priority of the recognition process using the point cloud cluster and the collision grace time of each point cloud cluster.

[0102] In the second embodiment, the in-vehicle information processing device includes an ECU Type B 120 (controller).

[0103] In the second embodiment, the same effect as in the first embodiment can be obtained, and in addition, the priority of the recognition processing is determined for the monitoring directions in the forward and backward directions where the relative speed between the vehicle 10 and other vehicles is likely to be high, and the weighting for the left and right sides is determined according to the behavior and state of the vehicle 10, thereby reducing the cost of the sensors required for point cloud acquisition and ensuring the calculation performance of the ECU by limiting the Lidar point cloud processing to the forward and backward directions which are important as sensing targets.

[0104] Example 3 Next, a third embodiment of the present invention will be described.

[0105] Fig. 10 shows an example of the connection configuration of a system that uses multiple ECUs with signal processing functions including recognition processing of data captured from external sensors, based on the configuration shown in Example 2. The configuration of multiple cameras and multiple LiDARs, which are external sensors, is the same as that of Example 2.

[0106] In the configuration shown in FIG. 10, the ECUs having signal processing functions including recognition processing are made up of a front sensing ECU 160 (first control unit), a rear sensing ECU 170 (second control unit), and an integrated ECU 150 (integrated control unit).

[0107] The front sensing ECU 160 is directly connected to the camera F310, the camera FL320, the camera FR330, and the LiDAR F210, and mainly performs external environment sensing processing in front of the host vehicle 10.

[0108] The rear sensing ECU 170 is directly connected to the camera R360, the camera RL340, the camera RR350, and the LiDAR R260, and mainly performs external environment sensing processing behind the host vehicle 10.

[0109] Taking into consideration the amount of sensing data output per unit time from each external sensor, each external sensor is directly connected to front sensing ECU 160 or rear sensing ECU 170.

[0110] The integrated ECU 150 is connected to the front sensing ECU 160, rear sensing ECU 170, and other ECUs mounted on the host vehicle 10 via a CAN or an in-vehicle Ethernet. In addition to integrating the external recognition results processed by the front sensing ECU 160 and rear sensing ECU 170, the integrated ECU 150 determines the priority of recognition processing for each direction of the host vehicle 10 based on information about the point cloud clusters extracted by the front sensing ECU 160 and rear sensing ECU 170 and the behavior and state of the vehicle from other ECUs mounted on the host vehicle 10, and makes adjustments to balance the processing load of the recognition processing of the front sensing ECU 160 and rear sensing ECU 170.

[0111] The front sensing ECU 160 and the rear sensing ECU 170 are directly connected via an inter-ECU image transfer compatible communication channel 165, which is a high-speed communication channel compatible with the transfer of moving image data acquired by multiple cameras. The inter-ECU image transfer compatible communication channel 165 uses, for example, high-speed serial communication. In this case, the communication technology used for digitally connecting cameras and ECUs may also be used. Even if the inter-ECU image transfer compatible communication channel 165 requires two-way communication of moving image data, it may be divided into two channels for data transfer from the front sensing ECU 160 to the rear sensing ECU 170 and data transfer in the opposite direction, and a unidirectional communication channel may be used for each.

[0112] An example of a processing flow in the configuration shown in Fig. 10 is shown in Fig. 11. Fig. 11 shows the processing separately for the front sensing ECU 160, the rear sensing ECU 170, and the integrated ECU 150. Each processing represents processing for one frame, and is repeated for each frame.

[0113] In the Front sensing ECU processing performed by the Front sensing ECU 160, the processing from the LiDAR F point cloud data acquisition step S810 to the LiDAR F point cloud cluster collision grace period calculation step S850 is the same processing as the LiDAR point cloud data acquisition step S710 to the collision grace period calculation step S750 described using Figure 5 in Example 1, and is performed on the point cloud data acquired by the LiDAR F210.

[0114] In LiDAR F point cloud cluster collision grace period notification step S860, the range of the direction in which each point cloud cluster exists is calculated for each point cloud cluster, and this is notified to the integrated ECU 150 together with the collision grace period for that point cloud cluster. The notified data is referenced in image recognition priority setting step S960 performed by the integrated ECU 150. The notification is performed using a communication path such as a CAN that connects the Front sensing ECU 160 and the integrated ECU 150. Regarding orientation, in this third embodiment, the horizontal direction is divided and images are acquired by multiple cameras, so when performing recognition processing on image data acquired by each camera, only horizontal orientation information is required. However, when a configuration using a separate camera in the vertical direction is included, vertical information is also required. Furthermore, when images captured by individual cameras are divided horizontally and recognition processing is performed separately for the top and bottom, vertical information is also required to determine the priority of recognition processing for each divided image area.

[0115] In the LiDAR F point cloud cluster recognition processing step S870, recognition processing is performed on each point cloud cluster by referencing the point cloud data that constitutes the point cloud cluster. This recognition processing detects the type of object and removes point cloud clusters that appear to be noise based on the spatial positional relationship of each point data that constitutes the point cloud cluster and the distribution of signal intensity of the reflected laser light. The recognition results are sent to the integrated ECU 150 in the recognition result integration processing step S980. The transmission is performed using a communication path such as CAN that connects the Front sensing ECU 160 and the integrated ECU 150.

[0116] In step S880 for setting image data transfer to the Rear sensing ECU 170, based on the result of step S970 for specifying the image transfer path mode of the integrated ECU 150, settings are made to send the image data required by the Rear sensing ECU 170 to the Rear sensing ECU 170 via the inter-ECU image transfer compatible communication path 165, regarding the image data acquired by the camera F310, camera FL320, and camera FR330 directly connected to the Front sensing ECU 160 and the data generated based on that image data.

[0117] In F directly connected camera image data acquisition S890, image data is acquired from the camera F310, the camera FL320, and the camera FR330 that are directly connected to the front sensing ECU 160.

[0118] The image data acquired in F directly connected camera image data acquisition step S890 is corrected for image distortion, image brightness correction, color reproduction and correction, image misalignment, etc. in F direct camera image correction processing step S892. At this time, corrections due to individual differences between each camera and errors in installing the camera on the vehicle are also performed. Regarding color, when using image data acquired by an image sensor that uses color filters with different transmission wavelength characteristics such as red, green, and blue that vary depending on the pixel, color restoration is performed using multiple surrounding pixels.

[0119] In the direct camera image correction processing step S892, an image acquired from a certain camera may be divided into multiple image regions as needed, and the multiple images may be divided and processed. Also, different image region divisions may be performed for each camera. For example, when an image is acquired using a fisheye lens with a wide angle of view on a certain camera, distortion differs greatly between the center and periphery of the image. Therefore, it is conceivable to divide the image into a center region and multiple peripheries for that camera, cut out each region as a separate image, and perform appropriate distortion correction for each region.

[0120] Of the image data corrected in step S892 for direct camera image correction, the image data required by rear sensing ECU 170 is transmitted to rear sensing ECU 170 based on the settings made in step S880 for setting image data transfer to rear sensing ECU 170.

[0121] In the F directly connected camera image recognition processing step S900, recognition processing is performed on image data acquired by the camera F310, the camera FL320, and the camera FR330 that are directly connected to the front sensing ECU 160, for image data that should be recognized and processed by the front sensing ECU 160. The method, content, and processing order of each image data item depend on the result of the image recognition calculation resource amount determination step S965 by the integrated ECU 150.

[0122] The recognition processing results of the F directly connected camera image recognition processing step S900 are sent to the recognition result integration processing step S980 of the integrated ECU 150. The transmission is performed using a communication path such as a CAN that connects the Front sensing ECU 160 and the integrated ECU 150. Even for images acquired by a camera directly connected to the Front sensing ECU 160, recognition processing is not performed on image data for which recognition processing is left to the Rear sensing ECU 170.

[0123] In the F indirectly connected camera image data acquisition step S910, the image data required by the Front sensing ECU 160 among the image data acquired from the camera R360, the camera RL340, and the camera RR350 is received via the communication path 165 for image transfer between ECUs.

[0124] In the F indirectly connected camera image recognition processing step S920, recognition processing is performed on the image data to be recognized from the image data received in the F indirectly connected camera image data acquisition step S910. The method and content of the recognition processing for each image data and the processing order for each image data are determined according to the result of the image recognition calculation resource amount determination step S965 by the integrated ECU 150.

[0125] The recognition processing results of the F indirectly connected camera image recognition processing step S920 are transmitted to the recognition result integration processing step S980 of the integrated ECU 150. The transmission is performed using a communication path such as a CAN that connects the front sensing ECU 160 and the integrated ECU 150.

[0126] The rear sensing ECU processing performed by the rear sensing ECU 170 is equivalent to the front sensing ECU processing, except that the directly connected cameras are camera R360, camera RL340, and camera RR350, and the indirectly connected cameras (cameras directly connected to the front sensing ECU 160) are camera F310, camera FL320, and camera FR330.

[0127] In the integrated ECU processing performed by the integrated ECU 150, first, in a vehicle state acquisition step S950, information on the behavior and state of the vehicle 10 required for prioritizing and weighting the recognition processing is acquired. The behavior and state information to be acquired includes the gear state (forward or reverse), the steering state, the turn signal lighting setting, etc.

[0128] In the image recognition priority setting step S960, the priority of the recognition processing for each image to be recognized is calculated using the methods shown in Examples 1 and 2, etc., from the direction of the point cloud cluster detected and the collision grace period obtained based on the sensing results of LiDAR F210 and LiDAR R260 obtained in the LiDAR F point cloud cluster collision grace period notification step S860 and the LiDAR R point cloud cluster collision grace period notification step S865, and the information obtained in the vehicle state acquisition step S950.

[0129] Possible images to be subjected to the recognition process include images acquired by each camera, images divided in association with image correction in F directly connected camera image correction processing step S892 and R directly connected camera image correction processing step S897, and images divided and handled when performing image recognition processing. The image data unit for calculating the priority is determined depending on the unit of image data handled in the recognition process.

[0130] In image recognition computation resource amount determination step S965, the amount of computation resources for recognition processing and the order of recognition processing are determined for each image data for recognition processing based on the result of image recognition priority setting step S960, in the same manner as recognition computation resource amount determination step S745 described in embodiment 1. However, consideration is also given to the division of recognition processing between front sensing ECU 160 and rear sensing ECU 170, and the amount and combination of image data that can be transferred via inter-ECU image transfer compatible communication path 165. The order of recognition processing is also determined for each of front sensing ECU 160 and rear sensing ECU 170.

[0131] The contents determined in image recognition calculation resource amount determination step S965 are transmitted via a path such as CAN to F directly connected camera image recognition processing step S900 and F indirectly connected camera image recognition processing step S920 performed by Front sensing ECU 160, R directly connected camera image recognition processing step S905 and R indirectly connected camera image recognition processing step S925 performed by Rear sensing ECU 170, and are reflected in each image recognition process.

[0132] In the image transfer path mode designation step S970, based on the determination made in the image recognition calculation resource amount determination step S965, transfer setting information is transmitted to the image data transfer setting step S880 to the rear sensing ECU 170 and the image data transfer setting step S885 to the front sensing ECU 170 so that each ECU can refer to the image data required for the recognition processing.

[0133] As a result, based on the contents determined in the image transfer path mode designation step S970, the front sensing ECU 160 and the rear sensing ECU 170 transmit the image data acquired by the camera directly connected to each ECU that is required by the other ECU via the inter-ECU image transfer compatible communication path 165.

[0134] In recognition result integration processing step S980, the recognition results of LiDAR F point cloud cluster recognition processing step S870, F directly connected camera image recognition processing step S900, F indirectly connected camera image recognition processing step S920, LiDAR R point cloud cluster recognition processing step S875, R directly connected camera image recognition processing step S905, and R indirectly connected camera image recognition processing step S925 are integrated to obtain a final external environment detection result, which is provided for driving planning and decision-making processing required for autonomous driving and driving assistance.

[0135] The integration of recognition results involves combining recognition results from various directions relative to the vehicle 10, checking the consistency of recognition results from multiple external sensors, and selecting or excluding recognition results where inconsistencies occur.

[0136] 12 is a diagram showing an example of image data transfer settings via the inter-ECU image transfer compatible communication path 165 in image transfer path mode designation step S970. Fig. 12 shows an example of transfer settings made in units of image data acquired by each camera.

[0137] In FIG. 12, only when a situation arises in which recognition processing has a high priority in one of the eight directions around the vehicle 10, image data is transferred via the inter-ECU image transfer compatible communication path 165, and recognition processing for images acquired by cameras that are not directly connected is performed by one or both of the front sensing ECU 160 and the rear sensing ECU 170.

[0138] The operation of each row shown in Fig. 12 will be described below. When the recognition process priority for the front direction of the vehicle 10 is high, the Front sensing ECU 160 performs recognition process only on image data acquired by the camera F310, and recognition process for image data acquired by cameras other than the camera F310 is performed by the Rear sensing ECU 170. Therefore, the image data acquired by the camera FL320 and the camera FR330 (strictly speaking, data obtained by performing image correction process on image data acquired by the cameras; the same applies hereinafter) is transmitted to the Rear sensing ECU 170 via the inter-ECU image transfer compatible communication path 165.

[0139] When the recognition processing priority for the right front direction of the vehicle 10 is high, the Front sensing ECU 160 performs recognition processing only on image data acquired by the camera F310 and the camera FR330, and recognition processing for image data acquired by cameras other than the camera F310 and the camera FR330 is performed by the Rear sensing ECU 170. Therefore, the image data acquired by the camera FL320 is transmitted to the Rear sensing ECU 170 via the communication path 165 for image transfer between ECUs.

[0140] When the recognition processing priority for the right direction of the host vehicle 10 is high, the Front sensing ECU 160 performs recognition processing only on image data acquired by the camera FR330 and the camera RR350, and recognition processing for image data acquired by cameras other than the FR330 and the RR350 is performed by the Rear sensing ECU 170. Therefore, the image data acquired by the camera F310 and the camera FL320 is transmitted to the Rear sensing ECU 170 via the inter-ECU image transfer compatible communication path 165. In addition, the image data acquired by the camera RR350 is transmitted to the Front sensing ECU 160 via the inter-ECU image transfer compatible communication path 165.

[0141] By collecting and processing image data acquired by camera FR330 and camera RR350, which have a high recognition priority, in front sensing ECU 160, the entire right direction of vehicle 10 can be processed comprehensively by a single ECU, and there is an advantage that the consistency of the directions captured by both cameras can be confirmed by front sensing ECU 160. Furthermore, in order to accommodate high-speed driving, it is considered that front sensing ECU 160 uses an ECU with higher processing performance than rear sensing ECU 170, and such a configuration has the advantage that recognition processing for directions with a high recognition priority can be performed by front sensing ECU 160.

[0142] When the recognition processing priority for the right rear direction of the vehicle 10 is high, the rear sensing ECU 160 performs recognition processing only on image data acquired by the camera R360 and the camera RR350, and the recognition processing for image data acquired by cameras other than the camera R360 and the camera RR350 is performed by the front sensing ECU 160. Therefore, the image data acquired by the camera RL340 is transmitted to the front sensing ECU 160 via the communication path 165 for image transfer between ECUs.

[0143] When the recognition processing priority for the rear direction of the vehicle 10 is high, the rear sensing ECU 170 performs recognition processing only on image data acquired by the camera R360, and recognition processing for image data acquired by cameras other than the camera R360 is performed by the front sensing ECU 160. Therefore, the image data acquired by the camera RL340 and the camera RR350 is transmitted to the front sensing ECU 160 via the inter-ECU image transfer compatible communication path 165.

[0144] If the recognition processing priority is high for the left front, left direction, and left rear direction of the vehicle 10, the directions are the same except for the difference between the left and right directions, i.e., the right front, right direction, and right rear direction, respectively.

[0145] In the third embodiment, as in the second embodiment, if it is desired to more strictly determine the priority in the horizontal direction, it is conceivable to combine cameras FL320 and RL340 in the left direction and cameras FR330 and RR350 in the right direction, and use stereo vision in the areas where the angles of view overlap.

[0146] In stereo vision, it is necessary to refer to both sets of image data acquired by the combined cameras, so it is better to collect the image data of the paired cameras in one ECU and process them. Therefore, when performing processing related to left-right stereo vision, an ECU that will perform processing to obtain a point cloud cluster from the disparity information of each pixel of the image data is determined in advance, and the usage settings for the inter-ECU image transfer compatible communication path 165 are made so that the ECU can refer to the image data required for processing related to the point cloud cluster using stereo vision.

[0147] 13 shows the usage settings of the inter-ECU image transfer communication path 165 required to perform stereoscopic vision in the left and right directions. In FIG. 13, when the front sensing ECU 160 performs stereoscopic vision in the right direction and converts the results into point cloud cluster information, the image data acquired by the camera RR350 is always transmitted from the rear sensing ECU 170 to the front sensing ECU 160 via the inter-ECU image transfer communication path 165. When the rear sensing ECU 170 processes stereoscopic vision in the right direction and obtains point cloud cluster information from the results, the image data acquired by the camera FR330 is always transmitted from the front sensing ECU 160 to the rear sensing ECU 170 via the inter-ECU image transfer communication path 165. The image data acquired by the camera FL320 and the camera RL340 is handled in the same way for the left direction.

[0148] The ECU that obtains point cloud cluster information from the stereo vision and the results thereof may be the front sensing ECU 160 or the rear sensing ECU 170 for both the left and right directions, depending on the configuration of the front sensing ECU 160 and the rear sensing ECU 170. Also, different ECUs may process the left direction with the front sensing ECU 160 and the right direction with the rear sensing ECU 170. It is also possible to dynamically allocate processing taking into account the priority of image recognition processing.

[0149] Because acquisition of point cloud cluster information is required in image recognition priority calculation step S960, stereo vision and the process of calculating point cloud cluster information from the results must be performed before image recognition priority calculation step S960. Therefore, the point cloud cluster information is created using image data from the previous frame using stereo vision. Alternatively, the ECU that performs the process of calculating point cloud cluster information from stereo vision and the results can be fixed, and the camera image data required for stereo vision processing can be always sent to that ECU, and the processes up to the stereo vision processing can be performed at an early stage of processing, separate from the image recognition processing. Stereo vision itself is basically performed by local matching processing of paired images after image correction processing, so it can be processed at a stage before the recognition processing.

[0150] In the third embodiment, the in-vehicle information processing device includes a front sensing ECU 160 (first control unit) and a rear sensing ECU 170 (second control unit). .

[0151] In Example 3, too, it is conceivable to omit only the LiDAR R260, or both the LiDAR F210 and the LiDAR R260, and capture the area around the vehicle 10 that cannot be sensed by the LiDAR using two or more cameras by widening the angle of view of the necessary cameras, obtain a point cloud using distance information calculated from the disparity information of each pixel in stereoscopic vision of the image data obtained by the cameras, extract point cloud clusters from the point cloud, calculate the collision grace time for each point cloud cluster, and determine the priority of the recognition process.

[0152] Except for the left and right directions, the cameras that acquire image data required for stereo vision are directly connected to the same ECU. Therefore, if the ECU performs the stereo vision and the processing that determines point cloud cluster information from the results, there is no need to transfer image data via the inter-ECU image transfer compatible communication path 165 associated with the stereo vision processing.

[0153] According to the third embodiment, the system is equipped with a front sensing ECU 160 and a rear sensing ECU 170, and is configured so that recognition processing is performed by one ECU that is suitable for processing output from a camera relating to a high-priority recognition direction, and recognition processing is performed by another ECU for output from a camera relating to directions other than the high-priority recognition direction. As a result, it is possible to realize an in-vehicle information processing device, an in-vehicle information processing system, and an in-vehicle information processing method that can further reduce recognition delays for objects with high urgency.

[0154] Example 4 Next, a fourth embodiment of the present invention will be described.

[0155] Fig. 14 shows a configuration used in a relatively simple driving assistance system as Example 4. Example 4 has only a camera F310 and a LiDAR F210 as sensors for sensing the external world, and is equipped with an ECU Type C130 (controller) that has a function of recognizing the external world based on information obtained from these sensors.

[0156] The ECU Type C130 communicates with other ECUs installed in the vehicle 10 via an interface such as CAN. By connecting with other ECUs, it is possible to transmit the results of recognizing the outside world to ECUs that make driving decisions and control driving, and to obtain the driving status of the vehicle.

[0157] In the fourth embodiment, image data acquired by camera F310 is subjected to image correction processing such as image distortion correction, and then the image data is divided into three areas, camera angle of view FA411, camera angle of view FB412, and camera angle of view FC413, shown in Fig. 15. These areas correspond to area A511, area B512, and area C513 of the camera F image shown in Fig. 6 as an image.

[0158] Using point cloud data acquired by the LiDAR F210, point cloud clusters are extracted in the same manner as in the first to third embodiments described above, the collision grace time for each point cloud cluster is obtained, and the recognition process priority for each direction of the camera view angle FA411, the camera view angle FB412, and the camera view angle FC413 is calculated based on the direction in which each point cloud cluster is located. Also, recognition process is performed for each point cloud cluster.

[0159] The point cloud data used to create the point cloud cluster may be data obtained by sensing with millimeter wave radar or stereo vision with a camera, as in the above-described Examples 1 to 3. In particular, when stereo vision with a camera is used, the configuration shown in Fig. 14 can be modified to a configuration in which only one stereo camera is connected as an external sensor.

[0160] The weighting of the recognition process priority may also be performed based on the behavior and state of the vehicle, as in the first to third embodiments described above.

[0161] As shown in FIG. 6, the recognition process is performed on the image data of each of the areas A511, B512, and C513 in accordance with the final recognition process priority that reflects the weighting of the recognition process priority.

[0162] The recognition results based on the image data of each area and the point cloud data acquired by LiDAR undergo a recognition result integration process to become the final external recognition results, which are used to process driving plans and decisions required for driving assistance.

[0163] Even with a relatively simple external sensor configuration, as in Example 4 shown in Figure 14, by dividing the image acquired by the camera into multiple areas, recognition processing with adjusted priority according to the degree of urgency is possible.

[0164] By using the relative speed with respect to surrounding objects, even if an object is in the expected travel area, an object moving away from the vehicle 10, i.e., an object with little need to be recognized, can be excluded from the priority objects, thereby reducing the amount of calculation required for the recognition process. Furthermore, by determining the priority of the recognition process using the time until collision, it is possible to increase the recognition priority of an object with little time until collision even in cases where the object is approaching from a surrounding object, and to reduce delays in recognition of objects with high urgency.

[0165] In other words, even when there are many objects around the vehicle 10 or when there are fast-moving objects such as bicycles or automobiles, the priority for recognition processing can be set to appropriately reflect the necessity and urgency, making it possible to perform recognition processing efficiently.

[0166] In the fourth embodiment, the in-vehicle information processing device includes an ECU Type C (control unit).

[0167] In the fourth embodiment, the same effects as those in the first embodiment can be obtained.

[0168] In this specification, LiDAR has been mainly used as a means for measuring the distance from the vehicle 10 to an object and detecting the relative speed between the vehicle 10 and the object, but in the present invention, millimeter wave radar or stereo vision using multiple cameras may also be used.

[0169] Furthermore, even if the external world sensing is done by a method other than LiDAR, millimeter wave radar, or stereo vision using multiple cameras, as long as the means can acquire at least the existence, direction, and distance of an object before the recognition process, the priority of the recognition process can be determined based on the content of these information and the direction of external world sensing, so the sensing means can be any.

[0170] Furthermore, the ECU Type A110 shown in FIG. 4 includes an image correction processing unit 116, an image recognition processing unit 117, and a point cloud cluster recognition processing unit 118. However, the present invention can be implemented even if the image correction processing unit 116 is omitted, or even if either the image recognition processing unit 117 or the point cloud cluster recognition processing unit 118 is omitted.

[0171] Furthermore, it is also conceivable to perform processing similar to recognition processing as a means for measuring the distance from the vehicle 10 to an object or detecting the relative speed and distance between the vehicle 10 and the object. If the processing similar to recognition processing as a means for detecting the relative speed and distance is a simple recognition processing that is performed with a lighter load than the recognition processing on sensing data that is performed after the processing, it is possible to perform the final recognition processing efficiently. Here, the sensing data that is the target of the recognition processing does not need to be limited to camera images, and processing on sensing data obtained by other external sensors may also be performed.

[0172] The external sensor configurations and ECU configurations shown in this specification are examples, and other configurations are acceptable as long as they allow for adjustment of the recognition processing priority based on the techniques shown in this specification. The processing in the ECU can be implemented using any method, such as software, electronic circuits such as logic circuits, or a combination of software and electronic circuits. Furthermore, the ECU can take any form that can realize its functions, such as an SoC (System on Chip) or a board with electronic circuits mounted, rather than just being housed in a single box. [Explanation of symbols]

[0173] 10. Ego-vehicle, 20. Vehicle A, 30. Vehicle B, 40. Vehicle C, 110. ECU Type A, 111. External data acquisition unit, 112. Point cloud cluster generation unit, 113. Priority setting unit, 114. Computational resource amount determination unit, 115. Recognition result integration processing unit, 116. Image correction processing unit, 117. Image recognition processing unit, 118. Point cloud cluster recognition processing unit, 120. ECU Type B, 130. ECU Type C, 150. Integrated ECU, 160. Front sensing ECU, 165. Inter-ECU image transfer communication path, 170. Rear sensing ECU, 210. LiDAR F, 260. LiDAR R, 290. LiDAR A, 310...Camera F, 320...Camera FL, 330...Camera FR, 340...Camera RL, 350...Camera RR, 360...Camera R, 410...Camera angle of view F, 411...Camera angle of view FA, 412...Camera angle of view FB, 413...Camera angle of view FC, 420...Camera angle of view FL, 430...Camera angle of view FR, 440...Camera angle of view RL, 450...Camera angle of view RR, 460...Camera angle of view R, 511...Area A, 512...Area B, 513...Area C, 514...Area D, 515...Area E, 516...Area F, 517...Area G, 611...[A] (processing of area A), 612...[B, A, C] (processing of areas B, A and C), 616...[D] (processing of area D), 617...[E] (processing of area E), 618...[D, F] (processing of areas D and F), 619...[E, G] (processing of areas E and G)

Claims

1. an external data acquisition unit that acquires external data from an external sensor mounted on the vehicle; a point cloud cluster generation unit that performs clustering on a plurality of points constituting the external world data acquired from all or a part of the external world sensors to generate a point cloud cluster; a priority setting unit that sets a priority to the point cloud cluster or an external environment area in which the point cloud cluster exists, based on a relative speed of the point cloud cluster with respect to the vehicle calculated from the external environment data related to the point cloud cluster; a computational resource amount determination unit that determines the amount of computational resources for the recognition process of the external world data in accordance with the priority; an image correction processing unit that corrects an image of the outside world data acquired by the outside world data acquisition unit; an image recognition processing unit that recognizes the image corrected by the image correction processing unit based on the amount of calculation resources determined by the calculation resource amount determination unit; a point cloud cluster recognition processing unit that performs object type detection and noise removal on the point cloud cluster generated by the point cloud cluster generation unit, and performs a recognition process on the point cloud cluster; a recognition result integration processing unit that checks the consistency of the processing result of the point cloud cluster recognition processing unit and the processing result of the image recognition processing unit, selects or excludes inconsistent recognition results, integrates the processing result of the point cloud cluster recognition processing unit and the processing result of the image recognition processing unit, and outputs the integrated result as an external world recognition result; 1. An in-vehicle information processing device comprising: a control unit having:

2. 2. The information processing device according to claim 1, The priority setting unit uses the relative speed to calculate a collision grace period that represents the time required for the point cloud cluster to come into contact with the vehicle when it is assumed that the point cloud cluster will come into contact with the vehicle, and sets the priority to the point cloud cluster or an external area in which the point cloud cluster is located based on the collision grace period.

3. 3. The in-vehicle information processing device according to claim 1, 10. An in-vehicle information processing device, wherein all or some of the external sensors are cameras.

4. 4. The in-vehicle information processing device according to claim 3, The in-vehicle information processing device is characterized in that the computational resource amount determination unit determines the computational resource amount for the recognition process for each image area in the image captured by the camera according to the priority.

5. 4. The in-vehicle information processing device according to claim 3, The in-vehicle information processing device is characterized in that the computational resource amount determination unit changes at least one of the resolution of the image captured by the camera, the cycle of the recognition processing, and the logic of the recognition processing according to the priority.

6. 4. The in-vehicle information processing device according to claim 3, The external world data acquisition unit acquires a plurality of images captured by a plurality of the cameras, and the computational resource amount determination unit determines the amount of computational resources for recognition processing for each of the plurality of images according to the priority.

7. 4. The in-vehicle information processing device according to claim 3, The external environment data acquisition unit acquires a plurality of images captured by a plurality of the cameras, and the computational resource amount determination unit determines the amount of computational resources for recognition processing for each image area in the plurality of images according to the priority.

8. 3. The in-vehicle information processing device according to claim 1, The in-vehicle information processing device, characterized in that the external sensor has a camera and a LiDAR.

9. 3. The in-vehicle information processing device according to claim 1, The in-vehicle information processing device, characterized in that the external sensor includes a camera, a LiDAR for monitoring the front of the vehicle, and a LiDAR for monitoring the rear of the vehicle.

10. 3. The in-vehicle information processing device according to claim 1, The control unit a first control unit that senses an external environment in front of the vehicle; a second control unit that senses the external environment behind the vehicle; and The in-vehicle information processing device, characterized in that the external sensor includes a camera, a LiDAR for monitoring the front of the vehicle, and a LiDAR for monitoring the rear of the vehicle.

11. a plurality of external sensors that detect objects around the vehicle; A control unit; Equipped with The control unit an external data acquisition unit that acquires external data from the plurality of external sensors; a point cloud cluster generation unit that performs clustering on a plurality of points constituting the external world data acquired from all or a part of the external world sensors to generate a point cloud cluster; a priority setting unit that sets a priority to the point cloud cluster or an external environment area in which the point cloud cluster exists, based on a relative speed of the point cloud cluster with respect to the vehicle calculated from the external environment data related to the point cloud cluster; a computational resource amount determination unit that determines the amount of computational resources for the recognition process of the external world data in accordance with the priority; an image correction processing unit that corrects an image of the outside world data acquired by the outside world data acquisition unit; an image recognition processing unit that recognizes the image corrected by the image correction processing unit based on the amount of calculation resources determined by the calculation resource amount determination unit; a point cloud cluster recognition processing unit that performs object type detection and noise removal on the point cloud cluster generated by the point cloud cluster generation unit, and performs a recognition process on the point cloud cluster; a recognition result integration processing unit that checks the consistency of the processing result of the point cloud cluster recognition processing unit and the processing result of the image recognition processing unit, selects or excludes inconsistent recognition results, integrates the processing result of the point cloud cluster recognition processing unit and the processing result of the image recognition processing unit, and outputs the integrated result as an external world recognition result; An in-vehicle information processing system comprising:

12. Detecting objects around the vehicle using multiple external sensors; Acquire external world data from the plurality of external world sensors; The control unit of the in-vehicle information processing device clustering a plurality of points constituting the external world data acquired from all or a part of the external world sensors to generate a point cloud cluster; setting a priority for the point cloud cluster or an external environment area in which the point cloud cluster exists based on a relative speed of the point cloud cluster with respect to the vehicle calculated from the external environment data related to the point cloud cluster; determining an amount of computational resources for the recognition processing of the external world data according to the priority; Correcting the acquired image of the external world data; Recognizing the corrected image based on the amount of computational resources; performing object type detection and noise elimination on the point cloud cluster, and performing recognition processing on the point cloud cluster; The recognition processing result of the point cloud cluster and the recognized corrected image are checked for consistency with each other, and inconsistent recognition results are selected and excluded, and the recognition processing result of the point cloud cluster and the recognized corrected image are integrated and output as an external world recognition result.

10. An in-vehicle information processing method comprising:

Images