Information processing device and information processing method

Abstract

Provided is an information processing device which integrates and processes, for each period, pieces of object information acquired by a plurality of sensors that detect an object, and generates the integrated information, wherein the information processing device is characterized by comprising: a weight setting unit which sets weights for the pieces of object information; and an integration processing unit which integrates and processes the pieces of object information in a processing order determined by at least the weights and generates the integrated information, finishes the integration process in the current period according to the lapse of time or the number of objects pertaining to the pieces of object information provided to the integration process of the current period, wherein the weight setting unit sets the weights of the pieces of object information on the basis of an integration history indicating whether the pieces of object information are provided to the integration process for each period.

Description

[Technical Field]

[0001] The present invention relates to an information processing device and an information processing method for integrally processing targets obtained by sensors. [Background technology]

[0002] Driver assistance systems and autonomous driving systems have been developed to achieve various goals, such as reducing traffic accidents, easing driver stress, improving fuel efficiency to reduce environmental impact, and providing transportation for vulnerable road users to realize a sustainable society. These driver assistance systems and autonomous driving systems are equipped with multiple sensors in the vehicle to monitor the area around the vehicle in place of the driver. In addition, systems have been developed that use the recognition results of multiple sensors installed in the vehicle to automatically brake for specific targets such as pedestrians and other vehicles.

[0003] The following prior art exists as background art in this technical field: Patent Document 1 describes an electronic control device having a priority assignment unit that assigns priorities to sensed data in accordance with the external situation and the situation of the vehicle itself, a priority determination unit that dynamically changes and determines the priorities to be assigned to the data, a data management unit that stores the prioritized data, an application execution unit, and a data selection unit that selects data to be passed from the data management unit to the application execution unit in accordance with the priority (see abstract). [Prior art documents] [Patent documents]

[0004] [Patent Document 1] International Publication No. 2020 / 066305 Summary of the Invention [Problem to be solved by the invention]

[0005] However, while Patent Document 1 describes determining a priority for each target and processing targets with higher priorities, it does not clarify the conditions for terminating the processing for this function. For example, in a scene where the number of detected targets increases, such as when turning right or left at an intersection, and there are many targets important to the control decisions of the driving control device (e.g., pedestrians, oncoming vehicles, etc.), the number of targets to be integrated increases, increasing the processing load and preventing the processing from being completed within a given execution period. In this case, the writing of the results to the database, which is performed as post-processing after the integration process, may terminate abnormally, or previous values ​​may remain when writing, causing control decisions to be made based on the previous values, which could lead to the driving control device making incorrect control decisions and causing automatic braking malfunctions.

[0006] Furthermore, Patent Document 1 describes that for targets with low priority, only counting up is performed during periods when no processing is performed, but if no processing is performed on the target's position estimation, the reliability will remain low when the priority becomes high. As a result, there is a risk of errors in target linking and position estimation during subsequent integration processing.

[0007] In view of the above circumstances, there has been a demand for a method for completing integration processing within a processing cycle even when a large amount of target information is input. [Means for solving the problem]

[0008] In order to solve the above problem, an information processing device according to one aspect of the present invention is an information processing device that generates integrated information by periodically integrating object information acquired by a plurality of sensors that detect objects, and includes: a weight setting unit that sets weights for the object information; and an integration processing unit that generates the integrated information by integrating the object information in a processing order determined at least by the weights, and terminates the integration processing for the current cycle depending on the passage of time or the number of objects related to the object information that has been subjected to the integration processing for the current cycle. The weight setting unit sets the weights for the object information based on an integration history that indicates whether the object information has been subjected to the integration processing for the current cycle. [Effects of the Invention]

[0009] According to at least one aspect of the present invention, even if a large amount of target information is input, the integration process can be completed within the processing period based on the specified processing conditions. Problems, configurations, and effects other than those described above will become apparent from the following description of the embodiments. [Brief explanation of the drawings]

[0010] [Figure 1] 1 is a functional block diagram of an information processing device according to a first embodiment of the present invention, having a function of performing integration processing in order of weight based on integration history. [Figure 2] 1 is a block diagram showing an example of the hardware configuration of an information processing device according to a first embodiment of the present invention. [Figure 3] 5 is a flowchart showing an example of a procedure for weight setting processing by the information processing device according to the first embodiment of the present invention. [Figure 4] 5 is a flowchart showing an example of the procedure of integration processing by an integration processing unit of the information processing device according to the first embodiment of the present invention. [Figure 5] FIG. 10 is a diagram showing an example of a case where targets for integration processing (main processing) are determined based on the number of targets according to the first embodiment of the present invention. [Figure 6] FIG. 2 is a diagram showing an example of a case where a target of integration processing (main processing) is determined using a threshold value according to the first embodiment of the present invention. [Figure 7] FIG. 1 is a diagram showing an overview of tracker management (update) in an information processing device according to a first embodiment of the present invention. [Figure 8] FIG. 1 is a diagram showing an overview of tracker management (new registration) in an information processing device according to a first embodiment of the present invention; [Figure 9] FIG. 10 is a conceptual diagram showing an information processing method of only counting up targets with low priority according to the prior art; [Figure 10] FIG. 1 is a conceptual diagram (1) showing a method for counting up and time synchronization for targets with low priority in an information processing device according to a first embodiment of the present invention. [Figure 11]FIG. 1 is a diagram (1) showing an example of integration processing at a certain integration execution time in an information processing device according to a first embodiment of the present invention. [Figure 12] FIG. 10 is a conceptual diagram (2) showing a method for counting up and time synchronization for targets with low priority in the information processing device according to the first embodiment of the present invention. [Figure 13] FIG. 10 is a diagram (2) showing an example of integration processing at a certain integration execution time in the information processing device according to the first embodiment of the present invention. [Figure 14] 10 is a flowchart illustrating an example of a procedure for integration processing including simplified integration by an information processing device according to a second embodiment of the present invention. [Figure 15] FIG. 10 is a conceptual diagram showing a method for performing simple integration in addition to counting up and time synchronization for targets with low priority in an information processing device according to a second embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0011] Hereinafter, examples of modes for carrying out the present invention (hereinafter referred to as "embodiments") will be described with reference to the accompanying drawings. In this specification and the accompanying drawings, identical components or components having substantially the same functions will be assigned the same reference numerals, and redundant explanations will be omitted. Furthermore, when there are multiple components having the same or similar functions, they may be described using the same reference numerals with different subscripts. Furthermore, when it is not necessary to distinguish between these multiple components, the subscripts may be omitted in the description.

[0012] First Embodiment [Configuration of information processing device] First, the configuration of an information processing device according to a first embodiment of the present invention will be described with reference to Fig. 1. Fig. 1 is a functional block diagram of an information processing device according to the first embodiment of the present invention that has a function of performing integration processing in order of weight based on integration history.

[0013] 1, the information processing device 1 includes a pre-processing unit 10, a weight setting unit 20, and an integration processing unit 30. The information processing device 1 receives output signals from an external sensor 2 and a vehicle behavior detection sensor 3. The information processing device 1 is also connected to a driving control device 5 via a post-processing unit 4.

[0014] The external sensor 2 is one or more sensors that detect targets around the vehicle. As an example, the external sensor 2 includes a camera (visible light, near-infrared, mid-infrared, or far-infrared camera), millimeter-wave radar, LiDAR (Light Detection and Ranging), sonar, a TOF (Time of Flight) sensor, or a sensor that combines these. The detection information of the external sensor 2 includes at least the ID, position, speed, and object type of the target. The target is a point of interest obtained from the information detected by the external sensor 2, and is not limited to moving objects such as humans and vehicles, or structures, but may also include driving lines, holes, light, or its reflections. Note that the external sensor 2 may also be simply referred to as a "sensor."

[0015] The vehicle behavior detection sensor 3 is a group of sensors that detect the speed, yaw rate, and steering angle of the vehicle. For example, the vehicle behavior detection sensor 3 includes a wheel speed sensor, an acceleration sensor, a yaw rate sensor, a steering angle sensor, etc.

[0016] The preprocessing unit 10, the weight setting unit 20, and the integration processing unit 30 will be described in detail later with reference to FIGS.

[0017] [Hardware configuration of information processing device] The information processing device 1 (electronic control device) and the external sensor 2 include a computer (microcontroller) including an arithmetic unit, a memory, and an input / output unit.

[0018] The arithmetic unit includes a processor and executes a program stored in a memory. Some of the processing performed by the arithmetic unit when executing the program may be executed by another arithmetic unit such as an MPU (Micro-Processing Unit). Furthermore, hardware such as an FPGA (Field Programmable Gate Array) or an ASIC (Application Specific Integrated Circuit) may be used as the other arithmetic unit.

[0019] The memory includes ROM, which is a non-volatile storage element, and RAM. ROM stores unchanging programs (e.g., BIOS (Basic Input / Output System)). RAM is a high-speed, volatile storage element such as DRAM (Dynamic Random Access Memory) or a non-volatile storage element such as SRAM (Static Random Access Memory), and stores programs executed by the computing device and data used when the programs are executed.

[0020] The input / output device is an interface that transmits the processing contents of the electronic control device and sensors to the outside and receives data from the outside according to a predetermined protocol. The programs executed by the arithmetic unit are stored in a non-volatile memory, which is a non-transitory storage medium of the electronic control unit or the sensor.

[0021] Fig. 2 is a block diagram showing an example of the hardware configuration of the information processing device 1. A calculator 40 shown in Fig. 2 is hardware used as a so-called computer.

[0022] The computer 40 includes a CPU (Central Processing Unit) 41, a ROM (Read Only Memory) 42, a RAM (Random Access Memory) 43, a non-volatile storage 46, and a network interface 47, which are all connected to a bus.

[0023] The CPU 41 is an example of a processor as a computing device. The ROM 42 and RAM 43 are examples of memory. The nonvolatile storage 46 is a nonvolatile storage element with a larger capacity than the memory. A program for realizing each function of the embodiment of the present invention is stored in the nonvolatile storage 46. The nonvolatile storage 46 is an example of a computer-readable non-transitory recording medium. The program may be stored in the ROM 42.

[0024] The network interface 47 is configured by a communication device that controls communication with other devices. The network interface 47 is an example of an input / output device. The function of each block of the information processing device 1 (FIG. 1) will be described in detail below.

[0025] [Preprocessing section] The pre-processing unit 10 receives as input the target information of the above-mentioned targets detected by the external sensor 2, the vehicle behavior information detected by the vehicle behavior detection sensor 3, and the target information of the fusion target (also called "tracker") which is the result of integrating and processing the target information of multiple targets in the previous cycle, and converts the target information of the above-mentioned targets and the vehicle behavior information into a predetermined unified format. Examples of the conversion process into the predetermined unified format include data conversion (e.g., unit conversion, coordinate conversion), additional information calculation (e.g., target reliability calculation), and time synchronization.

[0026] The target information of the target detected by the external sensor 2 includes at least the position, speed, target ID, and object type. The target ID is assigned in the cycle in which the target is first detected, and the same code is assigned thereafter when the same target is tracked in the previous cycle. The format also includes at least "time" and "position (coordinates)."

[0027] Regarding "time," this refers to estimating the target information of the target at the integration execution time by taking into consideration the time difference between the time detected by the external sensor 2 and the integration execution time, using the time difference. For example, in the case of "position," the calculation is performed as "target position at sensor integration execution time = position at sensor detection time + (time difference x speed at sensor detection time)."

[0028] Furthermore, for fusion targets, since the "time difference = integration execution period", the calculation is, for example, "target position at sensor integration execution time = position at previous execution time + (integration execution period × target speed at previous execution)".

[0029] Furthermore, the position and speed may be estimated taking into consideration the turning behavior of the host vehicle using the host vehicle's speed and steering angle or yaw rate. Regarding coordinates, for example, the target information of the target, which has the sensor installation position as the origin, the forward direction (front-rear direction) of the sensor as the x-axis, and the leftward direction (left-right direction) of the sensor as the y-axis, is converted into target information based on a coordinate system, which has the center of the host vehicle as the origin, the forward direction (front-rear direction) of the host vehicle as the x-axis, and the leftward direction (left-right direction) of the host vehicle as the y-axis. The pre-processing unit 10 outputs the format-converted target information to the weight setting unit 20.

[0030] [Weight setting section] The weight setting unit 20 receives the target information of the target output from the preprocessing unit 10 as input, and sets weights using the target information of the target based on the flowchart shown in FIG.

[0031] FIG. 3 is a flowchart showing an example of a procedure for weight setting processing by the information processing device 1. First, the weight setting unit 20 calculates weights for target information based on preprocessed information (target information, host vehicle behavior information) output from the preprocessing unit 10 (S100). In this weight calculation, weights are calculated for target information required by an application that uses the output results of the information processing device 1. If this application is AEB (Autonomous Emergency Brake), for example, the weight of the target information is set by calculating indicators such as the distance from the host vehicle and TTC (Time To Collision) using the target information of the detected target. This weight setting may be set not only for a single indicator but also for multiple indicators, such as the distance from the host vehicle and whether the target type is a vehicle. In this case, weight coefficients are multiplied according to the importance of the multiple indicators, and the sum is calculated. Note that the targets for which weights are set are targets obtained by individual sensors (sensor targets) and fusion targets.

[0032] Next, after setting weights for all targets, the value of a counter indicating the number of times (number of periods) that the fusion target (tracker) has been excluded from integration processing is referenced to determine whether the number of times that the tracker has been excluded from integration processing is equal to or greater than a specified value (S110). If the number of times that the tracker has been excluded from integration processing is equal to or greater than the specified value (YES determination in S110), weight adjustment is performed so that the tracker will be included in integration processing in the current period (S120). In weight adjustment, the larger the weight, the higher the priority of integration processing, so processing is performed to add the value of "weight due to not having been included in integration processing for a period of equal to or greater than the threshold value" to the weight of the target that meets the condition.

[0033] Then, the counter for the tracker whose weight has been adjusted is reset to a zero value (S130).

[0034] If the number of times that the image was excluded from the integration process in step S110 is less than the specified value (NO determination in S110), or after the process of step S130, this process ends.

[0035] [Integration Processing Section] The integration processing unit 30 receives target information of targets that have been preprocessed by the preprocessing unit 10 and weighted by the weight setting unit 20 as input, and performs integration processing using the target information of targets detected by multiple sensors.

[0036] Fig. 4 is a flowchart showing an example of the procedure of integration processing by the integration processing unit 30 of the information processing device 1. As shown in Fig. 1, the integration processing unit 30 has at least three processing blocks. That is, the integration processing unit 30 includes a grouping unit 30a that groups multiple pieces of target information among targets detected by trackers or sensors, an integration unit 30b that generates target information for the grouped targets (integration processing), and a tracker management unit 30c that newly registers, updates, and deletes fusion targets.

[0037] Before performing the processing by these processing blocks, the integration processing unit 30 determines the target to be integrated based on the specified number of targets (S200), and repeats the same processing in order of priority. Here, the processing for an arbitrary target is described. This priority is the same concept as the priority described in Patent Document 1, and the larger the set weight, the higher the priority of the integration processing.

[0038] After the process of step S200, the integration processing unit 30 determines whether the target object is a target object for integration processing (S210). If the target object is not a target object for integration processing (NO determination in S210), the process proceeds to tracker management in step S230. On the other hand, if the target object is a target object for integration processing (YES determination in S210), the process proceeds to the main process (grouping, integration processing) in step S220.

[0039] (How to determine the target for integration processing (main processing)) Here, a method for determining targets for the integration process (the main process in step S220) according to this embodiment will be described with reference to FIGS.

[0040] FIG. 5 is a diagram showing an example of determining targets for integration processing (main processing) based on their number. Assuming that the predefined condition for targets for integration processing is a predetermined number of targets, this method determines targets for integration processing (main processing) based on the predetermined number of targets. The predetermined number of targets is the number of targets specified for each priority. In this example, high priority is set to three targets, and medium priority is set to five targets. FIG. 5 shows an example in which weights of 100 to 20 and priorities of 1 to 9 are assigned to targets 1 to 9, respectively. Since the main integration processing is performed for the three targets in descending order of priority, targets 1 to 3 corresponding to priorities 1 to 3 are the targets for main integration processing. Note that five targets 4 to 8, which are the targets of medium priority, are the targets for simplified integration processing, which will be described in the second embodiment. Target 9, which has a low priority, is not the target for either the main processing or the simplified integration processing.

[0041] In this way, k targets with the highest weights may be classified as high priority targets, j targets as medium priority targets, and the rest as low priority targets. While the specified number of targets (number) is shown as an example of the specified condition in Figures 4 and 5, the specified condition is not limited to this example and may be a threshold value. An example of determining targets for integration processing (main processing) using a threshold value as the specified condition will be described below with reference to Figure 6.

[0042] FIG. 6 is a diagram showing an example of a method for determining targets for integration processing (main processing) using a threshold value. Assuming that the predefined condition for targets for integration processing is a threshold value, this method determines targets for integration processing (main processing) based on the threshold value. In this example, for example, when the threshold value is expressed as 0 to 100, the high priority threshold is set to 60 or more, and the medium priority threshold is set to 30 or more. Similarly to FIG. 5, FIG. 6 shows an example in which weights of 100 to 20 and priorities of 1 to 9 are assigned to targets 1 to 9, respectively. Since the main integration processing is performed for targets with a weight equal to or greater than the threshold "60," which corresponds to high priority, targets 1 to 5 with weights of 100 to 60 (priorities 1 to 5) are the targets for the main integration processing. Note that three targets 6 to 8 with weights less than "60" to weights equal to or greater than "30," which correspond to medium priority, are the targets for the simplified integration processing described in the second embodiment. Target 9 with a low priority is not the target for either the main integration processing or the simplified integration processing.

[0043] Returning to the description of the flowchart in Fig. 4, in the main process of step S220, the grouping unit 30a determines whether or not the plurality of pieces of detection information for the target that is the subject of the integration process are detection information for the same target, using at least the position information among the plurality of pieces of detection information for the target. If the grouping unit 30a determines that the plurality of pieces of detection information are detection information for the same target, it determines a combination of the plurality of pieces of target information for the target to generate a fusion target (S220a).

[0044] When determining a combination, if processing is performed on target information based on a fusion target from the previous cycle, an identity determination is performed on the target information of the fusion target. Here, the identity determination for the estimated information of the fusion target is performed using at least the position of the target information of the fusion target and the positions of multiple target information relative to the target. For example, an error covariance matrix is ​​calculated from the sensor installation position and specification information, a Mahalanobis distance, which is a probabilistic distance, is calculated from the error covariance matrix, and the Mahalanobis distance is compared with a threshold to determine identity. In other words, in the integration process (main process), grouping is performed based on a threshold determination using a reliability distance (Mahalanobis distance) calculated from the error covariance matrix. If the target information of the fusion target is determined to be identical, the ID of the integration process result based on the corresponding multiple target information becomes the ID of the fusion target determined to be identical. The grouping unit 30a outputs the multiple target information of the targets grouped by fusion target ID to the integration unit 30b.

[0045] The integration unit 30b receives as input a plurality of pieces of detection information for the above-mentioned targets grouped by the grouping unit 30a, and performs a process (integration process) to estimate plausible target information based on the plurality of pieces of detection information for the above-mentioned grouped targets (S220b).

[0046] In one example of the integration method, for example, error characteristics may be given in advance as parameters for each external sensor 2, and the position and velocity of the sensor target with the smallest error characteristic among the grouped target information may be used as the position and velocity of the fusion target. Alternatively, in another example of the integration method, a covariance matrix may be calculated from the target information of the above targets based on the error characteristics, and the position and velocity calculated by probability averaging may be used. Furthermore, instead of giving the sensor error characteristics in advance, they may be estimated during integration, or error characteristics included in the target information of the external sensor 2 may be used. Furthermore, even if the fusion target includes only one detection result from a sensor, the integration unit 30b outputs it as the fusion target.

[0047] In the following explanation, even when a single sensor detection result is grouped, the expression "integration" as a fusion target is used. In this case, the position and velocity are the same as the sensor detection result, or the position and velocity of the target at the integration execution time estimated by the preprocessing unit 10 are used. The integrating unit 30b outputs the integration result to the tracker management unit 30c. After performing integration processing on all targets to be integrated, the integrating unit 30b counts up the number of times integration processing was not performed for trackers on which integration processing was not performed. The integrating unit 30b outputs target information of the integrated fusion target to the tracker management unit 30c.

[0048] Next, if the determination in step S210 is NO or after the processing of step S220b, the tracker management unit 30c manages targets. For example, the tracker management unit 30c receives the fusion target that has been subjected to the integration processing and is output from the integration unit 30b, and overwrites (updates) or newly registers the target information of the tracker according to the content of the integration processing (S230).

[0049] (Update of tracker target information) For example, in the integration process, if target information detected in the current cycle is linked to a tracker that existed in the previous cycle and integrated, the tracker management unit 30c updates the target information of the fusion target that has the same ID as the tracker.

[0050] 7 is a diagram showing an overview of tracker management (update) in the information processing device 1. In FIG. 7, the horizontal axis represents time, and an example is shown in which a fusion target T with a target ID of "ID1" exists at integration execution time t2. The position of the fusion target T with "ID1" is (x, y), and its velocities in the x and y directions are (vx, vy). Before the next integration execution time t3, the camera detection time and radar detection time arrive, and a target C detected by the camera and a target R detected by the radar are obtained. At integration execution time t3, the grouping unit 30a groups the target C and target R into the fusion target T with "ID1" that existed at the previous integration execution time t2, and the integrating unit 30b performs integration processing (generates new target information) on the grouped fusion target T, target C, and target R with "ID1". Then, the tracker management unit 30c updates the target information (position, velocity) of the fusion target T with "ID1".

[0051] (New tracker registration) Furthermore, in the integration process, if targets that were not linked to a tracker that existed in the previous cycle but were detected by the sensor are linked and integrated, the tracker management unit 30c adds the fusion targets as newly generated trackers.

[0052] FIG. 8 is a diagram illustrating an overview of tracker management (new registration) in the information processing device 1. In FIG. 8, the horizontal axis represents time, and an example is shown in which a fusion target T does not exist at integration execution time t1. Before the next integration execution time t2, the camera detection time and the radar detection time arrive, and a target C detected by the camera and a target R detected by the radar are obtained. At integration execution time t2, the grouping unit 30a groups the target C and the target R, and the integration unit 30b performs integration processing on the grouped targets C and R to generate a fusion target T and assigns a target ID (e.g., "ID1") to the fusion target T. Then, the tracker management unit 30c newly registers the fusion target T with "ID1" generated by the integration processing. Note that targets such as sensor targets and fusion targets and their target information are stored in the RAM 43 or non-volatile storage 46.

[0053] Furthermore, if, during the integration process, a tracker is not linked to any target detected by the sensor, and has not been linked in past integration processes, and the number of times that it has not been linked meets a specified value (corresponding to a NO judgment in step S110), the tracker management unit 30c deletes the tracker.

[0054] After completing the tracker management for the above target in step S230, the processing end determination unit 30d determines whether the processing of the next target will fit within the processing time (S240). If it is determined that the processing will fit within the processing time (YES determination in S240), the integration processing unit 30 determines whether the next target is a target for integration processing (S210). On the other hand, if it is determined that the processing will not fit within the processing time (NO determination in S240), the integration processing unit 30 ends the integration processing. The integration processing unit 30 outputs the target information of the above fusion target processed by the tracker management unit 30c to the post-processing unit 4. The integration processing unit 30 generates integrated information (fusion target) by integrating the object information (sensor target, fusion target) in a processing order determined at least by the weights, and ends the integration processing for the current cycle depending on the passage of time or the number of objects related to the object information used in the integration processing for the current cycle.

[0055] [Post-processing section] Returning to the description of the configuration of the information processing device 1 shown in Fig. 1, the post-processing unit 4 receives the target information of the fusion target output from the tracker management unit 30c and writes the target information of the fusion target into a database (not shown) of the system. The database of the system may be configured using the non-volatile storage 46 of the information processing device 1 or another memory. The post-processing unit 4 may be included in the information processing device 1 (for example, a process subsequent to the integrated processing unit 30).

[0056] The driving control device 5 controls the driving of the vehicle (host vehicle) on which the information processing device 1 is mounted, based on the target information of the above-mentioned targets written in the database by the information processing device 1. The driving control device 5 can be configured using an ECU.

[0057] [Processing end conditions] Next, the processing end conditions for the integration processing determined by the processing end determination unit 30d (FIG. 1) of the integration processing unit 30 will be described.

[0058] In the integrated processing unit 30, the processing end determination unit 30d determines whether the next target can be processed each time the processing of a target is completed. If it is determined that the next target can be processed, a predetermined process is performed on the next target, but if it is determined that the next target cannot be processed, the processing of the integrated processing unit 30 ends with the previous target, and the processing of the post-processing unit 4 is started.

[0059] Here, in step S210 described above, the integration processing unit 30 determines whether the next target is a target for which integration processing is to be performed, based on a threshold determination using a predetermined number of targets or weights. If the next target is a target for which integration processing is to be performed, the processing end determination unit 30d estimates the processing time required for the integration processing. For example, in addition to the processing time calculated in advance from the hardware specifications of the ECU and the matrix operations performed in the integration processing, the processing time required for the next integration processing is determined by taking into account an error in the processing time estimated in advance, taking into account an increase in load due to heat generation on the ECU, etc., and the sum of the maximum error and the processing time calculated in advance.

[0060] In addition, the processing completion determination unit 30d refers to the system's timer (not shown) and determines whether the series of processing steps (pre-processing, integration processing, and post-processing) performed by the information processing device 1 can be completed in the remaining time within the processing cycle, based on the time that has elapsed since the information processing device 1 started processing in the current cycle (specifically, the first pre-processing in the current cycle) and the processing time required for the next integration processing that was previously calculated.

[0061] If the processing end determination unit 30d determines that the series of processes can be completed within the processing cycle, the information processing device 1 performs processing related to the next integration process (pre-processing, integration process, post-processing). In this embodiment, the processing end determination unit 30d determines whether the series of processes can be completed by determining the end of processing based on the specified number of targets (quantity) or the remaining time within the processing cycle. Therefore, it is desirable to set the specified number of targets with a margin (predetermined margin) so that the series of processes can be completed within the processing cycle. If it is determined that the series of processes cannot be completed within the processing cycle, the information processing device 1 ends the processing. Furthermore, if the next target is not a target for which integration processing is to be performed, the integration processing ends at that point.

[0062] By determining the end of the process as described above, even if the number of targets detected by the sensor increases, the process can be completed within the processing time given to the fusion function.

[0063] In the second embodiment described below, if there are targets for simple integration, the processing completion determination unit 30d determines whether the series of processes can be completed by both determining processing completion based on the number of specified object targets (quantity) and determining processing completion based on the remaining time within the processing cycle.

[0064] Next, using Figures 9 and 10 to 13, we will explain the difference between the conventional technology that only counts up the cycles (number of times) in which integration processing was not performed for targets with low priority, and the present invention that performs time synchronization in addition to counting up.

[0065] [Traditional count-up method] 9 is a conceptual diagram showing an information processing method of the prior art in which only count-up is performed for targets with low priority. The x-axis represents the coordinate in the forward direction of the vehicle, and the y-axis represents the coordinate in the left direction of the vehicle.

[0066] Regarding the conventional technology, consider a scene in which there are targets C0 detected by a camera (hereinafter referred to as "camera targets") and R0 detected by a radar (hereinafter referred to as "radar targets") at time t0, and there is also a fusion target T0 generated by grouping the camera targets C0 and the radar targets R0, as shown in Fig. 9. In other words, the priority of each target at time t0 is high.

[0067] At time t1, the fusion target T0 at time t0 is determined to be a target with low priority by the weight setting unit 20 and the integration processing unit 30, and is a target that is not subjected to integration processing with the camera target C1 and the radar target R1 at time t1. Therefore, the target information of the fusion target at time t1 is the same as the target information of the fusion target T0 at time t0.

[0068] Thereafter, if the target is determined to be a low-priority target at time t2 and time t3, the fusion target T0 at time t0 is not integrated with the camera target C2 (C3) and radar target R2 (R3) at time t2 (time t3). Therefore, the target information of the fusion target at time t2 and time t3 becomes the same as the target information of the fusion target T0 at time t0.

[0069] At time t4, when the priority of the fusion target T0 becomes high and it becomes the target of integration processing, grouping will be performed for the fusion target T0. However, since the fusion target T0 has not been processed since time t0, the fusion target T0 has the same position and velocity at time t4 as it did at time t0. Therefore, it is determined that the target information of the fusion target T0 is not identical to the camera target C4 and radar target R4 at time t4, and the fusion target T0 is not grouped with the camera target C4 and radar target R4.

[0070] [How to count up and synchronize the time] Next, a difference from the embodiment of the present invention in that time synchronization is performed in addition to counting up the cycles (number of times) in which the integration process was not performed will be described.

[0071] 10 is a conceptual diagram (1) showing a method for counting up and time synchronization for targets with low priority in the information processing device 1. The x-axis represents coordinates in the forward direction of the vehicle, and the y-axis represents coordinates in the left direction of the vehicle. FIG. 11 is a diagram (1) showing an example of integration processing at a certain integration execution time in the information processing device 1.

[0072] In Figure 10, consider a scene in which there is a camera target C0 detected by a camera at time t0, a radar target R0 detected by a radar, and a fusion target T0 generated by grouping the camera target C0 and the radar target R0. In other words, the priority of each target at time t0 is high. Let's assume that the target ID of the fusion target T0 is "ID1."

[0073] At time t1, the fusion target T0 of "ID1" at time t0 is determined by the weight setting unit 20 and the integration processing unit 30 to be a target with a low priority, and is therefore not subjected to integration processing with the camera target C1 and radar target R1 at time t1. However, in this embodiment, the grouping unit 30a and integration unit 30b of the integration processing unit 30 perform time synchronization (update) for the fusion target T0 at time t0 and the camera target C1 and radar target R1 at time t1. In the time synchronization, the integration unit 30b estimates (updates) at least the position of the fusion target T1 of "ID1" at time t1 based on at least the position and velocity of the fusion target T0 of "ID1" generated at time t0. Similarly, at time t2, the position of the fusion target T2 of "ID1" at time t2 is estimated (updated) based on the target information of the fusion target T1 of "ID1" at the previous time.

[0074] 11 shows an example of time synchronization at time t3. At time t3, time synchronization (updating) is also performed for the fusion target T2 time-synchronized at time t2, and the camera target C3 and radar target R3 at time t3. That is, the integrating unit 30b estimates (updates) at least the position of the fusion target T3 of "ID1" at time t3 based on at least the position and velocity of the fusion target T2 of "ID1" generated at time t2.

[0075] 12 is a conceptual diagram (2) showing a method for counting up and time synchronization for targets with low priority in the information processing device 1. The x-axis represents coordinates in the forward direction of the vehicle, and the y-axis represents coordinates in the left direction of the vehicle. FIG. 13 is a diagram (2) showing an example of integration processing at a certain integration execution time in the information processing device 1.

[0076] 12 and 13 show the states of each target from time t3 to t4. At time t4, the position of the fusion target T3' with "ID1" at time t4 is also estimated (updated) based on the target information of the fusion target T3 with "ID1" at the previous time t3. As explained using FIGS. 10 and 11, from time t1 to time t3 when the target priority is low, the error between the sensor target and the registered fusion target can be reduced. Therefore, when the target priority becomes high again at time t4 and integration processing is performed, grouping and integration processing can be performed with the camera target C4 and radar target R4 detected at time t4 based on the target information of the fusion target T3' estimated by time synchronization until before time t4 (here, time t3). By such integration processing, the position of the fusion target T4 after integration processing at least at time t4 can be estimated (updated) from the target information of the fusion target T3', the camera target C4, and the radar target R4.

[0077] As described above, the information processing device (information processing device 1) according to this embodiment is an information processing device that generates integrated information by periodically integrating object information (sensor targets) acquired by multiple sensors that detect objects. The information processing device includes: a weight setting unit (weight setting unit 20) that sets weights for the object information (sensor targets, fusion targets); and an integration processing unit (integration processing unit 30) that integrates the object information (sensor targets, fusion targets) in a processing order determined at least by the weights to generate integrated information (fusion targets) and terminates the integration processing for the current cycle depending on the passage of time or the number of objects related to the object information subjected to the integration processing for the current cycle. The weight setting unit is configured to set weights for the object information based on an integration history (the number of times the object information was excluded from the integration processing) that indicates whether the object information was subjected to the integration processing for the current cycle.

[0078] According to the information processing device of this embodiment having such a configuration, even if a large amount of object information is input, the integration process for the current cycle can be completed within the processing cycle by terminating the integration process for the current cycle depending on the passage of time or the number of objects related to the object information used in the integration process for the current cycle.

[0079] <Second embodiment> The second embodiment is an example in which the information processing device 1 according to the first embodiment has a simplified integration function based on weight settings. The information processing device according to the second embodiment will be described below with reference to Fig. 14 and Fig. 15. In the second embodiment, differences from the first embodiment will be mainly described, and the same components will be assigned the same reference numerals and their description will be omitted.

[0080] FIG. 14 is a flowchart showing an example of the procedure of the integration process including the simplified integration by the information processing device 1 according to the second embodiment. 15 is a conceptual diagram showing a method for performing simple integration in addition to counting up and time synchronization for targets with low priority in the information processing device 1 according to the second embodiment. The x-axis represents the coordinate in the forward direction of the host vehicle, and the y-axis represents the coordinate in the left direction of the host vehicle.

[0081] In the information processing device 1 according to this embodiment, in the processing of the integration processing unit 30 shown in FIG. 14, not only is integration processing performed for targets with high priority (the main processing in step S220), but also simplified integration (step S310) is performed for targets with medium priority (subject to simplified integration).

[0082] First, in the integration processing unit 30 (Figure 1), before performing processing by each processing block, targets to be integrated (main processing) and targets to be simplified integrated are determined based on the weight of the targets (S200a), and the same processing is repeatedly performed in order of priority.

[0083] Next, the integration processing unit 30 determines whether the target object is a target object for integration processing (main processing) (S210). If the target object is a target object for integration processing (main processing) (YES determination in S210), the process proceeds to the main processing (grouping, integration processing) in step S220. On the other hand, if the target object is not a target object for integration processing (main processing) (NO determination in S210), the integration processing unit 30 determines whether the target object is a target object for simple integration (S300). Here, if the target object is not a target object for simple integration (NO determination in S300), the process proceeds to tracker management in step S230. On the other hand, if the target object is a target object for simple integration (YES determination in S300), the process proceeds to the main processing (simple integration, count-up) in step S310.

[0084] For example, as shown in Fig. 6, two different priority (weight) thresholds are set, and the priority of the target is compared with the threshold to determine whether the target is a target for integration processing (YES determination in S210) or a target for simplified integration (YES determination in S300). Also, as shown in Fig. 5, whether a target is a target for integration processing or a target for simplified integration may be determined based on the number of targets (predetermined number of targets) set in advance for each level of priority.

[0085] In the simplified integration of step S310, the integration processing unit 30 performs simplified integration processing, which has a smaller computational load and lower estimation accuracy than the simplified integration processing (step S310a). In the simplified integration of step S310a, the grouping unit 30a does not perform an error covariance matrix calculation as the simplified grouping, but instead, for example, determines that targets within a certain distance from the position of a fusion target are identical to the fusion target. Furthermore, as a simplified position estimation, the integration unit 30b may, for example, use the average value of the target information of the camera targets or radar targets determined to be identical by the simplified grouping as the position of the fusion target after the simplified processing. Alternatively, as another example of simplified position estimation, the integration unit 30b may use, for example, the target information of the radar targets as the x-coordinate and the target information of the camera targets as the y-coordinate, depending on the combination of sensor targets simply grouped in consideration of sensor characteristics.

[0086] After the process of step S310a, the integrating unit 30b counts up the number of times that the integration process has not been performed for the trackers for which the integration process has not been performed (step S310b).

[0087] Next, if the determination is NO in step S300, after the processing of step S220b or after the processing of step S310b, the tracker management unit 30c manages targets. For example, the tracker management unit 30c receives the integrated processed (main processing) or simply integrated fusion target output from the integration unit 30b as input, and overwrites (updates) or newly registers the target information of the tracker according to the content of the processing (S230).

[0088] After completing the tracker management in step S230, the processing end determination unit 30d determines whether the processing time for the next target object can be met (S240), and depending on the determination result, the processing proceeds to step S210 or ends the integration processing.

[0089] By performing this type of integration process (including this process and simplified integration), for fusion targets that are not in a high priority state (e.g., medium priority) at times t1 to t3 in FIG. 15, not only counting up but also time synchronization is performed, and position estimation is also compensated for by simplified integration (S310a). This makes it possible to perform correct integration processing using the simply integrated fusion target when a target that has a higher priority and is the subject of integration processing appears in a later processing cycle. In the example shown in FIG. 15, it can be confirmed that the contents of the target information (position and movement direction) of each sensor target and each fusion target T1 to T3 at times t1 to t3 are more consistent than the examples of sensor targets and fusion targets shown in FIGS. 10 and 12.

[0090] In the above-described embodiment, the on-board electronic control unit (ECU) calculates the correction value for the sensor coordinate transformation (i.e., the integration process), but the correction value for the sensor coordinate transformation may also be calculated by a computer communicatively connected to the vehicle.

[0091] As described above, in the information processing device (information processing device 1) according to this embodiment, the integration processing unit (integration processing unit 30) is configured to perform simplified integration, which is time synchronization and simplified integration processing, on object information (sensor targets, fusion targets) that is not subjected to integration processing (S310).

[0092] In this way, in this embodiment, by performing time synchronization and simple integration processing for targets with low priorities determined by weights, it is possible to estimate object information including positions at each time using a method different from integration processing. Therefore, even when the priority of object information changes from a low priority state to a high priority state, it is possible to perform integration processing with high accuracy.

[0093] Furthermore, in this embodiment, the integration processing unit (integration processing unit 30) is configured to perform simplified integration on object information (sensor targets, fusion targets) that has not been integrated after the integration process is completed. By performing simplified integration after the integration process is completed based on the priority determined by the weight, integration processes with higher processing priorities can be completed more reliably within the processing cycle than simplified integration.

[0094] In this embodiment, the integration processing unit (integration processing unit 30) may be configured to terminate the current cycle of integration processing in accordance with the passage of time, the number of objects related to the object information subjected to the current cycle of integration processing, and the number of objects related to the object information subjected to the current cycle of simplified integration. This allows the current cycle of integration processing to be terminated even if the simplified integration is in progress.

[0095] The present invention is not limited to the above-described embodiments, and various other applications and modifications are possible without departing from the spirit of the present invention as defined in the claims. For example, the above-described embodiments have been described in detail and specifically to clearly explain the present invention, and are not necessarily limited to those including all of the components described. Furthermore, it is possible to replace part of the configuration of one embodiment with a component of another embodiment. It is also possible to add a component of another embodiment to the configuration of one embodiment. It is also possible to add, replace, or delete other components from part of the configuration of each embodiment.

[0096] Furthermore, the above-described configurations, functions, processing units, etc. may be partially or entirely implemented in hardware by, for example, designing them as integrated circuits. Broadly defined processor devices such as FPGAs (Field Programmable Gate Arrays) and ASICs (Application Specific Integrated Circuits) may also be used as the hardware. Furthermore, information such as programs, tables, and files that implement the functions of the embodiments may be stored in a recording device such as a memory, a hard disk, or an SSD (Solid State Drive), or in a recording medium such as an IC card, an SD card, an optical disk, or a magneto-optical disk.

[0097] In the above-described embodiment, the control lines and information lines are those that are considered necessary for the explanation, and not all control lines and information lines in the product are necessarily shown. In reality, it can be considered that almost all components are connected to each other. [Explanation of symbols]

[0098] REFERENCE SIGNS LIST 1...information processing device, 2...external sensor, 3...vehicle behavior detection sensor, 4...post-processing section, 5...driving control device, 10...pre-processing section, 20...weight setting section, 30...integration processing section, 30a...grouping section, 30b...integration section, 30c...tracker management section, 30d...processing end determination section

Claims

1. An information processing device that periodically integrates object information acquired by a plurality of sensors that detect objects to generate integrated information, a weight setting unit that sets a weight to the object information; an integration processing unit that integrates the object information in a processing order determined by at least the weights to generate integrated information, and ends the integration processing in a current cycle in accordance with the passage of time or the number of objects related to the object information that has been subjected to the integration processing in a current cycle; The weight setting unit sets the weight of the object information based on an integration history indicating whether the object information has been subjected to integration processing for each period. Information processing device.

2. The integration processing unit performs time synchronization and simplified integration on the object information that has not been integrated. The information processing device according to claim 1 .

3. The integration processing unit performs the simplified integration on the object information that has not been integrated after the integration processing is completed. The information processing device according to claim 2 .

4. The integration processing unit ends the integration process of the current cycle in accordance with the elapse of time, the number of objects related to the object information subjected to the integration process of the current cycle, and the number of objects related to the object information subjected to the simplified integration of the current cycle. The information processing device according to claim 2 .

5. An information processing method for an information processing device that periodically integrates object information acquired by a plurality of sensors that detect objects to generate integrated information, the method comprising: a process of setting a weight for the object information; generating integrated information by integrating the object information in a processing order determined at least by the weights, and terminating the current cycle of the integration process in accordance with the passage of time or the number of objects related to the object information subjected to the current cycle of the integration process; In the process of setting the weight, the weight of the object information is set based on an integration history indicating whether the object information has been subjected to integration processing for each period. Information processing methods.

Images