Target recognition device and target recognition method

Abstract

A target recognition device for recognizing a target in an outside environment of a vehicle comprises: a first target recognition unit that recognizes a first target in a first observation area observed by a first outside environment observation device mounted on the vehicle; a second target recognition unit that recognizes a second target in a second observation area different from the first observation area and observed by a second outside environment observation device mounted on the vehicle; an overlapping area recognition determination unit that determines whether the first target and the second target are targets recognized in an overlapping area of the first observation area and the second observation area; a base axis conversion processing unit that, when the first target and the second target are determined to be the targets recognized in the overlapping area, converts recognition area information of the first target based on an observation axis representing an observation direction of the first outside environment observation device into first recognition area information based on a predetermined reference axis using an external parameter of the first outside environment observation device, and converts recognition area information of the second target based on an observation axis representing an observation direction of the second outside environment observation device into second recognition area information based on the predetermined reference axis using an external parameter of the second outside environment observation device; and an object integration processing unit that determines whether or not the first target and the second target are the same on the basis of the first recognition area information and the second recognition area information.

Description

Target recognition device and target recognition method

[0001] This invention relates to a target recognition device.

[0002] To reduce the burden on drivers and decrease traffic accidents, Level 1 and Level 2 Advanced Driver Assistance Systems (ADAS), which partially automate one or both of the operation of the accelerator, brakes, and steering wheel, have been developed and put into practical use. Furthermore, Level 3 and Level 4 Autonomous Driving (AD) technologies are being developed to support the mobility of the elderly and others, address driver shortages in the transportation industry, and improve productivity.

[0003] Advanced driver-assistance systems and autonomous driving technologies require technologies to detect objects outside the vehicle and recognize the detected objects as targets.

[0004] Japanese Patent Publication No. 2017-211765

[0005] When multiple cameras observe the area outside the vehicle, the same target may be detected by multiple cameras, leading to the recognition of multiple targets. It is necessary to integrate the targets detected by multiple cameras to recognize a target that matches the real-world environment.

[0006] A typical example of the invention disclosed in the present application is as follows. That is, a target recognition device that recognizes a target in the external environment of a vehicle, comprising: a first target recognition unit that recognizes a first target in a first observation region observed by a first external observation device mounted on the vehicle; a second target recognition unit that recognizes a second target in a second observation region that is different from the first observation region and is observed by a second external observation device mounted on the vehicle; a duplicate region recognition determination unit that determines whether the first target and the second target are targets recognized in a duplicate region of the first imaging region and the second imaging region; a reference axis conversion processing unit that, when it is determined that the first target and the second target are targets recognized in the duplicate region, converts the recognition region information of the first target with respect to a reference axis representing the observation direction of the first external observation device into first recognition region information with respect to a predetermined reference axis using the external parameters of the first external observation device, and converts the recognition region information of the second target with respect to a reference axis representing the observation direction of the second external observation device into second recognition region information with respect to the predetermined reference axis using the external parameters of the second external observation device; and an object integration processing unit that determines whether the first target and the second target are the same based on the first recognition region information and the second recognition region information.

[0007] According to one aspect of the present invention, the same object detected by two or more external observation devices can be integrated into one target. Problems, configurations, and effects other than those described above will be clarified by the description of the following embodiments.

[0008] These are diagrams showing the logical configuration of the target detection and recognition device of Example 1 of the present invention. These are diagrams showing the hardware configuration of the target detection and recognition device of Example 1 of the present invention. These are diagrams showing the recognition results by the first target recognition unit of Example 1 of the present invention. These are diagrams showing the recognition results by the second target recognition unit of Example 1 of the present invention. These are diagrams showing the position information of the target of Example 1 of the present invention. These are diagrams showing the position information of the target of Example 1 of the present invention. These are diagrams showing the position information of the target of Example 1 of the present invention. These are diagrams showing input examples to the target recognition unit of Example 1 of the present invention. These are diagrams showing input examples to the target recognition unit of Example 1 of the present invention. These are diagrams showing the coordinate transformation process of Example 1 of the present invention. These are diagrams showing the object integration process 1 of Example 1 of the present invention. These are diagrams showing the object integration process 2 of Example 1 of the present invention. These are diagrams showing the object integration process 2 of Example 1 of the present invention. These are diagrams showing the object integration process 2 of Example 1 of the present invention. These are diagrams showing the object integration process 3 of Example 1 of the present invention. These are diagrams showing the object integration process 3 of Example 1 of the present invention. These are diagrams showing the object integration process 3 of Example 1 of the present invention. These are diagrams showing the object integration process 4 of Example 1 of the present invention. These are diagrams showing the object integration process 4 of Example 1 of the present invention. These are diagrams showing the object integration process 4 of Example 1 of the present invention. These are diagrams showing the object integration process 4 of Example 1 of the present invention.

[0009] <Example 1> Figure 1 is a diagram showing the logical configuration of the target detection and recognition device 10 of Example 1 of the present invention.

[0010] The target detection and recognition device 10 of Example 1 includes a first target recognition unit 3A, a second target recognition unit 3B, a coordinate transformation processing unit 5, and an object integration processing unit 6.

[0011] The first target recognition unit 3A detects the target to be recognized from the external observation information input from the first external observation device 1A and outputs the first three-dimensional target information 4A, which is the result of recognizing the detected target. The second target recognition unit 3B detects the target to be recognized from the external observation information input from the second external observation device 1B and outputs the second three-dimensional target information 4B, which is the result of recognizing the detected target. The first external observation device 1A and the second external observation device 1B are, for example, cameras, and each external observation device differs in observation direction (optical axis direction), observation range (field of view), and observation accuracy (number of pixels). Generally, vehicles are equipped with multiple external observation devices with different observation directions (for example, a front camera, a right front camera, a left front camera, a rear camera, a right rear camera, and a left rear camera). Also, the observation ranges of each external observation device overlap, and one target is observed by multiple external observation devices. Details of the processing performed by the first target recognition unit 3A and the second target recognition unit 3B will be explained with reference to Figures 3 to 9. The external observation information input to the first target recognition unit 3A and the second target recognition unit 3B may include, in addition to the aforementioned images, signal information such as parallax represented by images and matrices, distance represented by images and matrices, point clouds, and observed intensity obtained from radar, etc.

[0012] The axis conversion processing unit 5 converts the observation information from the first external observation device 1A and the observation information from the second external observation device 1B into a state where the observation direction (optical axis, etc.) is parallel. Details of the processing performed by the axis conversion processing unit 5 will be explained with reference to Figure 10.

[0013] The object integration processing unit 6 integrates the three-dimensional target information 4A and the second three-dimensional target information 4B for each target. Details of the processing performed by the object integration processing unit 6 will be explained with reference to Figures 11 to 19.

[0014] Figure 2 shows the hardware configuration of the target detection and recognition device 10 according to Embodiment 1 of the present invention.

[0015] The target detection and recognition device 10 is composed of an electronic control unit (ECU) having a CPU (Central Processing Unit) 11, memory 12, communication device 13, input / output device 14, storage device 15, and secure device 16. The CPU 11, memory 12, communication device 13, input / output device 14, storage device 15, and secure device 16 are connected to each other via an internal signal line 19 such as a bus.

[0016] The CPU 11 is an arithmetic unit that executes programs stored in the memory 12. By executing predetermined arithmetic processes, the CPU 11 operates as a functional block that provides various functions of the vehicle control device 1 (for example, the first target recognition unit 3A, the second target recognition unit 3B, the axis conversion processing unit 5, and the object integration processing unit 6). The memory 12 has a volatile storage area that temporarily stores data used by the CPU 11 when executing programs. The communication device 13 connects to other vehicle control devices via an in-vehicle network. The input / output device 14 is an interface for inputting and outputting data to the vehicle control device 1. The storage device 15 is accessible by the CPU 11 and has a non-volatile storage area that includes a program area for storing programs executed by the CPU 11 and a data area for storing data used by the CPU 11 when executing programs.

[0017] The secure device 16 can use, for example, a storage area that is physically unrewritable, a storage area that can be written to only once, or a storage area on which access control such as user or process authentication is set. The secure device 16 may store keys for encryption and decryption, digital signatures for verifying programs, digital certificates, settings, verification values, identification information, etc., like an HSM (Hardware Security Module). Furthermore, the secure device 16 may perform encryption / decryption processing, verification processing, signature addition, random number generation, etc.

[0018] Figure 3 shows the recognition result by the first target recognition unit 3A, and Figure 4 shows the recognition result by the second target recognition unit 3B. Figures 5, 6, and 7 show the position information of the targets recognized by the target recognition units 3A and 3B.

[0019] The target recognition units 3A and 3B each detect targets to be recognized from external observation information input from external observation devices 1A and 1B, and output three-dimensional target information 4A and 4B, which are the results of recognizing the detected targets. The targets recognized by the target recognition units 3A and 3B include moving objects such as vehicles, pedestrians, bicycles, and motorcycles, as well as traffic lights, signs, poles, and other obstacles. The target recognition units 3A and 3B may recognize targets by methods such as DNN identification, segmentation, template matching using a model with the characteristics of the target to be recognized, and feature matching. The three-dimensional target information 4A and 4B output from the target recognition units 3A and 3B includes the three-dimensional position of the target and the region and field of view θCam1 in which the target was observed on the observation data of the external observation devices 1A and 1B. The target information includes, for example, the type of recognized target, a tracking ID to uniquely identify the target, the relative velocity of the target in the vehicle coordinate system (horizontal X-axis, depth Z-axis), and the direction θ from which the target is visible from the optical axis center Cam1, as shown in Figure 5. Car1-1 And so on.

[0020] In the example shown in Figure 3, the first target recognition unit 3A detects a vehicle from the camera image input from the first external observation device (forward camera) 1A, labels the detected vehicle with the tracking ID Car1-1, and sets a bounding box in the area where vehicle Car1-1 is located in the image. In the example shown in Figure 4, the second target recognition unit 3B detects a vehicle from the camera image input from the second external observation device (right front camera) 1B, labels each of the detected vehicles with the tracking IDs Car2-1 and Car2-2, and sets bounding boxes in the areas where each of the detected vehicles Car2-1 and Car2-2 is located in the image. In the examples shown in Figures 3 and 4, the vehicle Car1-1 detected by the external observation device 1A and the vehicle Car2-1 detected by the external observation device 1B are the same vehicle.

[0021] Figures 6 and 7 show the position information of targets recognized by the target recognition units 3A and 3B.

[0022] As shown in Figure 6, the first target recognition unit 3A takes the horizontal field of view of the external observation device 1A as θCam1 and moves to the right from the optical axis center Cam1 θ Car1-1 Vehicle Car1-1 is recognized at a distance of θ. In the image, the bounding box of the detected vehicle Car1-1 is to the right of the optical axis center Cam1. Car1-1 It is set at a distance of only that much.

[0023] As shown in Figure 7, the second target recognition unit 3B takes the horizontal field of view of the external observation device 1B as θCam2 and moves to the left from the optical axis center Cam2 θ Car2-1 Vehicle Car2-1 is recognized at a distance of θ. In the image, the bounding box of the detected vehicle Car2-1 is to the left of the optical axis center Cam2. Car2-1 It is set at a distance of only that much.

[0024] In Figures 6 and 7, the position of the bounding box should be determined according to the characteristics of the technology used in the observation device and object recognition unit, such as the coordinates of the center of the bounding box (for example, the intersection of the diagonals) or the lower end (for example, the midpoint of the line closest to the road surface).

[0025] Figure 8 shows an example of observation results input to the first target recognition unit 3A, and Figure 9 shows an example of observation results input to the second target recognition unit 3B.

[0026] The same problem occurs with targets detected by stereoscopic vision using multiple cameras. For example, as shown in Figure 8, the first target recognition unit 3A performs stereoscopic vision using images captured by a wide-angle camera F observing the front of the vehicle and images captured by a surround camera F observing the front of the vehicle. Also, as shown in Figure 9, the second target recognition unit 3B performs stereoscopic vision using images captured by a wide-angle camera FR observing the front right of the vehicle and images captured by a surround camera R observing the right side of the vehicle.

[0027] This necessitates determining whether the targets recognized from two or more stereoscopic views are identical.

[0028] Figure 10 shows the axis conversion process performed by the axis conversion processing unit 5.

[0029] The axis conversion processing unit 5 performs calculations related to the observation axes (optical axes in the case of cameras) of the external observation devices 1A and 1B, and converts the observation information of the first external observation device 1A and the observation information of the second external observation device 1B so that their observation directions (optical axes) are parallel.

[0030] For example, the direction of the optical axis between two cameras, which are external observation devices, is known as an external parameter. In this case, the coordinate region of the target captured in the image can be expressed as an angle from the optical axis center on the image. Therefore, the angle obtained from the coordinate region on the image is transformed so that the optical axis centers become parallel based on the difference in the orientation of the optical axis of the external parameter.

[0031] If the difference in camera placement is sufficiently small compared to the distance to the object, the angle from the optical axis center of the target can be used for object integration simply by aligning the orientation of the optical axis centers. Furthermore, by transforming these optical axis centers, the intersection point of the lines extending at the angle to the target on the image can be calculated. If a front-facing camera capturing the front of a vehicle is positioned on the Z-axis (depth axis of the axle), and a side camera capturing the right side of the vehicle is positioned on the X-axis, and the optical axis of the side camera is aligned with the optical axis of the front camera using the intersection point as the axis center, the two lines viewing the same object are expressed by the following equations: z = tanθfx + Zf and z = tanθsx + Zs

[0032] Since the camera's mounting position is known relative to the axis, Zf and Zs are known constants. Also, θf and θs can be calculated from reflections in the image. Therefore, by solving the simultaneous equations, the three-dimensional position of the intersection point of the two lines can be calculated and used for integration judgment.

[0033] For example, the optical axis center Cam1 of the external observation device 1A shown in Figure 6 is facing forward of the vehicle (in the positive Z-axis direction), while the optical axis center Cam2 of the observation information of the external observation device 1B shown in Figure 7 is facing to the right of the positive Z-axis direction of the vehicle. Therefore, the optical axis center Cam2 of the observation information of the external observation device 1B is converted to a direction parallel to the optical axis center Cam1 of the observation information of the external observation device 1A. Figure 10 shows the optical axis center Cam2' after conversion to be parallel to the optical axis center Cam1, and the converted image that matches the converted optical axis center Cam2'.

[0034] The object integration process performed by the object integration processing unit 6 is described below. In the object integration process, if three-dimensional information of the target (such as a 3D bounding box) is obtained, the angle from the optical axis center on the image of the captured target is used to determine whether the targets are the same.

[0035] If the angle between the two objects obtained in the aforementioned axis transformation process is below a predetermined threshold, the objects are determined to be the same object and the process of integrating the targets is executed (object integration process 1, 2).

[0036] In addition, separate from the above, if the difference in the three-dimensional position obtained by the target recognition unit 3A is less than or equal to a threshold, based on the intersection point in three-dimensional space of two lines obtained from angles on the image, the system may determine that the objects are the same and perform a process to merge the targets. Target merging may involve merging management information such as the ID assigned to the target, as well as position, velocity, orientation, etc. The merging method may, for example, adopt the one with the higher detection count based on the ID of the management information, or calculate the average of the two values.

[0037] Furthermore, based on the characteristics of the external observation device, targets observed by high-precision external observation devices may be selected. For example, it is advisable to use information on targets detected in images taken by a camera that are close to the optical axis and have minimal distortion.

[0038] The following is a concrete example of object merging.

[0039] <Object Integration Process 1> Figure 11 shows the object integration process 1 executed by the object integration processing unit 6.

[0040] When the distance to the target object to be detected is greater than a predetermined threshold and the object is sufficiently far from the vehicle, or when the distance between the two external observation devices 1A and 1B is less than a predetermined threshold and the two external observation devices 1A and 1B are sufficiently close, θ Car1-1 ≈θ' Car2-1 holds. Therefore, when any of the following determination criteria is satisfied, it can be determined that the target objects included in the two observation results are the same. The distance to the target object to be detected may use the value measured by a stereo camera or a radar. The installation positions (distances) of the external observation devices 1A and 1B are determined by design values.

[0041] Condition 1: The difference between θ Car1-1 (see Fig. 6) and θ' Car2-1 (see Fig. 10) is less than or equal to a predetermined threshold. Condition 2: Based on either θ Car1-1 or θ' Car2-1 as a reference, the other is within a predetermined range. In this case, it is advisable to use a high-precision external observation device as a reference. For example, it is advisable to use a camera close to the optical axis center before the base axis conversion as a reference.

[0042] <Object integration process 2> Figs. 12, 13, and 14 are diagrams showing the object integration process 2 executed by the object integration unit 6. The object integration process 2 can be applied when the target object to be detected is sufficiently far from the vehicle, or when the distance between the two external observation devices 1A and 1B is sufficiently close.

[0043] The object integration process shown in Figs. 12, 13, and Fig. 14 may perform the determination shown in Fig. 11 using the left and right positions in the observation image of the detection area (bounding box) of the target object. For example, as shown in Fig. 12, the left end of the detection area (bounding box) of the vehicle Car1-1 in the image observed by the external observation device 1A is θ Car1-1L away from the optical axis center Cam1 in the right direction, and the right end is θ Car1-1R away from the optical axis center Cam1 in the right direction. Similarly, as shown in Fig. 13, the left end of the detection area (bounding box) of the vehicle Car2-1 in the image observed by the external observation device 1B is θ' Car2-1L away from the optical axis center Cam2 in the right direction, and the right end is θ' Car2-1R away from the optical axis center Cam2 in the right direction.

[0044] As shown in Figure 14, θ Car1-1L , θ Car1-1R θ' Car2-1L , and θ' Car2-1R θ is defined Car1-1L and θ' Car2-1L The difference is less than or equal to a predetermined threshold, and θ Car1-1R and θ' Car2-1R If the difference between θ is below a predetermined threshold, it can be determined that the targets included in the two observation results are the same. Car1-1L and θ' Car2-1L The difference is less than or equal to a predetermined threshold, or θ Car1-1R and θ' Car2-1R If the difference between the two observation results is below a predetermined threshold, it may be determined that the targets included in the two observation results are the same.

[0045] θ Car1-1L ≒θ' Car2-1L , or θ Car1-1R ≒θ' Car2-1R If this condition is met, it can be determined that the target being detected is the same object because it is sufficiently far from the vehicle, or the distance between the two external observation devices 1A and 1B is sufficiently close.

[0046] The aforementioned object integration processes 1 and 2 use large thresholds when the objects are close together and the distance between the two observation devices is large, which reduces the accuracy of identifying identical objects. Therefore, it is preferable to use the object integration process shown below.

[0047] <Object Integration Processing 3> Figures 15, 16, and 17 show the object integration processing 3 executed by the object integration processing unit 6.

[0048] For example, as shown in Figure 15, the estimated position of vehicle Car1-1 and the angle θ from the optical axis center Cam1 in the image are shown. Car1-1 Then, using the mounting position information of the external observation device 1A, the position of vehicle Car2-1 (angle θ' in the image) as captured by the external observation device 1B is determined. Car1-1) is calculated. As shown in Figure 16, the relationship between the mounting positions of the external observation device 1A and the external observation device 1B in the X-axis and Z-axis directions, Xc1-c2 and Zc1-c2, is known. Since the estimated values ​​of Xc1 and Zc1 are obtained, the angle θ' from the optical axis center Cam1 can be calculated using the principle of triangulation. Car1-1 The calculated angle θ' can be calculated. Car1-1 This is the angle from the optical axis center Cam2 as seen from the second external observation device 1B. Then, as shown in Figure 17, the calculated θ' Car1-1 ≒θ' Car2-1 If this condition is met, it can be determined that the targets included in the two observation results are the same.

[0049] In the aforementioned object integration process 3, the data is converted from the first external observation device 1A to the second external observation device 1B. However, as mentioned in object integration process 1, it is preferable to use a high-precision external observation device as the reference. Object integration process 3 may also be combined with object integration process 2.

[0050] <Object Integration Process 4> In the object integration process 3 described above, the error in the estimated position of Car1-1 causes θ' Car1-1 ≒θ' Car2-1 This may not be true. However, the vehicle Car1-1 that is the target of recognition is at position (Xθ' Car1-1 , z) and position (x, Zθ' Car1-1 It exists in the vicinity of the straight line connecting the two points. Therefore, the object integration process 4 shown below may be used.

[0051] Figures 18, 19, 20, and 21 show the object integration process 4.

[0052] As shown in Figure 18, for vehicle Car1-1 included in the observation results from the first external observation device 1A, the depth position is estimated value z Car1 , direction θ' Car1-1 The horizontal position in this case is angle Xθ' Car1-1 = zCar1 × tanθ' Car1-1 It is represented as such, and the horizontal position is estimated by x Car1 , direction θ Car1-1 The depth position in this case is Zθ' Car1-1 = x Car1 ÷tanθ' Car1-1It is represented as follows. The object of recognition, Car1-1, is at position (Xθ' Car1-1 , z Car1 ) and position (x Car1 , Zθ' Car1-1 It exists in the vicinity of the line connecting the two points.

[0053] Similarly, as shown in Figure 19, for vehicle Car2-1 included in the observation results from the second external observation device 1B, the depth position is estimated value z Car2 , direction θ' Car2-1 The horizontal position in this case is angle Xθ' Car2-1 = z Car2 ×tanθ' Car2-1 It is represented as such, and the horizontal position is estimated by x Car2 , direction θ Car2-1 The depth position in this case is Zθ' Car2-1 = x Car2 ÷tanθ' Car2-1 It is represented as follows. The object of recognition, Car2-1, is at position (Xθ'). Car2-1 , z Car2 ) and position (x Car2 , Zθ' Car1-1 It exists in the vicinity of the line connecting the two points.

[0054] Then, as shown in Figure 20, the aforementioned position (Xθ' Car1-1 , z Car1 ) and position (x Car1 , Zθ' Car1-1 A rectangle with a straight line connecting the two corners as its diagonal, and position (Xθ' Car2-1 , z Car2 ) and position (x Car2 , Zθ' Car1-1 A rectangle is generated with the line connecting the two points as its diagonal. If the overlap of the two rectangles is greater than a predetermined value, it can be determined that the targets included in the two observation results are the same.

[0055] Furthermore, as shown in Figure 21, for example, if the overlap between the first ellipse, whose foci are the two straight line ends derived from the observation results of the first external observation device 1A, and the second ellipse, whose foci are the two straight line ends derived from the observation results of the second external observation device 1B, is greater than a predetermined value, it can be determined that the targets included in the two observation results are the same.

[0056] In this way, a figure is created from each observation result based on two lines estimated in the image, and if the figure created from each observation result falls within a predetermined range, it can be determined that the targets included in the two observation results are the same.

[0057] In the object integration processes 1 to 4 described above, the type of object obtained by the first object recognition unit 3A and the second object recognition unit 3B may be considered when determining if two objects are identical. For example, if the type of object (vehicle, motorcycle, pedestrian, etc.) is known, objects of a different type will not be determined to be identical. However, if the object type is easily mistaken and the reliability of the object type is low, the determination of whether the objects are identical may be made.

[0058] Furthermore, if the movement of the targets (e.g., speed, direction of movement) is known, this information may be used to reject the identification. For example, targets moving in different directions and passing each other should not be determined to be the same.

[0059] These judgments can be made based on the confidence level of each element (for example, the number of times it has been recognized consecutively).

[0060] The correlation of features such as texture and color of identical targets may be compared with a predetermined threshold, and if the correlation is high, it may be determined whether the targets are identical.

[0061] Additionally, if the front, sides, and rear of the vehicle are known, it is advisable to convert the area captured by each camera.

[0062] If the head, body, arms, and legs of pedestrians and other objects can also be identified, it is advisable to similarly convert the area captured by each camera.

[0063] When segmentation allows for the division of regions pixel by pixel, the densest information is available, enabling accurate identification of identical objects. Alternatively, the proportion of the visible area can be used to determine if objects of the same type are identical, even if they overlap.

[0064] Furthermore, while wide-angle camera F, which observes the front of the vehicle, detects part of the rear and side of the vehicle in its images, if wide-angle camera FR detects part of the rear of the vehicle, wide-angle camera FR does not detect the side of the vehicle. Therefore, the information on the rear of the vehicle detected by wide-angle camera F and the rear of the vehicle detected by wide-angle camera FR (e.g., the angle from the optical axis center) can be converted into a base axis to determine if they are the same target. This method allows for a more accurate determination of whether they are the same target, as it does not include the side of the vehicle.

[0065] As described above, in the target detection and recognition device 10 of the embodiment of the present invention, the position on the image of a target detected by two or more external observation devices (e.g., cameras) with different field angles and non-parallel optical axes can be uniquely determined. Therefore, using the installation parameters of the mounting position of the external observation devices, the angle from the optical axis center of the target in the image captured by one external observation device is converted to the angle from the optical axis center of the target in the image captured by another external observation device, and the detected targets are associated. That is, targets whose angles from the optical axis center match after conversion can be determined to be the same object. According to the method of this embodiment, regardless of the position detection accuracy of the monocular camera, the same object detected by two or more external observation devices can be associated and integrated into a single target.

[0066] It should be noted that the present invention is not limited to the embodiments described above, but includes various modifications and equivalent configurations within the spirit of the attached claims. For example, the embodiments described above are described in detail to make the present invention easier to understand, and the present invention is not necessarily limited to having all of the described configurations. Furthermore, some of the configurations of one embodiment may be replaced with those of another embodiment. Furthermore, some of the configurations of one embodiment may be added to those of another embodiment. Furthermore, some of the configurations of each embodiment may be added, deleted, or replaced with those of other embodiments.

[0067] Furthermore, each of the aforementioned configurations, functions, processing units, and processing means may be implemented in hardware, for example, by designing them as integrated circuits, or they may be implemented in software by having a processor interpret and execute programs that realize each function.

[0068] Information such as programs, tables, and files that implement each function can be stored in memory, hard disks, SSDs (Solid State Drives), or recording media such as IC cards, SD cards, and DVDs.

[0069] Furthermore, the control lines and information lines shown are those deemed necessary for explanation purposes and do not necessarily represent all control lines and information lines required for implementation. In reality, it can be assumed that almost all components are interconnected.

Claims

1. A target recognition device for recognizing targets in the external environment of a vehicle, comprising: a first target recognition unit that recognizes a first target in a first observation area observed by a first external observation device mounted on the vehicle; a second target recognition unit that recognizes a second target in a second observation area different from the first observation area, observed by a second external observation device mounted on the vehicle; and an overlapping area recognition determination unit that determines whether the first target and the second target are targets recognized in the overlapping area of ​​the first observation area and the second observation area. A target recognition device comprising: a base axis conversion processing unit that, when it is determined that the first target and the second target are targets recognized in the overlapping region, converts the recognition region information of the first target, based on the observation axis representing the observation direction of the first external observation device, into first recognition region information based on a predetermined reference axis using the external parameters of the first external observation device, and converts the recognition region information of the second target, based on the observation axis representing the observation direction of the second external observation device, into second recognition region information based on the predetermined reference axis using the external parameters of the second external observation device; and an object integration processing unit that determines whether the first target and the second target are the same based on the first recognition region information and the second recognition region information.

2. A target recognition device according to claim 1, wherein the first recognition area information includes at least one of a first recognition position indicating the recognition position of the first target with respect to the reference axis, a first recognition angle indicating the recognition angle of the first target with respect to the reference axis, and a first recognition area indicating the recognition area of ​​the first target with respect to the reference axis, and the second recognition area information includes at least one of a second recognition position indicating the recognition position of the second target with respect to the reference axis, a second recognition angle indicating the recognition angle of the second target with respect to the reference axis, and a second recognition area indicating the recognition area of ​​the second target with respect to the reference axis.

3. A target recognition device according to claim 2, wherein the object integration processing unit determines that the first target and the second target are the same when the difference between the first recognition angle and the second recognition angle is less than a predetermined threshold.

4. A target recognition device according to claim 3, wherein each of the first recognition angle and the second recognition angle includes one or both of a horizontal angle from the observation axis in the image and a vertical angle from the observation axis in the image, and the object integration processing unit determines that the first target and the second target are the same if one or both of the horizontal angle difference and the vertical angle difference are less than a predetermined threshold.

5. A target recognition device according to claim 4, wherein the horizontal angle from the observation axis is at least one of the angle from the observation axis to the left end of the recognition area and the angle from the observation axis to the right end of the recognition area, the vertical angle from the observation axis is at least one of the angle from the observation axis to the upper end of the recognition area and the angle from the observation axis to the lower end of the recognition area, the object integration processing unit calculates the horizontal angle difference from the observation axis using at least one of the angle from the observation axis to the left end of the recognition area and the angle from the observation axis to the right end of the recognition area, calculates the horizontal angle difference from the observation axis using at least one of the angle from the observation axis to the upper end of the recognition area and the angle from the observation axis to the lower end of the recognition area, and determines that the first target and the second target are the same if the angle difference is smaller than a predetermined threshold using one or more combinations of the calculated angle differences.

6. A target recognition device according to claim 1, wherein the object integration processing unit determines that the first target and the second target are the same, and integrates the first target and the second target as a single target.

7. A target recognition method performed by a target recognition device for recognizing targets in the external environment of a vehicle, wherein the target recognition device comprises a computing device for performing predetermined calculation processing and a storage device accessible by the computing device, and the target recognition method comprises: a first target recognition procedure in which the computing device recognizes a first target in a first observation area observed by a first external observation device mounted on the vehicle; a second target recognition procedure in which the computing device recognizes a second target in a second observation area different from the first observation area, observed by a second external observation device mounted on the vehicle; and an overlapping area recognition determination procedure in which the computing device determines whether the first target and the second target are targets recognized in an overlapping area between the first observation area and the second observation area. A target recognition method comprising: a base axis conversion procedure in which, when the computing device determines that the first target and the second target are targets recognized in the overlapping region, the computing device converts the recognition region information of the first target, based on the observation axis representing the observation direction of the first external observation device, into first recognition region information based on a predetermined reference axis using the external parameters of the first external observation device, and converts the recognition region information of the second target, based on the observation axis representing the observation direction of the second external observation device, into second recognition region information based on the predetermined reference axis using the external parameters of the second external observation device; and an object integration procedure in which the computing device determines, based on the first recognition region information and the second recognition region information, whether the first target and the second target are the same.

Images