Information processing system, information processing device, computer-readable recording medium, and information processing method

Abstract

The information processing system (100) comprises: a distance measurement processing unit (101) for detecting the distance and direction of each of a plurality of targets, i.e., a plurality of distance measurement targets, and generating distance measurement information indicating the distance and direction of each of the plurality of distance measurement targets; an image processing unit (104) for capturing an image, generating image data indicating the image, determining the distance, direction and category of a target, i.e., an image capture target, contained in the image, and generating image capture information indicating the distance, direction and category of the image capture target; and a control unit (114) for determining a provisional value indicating the size of the plurality of distance measurement targets using the image capture information, determining a plurality of areas, i.e., a plurality of provisional areas, into which the plurality of distance measurement targets are projected in the image according to the provisional value and the distance measurement information, and calculating a coincidence probability indicating the possibility that the image capture target coincides with each of the plurality of distance measurement targets using the size of overlap between each of the plurality of provisional areas and an area, i.e., an object area, in which the image capture target is captured in the image.

Description

Technical Field

[0001] The present invention relates to an information processing system, an information processing device, a computer-readable recording medium, and an information processing method. Background Art

[0002] In vehicle control systems such as driver assistance systems and autonomous driving systems, the detection accuracy of sensors is improved by using multiple sensors for supplementation or redundancy.

[0003] For example, Patent Document 1 discloses an object detection device that performs sensor fusion using a radar sensor device and a camera sensor device.

[0004] In a conventional object detection device, when a target detected by a radar sensor device is included within a threshold range corresponding to the lateral width of a target detected by a camera sensor device, the target detected by the camera sensor device is determined to be the same as the target detected by the radar sensor device.

[0005] Prior art literature

[0006] Patent Literature

[0007] Patent Document 1: Japanese Patent Application Publication No. 2014-6123 Summary of the invention

[0008] Problems to be solved by the invention

[0009] However, in the existing object detection device, if the target detected by the radar sensor device is included in the range of the threshold value corresponding to the lateral width of the target detected by the camera sensor device, any distant target is judged to be the same, and therefore, a judgment error sometimes occurs in which different targets are judged to be the same target.

[0010] Therefore, one or more aspects of the present invention aim to reduce errors in determining the identity of a target.

[0011] Means for solving problems

[0012] An information processing system according to one embodiment of the present invention is characterized in that the information processing system comprises: a distance measurement processing unit that detects the distance and direction of each of a plurality of distance measurement targets existing in a detection range, and generates distance measurement information indicating the distance and direction of each of the plurality of distance measurement targets; an image processing unit that captures an image in such a manner that at least a portion of an image capture range overlaps with the detection range, generates image data indicating the image, determines the distance, direction, and category of an image capture target included in the image, and generates image capture information indicating the distance, direction, and category of the image capture target; and a coincidence probability calculation unit that determines a provisional value indicating the size of the plurality of distance measurement targets using the image capture information, determines a plurality of areas in which the plurality of distance measurement targets are projected in the image, namely, a plurality of provisional areas, according to the provisional value and the distance measurement information, and calculates a coincidence probability indicating the possibility that the image capture target coincides with each of the plurality of distance measurement targets using the size of overlap between each of the plurality of provisional areas and an area in which the image capture target is captured in the image, namely, an object area.

[0013] An information processing device according to one embodiment of the present invention is characterized in that the information processing device includes: a communication interface unit that obtains distance measurement information indicating the distance and direction of each of a plurality of targets, i.e., a plurality of distance measurement targets, existing within a detection range, image data indicating an image captured in such a manner that at least a portion of a camera range overlaps with the detection range, and imaging information indicating the distance, direction, and type of a target, i.e., an imaging target, contained in the image; and a coincidence probability calculation unit that determines a provisional value indicating the size of the plurality of distance measurement targets using the imaging information, determines a plurality of regions, i.e., a plurality of provisional regions, into which the plurality of distance measurement targets are projected in the image according to the provisional value and the distance measurement information, and calculates a coincidence probability indicating the possibility that the imaging target coincides with each of the plurality of distance measurement targets using the size of overlap between each of the plurality of provisional regions and a region, i.e., a target region, in which the imaging target is captured in the image.

[0014] A program according to one embodiment of the present invention is characterized in that the program causes a computer to function as: a communication interface unit that obtains ranging information indicating the distance and direction of each of a plurality of targets, i.e., a plurality of ranging targets, existing within a detection range, image data indicating an image captured in such a manner that at least a portion of a capturing range overlaps with the detection range, and imaging information indicating the distance, direction, and type of a target, i.e., an imaging target, contained in the image; and a coincidence probability calculation unit that determines a provisional value indicating the size of the plurality of ranging targets using the imaging information, determines a plurality of regions, i.e., a plurality of provisional regions, into which the plurality of ranging targets are projected in the image according to the provisional value and the ranging information, and calculates a coincidence probability indicating the possibility that the imaging target coincides with each of the plurality of ranging targets using the size of overlap between each of the plurality of provisional regions and a region, i.e., a target region, in which the imaging target is captured in the image.

[0015] An information processing method according to one embodiment of the present invention is characterized by detecting the distance and direction of each of a plurality of targets, i.e., a plurality of ranging targets, present in a detection range, generating ranging information indicating the distance and direction of each of the plurality of ranging targets, capturing an image in such a manner that at least a portion of a capturing range overlaps with the detection range, generating image data indicating the image, determining the distance, direction, and category of a target, i.e., a captured target, contained in the image, generating captured image information indicating the distance, direction, and category of the captured target, determining a provisional value indicating the size of the plurality of ranging targets using the captured image information, determining a plurality of regions, i.e., a plurality of provisional regions, into which the plurality of ranging targets are projected in the image according to the provisional value and the ranging information, and calculating a coincidence probability indicating the possibility that the captured target coincides with each of the plurality of ranging targets using a size of overlap between each of the plurality of provisional regions and a region, i.e., a target region, in which the captured target is captured in the image.

[0016] Effects of the Invention

[0017] According to one or more aspects of the present invention, it is possible to reduce errors in determining the identity of a target. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 This is a block diagram schematically showing the configuration of a vehicle control system according to the first to fifth embodiments.

[0019] Figure 2 It is a schematic diagram showing a provisional value table as an example of provisional value information.

[0020] Figure 3 This is a block diagram schematically showing the configuration of a control unit in Embodiments 1 to 3.

[0021] Figure 4 This is a schematic diagram for explaining a target indicated by distance measurement information and a target indicated by imaging information in the first embodiment.

[0022] Figure 5 This is a schematic diagram showing an image in which a distance measurement target is projected onto an image captured by an imaging unit in the first embodiment.

[0023] Figure 6 This is a schematic diagram showing an image in which a provisional region corresponding to a distance measurement target is projected onto an image captured by an imaging unit in the first embodiment.

[0024] Figure 7 This is a block diagram showing a hardware configuration example of a vehicle control system.

[0025] Figure 8 This is a flowchart showing the processing in the information processing device according to the first embodiment.

[0026] Fig. 9 This is a schematic diagram for explaining a target indicated by distance measurement information and a target indicated by imaging information in the third embodiment.

[0027] Fig.10 This is a schematic diagram showing an image in which a distance measurement target is projected onto an image captured by an imaging unit in the third embodiment.

[0028] Fig.11 This is a block diagram schematically showing the configuration of a control unit in the fourth embodiment.

[0029] Fig.12 This is a flowchart showing the processing in the information processing device according to the fourth embodiment.

[0030] Fig.13 This is a block diagram schematically showing the configuration of a control unit in the fifth embodiment.

[0031] Fig.14 This is a schematic diagram for explaining a target represented by distance measurement information and a target represented by imaging information in the fifth embodiment.

[0032] Fig.15 This is a schematic diagram showing an image in which a provisional region corresponding to a distance measurement target is projected onto an image captured by an imaging unit in the fifth embodiment.

[0033] Fig.16 This is a diagram showing the range of an image captured by an imaging unit, a distance measurement target, and an imaging target in a first modification of Embodiments 1 to 5 as viewed from above.

[0034] Fig.171 is a schematic diagram showing a tentative area of ​​a distance measurement target in the first modification.

[0035] Fig.18 This is a diagram showing the range of an image captured by the imaging unit, the distance measurement target, and the imaging target in the second modified example of the first to fifth embodiments as viewed from above.

[0036] Fig.19 This is a schematic diagram showing an image in which a provisional region corresponding to a distance measurement target is projected onto an image captured by an imaging unit in a second modification. DETAILED DESCRIPTION

[0037] Implementation Method 1

[0038] Figure 1 1 is a block diagram schematically showing the configuration of a vehicle control system 100 as an information processing system according to the first embodiment.

[0039] The vehicle control system 100 includes a distance measurement processing unit 101 , an image capture processing unit 104 , a vehicle control unit 107 , and an information processing device 110 .

[0040] The vehicle control system 100 is mounted on a vehicle (not shown). The vehicle is a car, a train, or the like.

[0041] The distance measurement processing unit 101 detects the distance and direction of each of the multiple distance measurement targets existing in the detection range, and generates distance measurement information indicating the distance and direction of each of the multiple distance measurement targets. The generated distance measurement information is provided to the information processing device 110.

[0042] The distance measurement processing unit 101 includes a distance measurement unit 102 and a distance measurement control unit 103 .

[0043] The distance measuring unit 102 measures the distance of the target. The distance measuring unit 102 provides the measurement result to the distance measuring control unit 103. For example, the distance measuring unit 102 may measure the distance of the target by a known method such as using millimeter waves or pulse lasers.

[0044] The distance measurement control unit 103 generates distance measurement information indicating the distance and direction of the detected target based on the detection result in the distance measurement unit 102. Then, the distance measurement control unit 103 provides the generated distance measurement information to the information processing device 110.

[0045] The image processing unit 104 captures an image in such a manner that at least a portion of the image capturing range overlaps with the detection range of the distance measuring processing unit 101, and generates image data representing the image. Then, the image processing unit 104 determines the distance, direction, and category of the object contained in the captured image, i.e., the image capturing target, and generates image capturing information representing the distance, direction, and category of the image capturing target. The generated image capturing information is provided to the information processing device 110.

[0046] The image capture processing unit 104 includes an image capture unit 105 and an image capture control unit 106 .

[0047] The imaging unit 105 captures an image of a target and supplies image data representing the captured image to the imaging control unit 106 .

[0048] The camera control unit 106 determines the target included in the image represented by the image data provided by the camera unit 105, and determines the distance, direction, and category of the target. In addition, when a plurality of targets are included in the image, the camera control unit 106 determines the distance, direction, and category of each target. Here, the camera control unit 106 can determine the distance, direction, and category of the target by using a known method such as parallax or template matching. Then, the camera control unit 106 provides the camera information and image data indicating the determined distance, direction, and category for each target to the information processing device 110.

[0049] The vehicle control unit 107 generates vehicle information which is information related to the travel of the vehicle on which the vehicle control system 100 is mounted, and provides the vehicle information to the information processing device 110 .

[0050] Here, the vehicle information indicates the steering angle, speed, or yaw rate of the vehicle, etc. In the first embodiment, the information processing device 110 does not use the vehicle information, and therefore, the vehicle control unit 107 may not be included.

[0051] The information processing device 110 performs processing for specifying the distance and direction of a target to be detected in the vehicle control system 100 .

[0052] The information processing device 110 includes a communication interface unit (hereinafter referred to as a communication I / F unit) 111 , an in-vehicle network interface unit (hereinafter referred to as an in-vehicle NWI / F unit) 112 , a storage unit 113 , and a control unit 114 .

[0053] The communication I / F unit 111 communicates with the distance measurement processing unit 101 and the image processing unit 104. For example, the communication I / F unit 111 obtains distance measurement information from the distance measurement processing unit 101 and provides the distance measurement information to the control unit 114. In addition, the communication I / F unit 111 obtains image information and image data from the image processing unit 104 and provides the image information and image data to the control unit 114.

[0054] The in-vehicle NWI / F unit 112 communicates with the vehicle control unit 107. For example, the in-vehicle NWI / F unit 112 acquires vehicle information from the vehicle control unit 107 and provides the vehicle information to the control unit 114.

[0055] The storage unit 113 stores information and programs necessary for processing in the information processing device 110. For example, the storage unit 113 stores provisional value information in which the type of an object and a provisional value indicating the size of the object are associated with each other.

[0056] Figure 2 1 is a schematic diagram showing a provisional value table 113 a as an example of provisional value information.

[0057] The provisional value table 113 a is table information including a category column 113 b , a width column 113 c , and a height column 113 d .

[0058] The category column 113b stores the category of the object.

[0059] The width column 113c stores the width of the object.

[0060] The height column 113d stores the height of the object.

[0061] The temporary value table 113 a can determine the size of the object consisting of the width and height of the object as temporary values.

[0062] return Figure 1 The control unit 114 controls the processing in the information processing device 110. For example, the control unit 114 determines whether the target represented by the distance measurement information provided by the distance measurement processing unit 101 and the target represented by the image capture information provided by the image capture processing unit 104 are the same, and if they are the same, combines the distance and direction represented by the distance measurement information with the distance and direction represented by the image capture information.

[0063] Figure 3 This is a block diagram schematically showing the configuration of the control unit 114 in the first embodiment.

[0064] The control unit 114 includes a coincidence probability calculation unit 115 , a combination target determination unit 116 , and a combination unit 117 .

[0065] The coincidence probability calculation unit 115 determines a provisional value indicating the size of the plurality of distance measurement targets using the imaging information from the imaging processing unit 104, and determines a plurality of areas, i.e., a plurality of provisional areas, in which the plurality of distance measurement targets are projected in the image represented by the image data, according to the provisional value and the distance measurement information from the distance measurement processing unit 101. Then, the coincidence probability calculation unit 115 calculates a coincidence probability indicating the possibility that the imaging target and each of the plurality of distance measurement targets coincide with each other using the size of overlap between each of the plurality of provisional areas and the target area, i.e., the area in which the imaging target is captured in the image.

[0066] For example, the consistency probability calculation unit 115 calculates the consistency probability in the following manner: the larger the size of the portion where each of the multiple tentative regions overlaps with the target region, the larger the consistency probability. Specifically, the consistency probability calculation unit 115 calculates the consistency probability in the following manner: the larger the area of ​​the portion where each of the multiple tentative regions overlaps with the target region, the larger the consistency probability. In addition, the consistency probability calculation unit 115 may also calculate the consistency probability in the following manner: the larger the lateral width of the portion where each of the multiple tentative regions overlaps with the target region, the larger the consistency probability.

[0067] Figure 4 This is a schematic diagram for explaining a target indicated by distance measurement information and a target indicated by imaging information in the first embodiment.

[0068] Figure 4 This is a diagram showing the range of an image captured by the imaging unit 105 , the distance measurement target, and the imaging target as viewed from above.

[0069] An image is captured in an imaging range A1 to A2 having a certain field angle relative to the position of the lens of the imaging unit 105 , that is, the lens position P.

[0070] The imaging range A1 to A2 includes the imaging target C1 and the imaging target C2.

[0071] Furthermore, it is assumed that the distance measuring target R1, the distance measuring target R2, and the distance measuring target R3 are detected in the imaging range A1 to A2.

[0072] In addition, Figure 4 In FIG. 1 , the range of B1 to B2 is assumed to be the detection range of the distance measuring unit 102. Figure 4 In the figure, the imaging ranges A1 to A2 include the detection ranges B1 to B2, but at least a part of them may overlap.

[0073] Figure 5 1 is a schematic diagram showing an image IM1 in which the distance measuring target R1 , the distance measuring target R2 , and the distance measuring target R3 are projected onto an image captured by the imaging unit 105 in the first embodiment.

[0074] The imaging target C1 and the imaging target C2 are shown in the image captured by the imaging unit 105 , and the distance measuring target R1 , the distance measuring target R2 , and the distance measuring target R3 are projected in the direction indicated by the distance measuring information in the image.

[0075] here, Figure 5 The region of the imaging target C1 and the region of the imaging target C2 shown are target regions, respectively.

[0076] In the above situation, the coincidence probability calculation unit 115 calculates the coincidence probability so as to identify the coincident distance measurement target for each imaging target.

[0077] In addition, hereinafter, one imaging target that has been determined to be consistent with the distance measurement target is also referred to as a target imaging target.

[0078] Specifically, the coincidence probability calculation unit 115 specifies the type of the target imaging object based on the imaging information, and specifies the size corresponding to the specified type based on the provisional value table 113 a .

[0079] The coincidence probability calculation unit 115 projects a tentative region corresponding to the distance measurement target on the image captured by the imaging unit 105 in a size corresponding to the distance to the distance measurement target based on the determined size.

[0080] Figure 6 This is a schematic diagram showing an image IM2 in which a tentative region T1 corresponding to the distance measuring target R1 , a tentative region T2 corresponding to the distance measuring target R2 , and a tentative region T3 corresponding to the distance measuring target R3 are projected onto an image captured by the imaging unit 105 in the first embodiment.

[0081] exist Figure 6 , the case where the types of both the image capturing object C1 and the image capturing object C2 are the same. Here, the case where both the image capturing object C1 and the image capturing object C2 are cars (front view) is shown.

[0082] like Figure 6 As shown, the tentative area T1, tentative area T2, and tentative area T3 are tentative areas corresponding to the vehicle (front), respectively, but their sizes are different according to the distances at which the ranging targets R1, R2, and R3 are detected. In addition, the size indicated by the tentative value table 113a is converted according to the size of the image and the distance of the ranging target, thereby being able to calculate the size of the tentative area. That is, the sizes of the tentative area T1, T2, and T3 are the sizes when it is assumed that the targets of the sizes indicated by the tentative value table 113a are reflected in the image at their respective distances.

[0083] In addition, regarding the imaging object C1 and the imaging object C2, the outer frames of the objects included in the images may be detected, and the outer frames of the imaging object C1 and the imaging object C2 may be approximated by quadrilaterals.

[0084] Then, the coincidence probability calculation unit 115 calculates a coincidence probability whose value becomes larger as the overlap between the target imaging object and the tentative region becomes larger.

[0085] Here, the agreement probability calculation unit 115 calculates the agreement probability using the following formula (1).

[0086]

Mathematical formula 1

[0087]

[0088] In addition, R represents the area or lateral width of the tentative region in the captured image, and C represents the area or lateral width of the object being imaged in the captured image. If R is the area of ​​the tentative region, C also becomes the area of ​​the object being imaged, and if R is the lateral width of the tentative region, C also becomes the lateral width of the imaged region.

[0089] The numerator of the formula (1) is the area or lateral width of the overlapping portion between the area or lateral width of the tentative region and the area or lateral width of the target imaging object in the captured image.

[0090] Therefore, the formula (1) is obtained by dividing the size of the overlapping portion between the tentative area and the target imaging object in the captured image by the size of the target imaging object in the captured image.

[0091] For example, in Figure 6 In the example of , when the target imaging object is the imaging object C1, the coincidence probability is calculated based on the size of the imaging object C1 and the sizes of the tentative regions T1, T2, and T3.

[0092] Furthermore, when the target imaging object is the imaging object C2, the probability of coincidence is calculated based on the size of the imaging object C2 and the sizes of the tentative region T1, the tentative region T2, and the tentative region T3.

[0093] return Figure 3 The combining target determining unit 116 determines the distance measuring target with the highest probability of coincidence calculated by the coincidence probability calculating unit 115 as the target to be combined with the target imaging target, that is, the combining target. Here, the distance measuring target determined as the combining target is also referred to as the target distance measuring target.

[0094] The combining unit 117 combines the distance and direction indicated by the imaging information of the target imaging object and the distance and direction indicated by the target ranging object, thereby setting the combined value as an output value.

[0095] Here, the method of combining may be a known method, but, for example, any one of the distance indicated by the imaging information and the distance indicated by the object ranging target, or any one of the direction indicated by the imaging information and the direction indicated by the object ranging target may be selected. In addition, the distance indicated by the imaging information and the distance indicated by the object ranging target may be weighted in a predetermined manner and added or multiplied, or the direction indicated by the imaging information and the direction indicated by the object ranging target may be weighted in a predetermined manner and added or multiplied.

[0096] Figure 7 This is a block diagram showing a hardware configuration example of the vehicle control system 100 according to the first embodiment.

[0097] The vehicle control system 100 includes a distance measuring sensor 140 , a distance measuring sensor ECU (Electronic Control Unit) 141 , a camera 142 , a camera ECU 143 , a vehicle control ECU 144 , and the information processing device 110 .

[0098] The information processing device 110 includes a communication I / F 145 , a CAN (Controller Area Network) I / F 146 , a memory 147 , and a processor 148 .

[0099] Figure 1 The distance measuring unit 102 shown is implemented by a distance measuring sensor 140. The distance measuring sensor 140 is, for example, a millimeter wave radar having a transmission antenna for transmitting millimeter waves and a reception antenna for receiving millimeter waves, or a Lidar (Light Detection and Ranging) for measuring distance using laser light.

[0100] Figure 1 The distance measurement control unit 103 shown is realized by the distance measurement sensor ECU 141 .

[0101] Figure 1 The imaging unit 105 shown is implemented by a camera 142 as an imaging device.

[0102] Figure 1 The imaging control unit 106 shown is realized by the camera ECU 143 .

[0103] Figure 1 The vehicle control unit 107 shown is implemented by the vehicle control ECU 144 .

[0104] Figure 1 The communication I / F unit 111 shown is implemented by a communication I / F 145 .

[0105] Figure 1 The in-vehicle NWI / F unit 112 shown is implemented by CANI / F 146 .

[0106] Figure 1 The storage unit 113 shown is implemented by a memory 147 .

[0107] Figure 1 The control unit 114 shown can be configured by a processor 148 such as a CPU (Central Processing Unit) executing a program stored in a memory 147. Such a program can be provided via a network or recorded in a recording medium. That is, such a program can also be provided as a program product, for example.

[0108] As described above, the information processing device 110 can be realized by a so-called computer.

[0109] Figure 8 This is a flowchart showing the processing in information processing device 110 according to Embodiment 1.

[0110] The communication I / F unit 111 acquires the distance measurement information from the distance measurement processing unit 101 ( S10 ). The acquired distance measurement information is provided to the control unit 114 .

[0111] The communication I / F unit 111 acquires imaging information and image data from the imaging processing unit 104 ( S11 ). The acquired imaging information and image data are supplied to the control unit 114 .

[0112] The coincidence probability calculation unit 115 specifies one imaging target as a target imaging target from among the imaging targets indicated by the provided imaging information ( S12 ).

[0113] Then, the coincidence probability calculation unit 115 refers to the supplied imaging information to determine the category corresponding to the determined target imaging object, and determines the width and height which are provisional values ​​corresponding to the category stored in the storage unit 113 ( S13 ).

[0114] The coincidence probability calculation unit 115 applies the width and height determined in step S13 to the distance measurement target indicated by the distance measurement information acquired in step S10 , thereby determining the size of the distance measurement target ( S14 ).

[0115] The coincidence probability calculation unit 115 arranges the distance measurement target of the size determined in step S14 in the image represented by the image data acquired in step S11 according to the corresponding direction and distance represented by the distance measurement information, thereby determining the tentative region of the distance measurement target in the image ( S15 ).

[0116] The coincidence probability calculation unit 115 calculates the coincidence probability for each ranging target based on the size of overlap between the target imaging target in the image and the provisional region of the ranging target ( S16 ).

[0117] Next, the merging target determination unit 116 determines the distance measurement target having the highest probability of coinciding with the target imaging target as the target merging target based on the probability of coincidence calculated in step S16 ( S17 ).

[0118] Next, the combining unit 117 combines the distance and direction of the target imaging object and the distance and direction of the target ranging object, thereby generating an output value ( S18 ).

[0119] Then, the coincidence probability calculation unit 115 determines whether all the imaging targets indicated by the imaging information are determined as the target imaging targets (S19). If all the imaging targets are determined as the target imaging targets (S19: Yes), the process ends. If there are still undetermined imaging targets (S19: No), the process returns to step S12. In step S12, the coincidence probability calculation unit 115 determines the imaging targets that have not yet been determined as the target imaging targets as the target imaging targets.

[0120] In addition, the order of the processing of step S10 and step S11 may be switched.

[0121] As described above, according to Embodiment 1, the category of the target can be determined based on the captured image, the size of the target measured based on the determined category can be determined, and the size can be changed based on the measured distance. Therefore, it is possible to appropriately judge whether the targets are consistent based on the measured distance. As a result, it is possible to reduce errors in judging the identity of the target. For example, when the size of the target included in the image in the direction measured is used as the size of the object measured, the overlap changes dramatically when the field of view of the camera unit 105 is missed or blocked. In this regard, by determining the size based on the category recognized from the image, it is possible to prevent such a sharp change in overlap.

[0122] Implementation Method 2

[0123] like Figure 1 As shown, the vehicle control system 200 according to the second embodiment includes a distance measurement processing unit 101 , an image capture processing unit 104 , a vehicle control unit 107 , and an information processing device 210 .

[0124] The distance measurement processing unit 101 , the imaging processing unit 104 , and the vehicle control unit 107 in the vehicle control system 200 of the second embodiment are the same as the distance measurement processing unit 101 , the imaging processing unit 104 , and the vehicle control unit 107 in the vehicle control system 100 of the first embodiment.

[0125] The information processing device 210 includes a communication I / F unit 111 , an in-vehicle NWI / F unit 112 , a storage unit 113 , and a control unit 214 .

[0126] The communication I / F unit 111 , the in-vehicle NWI / F unit 112 , and the storage unit 113 of the information processing device 210 of the second embodiment are the same as those of the information processing device 110 of the first embodiment.

[0127] The control unit 214 controls the processing in the information processing device 210. For example, the control unit 214 determines whether the target indicated by the distance measurement information provided by the distance measurement processing unit 101 and the target indicated by the image pickup information provided by the image pickup processing unit 104 are the same, and if they are the same, combines the distance and direction indicated by the distance measurement information with the distance and direction indicated by the image pickup information.

[0128] like Figure 3 As shown, the control unit 214 includes a coincidence probability calculation unit 215 , a combination target determination unit 116 , and a combination unit 117 .

[0129] The combination target determination unit 116 and the combination unit 117 of the control unit 214 in the second embodiment are the same as the combination target determination unit 116 and the combination unit 117 of the control unit 114 in the first embodiment.

[0130] The coincidence probability calculation unit 215 calculates a coincidence probability indicating the possibility that the target indicated by the distance measurement information and the target indicated by the imaging information coincide with each other. The coincidence probability calculation unit 215 in the second embodiment differs from the coincidence probability calculation unit 115 in the first embodiment in the calculation method of the coincidence probability.

[0131] In implementation mode 2, the consistency probability calculation unit 215 calculates the consistency probability in the following manner: the larger the size of the overlapping portion between each of the multiple temporary areas and the target area, the larger the consistency probability; and the closer the distance between each of the multiple ranging targets and the distance to the camera target, the larger the consistency probability.

[0132] Here, the agreement probability calculation unit 215 calculates the agreement probability using the following formula (2).

[0133]

Mathematical formula 2

[0134]

[0135] In addition, R_C is the distance of the target imaging object, and R_R is the distance of the distance measurement object. In addition, α and β are weighting coefficients, which are determined in advance.

[0136] For example, using a target represented by distance measurement information and a target represented by imaging information, Figure 4 The situation shown is described.

[0137] When the target imaging target is imaging target C2, R_C is the distance of imaging target C2, which is included in the imaging information. Also, R_R is the distance of each of distance measurement target R1, distance measurement target R2, and distance measurement target R3, which is included in the distance measurement information.

[0138] As described above, in Embodiment 2, a value corresponding to the distance of the detected target is added to the value for determining whether the target matches or not, so it is possible to more appropriately determine whether the target matches or not. This can reduce errors in determining the identity of the target.

[0139] Implementation 3

[0140] like Figure 1 As shown, the vehicle control system 300 according to the third embodiment includes a distance measurement processing unit 101 , an image capture processing unit 104 , a vehicle control unit 107 , and an information processing device 310 .

[0141] The distance measurement processing unit 101 , the imaging processing unit 104 , and the vehicle control unit 107 in the vehicle control system 300 of the third embodiment are the same as the distance measurement processing unit 101 , the imaging processing unit 104 , and the vehicle control unit 107 in the vehicle control system 100 of the first embodiment.

[0142] The information processing device 310 includes a communication I / F unit 111 , an in-vehicle NWI / F unit 112 , a storage unit 113 , and a control unit 314 .

[0143] The communication I / F unit 111 , the in-vehicle NWI / F unit 112 , and the storage unit 113 of the information processing device 310 of the third embodiment are the same as those of the information processing device 110 of the first embodiment.

[0144] The control unit 314 controls the processing in the information processing device 310. For example, the control unit 314 determines whether the target indicated by the distance measurement information provided by the distance measurement processing unit 101 is the same as the target indicated by the image pickup information provided by the image pickup processing unit 104, and if they are the same, combines the distance and direction indicated by the distance measurement information with the distance and direction indicated by the image pickup information.

[0145] like Figure 3 As shown, the control unit 314 includes a coincidence probability calculation unit 315 , a combination target determination unit 116 , and a combination unit 117 .

[0146] The combination target determination unit 116 and the combination unit 117 of the control unit 314 in the third embodiment are the same as the combination target determination unit 116 and the combination unit 117 of the control unit 114 in the first embodiment.

[0147] The coincidence probability calculation unit 315 calculates a coincidence probability indicating the possibility that the target indicated by the distance measurement information coincides with the target indicated by the image pickup information. The coincidence probability calculation unit 315 in the third embodiment differs from the coincidence probability calculation unit 115 in the first embodiment in the calculation method of the coincidence probability.

[0148] In implementation mode 3, the consistency probability calculation unit 315 calculates the consistency probability in the following manner: the larger the size of the portion where each of the multiple tentative areas overlaps with the target area, the larger the consistency probability; and, when multiple ranging targets are projected onto an image represented by image data, the closer the distance between the camera target and each of the multiple ranging targets, the larger the consistency probability.

[0149] Fig. 9 This is a schematic diagram for explaining a target indicated by distance measurement information and a target indicated by imaging information in the third embodiment.

[0150] Fig. 9 This is a diagram showing the range of an image captured by the imaging unit 105 , the distance measurement target, and the imaging target as viewed from above.

[0151] An image is captured in an imaging range A1 to A2 having a certain field angle relative to the position of the lens of the imaging unit 105 , that is, the lens position P.

[0152] The imaging range A1 to A2 includes the imaging target C1 and the imaging target C2.

[0153] Furthermore, it is assumed that the distance measuring target R1, the distance measuring target R2, and the distance measuring target R4 are detected in the imaging range A1 to A2.

[0154] In the above situation, the coincidence probability calculation unit 315 calculates the coincidence probability so as to identify the coincident distance measurement target for each imaging target.

[0155] Specifically, the coincidence probability calculation unit 315 specifies the type of the target imaging object based on the imaging information, and specifies the size corresponding to the specified type based on the provisional value table 113 a .

[0156] Based on the determined size, the coincidence probability calculation unit 315 projects a tentative region corresponding to the distance measurement target with a size corresponding to the distance measurement target onto the image captured by the imaging unit 105. The processing up to this point is the same as that performed by the coincidence probability calculation unit 115 in the first embodiment.

[0157] Next, the coincidence probability calculation unit 315 calculates the distance from the target imaging object to each of the distance measurement targets.

[0158] Fig.10 1 is a schematic diagram showing an image IM3 in which the distance measuring target R1 , the distance measuring target R2 , and the distance measuring target R4 are projected onto an image captured by the imaging unit 105 in the third embodiment.

[0159] The imaging target C1 and the imaging target C2 are shown in the image captured by the imaging unit 105 , and the distance measuring target R1 , the distance measuring target R2 , and the distance measuring target R4 are projected in the direction indicated by the distance measuring information in the image.

[0160] For example, when the target imaging target is imaging target C1, the coincidence probability calculation unit 315 calculates the distance from the center point PC1 which is a predetermined point in imaging target C1 to each of the distance measuring targets R1, R2, and R4.

[0161] Furthermore, when the object to be imaged is the imaged object C2, the coincidence probability calculation unit 315 calculates the distance from the predetermined point in the imaged object C2, i.e., the center point PC2, to each of the distance measuring targets R1, R2, and R4. Here, the predetermined point is the center point, but it may be, for example, the centroid or other point.

[0162] Then, the coincidence probability calculation unit 315 calculates a coincidence probability whose value becomes larger as the overlap between the target imaging object and the temporary region becomes larger and the distance to the target imaging object becomes shorter.

[0163] Here, the agreement probability calculation unit 315 calculates the agreement probability using the following formula (3).

[0164]

Mathematical formula 3

[0165]

[0166] In addition, u_C is the pixel position of a predetermined point of the target imaging object in the image, and u_R is the pixel position of the distance measurement target. Furthermore, |u_C-u_R| is the number of pixels (distance) between the target imaging object and the distance measurement target.

[0167] In addition, Max is the pixel position at the left end of the image, u Min is the pixel position at the right end of the image, u Max -u Min +1 is the number of pixels (length) in the horizontal direction of the image.

[0168] As described above, according to Embodiment 3, a value corresponding to the distance of the target in the captured image is added to the value for determining whether the target matches, so it is possible to more appropriately determine whether the target matches. This can reduce errors in determining the identity of the target.

[0169] Implementation 4

[0170] like Figure 1As shown, the vehicle control system 400 according to the fourth embodiment includes a distance measurement processing unit 101 , an image capture processing unit 104 , a vehicle control unit 107 , and an information processing device 410 .

[0171] The distance measurement processing unit 101 , the imaging processing unit 104 , and the vehicle control unit 107 in the vehicle control system 400 of the fourth embodiment are the same as the distance measurement processing unit 101 , the imaging processing unit 104 , and the vehicle control unit 107 in the vehicle control system 100 of the first embodiment.

[0172] The information processing device 410 includes a communication I / F unit 111 , an in-vehicle NWI / F unit 112 , a storage unit 113 , and a control unit 414 .

[0173] The communication I / F unit 111 , the in-vehicle NWI / F unit 112 , and the storage unit 113 of the information processing device 410 of the fourth embodiment are the same as those of the information processing device 110 of the first embodiment.

[0174] The control unit 414 controls the processing in the information processing device 410. For example, the control unit 414 determines whether the target indicated by the distance measurement information provided by the distance measurement processing unit 101 and the target indicated by the image pickup information provided by the image pickup processing unit 104 are the same, and if they are the same, combines the distance and direction indicated by the distance measurement information with the distance and direction indicated by the image pickup information.

[0175] Fig.11 This is a block diagram schematically showing the configuration of the control unit 414 in the fourth embodiment.

[0176] The control unit 414 includes a coincidence probability calculation unit 415 , a combination target determination unit 116 , a combination unit 117 , and a reliability calculation unit 418 .

[0177] The combination target determination unit 116 and the combination unit 117 of the control unit 414 in the fourth embodiment are the same as the combination target determination unit 116 and the combination unit 117 of the control unit 114 in the first embodiment.

[0178] The reliability calculation unit 418 calculates the reliability of the distance and direction of the imaging target indicated by the imaging information and the distance and direction of each of the plurality of distance measurement targets indicated by the distance measurement information.

[0179] For example, the reliability calculation unit 418 uses the direction and distance of the target imaging target and the direction and distance of each of the plurality of distance measurement targets as detection items, and performs calculation using a Kalman filter.

[0180] Specifically, the reliability calculation unit 418 acquires the direction and distance of the target imaging target as the observation value. In addition, the reliability calculation unit 418 acquires the direction and distance of each of the plurality of distance measurement targets indicated by the distance measurement information as the observation value.

[0181] Then, the reliability calculation unit 418 uses the observation value as input and calculates the detection value of each detection item using the Kalman filter.

[0182] For example, the reliability calculation unit 418 calculates the detection value by using a Kalman filter for the target motion model represented by the following equation (4) and the target observation model represented by the following equation (5) regarding the detection item.

[0183]

Mathematical formula 4

[0184] X t|t-1 =F t|t-1 ·X t-1|t-1 +G tt-1 ·U t-1 (4)

[0185]

Mathematical formula 5

[0186] Z t =H t ·X t|t-1 +V t (5)

[0187] Here, X t|t-1 is the state vector at time t in time t-1. t|t-1 is the transition matrix at time t starting from time t-1. t-1t-1 is the current value of the state vector of the target at time t-1. t|t-1 is the driving matrix at time t starting from time t-1. t-1 The average value at time t-1 is 0 and based on the covariance matrix Q t-1 The normally distributed system noise vector Z t is the observation vector representing the observation value at time t. t is the observation function at time t. V t is the average value at time t of 0 and based on the covariance matrix R t The observation noise vector is a normally distributed vector.

[0188] When using the extended Kalman filter, the reliability calculation unit 418 performs prediction processing represented by the following equations (6) to (7) and smoothing processing represented by equations (8) to (13) on the detection item, thereby calculating the detection value.

[0189]

Mathematical formula 6

[0190]

[0191]

Mathematical formula 7

[0192] P t|t =F t|t-1 ·P t-1|t-1 ·F t|t-1 T +G t|t-1 Q t-1 ·G t|t-1 T (7)

[0193]

Mathematical formula 8

[0194] S t =H k ·P t|t-1 ·H k T +R t (8)

[0195]

Mathematical formula 9

[0196]

[0197]

Mathematical formula 10

[0198]

[0199]

Mathematical formula 11

[0200]

[0201]

Mathematical formula 12

[0202]

[0203]

Mathematical formula 13

[0204] P t|t =(IK t ·H t )·P t|t-1 (13)

[0205] Here, X^ t|t-1 is the prediction vector at time t in time t-1. t-1|t-1 is the smoothing vector at time t-1. t|t-1 is the prediction error covariance matrix at time t in time t-1. t-1|t-1 is the smoothed error covariance matrix at time t-1. t is the residual covariance matrix at time t. t is the Mahalanobis distance at time t. K tis the Kalman gain at time t. X^ t|t is a smooth vector at time t, representing the detection value of each detection item at time t. t|t is the smoothed error covariance matrix at time t. I is the identity matrix. In addition, the superscript T in the matrix indicates that it is a transposed matrix, and -1 indicates that it is an inverse matrix.

[0206] The reliability calculation unit 418 calculates the Mahalanobis distance θ t , Kalman gain K t and the smooth vector X^ at time t t|t The various data obtained by such calculations are written into the storage unit 113 .

[0207] Next, the reliability calculation unit 418 calculates the Mahalanobis distance at the corresponding time between the observation values ​​of the multiple distance measurement targets obtained based on the distance measurement information and the observation value of the target imaging target obtained based on the imaging information. The calculation method of the Mahalanobis distance here is different from the calculation method of the Mahalanobis distance described above only in the data used as the calculation object.

[0208] When the Mahalanobis distance is equal to or smaller than the threshold, the reliability calculation unit 418 assumes that the observation values ​​obtained based on the distance measurement information and the imaging information are observation values ​​obtained by observing the same object, and classifies these observation values ​​into the same group.

[0209] Next, the reliability calculation unit 418 calculates the reliability of the detection value of the target detection item calculated as described above, using each of the plurality of detection items as a target detection item.

[0210] Specifically, the reliability calculation unit 418 obtains the Mahalanobis distance between the observed value of the object detection item obtained from the distance measurement information and the imaging information and the predicted value of the object detection item at the target time predicted at the time before the target time used in calculating the above detection value. t|t When the Mahalanobis distance θ calculated as described above is read from the storage unit 113 t , and thus obtain its value.

[0211] In addition, the reliability calculation unit 418 obtains the Kalman gain obtained when calculating the detection value as described above. t|t When the Kalman gain K is calculated, the Kalman gain K is read from the storage unit 113. t , and thus obtain its value.

[0212] Then, the reliability calculation unit 418 uses the Mahalanobis distance θ t and the Kalman gain K t, calculates the reliability of the detection value of the object detection item calculated based on the observation value obtained from the distance measurement information and the camera information. Specifically, the reliability calculation unit 418 converts the Mahalanobis distance θ into t and the Kalman gain K t Multiply them together to calculate the reliability of the detection value of the object detection item.

[0213]

Mathematical formula 14

[0214]

[0215] Here, M X is the reliability related to direction X, M Y is the reliability related to distance Y. K X is the Kalman gain associated with direction X, K Y is the Kalman gain related to the distance Y.

[0216] In addition, the reliability calculation unit 418 may also calculate the Mahalanobis distance θ t and the Kalman gain K t After weighting at least one of them, the Mahalanobis distance θt and the Kalman gain K t Multiply to calculate reliability.

[0217] The coincidence probability calculation unit 415 calculates the coincidence probability when the reliability of all the multiple distance measurement targets is lower than a predetermined threshold value.

[0218] Specifically, when the reliability calculated as above is greater than or equal to the reliability functioning as a predetermined threshold value among the plurality of detection values ​​calculated as above, the coincidence probability calculation unit 415 selects the detection value with the highest reliability as the output value. High reliability means that the value obtained by multiplying the Mahalanobis distance and the Kalman gain is small.

[0219] Here, the reliability is used when selecting a detection value to be adopted from a plurality of detection values ​​calculated from each observation value set as the observation value for detecting the same object. Therefore, the reliability calculation unit 418 may calculate the reliability from each observation value classified into the groups as described above.

[0220] The coincidence probability calculation unit 415 calculates the coincidence probability in the same manner as in the first embodiment when the reliability calculated by the reliability calculation unit 418 is lower than the reliability functioning as a predetermined threshold value.

[0221] Fig.12 This is a flowchart showing the processing in the information processing device 410 according to the fourth embodiment.

[0222] The communication I / F unit 111 acquires the distance measurement information from the distance measurement processing unit 101 ( S20 ). The acquired distance measurement information is provided to the control unit 414 .

[0223] The communication I / F unit 111 acquires imaging information and image data from the imaging processing unit 104 ( S21 ). The acquired imaging information and image data are provided to the control unit 414 .

[0224] The coincidence probability calculation unit 415 specifies one target imaging target from the imaging targets indicated by the provided imaging information ( S22 ).

[0225] Next, the reliability calculation unit 418 calculates the reliability using the direction and distance corresponding to the determined target imaging object and the direction and distance corresponding to the distance measurement object indicated by the distance measurement information as observation values ​​( S23 ).

[0226] Then, the agreement probability calculation unit 415 determines whether all the calculated reliabilities are less than the reliability threshold value in at least one detection item (S24). If all the reliabilities in at least one detection item are less than the reliability threshold value (S24: Yes), the process proceeds to step S25, and if at least one reliability in all the detection items is greater than the reliability threshold value (S24: No), the process proceeds to step S31.

[0227] In step S25, the coincidence probability calculation unit 415 determines the category corresponding to the determined object of the image pickup by referring to the provided image pickup information, and determines the provisional values ​​corresponding to the category, that is, the width and height by referring to the provisional value table 113a stored in the storage unit 113.

[0228] The coincidence probability calculation unit 415 applies the width and height determined in step S25 to the distance measurement target indicated by the distance measurement information acquired in step S20 , thereby determining the size of the distance measurement target ( S26 ).

[0229] The coincidence probability calculation unit 415 arranges the distance measurement target of the size determined in step S26 in the image represented by the image data acquired in step S21 according to the corresponding direction and distance represented by the distance measurement information, thereby determining the tentative region of the distance measurement target in the image ( S27 ).

[0230] The coincidence probability calculation unit 415 calculates the coincidence probability for each ranging target based on the overlap between the target imaging target in the image and the provisional region of the ranging target ( S28 ).

[0231] Next, the combining target determination unit 116 determines the distance measurement target having the highest probability of matching the target imaging target as the combining target distance measurement target based on the probability of matching calculated in step S28 ( S29 ).

[0232] Next, the combining unit 117 combines the distance and direction of the target imaging object with the distance and direction of the target ranging object to generate an output value ( S30 ). Then, the process proceeds to step S32 .

[0233] On the other hand, in step S24, when at least one reliability among all the detection items is greater than the reliability that becomes the threshold value (S24: No), the process proceeds to step S31, and in step S31, the coincidence probability calculation unit 415 determines the detection value with the highest reliability among the detection items of the detection items as the output value. Then, the process proceeds to step S32.

[0234] In step S32, the coincidence probability calculation unit 415 determines whether all the imaging targets indicated by the imaging information are determined as the target imaging targets. If all the imaging targets are determined as the target imaging targets (S32: Yes), the process ends. If there are imaging targets that have not been determined (S32: No), the process returns to step S22. In step S22, the coincidence probability calculation unit 415 determines the imaging targets that have not been determined as the target imaging targets as the target imaging targets.

[0235] In addition, the order of the processing of step S20 and step S21 may be switched.

[0236] As described above, according to the fourth embodiment, only the detection values ​​with high reliability can be used directly as output values, so it is possible to reduce the judgment errors when judging the identity of the target.

[0237] In the fourth embodiment, when the reliability of all the multiple ranging targets is lower than the predetermined threshold, the agreement probability calculation unit 415 calculates the agreement probability in the same manner as in the first embodiment, but the fourth embodiment is not limited to this example. For example, the agreement probability calculation unit 415 may also calculate the agreement probability in the same manner as in the second or third embodiment.

[0238] Implementation method 5

[0239] like Figure 1 As shown, the vehicle control system 500 according to the fifth embodiment includes a distance measurement processing unit 101 , an image capture processing unit 104 , a vehicle control unit 107 , and an information processing device 510 .

[0240] The distance measurement processing unit 101 , the imaging processing unit 104 , and the vehicle control unit 107 in the vehicle control system 500 of the fifth embodiment are the same as the distance measurement processing unit 101 , the imaging processing unit 104 , and the vehicle control unit 107 in the vehicle control system 100 of the first embodiment.

[0241] The information processing device 510 includes a communication I / F unit 111 , an in-vehicle NWI / F unit 112 , a storage unit 113 , and a control unit 514 .

[0242] The communication I / F unit 111 , the in-vehicle NWI / F unit 112 , and the storage unit 113 of the information processing device 510 of the fifth embodiment are the same as those of the information processing device 110 of the first embodiment.

[0243] The control unit 514 controls the processing in the information processing device 510. For example, the control unit 514 determines whether the target indicated by the distance measurement information provided by the distance measurement processing unit 101 is the same as the target indicated by the image pickup information provided by the image pickup processing unit 104, and if they are the same, combines the distance and direction indicated by the distance measurement information with the distance and direction indicated by the image pickup information.

[0244] Fig.13 This is a block diagram schematically showing the configuration of the control unit 514 in the fifth embodiment.

[0245] The control unit 514 includes a coincidence probability calculation unit 515 , a connection target determination unit 116 , a connection unit 117 , and a travel track determination unit 519 .

[0246] The combination target determination unit 116 and the combination unit 117 of the control unit 514 in the fifth embodiment are the same as the combination target determination unit 116 and the combination unit 117 of the control unit 114 in the first embodiment.

[0247] The traveling track determination unit 519 determines the traveling track of the vehicle equipped with the vehicle control system 500. The traveling track determination unit 519 may determine the traveling track using a known method.

[0248] For example, the driving track determination unit 519 can determine the driving track by determining the lines for dividing the lane in which the vehicle is traveling based on the image represented by the image data from the camera processing unit 104. In addition, the driving track determination unit 519 can also determine the driving track of the vehicle based on the steering angle or yaw rate of the vehicle represented by the vehicle information obtained from the vehicle control unit 107.

[0249] When the image capturing object affects the travel track, the coincidence probability calculation unit 515 calculates the coincidence probability between the image capturing object and each of the plurality of distance measuring objects.

[0250] Specifically, the coincidence probability calculation unit 515 determines, in the image represented by the image data from the image processing unit 104, a photographing target that affects the travel track determined by the travel track determination unit 519 as an influencing photographing target. For example, when at least a portion of a target included in the image is included in the travel track, the coincidence probability calculation unit 515 determines the target as an influencing photographing target.

[0251] Then, the coincidence probability calculation unit 515 determines the target imaging target based on the influencing imaging target, and calculates the coincidence probability between the target imaging target and the distance measurement target. The processing here is the same as that in the first embodiment.

[0252] Fig.14 This is a schematic diagram for explaining a target represented by distance measurement information and a target represented by imaging information in the fifth embodiment.

[0253] Fig.14 This is a diagram showing the range of an image captured by the imaging unit 105 , the distance measurement target, and the imaging target as viewed from above.

[0254] An image is captured in an imaging range A1 to A2 having a certain field angle relative to the position of the lens of the imaging unit 105 , that is, the lens position P.

[0255] The imaging range A1 to A2 includes an imaging target C3.

[0256] Furthermore, it is assumed that the distance measuring target R5 and the distance measuring target R6 are detected in the imaging range A1 to A2.

[0257] exist Fig.14 In FIG. 5 , it is assumed that the travel track identification unit 519 detects the left line L1 and the right line L2 of the lane as the travel track of the vehicle.

[0258] Furthermore, since a portion of the imaging target C3 overlaps with the line L2, it becomes an influencing imaging target.

[0259] Fig.15 This is a schematic diagram showing an image IM4 in which a tentative region T5 corresponding to the distance measuring target R5 and a tentative region T6 corresponding to the distance measuring target R6 are projected onto an image captured by the imaging unit 105 in the fifth embodiment.

[0260] exist Fig.15 , a case where the imaging target C3 that is an influencing imaging target is a car (front view) is shown.

[0261] like Fig.15 As shown, the tentative area T5 and the tentative area T6 are tentative areas corresponding to the vehicle (front), respectively, but the sizes thereof are different according to the distances at which the distance measurement targets R5 and R6 are detected.

[0262] In the above situation, the coincidence probability calculation unit 515 calculates the coincidence probability based on the size of the overlap between the imaging target C3 determined as an influencing imaging target and each of the tentative regions T5 and T6.

[0263] As described above, according to the fifth embodiment, an object that affects the travel of a vehicle equipped with the vehicle control system 500 (for example, an object or a person on a preceding vehicle or a travel track) can be detected with high accuracy.

[0264] In the fifth embodiment described above, an example is shown in which the travel track determination unit 519 is added to the first embodiment, but the fifth embodiment is not limited to this example. For example, the travel track determination unit 519 can be added to the second to fourth embodiments.

[0265] In the above-described embodiments 1 to 5, the probability of coincidence between each ranging target represented by the ranging information and the object camera target is calculated, but the embodiments 1 to 5 are not limited to such examples. For example, a plurality of ranging targets may be collected to generate a tentative area. Specifically, when a plurality of ranging targets are in an adjacent relationship, such as when the distance between the plurality of ranging targets is below a predetermined threshold value, or when the position of a ranging target in an image is included in the tentative area of ​​another ranging target, the coincidence probability calculation unit 115 to 515 can set such a plurality of ranging targets as a single ranging target collected. In addition, the single ranging target collected is also referred to as a collected ranging target.

[0266] In other words, when two or more ranging targets among the plurality of ranging targets are adjacent to each other, the coincidence probability calculation units 115 to 515 may determine a combined ranging target that combines the two or more ranging targets into one, and calculate the coincidence probability between the captured target and the combined ranging target.

[0267] Fig.16 This is a diagram showing the range of an image captured by the imaging unit 105 , the distance measurement target, and the imaging target in the first modified example of the first to fifth embodiments as viewed from above.

[0268] An image is captured in an imaging range A1 to A2 having a certain field angle relative to the position of the lens of the imaging unit 105 , that is, the lens position P.

[0269] The imaging range A1 to A2 includes the imaging target C1 and the imaging target C2.

[0270] Furthermore, it is assumed that the distance measuring target R2, the distance measuring target R3, and the distance measuring targets R7 to R9 are detected in the imaging ranges A1 to A2.

[0271] Fig.171 is a schematic diagram showing a tentative region T7 of a distance measuring target R7 , a tentative region T8 of a distance measuring target R8 , and a tentative region T9 of a distance measuring target R9 in the first modification.

[0272] exist Fig.17 In the example shown, the provisional area T9 of the ranging target R9 includes other ranging targets R7 and R8, so the coincidence probability calculation units 115 to 515 determine one aggregated ranging target R# in which the ranging targets R7 to R9 are aggregated.

[0273] Here, the coincidence probability calculation unit 115-515 sets the representative point calculated based on the ranging targets R7-R9, that is, the center point, as the aggregated ranging target R#, but the first modified example is not limited to this example. It is also possible to select any one of the aggregated ranging targets R7-R9, for example, a ranging target R9 that includes the other ranging targets R7 and the ranging target R8 in the temporary area T9 as the aggregated ranging target.

[0274] Then, the coincidence probability calculation units 115 to 515 may calculate the coincidence probability between the target imaging object and the aggregate ranging object R#.

[0275] As described above, according to the first variant, when multiple distance measurement targets are detected from one object or one person, or when it is suitable to be treated as one distance measurement target, such as a bicycle and a person riding the bicycle, the multiple distance measurement targets can be aggregated into one. In addition, regarding the distance and direction of the aggregated distance measurement target, a representative value of the distance and direction of the aggregated multiple distance measurement targets, such as an average value or a median value, can be used.

[0276] In the above-described embodiments 1 to 5, when the distance to the distance measurement target is too short, the provisional area of ​​the distance measurement target may be too large. An example of taking measures against such a situation will be presented as a second modification.

[0277] For example, in the second modification, the coincidence probability calculation unit 115 to 515 may not calculate the coincidence probability between the image capturing target and at least one of the multiple distance measuring targets when the distance to the at least one distance measuring target is less than a predetermined threshold distance.

[0278] Fig.18 This is a diagram showing the range of an image captured by the imaging unit 105 , the distance measurement target, and the imaging target in the second modified example of the first to fifth embodiments as viewed from above.

[0279] An image is captured in an imaging range A1 to A2 having a certain field angle relative to the position of the lens of the imaging unit 105 , that is, the lens position P.

[0280] The imaging range A1 to A2 includes the imaging target C1 and the imaging target C2.

[0281] In addition, it is assumed that the distance measuring target R1, the distance measuring target R2, the distance measuring target R3, and the distance measuring target R10 are detected in the imaging range A1 to A2. The distance measuring target R10 is detected at a very close distance relative to the lens position P.

[0282] Fig.19 This is a schematic diagram showing an image IM5 in which a tentative region T1 corresponding to the ranging target R1, a tentative region T2 corresponding to the ranging target R2, a tentative region T3 corresponding to the ranging target R3, and a tentative region T10 corresponding to the ranging target R10 are projected onto an image captured by the camera unit 105 in the second variation.

[0283] Fig.19 The case where the types of both the image capturing object C1 and the image capturing object C2 are car (front view) is also shown.

[0284] like Fig.19 As shown, the distance measuring target R10 is detected at a very close distance, and therefore its provisional area T10 is very large, and the probability of coincidence between the imaging targets C1 and C2 and the distance measuring targets R1 to R3 cannot be properly calculated.

[0285] Therefore, in the second variant, for example Fig.18 As shown, a threshold distance RTh serving as a threshold is determined in advance. Then, the coincidence probability calculation units 115 to 515 do not calculate the coincidence probability between the distance measurement target indicated by the distance measurement information and the target imaging target and the distance measurement target being less than the threshold distance RTh.

[0286] As described above, according to the second modification, even when the distance to the distance measurement target is too close to properly calculate the coincidence probability, the coincidence probability can be properly calculated. For example, when an error occurs in distance measurement in the distance measurement processing unit 101, the second modification is effective.

[0287] In the above-described embodiments 1 to 5, the output value after the combination by the combination unit 117, 417 is output, but the embodiments 1 to 5 are not limited to this example. For example, the coincidence probability calculated by the coincidence probability calculation unit 115 to 515 may be output. In this case, the combination object determination unit 116, 416 and the combination unit 117, 417 can be omitted.

[0288] Description of symbols

[0289] 100, 200, 300, 400, 500: vehicle control system; 101: distance measurement processing unit; 102: distance measurement unit; 103: distance measurement control unit; 104: camera processing unit; 105: camera unit; 106: camera control unit; 107: vehicle control unit; 110, 210, 310, 410, 510: information processing device; 111: communication I / F unit; 112: in-vehicle NWI / F unit; 113: storage unit; 114, 214, 314, 414, 514: control unit; 115, 215, 315, 415, 515: consistency probability calculation unit; 116, 416: combination object determination unit; 117, 417: combination unit; 418: reliability calculation unit; 519: driving track determination unit.

Claims

1. An information processing system, characterized in that: The information processing system has: a distance measurement processing unit that detects the distance and direction of each of a plurality of targets, i.e., a plurality of distance measurement targets, present within a detection range, and generates distance measurement information indicating the distance and direction of each of the plurality of distance measurement targets; an image processing unit that captures an image in such a manner that at least a portion of an image capturing range overlaps with the detection range, generates image data representing the image, determines a distance, direction, and type of an object included in the image, i.e., an image capturing target, and generates image capturing information representing the distance, direction, and type of the image capturing target; and a coincidence probability calculation unit that uses the imaging information to determine a provisional value indicating the size of the plurality of ranging targets, determines a plurality of regions, i.e., a plurality of provisional regions, in which the plurality of ranging targets are projected in the image according to the provisional value and the ranging information, and calculates a coincidence probability indicating the possibility that the imaging target coincides with each of the plurality of ranging targets, i.e., a target region, by using a size of overlap between each of the plurality of provisional regions and a region, i.e., a target region, in which the imaging target is captured in the image.

2. The information processing system according to claim 1, characterized in that The agreement probability calculation unit calculates the agreement probability in such a manner that the larger the size of the portion where each of the plurality of tentative regions overlaps with the target region, the larger the agreement probability.

3. The information processing system according to claim 2, characterized in that: The agreement probability calculation unit calculates the agreement probability in such a manner that the larger the area of ​​a portion where each of the plurality of tentative regions overlaps with the target region, the larger the agreement probability.

4. The information processing system according to claim 2, characterized in that: The agreement probability calculation unit calculates the agreement probability in such a manner that the greater the lateral width of a portion where each of the plurality of tentative regions overlaps the target region, the greater the agreement probability.

5. The information processing system according to claim 1, characterized in that: The consistency probability calculation unit calculates the consistency probability in the following manner: the larger the size of the overlapping portion between each of the multiple temporary areas and the target area, the larger the consistency probability; and the closer the distance between each of the multiple ranging targets and the distance to the camera target, the larger the consistency probability.

6. The information processing system according to claim 1, characterized in that: The consistency probability calculation unit calculates the consistency probability in the following manner: the larger the size of the portion where each of the multiple temporary areas overlaps with the target area, the larger the consistency probability; and, when the multiple ranging targets are projected onto the image, the closer the distance between the camera target and each of the multiple ranging targets, the larger the consistency probability.

7. The information processing system according to any one of claims 1 to 6, characterized in that: The information processing system further includes a reliability calculation unit that calculates the reliability of the distance and direction of the imaging target indicated by the imaging information and the distance and direction of each of the plurality of ranging targets indicated by the ranging information. The agreement probability calculation unit calculates the agreement probability when the reliability of all of the plurality of ranging targets is lower than a predetermined threshold value.

8. The information processing system according to any one of claims 1 to 6, characterized in that: The information processing system further includes a travel track determination unit that determines a travel track of a vehicle on which the information processing system is mounted. When the image capturing object affects the travel track, the coincidence probability calculation unit calculates the coincidence probability between the image capturing object and each of the plurality of distance measuring objects.

9. The information processing system according to any one of claims 1 to 6, characterized in that: The information processing system also has: a combining target determining unit that determines, among the plurality of distance measurement targets, a distance measurement target having the highest probability of coincidence as a combining target; and A combining unit combines the distance and direction of the distance measurement target with the distance and direction of the combining object.

10. The information processing system according to any one of claims 1 to 6, characterized in that: The coincidence probability calculation unit determines the provisional value corresponding to the category included in the imaging information.

11. The information processing system according to any one of claims 1 to 6, characterized in that: When two or more ranging targets among the plurality of ranging targets are in an adjacent relationship, the coincidence probability calculation unit determines a collection ranging target in which the two or more ranging targets are collected into one, and calculates the coincidence probability between the imaging target and the collection ranging target.

12. The information processing system according to any one of claims 1 to 6, characterized in that: When the distance to at least one ranging target among the plurality of ranging targets is smaller than a predetermined threshold distance, the coincidence probability calculation unit does not calculate the coincidence probability between the imaging target and the at least one ranging target.

13. An information processing device, characterized in that: The information processing device has: a communication interface unit that acquires distance measurement information indicating the distance and direction of each of a plurality of targets existing within a detection range, i.e., a plurality of distance measurement targets, image data indicating an image captured in such a manner that at least a portion of an imaging range overlaps with the detection range, and imaging information indicating the distance, direction, and type of an object contained in the image, i.e., an imaging target; and a coincidence probability calculation unit that uses the imaging information to determine a provisional value indicating the size of the plurality of ranging targets, determines a plurality of regions, i.e., a plurality of provisional regions, in which the plurality of ranging targets are projected in the image according to the provisional value and the ranging information, and calculates a coincidence probability indicating the possibility that the imaging target coincides with each of the plurality of ranging targets, i.e., a target region, by using a size of overlap between each of the plurality of provisional regions and a region, i.e., a target region, in which the imaging target is captured in the image.

14. A computer-readable recording medium storing a computer program, characterized in that: When the computer program is executed by a processor, Detecting the distance and direction of each of the multiple targets, i.e., multiple ranging targets, present within the detection range, generating ranging information indicating the distance and direction of each of the plurality of ranging targets, capturing an image in such a manner that at least a portion of an imaging range overlaps with the detection range, generating image data representing the image, determining the distance, direction and type of the object contained in the image, i.e., the imaging target, and generating imaging information indicating the distance, direction and type of the imaging target, determining a provisional value representing the size of the plurality of ranging targets using the imaging information, Determining, according to the provisional value and the ranging information, a plurality of areas in which the plurality of ranging targets are projected in the image, namely, a plurality of provisional areas, The coincidence probability indicating the possibility that the imaging target coincides with each of the plurality of ranging targets is calculated using the size of overlap between each of the plurality of tentative regions and a target region in which the imaging target is captured in the image.

15. An information processing method, characterized in that: Detecting the distance and direction of each of the multiple targets, i.e., multiple ranging targets, present within the detection range, generating ranging information indicating the distance and direction of each of the plurality of ranging targets, capturing an image in such a manner that at least a portion of an imaging range overlaps with the detection range, generating image data representing the image, determining the distance, direction and type of the object contained in the image, i.e., the imaging target, and generating imaging information indicating the distance, direction and type of the imaging target, determining a provisional value representing the size of the plurality of ranging targets using the imaging information, Determining, according to the provisional value and the ranging information, a plurality of areas in which the plurality of ranging targets are projected in the image, namely, a plurality of provisional areas, The coincidence probability indicating the possibility that the imaging target coincides with each of the plurality of ranging targets is calculated using the size of overlap between each of the plurality of tentative regions and a target region in which the imaging target is captured in the image.

Images