Information processing system, information processing device, program, and information processing method

Abstract

This information processing system comprises: a distance measurement processing unit (2) that generates distance measurement information indicating a distance and a direction for each of a plurality of distance measurement targets; an imaging processing unit (3) that detects a plurality of imaging targets by capturing an image of an imaging range overlapping at least a part of a detection range of the distance measurement processing unit (2), generates type information indicating the type of each of the plurality of imaging targets, and identifies, as target regions, each of a plurality of regions in which the imaging target appears; a provisional region calculation unit (22) that calculates, as a provisional region, a region of the distance measurement target projected on the image, the region being determined on the basis of the distance measurement information, the aspect ratio of the target regions, and provisional values indicating the size of each of the plurality of distance measurement targets identified on the basis of the type information; and a match probability calculation unit (23) that calculates, on the basis of the target regions of the imaging target and the provisional region calculated for each distance measurement target, a match probability indicating the possibility that the imaging target and each of the plurality of distance measurement targets match.

Description

Information processing system, information processing device, program, and information processing method

[0001] The present disclosure relates to an information processing system, an information processing device, a program, and an information processing method.

[0002] A technique is known for improving the accuracy of object detection, identification, or discrimination by combining multiple sensors and using them as a composite sensor. For example, when determining the identity of a target detected by imaging and a target detected by ranging, it is desirable to be able to reduce the error of determining that different targets are the same target. Hereinafter, a target detected by imaging will be referred to as an imaged target. A target detected by ranging will be referred to as a ranging target.

[0003] Patent Document 1 discloses a target detection system that uses a camera and a radar to determine the identity of targets detected by the camera and the radar. The target detection system according to Patent Document 1 performs identity determination using a condition that an imaged target detected by the camera and a range-measured target detected by the radar are present within a certain distance range and a certain azimuth range.

[0004] Patent Document 2 discloses an information processing system that projects an area where a target is detected onto an image obtained by capturing an image, and calculates a match probability indicating the possibility that the captured target and the target are identical based on the degree of overlap between the area where the captured target is detected and the area where the target is detected. The information processing system disclosed in Patent Document 2 identifies the type of the captured target and determines a provisional value for the size of the target based on a table that corresponds the type to a value indicating the size of the captured target. In Patent Document 2, the value indicating the size of the target is the height value of the target and the width value of the target. The information processing system disclosed in Patent Document 2 calculates a provisional area where the target is detected based on the provisional value of the size of the target.

[0005] JP 2022-39102 A Patent No. 7154470 A

[0006] In order to determine the position error between sensors such as a camera and a radar, it is necessary to compare the positions of the target in the same coordinate space. However, Patent Document 1 does not disclose a method for determining the position error by comparing the position of the captured target and the position of the range-finding target on the image captured by the camera through coordinate transformation of the position of the range-finding target detected by the radar. Therefore, the technique disclosed in Patent Document 1 has a problem in that it is difficult to determine the identity of the target with high accuracy.

[0007] Furthermore, when a region where a target is detected is projected onto an image obtained by imaging, the height and width of the target projected in the image may vary depending on the target's attitude or orientation. However, according to Patent Document 2, since the size values ​​associated with each type are fixed, even if the target detected by imaging and the target detected by ranging are the same, depending on the target's attitude or orientation, a provisional region whose size is significantly different from the region where the target is detected by imaging may be calculated. For this reason, the technology of Patent Document 2 has the problem that it is difficult to accurately determine the identity of targets whose attitude or orientation can change.

[0008] The present disclosure has been made in consideration of the above, and aims to provide an information processing system that can determine the identity of a target with high accuracy.

[0009] In order to solve the above-mentioned problems and achieve the object, an information processing system according to the present disclosure includes a distance measurement processing unit that detects multiple targets present in a detection range as distance measurement targets and generates distance measurement information indicating the distance and direction of each of the multiple distance measurement targets, an imaging processing unit that captures an image of an imaging range that overlaps at least a part of the detection range of the distance measurement processing unit, detects the multiple targets captured in the image as imaged targets, generates type information indicating the type of each of the multiple imaged targets, and identifies each of multiple areas in the image in which the imaged targets are captured as a target area, and The system is equipped with a provisional area calculation unit that calculates, for each of the identified multiple image-taking targets, provisional values ​​indicating the size of each of the multiple distance-measuring targets identified based on type information of the image-taking targets, distance measurement information of the distance-measuring targets, and an area of ​​the distance-measuring target projected onto an image, determined based on the aspect ratio of the target area, as a provisional area for each of the identified multiple image-taking targets, and a match probability calculation unit that calculates a match probability indicating the possibility that the image-taking target and each of the multiple distance-measuring targets will match, based on the target area of ​​the image-taking target and the provisional area calculated for each distance-measuring target.

[0010] The information processing system according to the present disclosure has the effect of being able to determine the identity of targets with high accuracy.

[0011] 1 is a block diagram showing a schematic configuration of an information processing system according to embodiment 1; 2 is a block diagram showing a schematic configuration of an information processing unit of the information processing system according to embodiment 1; 3 is a diagram showing an example of provisional value information stored in a memory unit of the information processing system according to embodiment 1; 4 is a diagram for explaining a distance measurement target indicated by distance measurement information and an imaged target indicated by image information in embodiment 1; 5 is a first diagram for explaining an imaged target shown in an image shown by image data in embodiment 1; 6 is a first diagram showing an image in which the target area of ​​the imaged target and the position where the distance measurement target was detected are projected in embodiment 1; 7 is a first diagram showing an image in which the provisional area of ​​each distance measurement target is projected in embodiment 1; FIG. 1 is a diagram showing an example of a target area of ​​an imaged target identified by an information processing system according to embodiment 1; FIG. 2 is a flowchart showing a processing procedure by an information processing system according to embodiment 1; FIG. 3 is a diagram showing a method for determining a provisional area by a provisional area calculation unit of an information processing system according to embodiment 1; FIG. 4 is a diagram showing a method for determining a provisional area by a provisional area calculation unit of an information processing system according to embodiment 1;

[0012] An information processing system, an information processing device, a program, and an information processing method according to embodiments will be described in detail below with reference to the accompanying drawings.

[0013] First Embodiment. Figure 1 is a block diagram showing a schematic configuration of an information processing system 1 according to the first embodiment. In the first embodiment, the information processing system 1 is a vehicle control system that controls a vehicle. The information processing system 1 is mounted on a vehicle. The vehicle on which the information processing system 1 is mounted may be an automobile, a train, a construction vehicle, or the like. The vehicle is not shown in the figure. The information processing system 1 may also be applied to a ship control system that controls a ship. The ship control system is mounted on a ship.

[0014] The information processing system 1 includes a distance measurement processing unit 2, an image capture processing unit 3, a vehicle control unit 4, an information processing device 5, and an input / output unit 6. The distance measurement processing unit 2 and the information processing device 5 are connected to each other so that they can communicate with each other. The image capture processing unit 3 and the information processing device 5 are connected to each other so that they can communicate with each other. The vehicle control unit 4 and the information processing device 5 are connected to each other so that they can communicate with each other. The input / output unit 6 and the information processing device 5 are connected to each other so that they can communicate with each other.

[0015] The ranging processing unit 2 detects multiple targets within the detection range as ranging targets and generates ranging information. The ranging information indicates the distance and direction of each of the multiple ranging targets. The ranging processing unit 2 provides the generated ranging information to the information processing device 5. The ranging processing unit 2 includes a ranging unit 11 and a detection processing unit 12.

[0016] The ranging unit 11 detects targets present within a detection range. The detection range is defined as a range in which targets can be detected by the ranging unit 11. The ranging unit 11 measures the distance to the target and the direction of the target. The ranging unit 11 provides the measurement results to the detection processing unit 12. The ranging unit 11 measures the distance and direction of each of multiple targets using a known method such as millimeter waves or a pulsed laser.

[0017] The ranging unit 11 may use a ranging sensor such as LiDAR (Light Detection And Ranging) or TOF (Time Of Flight). Blocks showing the components of the ranging unit 11 are not shown. The ranging unit 11 may output, in addition to the distance and direction measurement results, speed information detected from Doppler information and received signal level information indicating signal strength, etc. In addition to the distance and direction measurement results, the speed information detected from Doppler information and received signal level information indicating signal strength, etc. may be used for processing by the information processing device 5.

[0018] The detection processing unit 12 generates distance measurement information indicating the distance and direction of the target based on the measurement result of the distance measurement unit 11. The detection processing unit 12 sends the generated distance measurement information to the information processing device 5.

[0019] The imaging processing unit 3 captures an image of an imaging range that overlaps at least a part of the detection range of the distance measurement processing unit 2, detects multiple targets captured in the image as imaged targets, and generates type information indicating the type of each of the multiple imaged targets. The imaging processing unit 3 also identifies each of multiple areas in the image in which the imaged targets are captured as a target area. The imaging processing unit 3 includes an imaging unit 13 and a recognition processing unit 14.

[0020] The imaging unit 13 captures an image of an imaging range that overlaps at least a part of the detection range of the distance measurement processing unit 2. The image captured by the imaging unit 13 shows a target object present in the imaging range and the background of the target object. The imaging unit 13 provides image data representing the captured image to the recognition processing unit 14.

[0021] The recognition processing unit 14 detects a target object from an image represented by the image data. The recognition processing unit 14 detects the distance and direction of the captured target. The recognition processing unit 14 identifies the type of the captured target. The recognition processing unit 14 detects the distance and direction of the captured target and identifies the type of the target using a known method such as parallax or pattern matching.

[0022] For example, image artificial intelligence (AI) or other recognition and classification techniques may be used to detect the distance and direction of the imaged target and to identify the type of the imaged target. Note that object detection processing from an image containing general object information may be used to detect the type and position of the imaged target. The object detection processing may be performed using machine learning techniques such as, but not limited to, a convolutional neural network (CNN), a random forest, or a support vector machine.

[0023] The recognition processing unit 14 generates imaging information indicating the detected distance and direction and the identified type. The imaging information includes type information indicating the type of target captured in the image. Furthermore, by using an inference model that has previously learned images including the captured target, the accuracy of the detection, recognition, and discrimination can be improved.

[0024] The recognition processing unit 14 identifies a target area where an imaged target detected from the image exists. The target area is, for example, a rectangular area such as a bounding box obtained by object detection and capturing the target boundary. The target area is identified by four coordinates in the image. Each of the four coordinates indicates the position of a corner of the rectangle that is the shape of the target area. The recognition processing unit 14 generates target area information that indicates the target area and includes information on the four coordinates. The recognition processing unit 14 sends imaging information indicating the type or class (described later), the target area information, and image data to the information processing device 5. Note that the method for identifying the target area is not limited to the above method. Any method can be used to identify the target area, depending on the optical system or image conditions of the imaging unit 13.

[0025] The vehicle control unit 4 generates vehicle information, which is information related to the traveling of the vehicle in which the information processing system 1 is installed, and provides the generated vehicle information to the information processing device 5. The vehicle information is information indicating the steering angle of the vehicle, the speed of the vehicle, the yaw rate of the vehicle, etc.

[0026] The information processing device 5 performs processing to identify the distance and direction of a target object to be detected by the information processing system 1. The information processing device 5 includes a first communication interface unit 15, a vehicle interface unit 16, an information processing unit 17, a second communication interface unit 18, and a storage unit 19. Hereinafter, the first communication interface unit will be referred to as a first communication I / F unit, the vehicle interface unit will be referred to as a vehicle I / F unit, and the second communication interface unit will be referred to as a second communication I / F unit.

[0027] The first communication I / F unit 15 communicates with each of the ranging processing unit 2 and the imaging processing unit 3. The first communication I / F unit 15 receives ranging information sent from the ranging processing unit 2. The first communication I / F unit 15 provides the received ranging information to the information processing unit 17. The first communication I / F unit 15 receives imaging information, target area information, and image data sent from the imaging processing unit 3. The first communication I / F unit 15 provides the received imaging information, target area information, and image data to the information processing unit 17.

[0028] Furthermore, the information processing unit 17 transmits control signals and the like required by the ranging processing unit 2 to the ranging processing unit 2. The information processing unit 17 transmits control signals and the like required by the image capturing processing unit 3 to the image capturing processing unit 3. As the input / output I / F that each of the ranging processing unit 2, image capturing processing unit 3, and information processing unit 17 has, a serial communication interface such as I2C (registered trademark), SPI (Serial Peripheral Interface), or USB (Universal Serial Bus), or a transmission / reception I / F such as MIPI-CSI is used depending on the information or form detected by the ranging unit 11 or the image capturing unit 13.

[0029] The vehicle I / F unit 16 communicates with the vehicle control unit 4. The vehicle I / F unit 16 receives vehicle information sent from the vehicle control unit 4. The vehicle I / F unit 16 provides the received vehicle information to the information processing unit 17. The vehicle I / F unit 16 may output the detection information to the vehicle control unit 4 via an I / F such as a CAN (Controller Area Network).

[0030] The information processing unit 17 controls the information processing device 5. The information processing unit 17 executes various processes. For example, the information processing unit 17 determines whether a distance measurement target indicated by distance measurement information provided from the distance measurement processing unit 2 and an imaged target indicated by imaged information provided from the image processing unit 3 are the same target. If the distance measurement target and the imaged target are the same target, the information processing unit 17 combines the distance and direction indicated by the distance measurement information with the distance and direction indicated by the imaged information. The information processing unit 17 provides the distance and direction values ​​obtained by the combination to the second communication I / F unit 18. The configuration of the information processing unit 17 will be described in detail later.

[0031] The second communication I / F unit 18 communicates with the input / output unit 6. The second communication I / F unit 18 sends the distance and direction values ​​obtained by the combination to the input / output unit 6. The second communication I / F unit 18 may also send other information, such as an image, to the input / output unit 6 as necessary. The second communication I / F unit 18 receives information sent from the input / output unit 6, such as a control signal for controlling the information processing device 5 from the outside. The second communication I / F unit 18 provides the received information to the information processing device 5.

[0032] The storage unit 19 stores information and programs necessary for processing in the information processing device 5. The storage unit 19 also stores provisional value information in which the type of target and a provisional value indicating the size of the target are associated with each other. Pre-set provisional value information is stored in the storage unit 19. The provisional value information will be described later. The storage unit 19 may be used to store values ​​of distance and direction obtained by the information processing unit 17, output images of the imaging processing unit 3, etc. The storage unit 19 may be used as a buffer for temporarily storing input information for processing in the information processing unit 17 or information generated during the processing. The storage unit 19 may also be used to store log information, which is a history recorded by the processing of the information processing unit 17.

[0033] The input / output unit 6 accepts input operations to the information processing system 1 and sends the input information to the information processing device 5. For example, information necessary for processing in the information processing device 5 is input to the input / output unit 6. The input / output unit 6 sends the input information to the information processing device 5. The information sent to the information processing device 5 is stored in the storage unit 19 or processed in the information processing unit 17. The input / output unit 6 also receives information from the information processing system 1 and displays the received information on a screen. For example, the input / output unit 6 receives information from the information processing device 5, such as distance and direction values ​​and images, and outputs the received information to the outside. Alternatively, the input / output unit 6 displays the received values ​​as a GUI (Graphical User Interface).

[0034] Next, the configuration of the information processing unit 17 will be described. Fig. 2 is a block diagram schematically showing the configuration of the information processing unit 17 included in the information processing system 1 according to embodiment 1. Fig. 2 shows the information processing unit 17, the detection processing unit 12 of the ranging processing unit 2, the recognition processing unit 14 of the imaging processing unit 3, and the first communication I / F unit 15.

[0035] The information processing unit 17 includes an information acquisition unit 21, a provisional region calculation unit 22, a match probability calculation unit 23, a combination target determination unit 24, and a combination unit 25. The provisional region calculation unit 22 includes a provisional value identification unit 26 and a coordinate transformation mapping unit 27.

[0036] The information acquisition unit 21 acquires image data, image information of the imaged target, and position information of the imaged target from the imaging processing unit 3 via the first communication I / F unit 15. The imaging information includes type information of the imaged target and area information of the imaged target obtained by the object detection process by the recognition processing unit 14. The area information of the imaged target is information indicating the target area, which is the area of ​​the imaged target. The position information of the imaged target includes the distance and direction calculated from pixel position information of the identified target area.

[0037] The information acquisition unit 21 acquires distance measurement information of the distance measurement target from the distance measurement processing unit 2 via the first communication I / F unit 15. The distance measurement information is output from the detection processing unit 12 and is position information indicating the distance and direction of the distance measurement target. In addition to outputting position information indicating the distance and direction from the detection processing unit 12 to the information acquisition unit 21, the detection processing unit 12 may output speed information detected from Doppler information to the information acquisition unit 21, and may output received signal level information indicating signal strength, etc. to the information acquisition unit 21. The information processing unit 17 may utilize this information output from the detection processing unit 12 to the information acquisition unit 21. Note that in FIG. 2 , the arrow pointing from the information acquisition unit 21 to the match probability calculation unit 23 represents information sent from the information acquisition unit 21 to the match probability calculation unit 23. The information sent from the information acquisition unit 21 to the match probability calculation unit 23 may include speed information or received signal level information.

[0038] The detection processing unit 12 may output a ranging target ID (IDentifier), which is a target identifier assigned to the detection result of the ranging target by the ranging processing unit 2, to the information acquisition unit 21. The recognition processing unit 14 may output an imaged target ID, which is a target identifier assigned to the detection result of the imaged target by the imaging processing unit 3, to the information acquisition unit 21. Note that the information sent from the information acquisition unit 21 to the match probability calculation unit 23 may include the ranging target ID input from the detection processing unit 12 to the information acquisition unit 21, or may include the imaged target ID input from the recognition processing unit 14 to the information acquisition unit 21.

[0039] The detection processing unit 12 and the recognition processing unit 14 each detect, recognize, and extract output information from the RAW data. In the configuration shown in Figures 1 and 2, the detection processing unit 12 inside the ranging processing unit 2 detects, analyzes, and calculates ranging output information from the RAW data, and the recognition processing unit 14 inside the imaging processing unit 3 detects, recognizes, and extracts imaging output information from the RAW data. The information processing system 1 is not limited to this configuration. In the information processing system 1, RAW data may be input directly to the information acquisition unit 21, and the function of processing the RAW data may be realized by signal processing, image processing, or arithmetic processing in the information processing unit 17.

[0040] The provisional value specification unit 26 specifies the type of the imaged target based on the type information input to the information processing unit 17. The provisional value specification unit 26 reads out a provisional value associated with the same type as the specified type from the provisional value information stored in the storage unit 19. As a result, the provisional value specification unit 26 specifies a provisional value indicating the size of the ranging target. The provisional value specification unit 26 specifies a provisional value indicating the size of each of the multiple ranging targets for which identity determination with the imaged target is to be performed. The size, which is a provisional value, is set for each of the multiple ranging targets for which identity determination is to be performed by comparison with the imaged target whose type has been specified. Furthermore, the size, which is a provisional value, is set and specified for each of the multiple imaged targets to be detected.

[0041] The provisional area calculation unit 22 calculates a provisional area based on the provisional value identified based on the type information, the ranging information, and the aspect ratio of the target area. The provisional area is, for example, a rectangular area. The provisional area is the area of ​​the ranging target on the image when the coordinate transformation mapping unit 27 projects onto the captured image input to the information processing unit 17 the ranging position, which is the position where the ranging target is detected, and the size (area) of the ranging target assumed as the provisional value. The provisional area calculation unit 22 calculates a provisional area for each of the multiple ranging targets for which identity determination with the captured target is performed. The provisional area calculation unit 22 calculates the aspect ratio of the target area based on the target area information input to the information processing unit 17. The aspect ratio of the provisional area is set with reference to the aspect ratio of this target area.

[0042] The matching probability calculation unit 23 calculates a matching probability, which is a probability indicating a possibility that the captured target and each of the plurality of distance measurement targets will match, based on the target area that is compared with the provisional area calculated for each of the plurality of distance measurement targets and determined to be identical. The matching probability calculation unit 23 calculates the matching probability based on the size of the overlapping portion between each of the plurality of provisional areas and the target area.

[0043] The matching probability calculation unit 23 calculates the matching probability so that the larger the overlapping portion between the provisional region and the target region, the higher the matching probability. Specifically, the matching probability calculation unit 23 calculates the matching probability so that the larger the area of ​​the overlapping portion between the provisional region and the target region, the higher the matching probability. Alternatively, the matching probability calculation unit 23 may calculate the matching probability so that the larger the width of the overlapping portion between the provisional region and the target region, the higher the matching probability.

[0044] In the above description, the match probability calculation unit 23 calculates the match probability based on the degree of overlap between each of the plurality of provisional regions and the target region, but this is not limiting. The match probability calculation unit 23 may also calculate the match probability based on the position error between each of the plurality of provisional regions and the target region.

[0045] Alternatively, the match probability calculation unit 23 may calculate the match probability based on a combination of the size of the overlap between the provisional region and the target region and the position error. The match probability calculation unit 23 may calculate the match probability based on the size of the overlap between the provisional region and the target region and the position error, which is the difference between the distance and direction of each of the multiple range-finding targets and the distance and direction of the captured target. In this case, the match probability calculation unit 23 calculates the match probability so that the larger the overlap between the provisional region and the target region, the higher the match probability. The match probability calculation unit 23 also calculates the match probability so that the closer the distance between each of the multiple range-finding targets is to the distance of the captured target, the higher the match probability. The match probability calculation unit 23 may determine identity by adding information sent from the information acquisition unit 21 to the match probability calculation unit 23, such as speed information or received signal level from the ranging processing unit 2, the range-finding target ID assigned to the detection result by the ranging processing unit 2, or the captured target ID assigned to the detection result by the imaging processing unit 3, to the information for determination. That is, the matching probability calculation unit 23 may calculate the matching probability based on one or both of the size of overlap between the provisional area and the target area and the position error, the distance, speed, and received signal level detected by the ranging processing unit 2, and one or more of the target ID assigned to the detection result by the ranging processing unit 2 or the imaging processing unit 3.

[0046] The order in which the matching probability calculation unit 23 calculates the matching probability for each of the multiple imaged targets is arbitrary. For example, the matching probability calculation unit 23 may calculate the matching probability between the imaged target and each of the multiple distance-measurement targets for the multiple imaged targets detected in the imaging range, starting from the imaged target closest in distance from the imaging unit 13. The order in which the matching probability calculation unit 23 calculates the matching probability between the imaged target and each of the multiple distance-measurement targets is arbitrary. The matching probability calculation unit 23 may calculate the matching probability between the imaged target and the distance-measurement target for the multiple distance-measurement targets detected in the imaging range, starting from the distance-measurement target closest in distance from the distance measurement unit 11. Furthermore, a priority order may be set for each specified type.

[0047] The combining target determination unit 24 specifies, among the multiple ranging targets, a ranging target whose calculated matching probability for the provisional area is equal to or greater than a certain value as a target to be combined with the captured target. Alternatively, the combining target determination unit 24 may specify, among the multiple ranging targets, a ranging target whose calculated matching probability for the provisional area is the highest as a target to be combined with the captured target. Here, a ranging target specified as a target to be combined is referred to as a target ranging target.

[0048] The combining unit 25 executes a combining process that combines the distance and direction indicated in the imaging information of the imaged target with the distance and direction indicated by the target distance measurement target. The combining unit 25 outputs the distance and direction values ​​obtained by the combining process. The combining unit 25 may output at least one of type information and size information indicating the size of the imaged target along with the distance and direction values ​​obtained by the combining process. The combining unit 25 may also output an image or video obtained by adding the combined position information, type information, and area information to the captured image. Furthermore, the combining unit 25 may output the imaged target ID or distance measurement target ID, which is a target ID, speed information from the distance measurement processing unit 2, or a received signal level.

[0049] In this way, when the match probability between the imaged target and the distance-measured target is equal to or greater than a certain value, the information processing unit 17 considers the imaged target and the distance-measured target to be the same, and combines the distance and direction of the imaged target with the distance and direction of the distance-measured target. The information processing unit 17 outputs the distance and direction values ​​obtained by the combining process.

[0050] Here, a known method can be applied to the combination by the combination unit 25. For example, the combination unit 25 may apply a predetermined weight to each of the values ​​indicating the distance of the imaged target and the value indicating the distance of the distance-measured target, and combine the distance of the imaged target and the distance of the distance-measured target by adding or multiplying the weighted values ​​together. The combination unit 25 may apply a predetermined weight to each of the values ​​indicating the direction of the imaged target and the value indicating the direction of the distance-measured target, and combine the direction of the imaged target and the direction of the distance-measured target by adding or multiplying the weighted values ​​together. Furthermore, the distance and direction may be determined by selecting either the imaged target or the distance measurement information depending on the environmental conditions in the environment in which the vehicle exists or the operating conditions of the information processing system 1.

[0051] The information processing system 1 detects surrounding objects and outputs object information including the position, speed, type information, and target ID of the detected object to an external device via the input / output unit 6. For example, the input / output unit 6 may generate a display image for displaying the object information and output the display image to a display device (not shown).

[0052] FIG. 3 is a diagram showing an example of provisional value information stored in the storage unit 19 of the information processing system 1 according to the first embodiment. In the example shown in FIG. 3, each of the height value, width value, and area value is a provisional value indicating the size of the target. In the provisional value information, each provisional value is associated with a target type. In the information processing device 5, provisional values ​​corresponding to each of the multiple types are set in advance as provisional value information. The provisional value identification unit 26 identifies a provisional value corresponding to the type indicated in the type information from among the provisional values ​​for each of the multiple types.

[0053] In the example shown in FIG. 3, a target whose type is "automobile" has a height of "1.7 m," a width of "3.0 m," and an area of ​​"5.1 m." 2 Each provisional value of "automobile" shown in FIG. 3 indicates a standard size of the side of an automobile. Also, for a target whose type is "person", the provisional values ​​of height "1.7 m", width "0.7 m" and area "1.2 m" are associated. 2 " is associated with each provisional value. Each provisional value of "person" shown in Figure 3 is the standard size of a standing person. In the example shown in Figure 3, each provisional value of "person" is based on a standard-sized adult. In the provisional values ​​set for "person", it can be said that the standard posture is standing. Note that the types and provisional values ​​in the provisional value information shown in Figure 3 are examples. The types for which provisional values ​​are set are not limited to those shown in Figure 3. Furthermore, each provisional value for each type is not limited to the values ​​shown in Figure 3 and may be arbitrary.

[0054] Next, examples of calculating the matching probability in two cases will be described. In each case, the target is assumed to be a person. In the first case, a person in a standing position is photographed. In the second case, a person lying on the ground is photographed. FIGS. 4 to 7 are diagrams for explaining the first case. FIGS. 8 to 10 are diagrams for explaining the second case.

[0055] 4 is a diagram for explaining a distance measurement target indicated by distance measurement information and an imaged target indicated by image capturing information in embodiment 1. Fig. 4 is a diagram showing the image capturing range captured by the image capturing unit 13, the distance measurement target, and the imaged target as viewed from above in the first case.

[0056] The imaging unit 13 captures an image of an imaging range A1-A2, which is a range of a certain angle of view, from a lens position P. The lens position P is the position of the lens of the imaging unit 13. The distance measurement unit 11 detects targets in a detection range B1-B2. In the example shown in FIG. 4, the imaging range A1-A2 includes the detection range B1-B2 and is a range wider than the detection range B1-B2. In the first embodiment, the imaging range A1-A2 may be a range that overlaps at least a portion of the detection range B1-B2.

[0057] 4, two image capture targets C1 and C2 are located within an image capture range A1-A2, and three distance measurement targets R1, R2, and R3 are detected within a detection range B1-B2.

[0058] FIG. 5 is a first diagram for explaining imaged objects captured in an image represented by image data in the first embodiment. FIG. 5 can also be said to be a diagram of the image capturing range A1-A2 shown in FIG. 4 viewed from the lens position P in the first case. FIG. 5 schematically shows three imaged objects C1, C2, and C3 captured in the image in the first case. Each of the imaged objects C1, C2, and C3 is a person standing. Each of the imaged objects C1 and C3 is captured facing directly toward the imaging unit 13, i.e., facing forward. The imaged object C2 is captured facing a direction 90 degrees different from the direction of the imaging unit 13, i.e., facing sideways, as viewed from the imaged object C2.

[0059] The recognition processing unit 14 detects each of the captured targets C1, C2, and C3 from the image represented by the image data. The recognition processing unit 14 identifies the type of each of the captured targets C1, C2, and C3 as a person. The captured target C3 is not shown in FIG. 4. The recognition processing unit 14 identifies the target area in which each of the captured targets C1, C2, and C3 is captured. The recognition processing unit 14 detects the distance and direction of each of the captured targets C1, C2, and C3 based on the identified target area.

[0060] FIG. 6 is a first diagram schematically illustrating an image in which the target area of ​​an imaged target and the position where the distance measurement target was detected are projected in the first embodiment. Image IM1 shown in FIG. 6 is an image represented by image data in the first case, in which detection points indicating the positions where the distance measurement targets R1, R2, and R3 were detected are projected. Target areas D1 and D2 are the target areas of the imaged targets C1 and C2, respectively. The target areas D1 and D2 represent the ranges of the imaged targets C1 and C2 in the image represented by the image data by rectangular areas. In image IM1 shown in FIG. 6, the imaged targets C1 and C2 are represented by the target areas D1 and D2. The target areas are indicated by detection frames, which are rectangular frames representing the boundaries of targets in image space captured by object detection, for example. Each target area D1 and D2 can be said to be a rectangular approximation of the area of ​​each imaged target C1 and C2. The target area of ​​the imaged target C3 is not shown in FIG.

[0061] 7 is a first diagram schematically showing an image onto which the provisional areas of each distance measurement target are projected in embodiment 1. Image IM2 shown in FIG. 7 is obtained by projecting provisional areas T1, T2, and T3 onto image IM1 shown in FIG. 6. Provisional area T1 is the provisional area calculated for distance measurement target R1. Provisional area T2 is the provisional area calculated for distance measurement target R2. Provisional area T3 is the provisional area calculated for distance measurement target R3.

[0062] The provisional value determination unit 26 determines that the types of the distance measurement targets R1, R2, and R3 are "people" based on the type information input to the information processing unit 17. The provisional value determination unit 26 reads out provisional values ​​for people from the provisional value information shown in FIG. 3. The provisional value determination unit 26 determines the height of "1.7 m", the width of "0.7 m", and the area of ​​"1.2 m" as provisional values ​​for each of the distance measurement targets R1, R2, and R3. 2 Identify each value of ".

[0063] The provisional area calculation unit 22 determines provisional areas T1, T2, and T3 for the distance measurement targets R1, R2, and R3 based on the provisional values ​​of the distance measurement targets R1, R2, and R3, the distance measurement information for the distance measurement target R1, and the imaged targets C1 and C2. As shown in FIG. 7 , the sizes of the provisional areas T1, T2, and T3 vary depending on the distances at which the distance measurement targets R1, R2, and R3 are detected. For example, when calculating the match probability with the imaged target C1, the provisional areas T1, T2, and T3 for the distance measurement targets R1, R2, and R3 are determined by the size, which is a provisional value determined based on the type of the imaged target C1, and the aspect ratio of the target area D1. Similarly, when calculating the match probability with the imaged target C2, the provisional areas are determined by the size, which is a provisional value determined based on the type of the imaged target C2, and the aspect ratio of the target area D2.

[0064] The provisional area calculation unit 22 acquires the position of the ranging target. The coordinate transformation mapping unit 27 calculates the size of the provisional area on the image by converting the provisional value indicated in the provisional value information according to the distance of the ranging target. The size of each provisional area T1, T2, T3 is the size that would occur if a target of the same size as the provisional value were projected onto the image at the distance of each ranging target R1, R2, R3. The coordinate transformation mapping unit 27 then determines the height, width, and area of ​​the provisional area on the image. The coordinate transformation mapping unit 27 projects the area of ​​the ranging target of the calculated size onto the image as a provisional area at the position of the detected ranging target. In this way, the provisional area calculation unit 22 calculates the provisional area.

[0065] The match probability calculation unit 23 calculates a match probability for each provisional region T1, T2, T3 and each target region D1, D2 such that the greater the overlap between the provisional region and the target region, the greater the match probability. The match probability calculation unit 23 also calculates a match probability that the smaller the positional error between the distance measurement target and the imaged target. For the imaged target C1, the match probability calculation unit 23 calculates a match probability between the imaged target C1 and each distance measurement target R1, R2, R3 based on the overlap between the target region D1 and each provisional region T1, T2, T3, the size of the overlap, and the positional error of each region. For the imaged target C2, the match probability calculation unit 23 calculates a match probability between the imaged target C2 and each distance measurement target R1, R2, R3 based on the overlap between the target region D2 and each provisional region T1, T2, T3, the size of the overlap, and the positional error of each region.

[0066] The combining target determining unit 24 specifies, for each of the captured targets C1 and C2, a target distance measurement target among the distance measurement targets R1, R2, and R3 whose coincidence probability is equal to or greater than a certain value, as a target distance measurement target.

[0067] As shown in FIG. 7 , the target region D1 and the provisional region T1 overlap each other. There is no overlap between the target region D1 and each of the provisional regions T2 and T3. In the first case, the provisional value indicates the size of a standing person, and since the captured target C1 is a standing person, the area of ​​the overlapping portion between the provisional region T1 and the target region D1 is large. Therefore, a large value is calculated as the match probability between the captured target C1 and the range-finding target R1. Here, it is assumed that the match probability between the captured target C1 and the range-finding target R1 is equal to or greater than a certain value. The combination target determination unit 24 identifies the range-finding target R1 as the target range-finding target, as the same target that matches the captured target C1.

[0068] As shown in FIG. 7 , the target region D2 and the provisional region T2 overlap each other, and the target region D2 and the provisional region T3 overlap each other. There is no overlap between the target region D2 and the provisional region T1. Because the target region D2 overlaps with each of the provisional regions T2 and T3, the combining target determination unit 24 identifies one of the two ranging targets R2 and R3 that has a higher probability of matching with the captured target C2. Here, based on the positional error between the captured target C2 and the ranging target R2 and the positional error between the captured target C2 and the ranging target R3, it is assumed that the probability of matching between the captured target C2 and the ranging target R3 is greater than the probability of matching between the captured target C2 and the ranging target R3. Also, it is assumed that the probability of matching between the captured target C2 and the ranging target R2 is equal to or greater than a certain value. The combining target determining unit 24 identifies the distance measurement target R2 as the target object that matches the image capture target C2 and is the same target object.

[0069] Fig. 8 is a second diagram for explaining imaged targets captured in an image represented by image data in embodiment 1. Fig. 8 can also be said to be a diagram of the image capturing range A1-A2 shown in Fig. 4 viewed from lens position P in the second case. Fig. 8 schematically shows three imaged targets C2, C3, and C4 captured in the image in the second case.

[0070] The imaged target C2 shown in Fig. 8 is assumed to have the same posture, orientation, and position as the imaged target C2 shown in Fig. 5. The imaged target C3 shown in Fig. 8 is assumed to have the same posture, orientation, and position as the imaged target C3 shown in Fig. 5. The imaged target C4 shown in Fig. 8 is assumed to have the same position as the imaged target C1 shown in Fig. 5. While the imaged target C1 shown in Fig. 5 is in an upright position, the imaged target C4 shown in Fig. 8 is assumed to be lying on the ground.

[0071] The recognition processing unit 14 detects each of the captured targets C2, C3, and C4 from the image represented by the image data. The recognition processing unit 14 identifies the type of each of the captured targets C2, C3, and C4 as a person. The recognition processing unit 14 identifies the target area in which each of the captured targets C2, C3, and C4 is captured. The recognition processing unit 14 detects the distance and direction of each of the captured targets C2, C3, and C4 based on the identified target area. The handling of the captured targets C2 and C3 in the second case is the same as in the first case. Below, the following will mainly describe the differences between the processing of the captured target C4 in the second case and the processing of the captured target C1 in the first case.

[0072] FIG. 9 is a second diagram schematically illustrating an image projecting the target area of ​​an imaged target and the positions at which the ranging targets were detected in embodiment 1. Image IM3 shown in FIG. 9 is an image represented by image data in the second case, with detection points indicating the positions at which ranging targets R2, R3, and R4 were detected projected onto it. Target area D4 is the target area of ​​the imaged target C4. In image IM3 shown in FIG. 9, the imaged target C4 is represented by the target area D4. The target area is indicated by a detection frame, which is a rectangular frame that represents the boundary of the target in image space captured by object detection, for example. It can be said that target area D4 approximates the area of ​​the imaged target C4 to a rectangular area.

[0073] Fig. 10 is a second diagram schematically illustrating an image onto which the provisional areas of each distance measurement target are projected in embodiment 1. Image IM4 shown in Fig. 10 is obtained by projecting provisional areas T2, T3, and T4 onto image IM3 shown in Fig. 9. Provisional area T4 is a provisional area calculated for distance measurement target R4.

[0074] The provisional value determination unit 26 determines that the type of the distance measurement target R4 is "person" based on the type information input to the information processing unit 17. The provisional value determination unit 26 reads out provisional values ​​for the person from the provisional value information shown in FIG. 3. The provisional value determination unit 26 determines the provisional values ​​of the distance measurement target R4 as "person": height "1.7 m", width "0.7 m", and area "1.2 m". 2 Identify each value of ".

[0075] In the second case, the provisional value for the person shown in FIG. 3 indicates the size of a standing person, while the imaged target C4 is a person who is lying down. Here, it is assumed that the provisional area for the imaged target C4 is calculated for each ranging target using the specified provisional value without referring to the aspect ratio of the target area D4 as in the conventional case. This case will be described as a comparative example of the first embodiment.

[0076] FIG. 11 is a diagram illustrating a comparative example of the first embodiment. The specified provisional values ​​are a height of 1.7 m and a width of 0.7 m. In the image IM4a shown in FIG. 11, a provisional region T4a for the captured target C4 is calculated for each ranging target using the specified provisional values ​​without referring to the aspect ratio of the target region D4, and then projected. In this case, the area of ​​the overlapping portion between the provisional region T4a for the ranging target R4 and the target region D4 is significantly smaller than the area of ​​the overlapping portion between the provisional region T1 and the target region D1 in FIG. 7.

[0077] As described above, the conventional provisional region T4a has a fixed aspect ratio size determined by provisional values, and therefore cannot capture the region of the imaged target C4, which is in a fallen posture, and the area of ​​the portion where the provisional region T4a and the target region D4 overlap becomes small. Because the area of ​​the portion where the provisional region T4a and the target region D4 overlap becomes small, the probability of coincidence between the imaged target C4 and the distance-measurement target R4 does not exceed a certain value, and it is determined that the imaged target C4 and the distance-measurement target R4 are not the same object.

[0078] In contrast to the comparative example described above, in the first embodiment, the provisional area calculation unit 22 compares and references the aspect ratio of the provisional area T4 calculated based on the provisional values ​​for the ranging target R4 with the aspect ratio of the target area D4. The aspect ratio of the provisional area calculated based on the provisional values ​​for the ranging target R4 is the ratio of the height value to the width value of the provisional size of the ranging target R4. In this example, the aspect ratio of the provisional area T4 calculated based on the provisional values ​​for the ranging target R4 matches the aspect ratio of the target area D4.

[0079] For example, the provisional area calculation unit 22 calculates a provisional area T4 for the distance measurement target R4 so that the provisional size calculated based on the area is met as the provisional value for the distance measurement target R4, and changes the shape of the provisional area T4 to match the shape of the target area D4. The provisional area calculation unit 22 calculates a provisional area T4 for the distance measurement target R4 with a provisional size calculated based on the provisional value and corresponding to the shape of the target area D4 of the captured target C4. In this manner, the provisional area calculation unit 22 calculates each provisional area based on the provisional value specified based on the type information, the distance measurement information, and the aspect ratio of the target area. Similarly, when height and width are used as provisional values, the provisional area calculation unit 22 calculates a provisional area according to an aspect ratio based on the height and width values.

[0080] The matching probability calculation unit 23 calculates the matching probability between the target region D4 and each of the distance-detection targets R2, R3, and R4. Here, it is assumed that the matching probability between the image-captured target C4 and the distance-detection target R4 is equal to or greater than a certain value. The combined target determination unit 24 identifies the distance-detection target R4 as a distance-detection target of the same target as the image-captured target C4.

[0081] The information processing device 5 calculates a provisional area based on the provisional value identified based on the type information, the ranging information, and the aspect ratio of the target area using the provisional area calculation unit 22. This allows the information processing device 5 to determine the identity with high accuracy by referring to the aspect ratio of the target area of ​​the captured target, even if the attitude of the target has changed from the attitude used as the reference in the provisional value.

[0082] Fig. 12 is a diagram showing examples of target regions of imaged targets identified by the information processing system 1 according to embodiment 1. Fig. 12 shows examples of target regions identified for various postures of a person. Each of imaged targets C11-C15 is an example of an imaged target whose type is "person." In Fig. 12, a rectangle surrounding each of imaged targets C11-C15 represents a target region.

[0083] In the object detection process performed in the recognition processing unit 14 on the image acquired by the imaging processing unit 3, for example, object detection process using image AI, an area corresponding to the posture and orientation of the target can be captured as the detected target area. By referring to the aspect ratio of the detected target area, the match probability calculation unit 23 can determine the degree of match in accordance with the posture and orientation of the target and in a strict manner.

[0084] Next, a description will be given of the processing procedure by the information processing system 1. Fig. 13 is a flowchart showing the processing procedure by the information processing system 1 according to embodiment 1. Fig. 13 mainly shows a flow of a process for determining whether or not the detected targets of the ranging processing unit 2 and the imaging processing unit 3 match each other for each detection result, and an integration process when a match is found.

[0085] In step S1, the first communication I / F unit 15 shown in FIG. 1 acquires an image, imaging information, and location information from the imaging processing unit 3. The information acquisition unit 21 shown in FIG. 2 acquires the image, imaging information, and location information from the first communication I / F unit 15. As a result, the information processing device 5 collects the image, imaging information, and location information. The imaging information acquired by the information acquisition unit 21 includes target type information and area information. The information acquisition unit 21 provides the acquired image, imaging information, and location information to the information processing unit 17. In addition, an imaged target ID assigned to the imaged target, which is the detection result by the imaging processing unit 3, may be provided from the information acquisition unit 21 to the information processing unit 17.

[0086] In step S2, the match probability calculation unit 23 selects a target captured object for which match is to be determined. The captured object selected as the target captured object is one captured object that is compared with each of the multiple range-finding objects based on the match probability to determine whether the target is the same as the range-finding object. The recognition processing unit 14 identifies one of the multiple captured objects captured in the image as the target captured object. The captured object selected as the target captured object is determined based on the order of closest distance from the imaging unit 13 or the order of type, i.e., based on the operation mode of the information processing system 1. The order of closest distance from the imaging unit 13 can be said to be the order of increasing risk of collision with a vehicle equipped with the information processing system 1. The order of type refers to the order of increasing priority in determining the risk of collision.

[0087] In step S3, the coincidence probability calculation unit 23 acquires the position of the target object on the image.

[0088] In step S4, the coincidence probability calculation unit 23 acquires the type of the target object being imaged.

[0089] In step S5, the coincidence probability calculation unit 23 acquires the captured image and the target area of ​​the target object.

[0090] In step S6, the first communication I / F unit 15 acquires ranging information from the ranging processing unit 2. The ranging information acquired from the ranging processing unit 2 is position information indicating the distance and direction of the ranging target. The information acquisition unit 21 acquires ranging information for all object detection points from the first communication I / F unit 15. This allows the information processing device 5 to collect ranging information. The information acquisition unit 21 provides the acquired ranging information to the provisional area calculation unit 22. The information acquisition unit 21 may simultaneously acquire speed information or received signal level information obtained by the ranging processing unit 2 via the first communication I / F unit 15, along with the ranging information. The speed information, received signal level information, or ranging target ID acquired by the information acquisition unit 21 from the ranging processing unit 2 is provided to the match probability calculation unit 23 together with the ranging information of the ranging target. Detection information other than distance and direction, such as speed information, received signal level information, or range target ID, is also used as additional information for identifying and specifying the target.

[0091] In step S7, the provisional area calculation unit 22 selects a ranging target to be subjected to identity determination with the imaged target selected in step S2, and acquires position information of the detection point, which is the position where the ranging target was detected, from the information acquisition unit 21. Such position information includes information such as the distance to the detection point, the direction of the detection point, or coordinates indicating the physical location of the detection point. The ranging targets selected as targets for identity determination are determined in order of proximity from the ranging unit 11, or in order of proximity from the imaged target to be compared for identity, i.e., based on the manner in which the information processing system 1 is operated.

[0092] In step S8, the provisional value determination unit 26 determines a provisional value of the ranging target from the type information of the target to be imaged, the coincidence of which is to be determined. The provisional value determination unit 26 determines the provisional value of the ranging target by reading out the provisional value associated with the same type as the type acquired in step S4 from the provisional value information stored in the storage unit 19. The provisional value determination unit 26 similarly determines provisional values ​​for multiple ranging targets detected in the imaging range.

[0093] In step S9, the provisional area calculation unit 22 calculates the provisional area of ​​the ranging target. The coordinate transformation mapping unit 27 calculates the size of the provisional area by converting the provisional value determined in step S8 according to the position of the ranging target acquired in step S7. The coordinate transformation mapping unit 27 also calculates the aspect ratio of the provisional area. For example, when an area is used as the provisional value, an area equivalent to that area is calculated according to the aspect ratio of the target area of ​​the target imaged. The provisional shape and size of the physical ranging target are defined by that area. Similarly, when a height and a width are used as the provisional values, the provisional shape and size are defined according to the aspect ratio based on that height and that width.

[0094] The coordinate transformation mapping unit 27 calculates the position of the target on the image based on the position information acquired in step S7. The coordinate transformation mapping unit 27 projects a provisional area of ​​the calculated size onto the position of the target on the image. That is, the coordinate transformation mapping unit 27 projects and converts the provisional shape and size of the target calculated using the provisional value (size) specified by the type of the imaged target and the aspect ratio of the target area onto the image space according to the distance to the target. As a result, the coordinate transformation mapping unit 27 calculates the provisional area as coordinates (pixel values) indicating the pixel position. The coordinate transformation mapping unit 27 provides information indicating the provisional area calculated for each of the multiple targets to the match probability calculation unit 23.

[0095] The coordinate transformation mapping unit 27 may project and transform the position of the target detected in advance onto the image space, and define the shape of the target area of ​​the imaged target as a provisional area of ​​the target in the image space. This prevents changes in the shape of the area due to coordinate transformation or distortion of the optical system, and makes it possible to map the provisional area to a shape that is closer to the target area in the image space.

[0096] In step S10, the match probability calculation unit 23 calculates the match probability between the target imaged target and each of the multiple distance measurement targets. The match probability calculation unit 23 calculates the match probability between the target imaged target and each of the multiple distance measurement targets based on the target area acquired in step S5 and the provisional area for each of the multiple distance measurement targets calculated in step S9. The match probability calculation unit 23 provides the calculated match probability value to the combination target determination unit 24.

[0097] In step S11, the match probability calculation unit 23 determines whether the match probability with the target image target has been calculated for all distance measurement targets detected within the detection range of the distance measurement processing unit 2. If there are distance measurement targets for which the match probability with the target image target has not yet been calculated (step S11, No), the information processing system 1 returns to step S7. The information processing system 1 executes steps S7 to S10 for distance measurement targets for which the match probability with the target image target has not yet been calculated. On the other hand, if the match probability with the target image target has been calculated for all distance measurement targets detected within the detection range of the distance measurement processing unit 2 (step S11, Yes), the information processing system 1 proceeds to step S12.

[0098] In step S12, the combining target determination unit 24 identifies a ranging target that is determined to be identical to the target imaged target and to be combined with the target imaged target based on the matching probability calculated in step S10. The combining target determination unit 24 determines, among the multiple ranging targets, ranging targets with a matching probability equal to or greater than a certain value as identical to the target imaged target, and identifies them as target ranging targets to be combined with the target imaged target. The ranging target with the highest matching probability is selected as the ranging target to be combined. Alternatively, when multiple ranging targets that satisfy a certain level of matching probability are detected, the average of the object detection points, which are the detection points of these ranging targets, may be calculated, and a collective detection point representing this average may be obtained. In this way, the selection of ranging targets and the combination process may be performed according to the operation. The combining target determination unit 24 provides information indicating the identified target ranging targets to the combining unit 25.

[0099] In step S13, the combining unit 25 performs a combining process. The combining unit 25 performs the combining process to combine the distance and direction indicated in the imaging information of the target imaged object with the distance and direction indicated in the ranging information of the target ranging object. For example, in the combining process, the combining unit 25 calculates the average value of the position information of the imaged object and the position information of the ranging object. Alternatively, in the combining process, the combining unit 25 may combine each piece of position information by adding or multiplying the values ​​of each piece of position information to which a predetermined weighting has been applied. Alternatively, the combining unit 25 may select one of the two pieces of position information depending on environmental conditions or operating conditions to determine the distance and direction. The combining unit 25 outputs the distance and direction values ​​obtained by the combining process.

[0100] In step S14, the match probability calculation unit 23 determines whether the match between all imaged targets detected in the imaging range and the distance measurement target has been determined. If there are imaged targets for which the match between the imaged targets and the distance measurement targets has not yet been determined (No in step S14), the information processing system 1 returns to steps S2 and S7. The information processing system 1 executes steps S2 to S5 for imaged targets for which the match between the imaged targets and the distance measurement targets has not yet been determined. The information processing system 1 also executes steps S7 to S9 for all distance measurement targets to be subjected to identity determination with the imaged targets. At this time, distance measurement targets that were determined to be identical to the imaged target previously selected in step S12 and subjected to the combining process are excluded from the identity determination. Thereafter, the information processing system 1 executes steps S10 to S14. On the other hand, if it is determined that all of the imaging targets detected in the imaging range match the distance measurement targets (Yes in step S14), the information processing system 1 ends the processing according to the procedure shown in FIG.

[0101] Fig. 14 is a first diagram for explaining a method for determining a provisional area by the provisional area calculation unit 22 of the information processing system 1 according to embodiment 1. Fig. 15 is a second diagram for explaining a method for determining a provisional area by the provisional area calculation unit 22 of the information processing system 1 according to embodiment 1.

[0102] FIG. 14 shows examples of imaged targets captured in an image captured by the imaging unit 13. Imaged targets C5 and C6 are examples of imaged targets captured in the image. In FIG. 14, a rectangle surrounding the imaged target C5 represents a target region D5. A rectangle surrounding the imaged target C6 represents a target region D6. These target regions D5 and D6 are given by bounding boxes obtained by, for example, object detection.

[0103] FIG. 15 shows a perspective view in which the positions corresponding to the target objects and the provisional sizes of the target objects are projected onto an image space free of distortion due to the optical system. Here, it is assumed that two target objects are detected by the distance measurement processing unit 2. The coordinate transformation mapping unit 27 identifies the positions where each target object is projected in the image based on the distance measurement information of each target object whose consistency is compared with the target object being imaged. By identifying the positions where each target object is projected, the coordinate transformation mapping unit 27 identifies the positions where provisional regions are projected in the image. The sizes of regions T5 and T6 represent the provisional sizes of the target objects detected at the positions of the target objects C5 and C6, respectively. In a space free of distortion due to the optical system, the size of the region representing the target object is a physical size calculated from the provisional value corresponding to the type of target object and the aspect ratio.

[0104] The provisional area calculation unit 22 determines an area T5 corresponding to the shape and size of the target to be projected onto the image based on the provisional value specified based on the type information and the aspect ratio of the target area D5. The provisional area calculation unit 22 determines an area T6 corresponding to the shape and size of the target to be projected onto the image based on the provisional value specified based on the type information and the aspect ratio of the target area D6.

[0105] 16 is a third diagram for explaining a method for determining a provisional area by the provisional area calculation unit 22 of the information processing system 1 according to the first embodiment. FIG. 16 shows provisional areas T5b and T6b obtained by projecting, by the coordinate transformation mapping unit 27, the areas indicating the provisional sizes of the respective distance measurement targets shown in FIG. 15 onto an image captured by the imaging unit 13 and having distortion due to the optical system. Due to the distortion due to the optical system occurring in the captured image, each provisional area T5b and T6b projected onto the image may be deformed to be thinner or have a distorted shape compared to the areas T5 and T6 shown in FIG. 15. Each provisional area T5b and T6b shown in FIG. 16 is thinner and has a distorted shape than the areas T5 and T6 shown in FIG. 15.

[0106] The coordinate transformation mapping unit 27 linearly approximates or corrects shapes that are deformed or distorted, such as the provisional regions T5b and T6b, or the complex shapes of the provisional regions, in the distorted image space. Alternatively, the coordinate transformation mapping unit 27 converts these shapes into simple rectangular shapes. In this manner, for convenience, the coordinate transformation mapping unit 27 may define the regions T5 and T6 shown in FIG. 15 as provisional regions. This reduces errors caused by distortions inherent in the optical system when the matching probability calculation unit 23 calculates the areas of the overlapping portions between the provisional regions T5b and T6b and the target regions D5 and D6, thereby reducing the processing load required for determining whether the provisional regions T5b and T6b match the captured target.

[0107] According to the first embodiment, the information processing system 1 includes a ranging processing unit 2, an imaging processing unit 3, a provisional area calculation unit 22, and a match probability calculation unit 23. The provisional area calculation unit 22 includes a provisional value determination unit 26 and a coordinate transformation mapping unit 27. The provisional area calculation unit 22 calculates, in the coordinate transformation mapping unit 27, a provisional area for each of the plurality of ranging objects, which is the area of ​​the ranging object on the image when the ranging object is projected onto the image, based on a provisional value indicating the size of each of the plurality of ranging objects, ranging information, and an aspect ratio of the target area. The match probability calculation unit 23 calculates a probability indicating the possibility that the captured target and each of the plurality of ranging objects will match, based on the provisional area and target area calculated for each of the plurality of ranging objects. The information processing system 1 calculates the provisional area based on the provisional value, the ranging information, and the aspect ratio of the target area, thereby enabling the information processing system 1 to flexibly respond to changes in attitude or orientation and perform highly accurate determination of identity, thereby enabling the information processing system 1 to highly accurately determine the identity of targets in a combined sensor that performs ranging and imaging.

[0108] Further, provisional values ​​corresponding to each of the multiple types of imaged targets are set. The provisional value specifying unit 26 specifies a provisional value corresponding to the type indicated in the type information from among the provisional values ​​for each of the multiple types. This allows the information processing system 1 to calculate provisional areas for each of the multiple types of targets.

[0109] The provisional value specifying unit 26 specifies at least one of the height value set corresponding to the type, the width value set corresponding to the type, and the area value set corresponding to the type as a provisional value, thereby enabling the information processing system 1 to calculate a provisional area for the ranging target.

[0110] Furthermore, the provisional area calculation unit 22 determines a provisional area corresponding to the shape and size of the target to be projected onto the image based on the provisional value and the aspect ratio of the target area in the coordinate transformation mapping unit 27. This allows the information processing system 1 to flexibly respond to changes in attitude or orientation and calculate the provisional area for the target.

[0111] The coincidence probability calculation unit 23 calculates the coincidence probability based on one or both of the size of the overlapping portion between the provisional region and the target region and the difference between the positions of the multiple range-measurement targets and the position of the captured target. This allows the information processing system 1 to calculate the coincidence probability as an index for determining the identity of the targets.

[0112] The match probability calculation unit 23 also calculates the match probability based on one or both of the size of the overlapping portion between the provisional area and the target area, the difference between the positions of the multiple range-finding targets and the position of the imaged target, and one or more of the distance, speed, and received signal level detected by the range-finding processing unit 2. This allows the information processing system 1 to calculate the match probability as an index for determining the identity of the targets.

[0113] The information processing system 1 also includes a combining target determining unit 24 that specifies, among the plurality of ranging targets, ranging targets whose coincidence probability calculated for the provisional region is equal to or greater than a certain value as targets to be combined with the captured target. This allows the information processing system 1 to determine ranging targets that are identical to the captured target and combine the ranging targets with the captured target.

[0114] Furthermore, the combining target determination unit 24 identifies the ranging target with the highest match probability calculated for the provisional region among the multiple ranging targets as the target to be combined with the captured target. This allows the information processing system 1 to determine which ranging target is identical to the captured target and combine the ranging target with the captured target. Furthermore, when multiple ranging targets that satisfy a certain level of match probability are detected, for example, when there are multiple object detection points within a certain distance, the ranging processing unit 2 determines that multiple object detection points have been detected due to multiple reflections from the target. The combining unit 25 can combine the ranging target with the captured target after performing a synthesis process, such as calculating the average of the multiple object detection points to obtain a collective detection point.

[0115] Embodiment 2. In Embodiment 1, the physical size of the target to be measured was determined based on a provisional value identified based on the type information of the captured target and the aspect ratio of the target area, and then the provisional area was defined as an area in the distorted image where the physical shape and size of the target to be measured were projected onto the detected position of the target to be measured. In Embodiment 2, the position where the target to be measured is projected in the image is identified, and a provisional area is determined at the identified position in image space, having the same aspect ratio as the target area in image space and a size calculated based on the provisional value.

[0116] 17 is a diagram for explaining a method for determining a provisional area by the provisional area calculation unit 22 of the information processing system 1 according to embodiment 2. The information processing system 1 according to embodiment 2 has the same configuration as the information processing system 1 according to embodiment 1. In embodiment 2, processing different from embodiment 1 will be mainly described.

[0117] 17 shows temporary areas T5 and T6 projected onto an image space captured by the imaging unit 13 and having distortion due to the optical system. Temporary area T5 is a temporary area of ​​the distance measurement target detected by the distance measurement processing unit 2, and corresponds to the position of the imaged target C5 shown in FIG. 14. Temporary area T6 is a temporary area of ​​the distance measurement target detected by the distance measurement processing unit 2, and corresponds to the position of the imaged target C6 shown in FIG. 14. Here, it is assumed that two distance measurement targets are detected by the distance measurement processing unit 2. As in the first embodiment, the coordinate transformation mapping unit 27 identifies the position where each distance measurement target is projected in the image based on the distance measurement information of each distance measurement target compared with the target imaged target for consistency.

[0118] The coordinate transformation mapping unit 27 identifies the position of each ranging object in the image space. In the second embodiment, the image space is assumed to have distortion due to the optical system. The position of each ranging object in the image space is a position identified by the coordinate transformation mapping unit 27 based on the ranging information of each ranging object. The coordinate transformation mapping unit 27 determines, at the position identified for one of the two ranging objects, a region having the same aspect ratio as the target region D5 shown in FIG. 12 and a size calculated by the provisional region calculation unit 22 based on provisional values, as a provisional region T5. The coordinate transformation mapping unit 27 determines, at the position identified for the other of the two ranging objects, a region having the same aspect ratio as the target region D6 shown in FIG. 14 and a size calculated by the provisional region calculation unit 22 based on provisional values, as a provisional region T6.

[0119] Furthermore, the coordinate transformation mapping unit 27 may further simplify the process and determine, without using a provisional value, an area that has the same aspect ratio as the target area of ​​the imaged target and that has been corrected by enlarging or reducing the size of the target area of ​​the imaged target as the provisional area of ​​the target to be measured. The correction is performed taking into account the difference between the position of the target to be measured and the position of the imaged target.

[0120] In this way, the coordinate transformation mapping unit 27 specifies the position on the image where the distance measurement target is projected based on the distance measurement information, and determines a provisional area at the specified position, which is an area having the same aspect ratio as the target area and a size calculated based on the provisional value. As a result, the coordinate transformation mapping unit 27 calculates a provisional area for each of the multiple distance measurement targets based on the provisional value, the distance measurement information, and the aspect ratio of the target area.

[0121] According to the second embodiment, the provisional region calculation unit 22 determines, at a position identified based on the ranging information, a provisional region having the same aspect ratio as the target region and a size calculated based on provisional values, using the coordinate transformation mapping unit 27. By determining a provisional region having the same aspect ratio as the target region and a size calculated based on provisional values, the information processing system 1 can prevent changes in the shape of the region due to distortion caused by coordinate transformation or optical system, and can map the provisional region to a shape in image space that more closely resembles the target region. Therefore, the information processing system 1 can flexibly respond to changes in attitude or orientation and perform highly accurate identity determination. This enables the information processing system 1 to determine the identity of targets with high accuracy.

[0122] Embodiment 3 In embodiment 3, an example will be described in which a provisional value for each of a plurality of attitudes of a target or a provisional value for each of a plurality of orientations of a target is set for each type. The information processing system 1 according to embodiment 3 has the same configuration as the information processing system 1 according to embodiment 1 or 2. In embodiment 3, processing different from embodiment 1 or 2 will be mainly described.

[0123] For at least one of the multiple types, provisional values ​​for each of multiple attitudes or multiple orientations of the target are set as provisional values ​​of the target of that type in the information processing device 5. The provisional value specifying unit 26 specifies a provisional value corresponding to the type indicated in the type information and the attitude or orientation of the imaged target.

[0124] FIG. 18 is a diagram showing a first example of provisional value information stored in the storage unit 19 of the information processing system 1 according to the third embodiment. In the first example shown in FIG. 18, provisional values ​​are identified by classes including the classification of attitude or orientation. In FIG. 18, the height value, width value, and area value are each provisional values ​​indicating the size of the target. Different values ​​are set as provisional values ​​depending on the classification of attitude or orientation. These classes are detected as different types (classes) by the recognition processing unit 14, and each detection result is output as different type information.

[0125] "Automobile 1" and "Automobile 2" shown in FIG. 18 are types representing automobiles. Here, "Automobile 1" and "Automobile 2" are each a class. "Automobile 1" is a class representing an automobile facing "front". "Automobile 2" is a class representing an automobile facing "side". In the first example, classes of the same type but different orientations or attitudes are identified to identify the provisional value of the target.

[0126] As shown in FIG. 18, the class “Car 1” has a height of “1.7 m”, a width of “1.5 m”, and an area of ​​“2.55 m”. 2 Each provisional value of "Automobile 1" is a value indicating the standard size of the front of an automobile. The class of "Automobile 2" has a height of "1.7 m", a width of "3.0 m", and an area of ​​"5.1 m". 2 Each provisional value of "Car 2" is a value indicating the standard size of the side of a car.

[0127] "Person 1," "Person 2," and "Person 3" shown in FIG. 18 are types that represent people. Here, "Person 1," "Person 2," and "Person 3" are each considered to be a class. "Person 1" is a class that represents a person whose posture is "standing." "Person 2" is a class that represents a person whose posture is "falling." "Person 3" is a class that represents a person whose posture is "squatting."

[0128] The class of "Person 1" has a height of 1.7m, a width of 0.7m, and an area of ​​1.2m 2 The provisional values ​​of "Person 1" are values ​​indicating the standard size of a standing person. The provisional values ​​of "Person 2" class are "height" "0.5 m", "width" "1.7 m", and "area" "0.85 m". 2 The provisional values ​​of "Person 2" are values ​​that indicate the standard size of a person who has fallen. The provisional values ​​of "Person 3" are the height "1.2 m", width "1.2 m", and area "1.4 m". 2" are associated with each provisional value. Each provisional value of "Person 3" is a value indicating the standard size of a person in a half-squat position. In the example shown in FIG. 18 , each provisional value of "Person 1," "Person 2," and "Person 3" is based on the standard size of an adult.

[0129] Fig. 19 is a diagram showing a second example of the provisional value information stored in the storage unit 19 included in the information processing system 1 according to Embodiment 3. In the second example shown in Fig. 19, the provisional value is identified based on the type and the aspect ratio.

[0130] "Automobile" shown in FIG. 19 is a type representing an automobile. In the provisional value information shown in FIG. 19, the "automobile" type has items "front" and "side" that represent the orientation set therein. In the "front" item of "automobile", provisional values ​​corresponding to the "front" are set, similar to those in the case of "automobile 1" in FIG. 18. In the "side" item of "automobile", provisional values ​​corresponding to the "side" are set, similar to those in "automobile 2" in FIG. 18.

[0131] "Person" shown in FIG. 19 is a type representing a person. In the provisional value information shown in FIG. 19, the type of "person" has items set for "standing," "falling," and "squatting" that represent postures. In the "standing" item of "person," provisional values ​​corresponding to "standing" are set, similar to those in the case of "person 1" shown in FIG. 18. In the "falling" item of "person," provisional values ​​corresponding to "falling" are set, similar to those in the case of "person 2" shown in FIG. 18. In the "squatting" item of "person," provisional values ​​corresponding to "squatting" are set, similar to those in the case of "person 3" shown in FIG. 18.

[0132] The recognition processing unit 14 detects the attitude or orientation of an imaged target detected from an image captured by the imaging unit 13. Specifically, for example, in object detection, the recognition processing unit 14 learns and recognizes targets with different attitudes as different classes, and detects targets of the same type but with different orientations or attitudes as different classes. Furthermore, when detecting a person, the attitude may be determined by inference from the image, such as skeletal estimation.

[0133] Alternatively, as post-processing of the image AI, an approximate posture is determined from the aspect ratio of the bounding box, which is the target area of ​​the imaged target obtained by object detection, and the determined posture information is output together with the image information. These processes may be performed by image analysis within the information processing unit 17 after the image information is acquired by the information processing unit 17, rather than by the image processing unit 3. The recognition processing unit 14 generates posture information indicating the detected posture or direction. The recognition processing unit 14 sends the image information including the type information and posture information, the target area information, and the image data to the information processing device 5.

[0134] The provisional value specifying unit 26 specifies the type of the imaged target based on the type information input to the information processing unit 17. The provisional value specifying unit 26 specifies the attitude or orientation of the imaged target based on the attitude information input to the information processing unit 17. The provisional value specifying unit 26 specifies a provisional value corresponding to the type indicated in the type information and the attitude or orientation indicated in the attitude information.

[0135] 20 is a diagram for explaining a method for specifying a provisional value by the provisional value specifying unit 26 of the information processing system 1 according to the third embodiment. Each of the imaged targets C16 and C17 is an example of an imaged target whose type is "automobile." In FIG. 17, a rectangle surrounding each of the imaged targets C16 and C17 represents a target area.

[0136] For the type information in Figure 18, the provisional value identification unit 26 identifies the type of the imaged target C16 as a ``front-facing vehicle'' and the type of the imaged target C17 as a ``side-facing vehicle'' based on the type information classified by posture and orientation.

[0137] When the provisional value information stored in the storage unit 19 is provisional value information according to the first example shown in Fig. 18, the provisional value identification unit 26 reads out each provisional value of "Automobile 1" for the imaged target C16 from the provisional value information shown in Fig. 18. The provisional value identification unit 26 reads out each provisional value of "Automobile 2" for the imaged target C17 from the provisional value information shown in Fig. 18.

[0138] 19 , the provisional value identification unit 26 identifies the type of each of the imaged targets C16 and C17 as "automobile" based on the type information. The provisional value identification unit 26 estimates the attitude and orientation from the aspect ratio (ratio of width to height) of the target area in the captured image, and identifies the orientation of the imaged target C16 as "front" based on attitude information that matches these. The provisional value identification unit 26 identifies the orientation of the imaged target C17 as "side" based on the attitude information.

[0139] When the provisional value information stored in the storage unit 19 is provisional value information according to the second example shown in Fig. 19 , the provisional value identification unit 26 reads out the provisional values ​​of "automobile" and "front" for the imaged target C16 from the provisional value information shown in Fig. 19. The provisional value identification unit 26 reads out the provisional values ​​of "automobile" and "side" for the imaged target C17 from the provisional value information shown in Fig. 19. In this way, the provisional value identification unit 26 identifies a provisional value corresponding to the type indicated in the type information and the orientation of the imaged target.

[0140] Next, an example of an imaged target whose type is a "person" will be described. Each of the imaged targets C11-C15 shown in Fig. 12 is an imaged target whose type is a "person." As shown by the target area represented as a rectangle surrounding each of the imaged targets C11-C15 in Fig. 12, each provisional value corresponding to a posture such as "standing," "falling," and "squatting" is read out.

[0141] In this way, provisional values ​​subdivided based on at least one of the target attitude and the target orientation are set as provisional value information. The provisional area calculation unit 22 can identify a provisional area based on the provisional value corresponding to the target attitude or the target orientation. The match probability calculation unit 23 can flexibly respond to changes in the attitude or orientation and calculate an accurate match probability.

[0142] In the above description, the attitude or orientation of the imaged target is detected from the image captured by the imaging unit 13. The provisional value identification unit 26 may identify the attitude or orientation of the imaged target based on the aspect ratio of the target area indicated in the target area information. In this case, the provisional area calculation unit 22 can also identify the provisional area based on the provisional value corresponding to the attitude or orientation of the target.

[0143] According to the third embodiment, a provisional value for each of a plurality of target attitudes or a provisional value for each of a plurality of target orientations is set as a provisional value for at least one of a plurality of types. The provisional value specification unit 26 specifies a provisional value corresponding to the type indicated in the type information and the attitude or orientation of the captured target. The information processing system 1 can flexibly respond to changes in attitude or orientation and perform highly accurate determination of identity. This enables the information processing system 1 to determine the identity of targets with high accuracy.

[0144] Next, a description will be given of the hardware configuration of the information processing system 1 according to the first to third embodiments. Fig. 21 is a block diagram showing an example of the hardware configuration of the information processing system 1 according to the first to third embodiments.

[0145] The information processing system 1 includes an information processing device 5, a distance measurement sensor 31, a distance measurement sensor ECU (Electronic Control Unit) 32, a camera 33, a camera ECU 34, a vehicle control ECU 35, and a PC (Personal Computer) 36. The information processing device 5 includes communication I / Fs 41 and 43, a CAN I / F 42, a processor 44, and a memory 45.

[0146] 1 is realized by using a distance measurement sensor 31. The distance measurement sensor 31 is, for example, a millimeter wave radar equipped with a transmitting antenna for transmitting millimeter waves and a receiving antenna for receiving millimeter waves, or a LiDAR that measures distances using laser light.

[0147] The distance measurement unit 11 is realized by using a distance measurement sensor 31. The detection processing unit 12 is realized by using a distance measurement sensor ECU 32. The imaging unit 13 is realized by using a camera 33 as an imaging device. The recognition processing unit 14 is realized by using a camera ECU 34. The vehicle control unit 4 is realized by using a vehicle control ECU 35. The information processing system 1 may be provided with a camera that functions as the imaging unit 13 and the recognition processing unit 14.

[0148] The information processing device 5 can be realized by a so-called computer. The first communication I / F unit 15 is realized by using a communication I / F 41. The vehicle I / F unit 16 is realized by using a CAN I / F 42. The second communication I / F unit 18 is realized by using a communication I / F 43. The storage unit 19 is realized by using a memory 45.

[0149] The information processing unit 17 is realized by a processing circuit, which is a circuit on which the processor 44 executes software. The information processing unit 17 is realized by software, firmware, or a combination of software and firmware. The software or firmware is written as a program and stored in the memory 45. The processor 44 reads and executes the program stored in the memory 45, thereby realizing the functions of the information processing unit 17. The processing circuit includes the processor 44 and the memory 45 for storing a program that results in the processing of the information processing device 5 being executed. The program stored in the memory 45 can also be said to cause the computer to execute the processing procedures and methods of the information processing device 5. The memory 45 is also used as temporary memory when the processor 44 executes various processes.

[0150] The processor 44 is a central processing unit (CPU), a micro control unit (MCU), a graphics processing unit (GPU), or a digital signal processor (DSP). The memory 45 is, for example, a non-volatile or volatile semiconductor memory such as a random access memory (RAM), a read only memory (ROM), a flash memory, an erasable programmable read only memory (EPROM), or an electrically erasable programmable read only memory (EEPROM (registered trademark)), a magnetic disk, a flexible disk, an optical disk, a compact disk, a mini disk, or a digital versatile disk (DVD).

[0151] The information processing device 5 may include an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array). The program stored in the memory 45 may be provided by being stored on a recording medium such as a CD (Compact Disc)-ROM or a DVD-ROM. The program may be provided by being stored on a computer connected to a network such as the Internet and downloaded via the network. The program may be provided or distributed via a network such as the Internet.

[0152] The input / output unit 6 is realized by using a PC 36. The PC 36 functions as a GUI, which is an input / output device. The PC 36 includes a processing circuit similar to the processing circuit included in the information processing device 5, and a communication I / F similar to the communication I / Fs 41 and 43. The PC 36 also includes an input device, which is a device for input, and a monitor that displays a screen. The input device includes, for example, a keyboard, a mouse, a keypad, or a touch panel. The monitor is, for example, an LCD (Liquid Crystal Display) or an organic EL (Electro-Luminescence) display.

[0153] The components of the information processing system 1 do not necessarily have to be physically configured as shown in the figure. The specific form of distribution and integration of the components is not limited to that shown in the figure. The components may be configured in a functionally or physically distributed manner in any unit, or may be configured in an integrated manner. For example, the functions of the information processing device 5 may be distributed across two or more devices.

[0154] In the above description, the information processing unit 17 is configured to output the distance and direction values ​​obtained by the combining process, but this is not limiting. The information processing unit 17 may be configured to output the match probability value calculated by the match probability calculation unit 23. In this case, the information processing unit 17 may be configured such that the combiner 24 and the combiner 25 are omitted.

[0155] In the first to third embodiments, the information processing system 1 is a vehicle control system. The functions of the information processing system 1 described in the first to third embodiments may be applied to systems other than vehicle control systems.

[0156] The configurations shown in the above embodiments are examples of the contents of the present disclosure. The configurations of each embodiment can be combined with other known technologies. The configurations of each embodiment can also be combined as appropriate. Part of the configuration of each embodiment can be omitted or modified without departing from the gist of the present disclosure.

[0157] Various aspects of the present disclosure are summarized below as appendices.

[0158] (Supplementary Note 1) A ranging processing unit that detects multiple targets present in a detection range as ranging targets and generates ranging information indicating the distance and direction of each of the multiple ranging targets; an imaging processing unit that captures an image of an imaging range that overlaps at least a part of the detection range of the ranging processing unit, detects the multiple targets captured in the image as imaging targets, generates type information indicating the type of each of the multiple imaging targets, and identifies each of multiple areas in the image where the imaging targets are captured as a target area; and a provisional area calculation unit that calculates, for each of the multiple ranging targets whose type has been identified, a provisional value indicating the size of each of the multiple ranging targets identified based on the type information of the imaging targets, the ranging information of the ranging targets, and an aspect ratio of the target area, and calculates a provisional area for each of the multiple ranging targets. and a match probability calculation unit that calculates a match probability indicating a possibility that the imaged target and each of the distance-measurement targets will match, based on the target area of ​​the imaged target and the provisional area calculated for each of the distance-measurement targets. (Supplementary Note 2) The information processing system according to Supplementary Note 1, wherein the provisional area calculation unit specifies at least one of a height value set corresponding to the type, a width value set corresponding to the type, and an area value set corresponding to the type as the provisional value. (Supplementary Note 3) The information processing system according to Supplementary Note 1 or 2, wherein the provisional area calculation unit calculates in advance a provisional size and shape of the distance-measurement target based on the provisional value and an aspect ratio of the target area, and determines the area of ​​the distance-measurement target projected on the image as the provisional area based on the distance measurement information and the provisional size and shape. (Supplementary Note 4) The information processing system described in Supplementary Note 1 or 2 is characterized in that the provisional area calculation unit calculates the position where the ranging target is projected in the image based on the ranging information, and determines the provisional area to be an area on the image that has the same aspect ratio as the target area and has a size calculated based on the provisional value.(Supplementary Note 5) The information processing system according to Supplementary Note 1 or 2, characterized in that the provisional area calculation unit calculates a position on the image where the ranging target is projected based on the ranging information, and determines, as the provisional area, an area on the image having the same aspect ratio as the target area and a size calculated based on a difference in position between the ranging target and the captured target. (Supplementary Note 6) The information processing system according to any one of Supplementary Notes 1 to 5, characterized in that the provisional values ​​corresponding to each of a plurality of the types are set, and the provisional area calculation unit includes a provisional value specification unit that specifies, from the provisional values ​​for each of the plurality of types, the provisional value corresponding to the type indicated in the type information. (Supplementary Note 7) The information processing system described in Supplementary Note 6 is characterized in that the imaging processing unit outputs the type information indicating each of the multiple types classified for each of the multiple attitudes of the imaged target or each of the multiple orientations of the imaged target, the provisional value classified for each of the multiple attitudes of the imaged target or the provisional value classified for each of the multiple orientations of the imaged target is set as the provisional value for at least one of the multiple types, and the provisional value identification unit identifies the provisional value corresponding to the type information defined for each attitude of the imaged target or each orientation of the imaged target. (Supplementary Note 8) The information processing system according to Supplementary Note 6, wherein the provisional value for at least one of the plurality of types is set to the provisional value classified for each of a plurality of attitudes of the imaged target or the provisional value classified for each of a plurality of orientations of the imaged target, and the provisional value specifying unit determines the attitude or orientation of the target based on an aspect ratio of the target area of ​​the imaged target in addition to the types of the plurality of imaged targets, and specifies the provisional value corresponding to the determined attitude or orientation. (Supplementary Note 9) The information processing system according to any one of Supplementary Notes 1 to 8, wherein the match probability calculation unit calculates the match probability based on one or both of a size of an overlapping portion between the provisional area and the target area and a difference between the position of each of the plurality of range-measured targets and the position of the imaged target.(Supplementary Note 10) The information processing system according to any one of Supplements 1 to 8, wherein the match probability calculation unit calculates the match probability based on one or more of the size of an overlapping portion between the provisional region and the target region, the difference between the positions of each of the plurality of range-finding targets and the position of the imaged target, the distance, speed, and received signal level detected by the range-finding processing unit, and a target identifier assigned to the detection result by the range-finding processing unit or the imaged processing unit. (Supplementary Note 11) The information processing system according to any one of Supplements 1 to 10, further comprising a combination target determination unit that specifies, among the plurality of range-finding targets, a range-finding target whose match probability calculated for the provisional region is equal to or greater than a certain value, as a target to be combined with the imaged target. (Supplementary Note 12) The information processing system according to Supplementary Note 11, wherein the combination target determination unit specifies, among the plurality of range-finding targets, a range-finding target whose match probability calculated for the provisional region is highest, as a target to be combined with the imaged target. (Supplementary Note 13) A communication interface unit that acquires ranging information indicating the distance and direction of each of a plurality of ranging targets, wherein a plurality of targets present in a detection range are detected as ranging targets, an image captured of an imaging range that overlaps at least a part of the detection range, type information indicating the type of each of the plurality of imaging targets, wherein the plurality of targets captured in the image are detected as imaging targets, and target area information indicating target areas that are each of a plurality of areas in the image in which the imaging targets are captured; a provisional area calculation unit that calculates, for each of the plurality of ranging targets, a provisional value indicating the size of each of the plurality of ranging targets identified based on the type information of the imaging targets, the ranging information of the ranging targets, and an aspect ratio of the target area, and calculates a provisional area for each of the plurality of ranging targets. and a match probability calculation unit that calculates a match probability indicating a possibility that the imaged target and each of the plurality of distance-measurement targets will match, based on the target area of ​​the imaged target and the provisional area calculated for each of the distance-measurement targets.(Supplementary Note 14) A processing step in which a computer acquires: a processing step in which a plurality of targets present in a detection range are detected as distance measurement targets, and distance measurement information indicating the distance and direction of each of the plurality of distance measurement targets; an image captured of an imaging range overlapping at least a part of the detection range; a processing step in which a plurality of targets captured in the image are detected as imaged targets, and type information indicating the type of each of the plurality of imaged targets; and target area information indicating target areas which are each of a plurality of areas in the image in which the imaged targets are captured; and a processing step in which, for each of the plurality of imaged targets whose type has been identified, a provisional value indicating the size of each of the plurality of distance measurement targets identified based on the type information of the imaged targets, the distance measurement information of the distance measurement targets, and an aspect ratio of the target area, and the area of ​​the distance measurement target projected on the image is determined as a provisional area for each of the plurality of distance measurement targets. and a processing step of calculating a matching probability indicating a possibility that the imaged target and each of the plurality of distance-measurement targets will match, based on the target area of ​​the imaged target and the provisional area calculated for each of the distance-measurement targets. (Supplementary Note 15) A processing step of detecting a plurality of targets present in a detection range as distance measurement targets, and acquiring distance measurement information indicating the distance and direction of each of the plurality of distance measurement targets, an image of an imaging range overlapping at least a part of the detection range, and detecting a plurality of targets captured in the image as imaged targets, and acquiring type information indicating the type of each of the plurality of imaged targets, and target area information indicating target areas which are each of a plurality of areas in the image in which the imaged targets are captured; and a processing step of calculating, for each of the plurality of distance measurement targets, a provisional value indicating the size of each of the plurality of distance measurement targets identified based on the type information of the imaged targets, the distance measurement information of the distance measurement targets, and an aspect ratio of the target area, and using the area of ​​the distance measurement target projected on the image as a provisional area for each of the plurality of distance measurement targets. and a processing step of calculating a match probability indicating a possibility that the imaged target and each of the plurality of distance-measured targets will match, based on the target area of ​​the imaged target and the provisional area calculated for each of the distance-measured targets.(Supplementary Note 16) An information processing system comprising: a millimeter-wave radar that detects targets present in a detection range; a camera that captures an image of an imaging range that overlaps at least a portion of the detection range of the millimeter-wave radar to detect the targets captured in the image, generates type information that indicates a type of the target detected from the image, and identifies a target area in which the target detected from the image exists; a provisional area calculation unit that estimates a size of the target detected by the millimeter-wave radar based on the type information, and calculates an area in which the target exists on the image as a provisional area based on the distance and direction of the target obtained from the millimeter-wave radar and an aspect ratio of the target area; and a match probability calculation unit that calculates a match probability between the target detected from the image and the target detected by the millimeter-wave radar based on the target area and the provisional area.

[0159] 1 Information processing system, 2 Distance measurement processing unit, 3 Imaging processing unit, 4 Vehicle control unit, 5 Information processing device, 6 Input / output unit, 11 Distance measurement unit, 12 Detection processing unit, 13 Imaging unit, 14 Recognition processing unit, 15 First communication I / F unit, 16 Vehicle I / F unit, 17 Information processing unit, 18 Second communication I / F unit, 19 Memory unit, 21 Information acquisition unit, 22 Provisional area calculation unit, 23 Matching probability calculation unit, 24 Combination target determination unit, 25 Combining unit, 26 Provisional value identification unit, 27 Coordinate transformation mapping unit, 31 Distance measurement sensor, 32 Distance measurement sensor ECU, 33 Camera, 34 Camera ECU, 35 Vehicle control ECU, 36 PC, 41, 43 Communication I / F, 42 CANI I / F, 44 Processor, 45 Memory.

Claims

1. A ranging processing unit that detects multiple targets present in a detection range as ranging targets and generates ranging information indicating the distance and direction of each of the multiple ranging targets; an imaging processing unit that captures an image of an imaging range that overlaps at least a portion of the detection range of the ranging processing unit, detects the multiple targets captured in the image as captured targets, generates type information indicating the type of each of the multiple captured targets, and identifies each of the multiple areas in the image in which the captured targets are captured as a target area; and a provisional area calculation unit that calculates, for each of the multiple ranging targets whose type has been identified, a provisional value indicating the size of each of the multiple ranging targets identified based on the type information of the captured targets, the ranging information of the ranging targets, and the aspect ratio of the target area, and calculates a provisional area for each of the multiple ranging targets, the provisional area being determined based on the provisional value indicating the size of each of the multiple ranging targets identified based on the type information of the captured targets, the ranging information of the ranging targets, and the aspect ratio of the target area. and a match probability calculation unit that calculates a match probability indicating a possibility that the imaged target and each of the plurality of distance-measured targets will match, based on the target area of ​​the imaged target and the provisional area calculated for each of the distance-measured targets.

2. The information processing system described in claim 1, characterized in that the provisional area calculation unit identifies at least one of the height value set corresponding to the type, the width value set corresponding to the type, and the area value set corresponding to the type as the provisional value.

3. The information processing system of claim 1 or 2, characterized in that the provisional area calculation unit calculates in advance the provisional size and shape of the ranging target based on the provisional value and the aspect ratio of the target area, and determines the area of ​​the ranging target projected onto the image as the provisional area based on the ranging information and the provisional size and shape.

4. The information processing system described in claim 1 or 2, characterized in that the provisional area calculation unit calculates the position where the ranging target is projected in the image based on the ranging information, and determines as the provisional area an area on the image that has the same aspect ratio as the target area and a size calculated based on the provisional value.

5. The information processing system of claim 1 or 2, characterized in that the provisional area calculation unit calculates the position where the ranging target is projected in the image based on the ranging information, and determines the provisional area to be an area on the image that has the same aspect ratio as the target area and a size calculated based on the difference in position between the ranging target and the imaged target.

6. An information processing system as described in any one of claims 1 to 5, characterized in that the provisional values ​​corresponding to each of the multiple types are set, and the provisional area calculation unit is provided with a provisional value identification unit that identifies the provisional value corresponding to the type indicated in the type information from among the provisional values ​​for each of the multiple types.

7. The information processing system according to claim 6, characterized in that the imaging processing unit outputs the type information indicating each of the multiple types classified for each of the multiple attitudes of the imaged target or for each of the multiple orientations of the imaged target, the provisional value classified for each of the multiple attitudes of the imaged target or the provisional value classified for each of the multiple orientations of the imaged target is set as the provisional value for at least one of the multiple types, and the provisional value identification unit identifies the provisional value corresponding to the type information defined for each attitude of the imaged target or for each orientation of the imaged target.

8. The information processing system of claim 6, wherein the provisional value for at least one of the plurality of types is set to a provisional value classified for each of a plurality of attitudes of the imaged target, or a provisional value classified for each of a plurality of orientations of the imaged target, and the provisional value identification unit determines the attitude or orientation of the target based on the aspect ratio of the target area of ​​the imaged target in addition to the types of the plurality of imaged targets, and identifies the provisional value corresponding to the determined attitude or orientation.

9. An information processing system according to any one of claims 1 to 8, characterized in that the matching probability calculation unit calculates the matching probability based on one or both of the size of the overlapping area between the provisional area and the target area and the difference between the positions of each of the multiple range-finding targets and the position of the imaged target.

10. An information processing system as described in any one of claims 1 to 8, characterized in that the matching probability calculation unit calculates the matching probability based on one or both of the size of the overlapping area between the provisional area and the target area, the difference between the positions of each of the multiple ranging targets and the position of the imaged target, the distance, speed and received signal level detected by the ranging processing unit, and one or more target identifiers assigned to the detection results by the ranging processing unit or the image processing unit.

11. An information processing system described in any one of claims 1 to 10, characterized in that it is provided with a combining target determination unit that identifies, among the multiple ranging targets, those ranging targets for which the matching probability calculated for the provisional area is equal to or greater than a certain value as targets to be combined with the imaging target.

12. The information processing system described in claim 11, characterized in that the combining target determination unit identifies, among the multiple ranging targets, the ranging target with the highest matching probability calculated for the provisional area as the target to be combined with the imaged target.

13. A communication interface unit that acquires distance measurement information indicating the distance and direction of each of a plurality of distance measurement targets, wherein a plurality of targets present in a detection range are detected as distance measurement targets, an image captured of an imaging range that overlaps at least a portion of the detection range, type information indicating the type of each of the plurality of imaged targets, wherein the plurality of targets captured in the image are detected as imaged targets, and target area information indicating target areas which are each of a plurality of areas in the image in which the imaged targets are captured; a provisional area calculation unit that calculates, for each of the plurality of distance measurement targets, a provisional value indicating the size of each of the plurality of distance measurement targets identified based on the type information of the imaged targets, the distance measurement information of the distance measurement targets, and the aspect ratio of the target area, and calculates a provisional area for each of the plurality of distance measurement targets, wherein the provisional area is determined based on the distance measurement information of the distance measurement targets and the aspect ratio of the target area. and a match probability calculation unit that calculates a match probability indicating a possibility that the imaged target and each of the plurality of distance-measured targets will match, based on the target area of ​​the imaged target and the provisional area calculated for each of the distance-measured targets.

14. A computer includes a processing step of detecting a plurality of targets present in a detection range as distance measurement targets, acquiring distance measurement information indicating the distance and direction of each of the plurality of distance measurement targets, an image of an imaging range overlapping at least a part of the detection range, detecting a plurality of targets captured in the image as imaged targets, and acquiring type information indicating the type of each of the plurality of imaged targets, and target area information indicating target areas which are each of a plurality of areas in the image in which the imaged targets are captured; and a processing step of calculating, for each of the plurality of imaged targets whose type has been identified, a provisional value indicating the size of each of the plurality of distance measurement targets identified based on the type information of the imaged targets, the distance measurement information of the distance measurement targets, and the aspect ratio of the target area, and using the area of ​​the distance measurement target projected on the image as a provisional area. and a processing step of calculating a matching probability indicating a possibility that the imaged target and each of the plurality of distance-measurement targets will match, based on the target area of ​​the imaged target and the provisional area calculated for each of the distance-measurement targets.

15. A processing step of detecting a plurality of targets present in a detection range as distance measurement targets, and acquiring distance measurement information indicating the distance and direction of each of the plurality of distance measurement targets, an image of an imaging range overlapping at least a part of the detection range, and detecting a plurality of targets captured in the image as imaged targets, and acquiring type information indicating the type of each of the plurality of imaged targets, and target area information indicating target areas which are each of a plurality of areas in the image in which the imaged targets are captured; a processing step of calculating, for each of the plurality of distance measurement targets, a provisional value indicating the size of each of the plurality of distance measurement targets identified based on the type information of the imaged targets, the distance measurement information of the distance measurement targets, and the aspect ratio of the target area, and using the area of ​​the distance measurement target projected on the image as a provisional area for each of the plurality of distance measurement targets; and a processing step of calculating a match probability indicating a possibility that the imaged target and each of the plurality of distance-measured targets will match, based on the target area of ​​the imaged target and the provisional area calculated for each of the distance-measured targets.

16. An information processing system comprising: a millimeter-wave radar that detects targets present within a detection range; a camera that captures an image of an imaging range that overlaps at least a portion of the detection range of the millimeter-wave radar to detect the targets captured in the image, generates type information indicating the type of the target detected from the image, and identifies a target area in which the target detected from the image exists; a provisional area calculation unit that estimates the size of the target detected by the millimeter-wave radar based on the type information, and calculates, as a provisional area, the area in which the target exists on the image based on the distance and direction of the target obtained from the millimeter-wave radar and the aspect ratio of the target area; and a match probability calculation unit that calculates a match probability between the target detected in the image and the target detected by the millimeter-wave radar based on the target area and the provisional area.

Images