Image recognition device, image recognition program and image recognition method

Abstract

To provide an image recognition device, an image recognition program, and an image recognition method that can appropriately determine whether or not recognition targets are the same target.SOLUTION: An image recognition device 30 that recognizes recognition targets from camera images includes: an acquisition unit 31 that acquires the camera images; an estimation unit 33 that detects the recognition targets from the camera images acquired by the acquisition unit 31 and estimates a class of the detected recognition targets; and a determination unit 34 that determines whether or not the recognition targets estimated by the estimation unit 33 to be of different classes are the same target. When a plurality of recognition targets estimated by the estimation unit 33 to be of different classes from a plurality of camera images captured within a predetermined time exist in a range of a predetermined distance, the determination unit 34 determines that they are the same target.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present disclosure relates to an image recognition device, an image recognition program, and an image recognition method. [Background technology]

[0002] Conventionally, there are image recognition devices that detect recognition targets such as people and vehicles from camera images. Such image recognition devices incorporate various innovations to improve the detection rate of recognition targets. For example, there is an image recognition device that determines that a recognition target recognized in a current frame is the same recognition target if the position of the recognition target is within a predetermined distance from the position of the recognition target recognized in a previous frame. Such an image recognition device is described, for example, in Patent Document 1. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2013-210705 Summary of the Invention [Problem to be solved by the invention]

[0004] In recent years, there have been technologies that use machine learning such as deep learning to generate inference models, which are used to detect objects to be recognized from camera images and estimate the class of the objects to be recognized.

[0005] However, some recognition targets, such as motorcycles, are prone to misclassification depending on the imaging direction and distance to the target. Even if the target is the same, if it is in a different class, the recognition target can be mistakenly determined to be a different object, causing problems.

[0006] The present invention has been made in consideration of the above circumstances, and its main object is to provide an image recognition device, an image recognition program, and an image recognition method that can appropriately determine whether recognition objects are the same object. [Means for solving the problem]

[0007] A first means for solving the above problem is an image recognition device that recognizes a recognition target from a camera image, and includes an acquisition unit that acquires the camera image, an estimation unit that detects a recognition target from the camera image acquired by the acquisition unit and estimates the class of the detected recognition target, and a determination unit that determines whether recognition targets estimated by the estimation unit to be of different classes are the same target.

[0008] As described above, since the determination unit is provided, even if it is estimated that the classes are different, the recognition objects are not uniformly determined to be different targets, and it is possible to appropriately determine whether the recognition objects are the same object or not.

[0009] A second means for solving the above problem is an image recognition program executed by an image recognition device (30, 130) that recognizes a recognition target from a camera image, which causes the image recognition device to execute an acquisition step of acquiring a camera image, an estimation step of detecting a recognition target from the camera image acquired by the acquisition step and estimating the class of the detected recognition target, and a determination step of determining whether or not recognition targets estimated to be of different classes by the estimation step are the same target.

[0010] As described above, since the judgment step is performed, even if it is estimated that the classes are different, the recognition objects are not uniformly judged to be different targets, and it is possible to appropriately judge whether the recognition objects are the same object or not.

[0011] A third means for solving the above problem is an image recognition method implemented by an image recognition device that recognizes a recognition target from a camera image, and includes an acquisition step of acquiring a camera image, an estimation step of detecting a recognition target from the camera image acquired by the acquisition step and estimating a class of the detected recognition target, and a determination step of determining whether recognition targets estimated to be of different classes by the estimation step are the same target.

[0012] As described above, since the method includes a judgment step, even if the recognition objects are estimated to be of different classes, they are not uniformly judged to be different targets, and it is possible to appropriately judge whether the recognition objects are the same object or not. [Brief explanation of the drawings]

[0013] [Figure 1] FIG. 1 is a block diagram showing the configuration of a driving assistance system. [Figure 2] FIG. 2 is a block diagram showing the functions of the image recognition device. [Figure 3] FIG. 10 is a diagram showing an example of a camera image. [Figure 4] FIG. 10 is a diagram for explaining a range of a predetermined distance. [Figure 5] FIG. 10 is a diagram for explaining a method for determining a class. [Figure 6] 10 is a flowchart of an image recognition process. [Figure 7] FIG. 10 is a block diagram showing the configuration of a driving assistance system according to a second embodiment. [Figure 8] FIG. 10 is a block diagram showing the functions of an image recognition device according to a second embodiment. [Figure 9] FIG. 10 is a diagram showing an example of a camera image. [Figure 10] 10 is a flowchart of image recognition processing according to a second embodiment. [Figure 11] 10 is a flowchart of image recognition processing according to a third embodiment. [Figure 12] FIG. 10 is a diagram for explaining a range of a predetermined distance in a modified example. DETAILED DESCRIPTION OF THE INVENTION

[0014] Hereinafter, embodiments of an image recognition device, an image recognition program, and an image recognition method according to the present disclosure will be described in detail with reference to the drawings. Note that, between the embodiments and modifications, the same or corresponding parts in the drawings are designated by the same reference numerals, and their descriptions will not be repeated in principle.

[0015] (First embodiment) 1 shows a driving assistance system 10 to which an image recognition device according to this embodiment is applied. The driving assistance system 10 is provided in a vehicle and performs driving assistance such as automatic driving.

[0016] 1, the driving assistance system 10 includes a monocular camera 20 as an imaging device, an image recognition device 30, and a monitor 40 as a display unit. The image recognition device 30 is connected to the monocular camera 20 and the monitor 40 via wired or wireless communication so as to be able to communicate with each other.

[0017] The monocular camera 20 is a camera that uses an imaging element such as a CCD or CMOS. The monocular camera 20 is disposed, for example, near the top edge of the vehicle's windshield, and captures (images) the surrounding environment including the road ahead of the vehicle. The monocular camera 20 is configured to capture video at a predetermined frame rate. The camera images captured by the monocular camera 20 are output to the image recognition device 30. Although the monocular camera 20 is provided, a compound eye camera may also be provided.

[0018] The image recognition device 30 is primarily composed of a microcomputer equipped with a processing unit 30a such as a CPU and a storage unit 30b including various types of memory. The functions provided by the microcomputer can be provided by software stored in a physical memory device and a computer executing the software, software alone, hardware alone, or a combination thereof. For example, when the microcomputer is provided by electronic circuits, which are hardware, the functions can be provided by digital circuits including multiple logic circuits or analog circuits. For example, the processing unit 30a of the microcomputer executes programs stored in a non-transitory tangible storage medium serving as the storage unit 30b. The programs include, for example, programs that realize the functions shown in FIG. 2. Execution of the programs results in the execution of a method corresponding to the programs. The storage unit 30b is, for example, a non-volatile memory. The programs stored in the storage unit 30b can be downloaded and updated via a communication network such as the Internet, for example, via OTA (Over the Air) or other means.

[0019] The image recognition device 30 has, for example, a function as an image acquisition unit 31, a function as an image processing unit 32, and a function as an estimation unit 33, as shown in FIG.

[0020] The image acquisition unit 31 acquires camera images (or image information relating to camera images, the same applies hereinafter) taken by the monocular camera 20 sequentially from the monocular camera 20.

[0021] Image processing unit 32 converts the camera image acquired by image acquisition unit 31 into an overhead image (bird's-eye view image). The conversion method may be a well-known method, for example, by performing perspective transformation to generate an overhead image of the shooting range of monocular camera 20 as viewed vertically from above. Note that the conversion is not limited to an overhead image, and camera images (two-dimensional images) may also be converted into three-dimensional images. That is, image processing unit 32 performs preprocessing on the camera image to clarify the features of the camera image and clarify the positional relationship in order to perform image recognition.

[0022] The estimation unit 33 detects a recognition target from the camera image acquired by the image acquisition unit 31 and estimates the class of the detected recognition target. The object detection method for detecting a recognition target from the camera image and the object recognition method for estimating the class of the detected recognition target may be well-known methods, such as template matching, a method using a convolutional neural network (CNN), or semantic segmentation. The estimation unit 33 of this embodiment uses an inference model generated based on deep learning to detect a recognition target from the camera image and estimate the class of the detected recognition target.

[0023] In this embodiment, detection and class estimation are performed based on a camera image, but detection and class estimation may also be performed based on an image that has been preprocessed by the image processing unit 32, such as an overhead image, a three-dimensional image, or image data converted from a camera image.

[0024] However, when estimating the class of a recognition target, the estimation unit 33 may make an incorrect class. For example, a motorcycle may have similar characteristics to a pedestrian depending on the shooting angle and the distance from the camera, and may be erroneously recognized as a pedestrian. This will be explained using the example shown in FIG. 3. The left side of FIG. 3 shows an image of a motorcycle in the distance, and the right side shows an image of the motorcycle approaching. It is difficult to distinguish the characteristics of the motorcycle from the image on the left side of FIG. 3, and the motorcycle may be erroneously recognized as a pedestrian. In this way, if the same object is determined to be a different object due to different classes, various problems may arise.

[0025] For example, when an identification graphic (such as a bicycle or pedestrian mark) indicating an estimated class is displayed on a monitor, a problem may occur in which the motorcycle or pedestrian suddenly disappears from the monitor, or the motorcycle and pedestrian suddenly switch places. Furthermore, when measuring the speed of a recognition target by tracking it over time, a problem may occur in which the speed cannot be measured accurately. Therefore, the image recognition device 30 of this embodiment has a function as a determination unit 34 and a function as a class determination unit 35. These functions will be described below.

[0026] The determination unit 34 determines whether or not the recognition targets estimated by the estimation unit 33 to be of different classes are the same target (same object). More specifically, when the estimation unit 33 detects multiple recognition targets of different classes from multiple camera images captured by the monocular camera 20 between the first time and the second time, and these recognition targets exist within a predetermined distance range, the determination unit 34 determines that the multiple recognition targets of different classes existing within the predetermined distance range are the same target.

[0027] The predetermined period from the first time to the second time may be any period, but is preferably a short period. For example, the predetermined period from the first time to the second time is within a range of about 0.5 to 3 seconds, or a period long enough to acquire 10 to 100 frames of camera images. The predetermined period may be changed depending on the speed of the vehicle in which the driving assistance system 10 is installed, or the frame rate of the monocular camera 20.

[0028] Furthermore, when those recognition targets are present within a predetermined distance range, it means that, for example, when viewed from a bird's-eye view, in the camera image acquired at the second time, i.e., the last camera image acquired in the time series, other recognition targets (recognition targets detected before the second time) are included within a predetermined distance range centered on each detected recognition target.

[0029] For example, consider the case where recognition target TA1 is detected from a camera image at time t1, recognition target TA2 is detected from a camera image at time t2, recognition target TA3 is detected from a camera image at time t3, recognition targets TA4 and TB4 are detected from a camera image at time t4, and recognition targets TA5 and TB5 are detected from a camera image at time t5. The positions where each recognition target TA1 to TA5 and TB4 to TB5 is detected are shown in overhead image 50 of FIG. 4. The positions of the recognition targets are indicated by black dots, and ranges of a predetermined distance are indicated by circles. The time series is time t1 → time t2 → time t3 → time t4 → time t5, and the time intervals between the time points are constant.

[0030] In this case, the determination unit 34 determines that the recognition targets TA1 to TA4 that exist within a predetermined distance range RA5 centered on the last detected recognition target TA5 are the same as the recognition target TA5, even though they belong to different classes. Similarly, the determination unit 34 determines that the recognition target TB4 that exists within a predetermined distance range RB5 centered on the last detected recognition target TB5 is the same as the recognition target TB5, even though they belong to different classes. On the other hand, the determination unit 34 determines that the recognition targets TB4 to TB5 are not the same as the recognition target TA5, even though they belong to the same class, because the recognition targets TB4 and TB5 do not exist within the predetermined distance range RA5 for the last detected recognition target TA5. Similarly, the determination unit 34 determines that the recognition targets TA1 to TA5 are not the same as the recognition target TB5, even though they belong to the same class, because the recognition targets TA1 to TA5 do not exist within the predetermined distance range RB5 for the last detected recognition target TB5.

[0031] The determination unit 34 determines that multiple recognition targets existing within a predetermined distance range are a group of recognition targets that are considered to be the same target, and if there are multiple such groups, identifies each group. In the example shown in Fig. 4, the determination unit 34 identifies a group of multiple recognition targets TA1 to TA5 of different classes that exist within a predetermined distance range RA5, and a group of multiple recognition targets TB4 to TB5 of different classes that exist within a predetermined distance range RB5.

[0032] When the determination unit 34 determines that a plurality of recognition targets of different classes are the same target, the class determination unit 35 determines the class of the plurality of recognition targets of different classes by majority vote.

[0033] Here, a specific example will be described. As shown in Fig. 5, a recognition object TA1 estimated to be of the class "pedestrian" is detected from a camera image at time t1, a recognition object TA2 estimated to be of the class "two-wheeled vehicle" is detected from a camera image at time t2, a recognition object TA3 estimated to be of the class "pedestrian" is detected from a camera image at time t3, a recognition object TA4 estimated to be of the class "two-wheeled vehicle" is detected from a camera image at time t4, and a recognition object TA5 estimated to be of the class "two-wheeled vehicle" is detected from a camera image at time t5. The time series is time t1 → time t2 → time t3 → time t4 → time t5, and the time intervals between the time points are constant.

[0034] Here, when the determination unit 34 determines that the recognition targets TA1 to TA5 are the same target, the class determination unit 35 determines, by majority vote, that the class of the recognition targets TA1 to TA5 is "two-wheeled vehicle."

[0035] Next, the flow of image recognition processing relating to detection of a recognition target and class estimation will be described with reference to Fig. 6. The image recognition processing is performed by the image recognition device 30 at predetermined intervals.

[0036] When the image recognition process is started, first, the image acquisition unit 31 of the image recognition device 30 sequentially acquires camera images taken by the monocular camera 20 from the monocular camera 20 (step S101). Step S101 corresponds to the acquisition step.

[0037] Next, the image processing unit 32 of the image recognition device 30 performs preprocessing on the camera image acquired by the image acquisition unit 31 (step S102). In step S102, for example, an overhead image is generated.

[0038] Next, the estimation unit 33 of the image recognition device 30 detects a recognition target from the camera image (or an image obtained by preprocessing the camera image) and estimates the class of the detected recognition target (step S103). Step S103 corresponds to the estimation step.

[0039] Then, the determination unit 34 of the image recognition device 30 determines whether the recognition targets estimated by the estimation unit 33 to be of different classes are the same target (step S104). The determination method is as described above. Step S104 corresponds to the determination step.

[0040] If the result of this determination is positive, that is, if it is estimated that the classes are different but it is determined that the recognition targets are the same target, the class determination unit 35 of the image recognition device 30 determines the class of the multiple recognition targets that are present within a predetermined distance and are of different classes by majority vote (step S105). Note that, as shown in FIG. 4, if there are multiple groups of recognition targets that are considered to be the same target, the class determination unit 35 determines a class for each group. In FIG. 4, the class determination unit 35 determines a class for the group of recognition targets TA1 to TA5 and the group of recognition targets TB4 to TB5. Step S105 corresponds to the class determination step.

[0041] On the other hand, if the determination result in step S104 is negative, that is, if the recognition targets are different, the class determination unit 35 determines the class estimated by the estimation unit 33 as the class of each recognition target (step S106).

[0042] The image recognition device 30 outputs the position of the detected recognition target and the recognition information relating to the class of each recognition target determined in step S105 or step S106 to the outside (step S107), and ends the image recognition process.

[0043] When this recognition information is input, the monitor 40 displays the position of the recognition target and the class of each recognition target on the screen.

[0044] According to the above embodiment, the following effects are achieved.

[0045] The system is provided with a determination unit 34 that determines whether or not the recognition targets estimated by the estimation unit 33 to be of different classes are the same target. This prevents the recognition targets from being uniformly determined to be different targets (objects) even when they are estimated to be of different classes, and allows appropriate determination as to whether or not the recognition targets are the same target.

[0046] Even if multiple recognition targets of different classes are detected from multiple camera images captured within a predetermined time, if the distance between those recognition targets is extremely close, there is a high possibility that they are the same target, and the class estimation is likely to be incorrect. Therefore, the determination unit 34 is configured to determine that multiple recognition targets estimated to be of different classes are the same target if they exist within a predetermined distance range.

[0047] More specifically, when multiple recognition targets of different classes are detected from multiple camera images captured by monocular camera 20 between the first time and the second time, and these recognition targets exist within a predetermined distance, determination unit 34 determines that the multiple recognition targets of different classes existing within the predetermined distance are the same target. This makes it possible to appropriately determine whether the recognition targets are the same target even when they are estimated to be of different classes.

[0048] In this embodiment, if a recognition target detected from the first time to just before the second time exists within a range of a predetermined distance centered on the recognition target detected at the second time, the judgment unit 34 judges that the recognition targets are the same target even if they are in different classes.

[0049] Incidentally, it is believed that the appropriate distance range in which the same object exists differs depending on the class of the recognition object. For example, since the moving speeds of pedestrians and vehicles are different, the appropriate distance range in which the same object exists is thought to differ. Therefore, the predetermined distance range in which multiple recognition objects of different classes are determined to be the same object is made different depending on the class. This makes it possible to improve the accuracy of determining whether the objects are the same or not, compared to when the predetermined distance range is set uniformly.

[0050] When multiple recognition targets of different classes are determined to be the same target, the class determination unit 35 determines the class based on detection information from multiple camera images acquired in time series. Specifically, when the determination unit 34 determines that multiple recognition targets of different classes are the same target, the class determination unit 35 determines the classes of the multiple recognition targets determined to be the same target by majority vote. This makes it possible to correctly re-determine the class of the recognition targets.

[0051] (Second embodiment) A second embodiment will be described, which is a partial modification of the configuration of the first embodiment. Note that the same components as those in the first embodiment are given the same reference numerals and their description will be omitted.

[0052] As shown in FIG. 7, in the second embodiment, the vehicle has multiple (four) cameras 121 to 124. Specifically, the vehicle is equipped with a front camera 121 whose imaging range is in front of the vehicle, a right side camera 122 whose imaging range is on the right side of the vehicle, a left side camera 123 whose imaging range is on the left side of the vehicle, and a rear camera 124 whose imaging range is on the rear of the vehicle. The front camera 121 and the rear camera 124 have a horizontal field of view of approximately 130 degrees with the fore-and-aft direction of the vehicle as the center (optical axis). The right side camera 122 and the left side camera 123 have a horizontal field of view of approximately 130 degrees with the direction orthogonal to the fore-and-aft direction of the vehicle as the center (optical axis). Note that the number, imaging range, imaging direction (direction of the optical axis), field of view, etc. of each of the cameras 121 to 124 may be changed as desired. However, in the second embodiment, the description will be made on the assumption that the imaging range of the front camera 121 partially overlaps with the imaging ranges of the right side camera 122 and left side camera 123, and that the imaging range of the rear camera 124 partially overlaps with the imaging ranges of the right side camera 122 and left side camera 123. Note that each of the cameras 121 to 124 may capture either still images or moving images.

[0053] The image recognition device 130 of the second embodiment, like the first embodiment, includes a processing unit 130a and a memory unit 130b, and the processing unit 130a realizes various functions shown in FIG. 8 by executing a program stored in the memory unit 130b.

[0054] As shown in FIG. 8, the image recognition device 130 of the second embodiment has a function as an image acquisition unit 131, a function as an image processing unit 132, a function as an estimation unit 133, a function as a judgment unit 134, and a function as a class determination unit 135.

[0055] The image acquisition unit 131 acquires camera images taken by the cameras 121 to 124 (or image information relating to each camera image, the same applies below).

[0056] The image processing unit 132 converts the camera images of the cameras 121 to 124 acquired by the image acquisition unit 131 into overhead images (bird's-eye images). The conversion method is the same as in the first embodiment.

[0057] The estimation unit 133 detects a recognition target for each camera image of the cameras 121 to 124 acquired by the image acquisition unit 131, and estimates the class of the detected recognition target. The object detection method for detecting a recognition target from a camera image and the object recognition method for estimating the class of the detected recognition target are the same as those in the first embodiment. Note that, when estimating the class of the recognition target, the estimation unit 133 in the second embodiment calculates a confidence level (reliability) of the estimated class. The confidence level tends to increase as the feature amount of the recognition target increases.

[0058] In the second embodiment, the imaging ranges of the cameras 121 to 124 partially overlap, but the class of the recognition target is estimated for each camera image. Even for the same target, there are directions in which the characteristics of the recognition target are more likely to appear and directions in which they are less likely to appear depending on the imaging direction, so different classes may be erroneously selected for the same target. This will be described in detail with reference to FIG. 9. The left side of FIG. 9 is a camera image taken by the front camera 121, and the right side is a camera image taken by the left side camera 123. As shown in FIG. 9, the feature amount of a motorcycle increases or decreases depending on the imaging direction. In FIG. 9, the camera image taken by the left side camera 123 (right image) has more features of the motorcycle than the camera image taken by the front camera 121 (left image), while the camera image taken by the front camera 121 has fewer features of the motorcycle. In this case, even for the same target, the class of the recognition target may be estimated as "pedestrian" from the camera image taken by the front camera 121, and the class of the recognition target may be estimated as "two-wheeled vehicle" from the camera image taken by the left side camera 123. If the same object is determined to be a different object because of the different classes, various problems may arise.

[0059] For example, when displaying identification figures (such as bicycle or pedestrian marks) indicating classes on the monitor 40, even if they are the same object, a motorcycle and a pedestrian may be displayed simultaneously (double) at the same location, causing confusion. Therefore, in the second embodiment, the determination unit 134 and the class determination unit 135 are configured as follows.

[0060] When a determination unit 134 detects multiple recognition targets of different classes from multiple camera images captured by multiple cameras 121 to 124 with different shooting ranges at the same time, and these recognition targets exist in the same position, the determination unit 134 determines that the multiple recognition targets of different classes are the same target.

[0061] Whether or not the recognition targets exist in the same position is determined, for example, from an overhead image generated by image processing unit 132. Note that although "same position" is used, it actually means the same position after taking into consideration various errors such as detection errors during recognition, installation errors in cameras 121 to 124, and errors due to distortion by the camera lenses. In other words, when multiple recognition targets of different classes exist within a predetermined distance range that takes these errors into consideration, determination unit 134 determines that these recognition targets are the same target.

[0062] When the determination unit 134 determines that multiple recognition targets of different classes are the same target, the class determination unit 135 determines the class of the multiple recognition targets of different classes based on the confidence level. For example, if the confidence level of the class "pedestrian" of the recognition target detected based on the camera image of the front camera 121 is 60%, while the confidence level of the class "two-wheeled vehicle" of the recognition target detected based on the camera image of the left side camera 123 is 90%, the class determination unit 135 determines the class "two-wheeled vehicle" with the higher confidence level as the class of the recognition target.

[0063] Next, the flow of image recognition processing related to detection of a recognition target and class estimation will be described with reference to Fig. 10. The image recognition processing is performed by the image recognition device 130 at predetermined intervals.

[0064] When the image recognition process is started, first, the image acquisition unit 131 of the image recognition device 130 acquires the camera images taken by the cameras 121 to 124 (step S201).

[0065] Next, the image processing unit 132 of the image recognition device 130 performs preprocessing on the multiple camera images acquired by the image acquisition unit 131, and generates an overhead image from the multiple camera images (step S202).

[0066] Next, the estimation unit 133 of the image recognition device 130 detects a recognition target for each of the multiple camera images and estimates the class of each detected recognition target (step S203). At that time, the estimation unit 133 also calculates the confidence level for each class.

[0067] Then, the determination unit 134 of the image recognition device 130 determines whether the recognition targets estimated by the estimation unit 133 to be of different classes are the same target (step S204). In step S204, when multiple recognition targets of different classes are detected from multiple camera images captured by multiple cameras 121 to 124 with different shooting ranges at the same time, and these recognition targets exist in the same position, the determination unit 134 determines that the multiple recognition targets of different classes are the same target.

[0068] If the result of this determination is positive, that is, if it is estimated that the classes are different but it is determined that the recognition targets are the same target, the class determination unit 135 of the image recognition device 130 determines the class of the multiple recognition targets of different classes based on the confidence level (step S205). Note that, as in the first embodiment, if there are multiple groups of recognition targets that are considered to be the same target, the class determination unit 135 determines a class for each group.

[0069] On the other hand, if the determination result in step S204 is negative, the class determination unit 135 determines the class estimated by the estimation unit 133 as the class of each recognition target (step S206).

[0070] The image recognition device 130 outputs the position of the detected recognition target and the recognition information relating to the class of each recognition target determined in step S205 or step S206 to the outside (step S207), and ends the image recognition process.

[0071] When this recognition information is input, the monitor 40 displays the position of the recognition target and the class of each recognition target on the screen.

[0072] According to the above embodiment, the following effects are achieved.

[0073] If the shooting range or shooting angle is different, the shape of the object and the feature amount will be different even if it is the same object, so the estimation unit 133 may make an incorrect class estimation. For this reason, even when multiple recognition objects of different classes are detected from multiple camera images captured by multiple cameras 121 to 124 with different shooting ranges at the same time, if these recognition objects exist in the same position, the determination unit 134 determines that they are the same object. This makes it possible to appropriately determine whether the recognition objects are the same object even if they are estimated to be of different classes.

[0074] Furthermore, when it is determined that multiple recognition targets of different classes are the same target, the class determination unit 135 determines the class of the recognition targets based on detection information from multiple camera images captured by the multiple cameras 121 to 124. In other words, when it is estimated that the classes are different but it is determined that the recognition targets are the same target, the class determination unit 135 of the image recognition device 130 determines the class of the multiple recognition targets of different classes based on the confidence level. This makes it possible to determine a more appropriate class.

[0075] (Third embodiment) A third embodiment will be described, which is a partial modification of the configuration of the first embodiment. Note that the same components as those in the first embodiment are given the same reference numerals and the description thereof will be omitted.

[0076] Combinations of classes that are easily confused are known empirically. For example, it is easy to mistakenly estimate the class between "motorcycle" and "pedestrian." Similarly, depending on the shooting angle and shooting distance, it is easy to mistakenly estimate the class between "bus," "truck," and "passenger car." Similarly, it is easy to mistakenly estimate the class between "stroller" and "pedestrian," between "motorcycle" and "stroller," between "motorcycle," "pedestrian," and "stroller," and between "stroller" and "child vehicle."

[0077] In the third embodiment, whether or not the objects are the same is determined taking into consideration combinations of classes that are likely to result in a class estimation error. The flow of image recognition processing in the third embodiment will be described below with reference to Fig. 11. In the image recognition processing in the third embodiment, the processing from steps S101 to S103 is the same as in the first embodiment.

[0078] The determination unit 34 of the image recognition device 30 determines whether or not multiple recognition targets of different classes exist within a predetermined distance range (step S301), similarly to step S104 in the first embodiment. Note that, similarly to the first embodiment, the determination unit 34 determines multiple recognition targets existing within a predetermined distance range as a group of recognition targets that are considered to be the same target, and if multiple such groups exist, identifies each group.

[0079] If the result of this determination is positive, the determination unit 34 determines whether the combination of classes existing within the range of the predetermined distance is a predetermined combination (step S302). Note that if there are multiple groups of recognition targets determined to be the same target, the determination unit 34 determines for each group whether the combination of classes of the recognition targets constituting the group is a predetermined combination.

[0080] As mentioned above, the predetermined combinations are combinations of classes that are likely to result in a mistaken class selection. For example, the combination of "motorcycle" and "pedestrian," the combination of "bus," "truck," and "passenger car," the combination of "stroller" and "pedestrian," the combination of "motorcycle" and "stroller," the combination of "motorcycle," "pedestrian," and "stroller," and the combination of "stroller" and "children's vehicle." Other combinations may be possible, and the types and number of classes in the combinations can be changed as desired.

[0081] If the determination result is positive, the class determination unit 35 of the image recognition device 30 determines the class of the multiple recognition targets of different classes by majority vote, similar to step S105 in the first embodiment. Note that, as shown in Fig. 4, if there are multiple groups of recognition targets that are considered to be the same target, the class determination unit 35 determines a class for each group.

[0082] On the other hand, if the determination result at step S301 or step S302 is negative, the class determination unit 35 determines the class estimated by the estimation unit 33 as the class of each recognition target (step S106).

[0083] The image recognition device 30 outputs the position of the detected recognition target and the recognition information relating to the class of each recognition target determined in step S105 or step S106 to the outside (step S107), and ends the image recognition process.

[0084] According to the above embodiment, the following effects are achieved.

[0085] As mentioned above, class combinations that are likely to result in incorrect class selection are empirically known. Therefore, when multiple recognition targets estimated to be of different classes exist within a predetermined distance, and the class combination of each recognition target existing within that distance is a predetermined class combination, the determination unit 34 determines that the targets are the same target. This enables more accurate determination.

[0086] It is to be noted that the third embodiment may be combined with the second embodiment. For example, in the image recognition processing of the second embodiment, if the determination result of step S204 is positive, step S305 may be performed.

[0087] In the third embodiment, step S302 and step S301 may be interchanged, and if a predetermined combination exists, it may be determined whether or not the recognition target that forms the combination exists within a predetermined distance range.

[0088] Furthermore, in the third embodiment, the class is determined by majority vote, but similar to the second embodiment, the class may be determined based on the degree of confidence.

[0089] (Variation) A modified example in which the configuration of the above embodiment is partially changed will be described below.

[0090] In the above embodiments, when multiple recognition targets of different classes are determined to be the same object, the class determination unit 35, 135 may determine the classes of the multiple recognition targets using an algorithm different from the algorithm used by the estimation unit 33, 133 when detecting and class-estimating the recognition targets. For example, if the estimation unit 33, 133 uses a convolutional neural network (CNN), the class determination unit 35, 135 may determine the class using semantic segmentation. The class determination unit 35, 135 may also determine the class using a combination of methods. For example, the class determination unit 35, 135 may weight the confidence levels, calculate a sum of the confidence levels for each class of recognition targets determined to be the same object, and compare the sums to determine the class. Alternatively, the confidence levels of each class calculated using the convolutional neural network may be compared with the confidence levels of each class calculated using semantic segmentation, and the class with the highest confidence level may be determined as the class of the recognition target.

[0091] The first embodiment and the second embodiment may be combined. For example, the estimation unit 33 in the first embodiment may calculate a confidence level for the class of each recognition target detected from multiple camera images acquired in time series by the monocular camera 20, similar to the estimation unit 133 in the second embodiment. Then, the class determination unit 35 in the first embodiment may determine the class of multiple different recognition targets based on the confidence level, similar to the class determination unit 135 in the second embodiment. For example, the average confidence level may be calculated for each estimated class, and the class with the highest average value may be determined as the class of the recognition target.

[0092] Specifically, a case will be described in which a recognition target TA1, whose class is "pedestrian" and whose confidence level is "60%," is detected from a camera image at time t1; a recognition target TA2, whose class is "motorcycle" and whose confidence level is "70%," is detected from a camera image at time t2; a recognition target TA3, whose class is "pedestrian" and whose confidence level is "70%," is detected from a camera image at time t3; a recognition target TA4, whose class is "motorcycle" and whose confidence level is "80%," is detected from a camera image at time t4; and a recognition target TA5, whose class is "motorcycle" and whose confidence level is "90%," is detected from a camera image at time t5. In this case, the average confidence level for the class "pedestrian" is "65%," and the average confidence level for the class "motorcycle" is "80%." Therefore, the class determination unit 35 determines "motorcycle" as the correct class. The class with the highest confidence level (in the above example, "motorcycle" with a confidence level of "90%) may be determined as the correct class.

[0093] Furthermore, the class determination unit 135 of the second embodiment may determine the class based on detection information from a plurality of camera images acquired in time series by the cameras 121 to 124. A specific example will be described. A case will be described in which a recognition object TA1 whose class is determined to be "pedestrian" is detected from the camera image of the front camera 121 at time point t1, a recognition object TA2 whose class is determined to be "two-wheeled vehicle" is detected from the camera image of the front camera 121 at time point t2, a recognition object TA3 whose class is determined to be "pedestrian" is detected from the camera image of the front camera 121 at time point t3, a recognition object TB1 whose class is determined to be "two-wheeled vehicle" is detected from the camera image of the left side camera 123 at time point t1, a recognition object TB2 whose class is determined to be "two-wheeled vehicle" is detected from the camera image of the left side camera 123 at time point t2, and a recognition object TB3 whose class is determined to be "two-wheeled vehicle" is detected from the camera image of the left side camera 123 at time point t3. In this case, the class determination unit 135 of the second embodiment may determine the class by majority vote, as in the first embodiment. In the above example, "two-wheeled vehicle" may be determined as the class. Note that, as in the second embodiment, the class may be determined based on the confidence level. For example, the class may be determined based on the average value of the confidence level for each class, or the class may be determined from the one with the highest confidence level.

[0094] The first embodiment may include the class determination unit 135 of the second embodiment. That is, a confidence level of a class may be set, and when it is determined that the objects are the same, the class determination unit 135 may determine a class based on the confidence level.

[0095] The determination unit 34, 134 in the above embodiments may determine that a recognition target of a different class exists within a predetermined distance range when another recognition target is included within a predetermined distance range centered on a certain recognition target and the certain recognition target is included within a predetermined distance range centered on the other recognition target.

[0096] For example, a case will be described in which the positions of recognition targets TA1 to TA3 of different classes are detected in an overhead image as shown in Fig. 12. In this case, if recognition targets TA2 and TA3 are included in a range RA1 of a predetermined distance centered on recognition target TA1, and recognition targets TA1 and TA3 are included in a range RA2 of a predetermined distance centered on recognition target TA2, and recognition targets TA1 and TA2 are included in a range RA3 of a predetermined distance centered on recognition target TA3, the determination unit 34 determines that recognition targets TA1 to TA3 of different classes are present within the range of the predetermined distance.

[0097] The determination unit 34 in the first embodiment sets a range of a predetermined distance centered on each recognition target detected at the second time (last), but the range of a predetermined distance may also be set centered on any of the recognition targets detected from the first time to just before the second time.

[0098] In the above embodiment, the image recognition device 30 inputs the processing results to the monitor 40, but the processing results may also be input to an external device other than the monitor 40. For example, the processing results may be input to a vehicle control device that performs automatic driving of the vehicle or performs driving assistance.

[0099] The controller and methods described herein may be implemented by a special-purpose computer configured with a processor and memory programmed to perform one or more functions embodied in a computer program. Alternatively, the controller and methods described herein may be implemented by a special-purpose computer configured with a processor configured with one or more dedicated hardware logic circuits. Alternatively, the controller and methods described herein may be implemented by one or more special-purpose computers configured with a processor and memory programmed to perform one or more functions in combination with a processor configured with one or more hardware logic circuits. Furthermore, the computer program may be stored as instructions executed by a computer on a computer-readable non-transitory storage medium.

[0100] The following describes technical ideas that can be derived from the above-described embodiment and modifications.

[0101] [Configuration 1] In an image recognition device (30, 130) that recognizes a recognition target from a camera image, an acquisition unit (31, 131) for acquiring camera images; an estimation unit (33, 133) that detects a recognition target from the camera image acquired by the acquisition unit and estimates a class of the detected recognition target; and a determination unit (34, 134) that determines whether or not the recognition targets estimated by the estimation unit to be of different classes are the same target.

[0102] [Configuration 2] The image recognition device according to configuration 1, wherein the determination unit determines that multiple recognition targets estimated by the estimation unit to be of different classes from multiple camera images captured within a predetermined time are the same target when they exist within a predetermined distance range.

[0103] [Configuration 3] 3. The image recognition device according to configuration 1 or 2, wherein when the estimation unit detects multiple recognition targets of different classes from multiple camera images captured by a predetermined camera between a first time and a second time, and the multiple recognition targets exist within a predetermined distance range, the determination unit determines that the multiple recognition targets of different classes existing within the predetermined distance range are the same target.

[0104] [Configuration 4] The image recognition device according to any one of configurations 1 to 3, wherein when the estimation unit detects multiple recognition targets of different classes from multiple camera images captured by multiple cameras with different shooting ranges at the same time, and the multiple recognition targets exist in the same position, the determination unit determines that the multiple recognition targets of different classes are the same target.

[0105] [Configuration 5] 5. The image recognition device according to any one of configurations 2 to 4, wherein the range of the predetermined distance varies depending on the class estimated by the estimation unit.

[0106] [Configuration 6] The image recognition device according to any one of configurations 1 to 5, further comprising: a class determination unit (35) that, when the determination unit determines that the plurality of recognition targets of different classes are the same target, determines the classes of the plurality of recognition targets of different classes based on detection information from a plurality of camera images acquired in time series.

[0107] [Configuration 7] The image recognition device according to any one of configurations 1 to 6, further comprising: a class determination unit (35) that, when the determination unit determines that the plurality of recognition targets of different classes are the same target, determines the classes of the plurality of recognition targets of different classes based on detection information from a plurality of camera images captured by a plurality of cameras.

[0108] [Configuration 8] The image recognition device according to any one of configurations 1 to 7, further comprising a class determination unit (35) that, when the determination unit determines that the plurality of recognition targets of different classes are the same target, determines the class of the plurality of recognition targets of different classes by majority vote.

[0109] [Configuration 9] the estimation unit is configured to set a confidence level of class estimation for each recognition target when detecting the recognition target from the camera image and estimating the class; The image recognition device according to any one of configurations 1 to 7, further comprising a class determination unit (135) that, when the determination unit determines that the plurality of recognition targets of different classes are the same object, determines the class of the plurality of recognition targets of different classes based on a degree of confidence.

[0110] [Configuration 10] The image recognition device according to any one of configurations 1 to 7, further comprising: a class determination unit that, when the determination unit determines that the plurality of recognition targets of different classes are the same target, determines the classes of the plurality of recognition targets of different classes using an algorithm different from the algorithm used by the estimation unit when detecting and estimating the class of the recognition targets.

[0111] [Configuration 11] The image recognition device according to any one of configurations 2 to 10, wherein the determination unit determines that a plurality of recognition targets estimated to be of different classes are the same target when they exist within a range of a predetermined distance and the combination of classes of the recognition targets existing within the range is a predetermined combination of classes.

[0112] [Configuration 12] An image recognition program executed by an image recognition device (30, 130) that recognizes a recognition target from a camera image, The image recognition device includes: an acquisition step of acquiring a camera image; an estimation step of detecting a recognition target from the camera image acquired by the acquisition step and estimating a class of the detected recognition target; a determination step of determining whether or not the recognition targets estimated to be of different classes by the estimation step are the same target.

[0113] [Configuration 13] An image recognition method implemented by an image recognition device that recognizes a recognition target from a camera image, an acquisition step of acquiring a camera image; an estimation step of detecting a recognition target from the camera image acquired by the acquisition step and estimating a class of the detected recognition target; a determining step of determining whether or not the recognition targets estimated to be of different classes by the estimating step are the same target. [Explanation of symbols]

[0114] 10...driving assistance system, 20...monocular camera, 30,130...image recognition device, 31,131...image acquisition unit, 32,132...image processing unit, 33,133...estimation unit, 34,134...judgment unit, 35,135...class determination unit, 40...monitor, 121...front camera, 122...right side camera, 123...left side camera, 124...rear camera.

Claims

1. In an image recognition device (30, 130) that recognizes a recognition target from a camera image, an acquisition unit (31, 131) that acquires a camera image; an estimation unit (33, 133) that detects a recognition target from the camera image acquired by the acquisition unit and estimates a class of the detected recognition target; and a determination unit (34, 134) that determines whether or not the recognition targets estimated by the estimation unit to be of different classes are the same target.

2. 2. The image recognition device according to claim 1, wherein the determination unit determines that multiple recognition targets estimated by the estimation unit to be of different classes from multiple camera images captured within a predetermined time are the same target when they exist within a predetermined distance range.

3. 2. The image recognition device according to claim 1, wherein when the estimation unit detects multiple recognition targets of different classes from multiple camera images captured by a predetermined camera between a first time and a second time, and the multiple recognition targets exist within a predetermined distance range, the determination unit determines that the multiple recognition targets of different classes existing within the predetermined distance range are the same target.

4. The image recognition device according to claim 3 , wherein the range of the predetermined distance varies depending on the class estimated by the estimation unit.

5. 2. The image recognition device according to claim 1, wherein when the estimation unit detects multiple recognition targets of different classes from multiple camera images captured by multiple cameras with different shooting ranges at the same time, and the multiple recognition targets exist in the same position, the determination unit determines that the multiple recognition targets of different classes are the same target.

6. The image recognition device according to any one of claims 1 to 5, further comprising: a class determination unit (35) that, when the determination unit determines that the plurality of recognition targets of different classes are the same target, determines the classes of the plurality of recognition targets of different classes based on detection information from a plurality of camera images acquired in time series.

7. The image recognition device according to any one of claims 1 to 5, further comprising: a class determination unit (135) that determines the classes of the plurality of recognition targets of different classes based on detection information from a plurality of camera images captured by a plurality of cameras when the determination unit determines that the plurality of recognition targets of different classes are the same target.

8. The image recognition device according to any one of claims 1 to 5, further comprising a class determination unit (35) that determines the classes of the plurality of recognition targets of different classes by majority vote when the determination unit determines that the plurality of recognition targets of different classes are the same target.

9. the estimation unit is configured to set a confidence level of class estimation for each recognition target when detecting the recognition target from the camera image and estimating the class; The image recognition device according to any one of claims 1 to 5, further comprising: a class determination unit (135) that determines the classes of the plurality of recognition targets of different classes based on a degree of confidence when the determination unit determines that the plurality of recognition targets of different classes are the same target.

10. The image recognition device according to any one of claims 1 to 5, further comprising a class determination unit that, when the determination unit determines that the plurality of recognition targets of different classes are the same target, determines the classes of the plurality of recognition targets of different classes using an algorithm different from the algorithm used by the estimation unit when detecting and estimating the class of the recognition target.

11. The image recognition device according to any one of claims 1 to 5, wherein the determination unit determines that a plurality of recognition targets estimated to be of different classes are the same target when they exist within a range of a predetermined distance and when the combination of classes of each recognition target existing within the range is a predetermined combination of classes.

12. An image recognition program executed by an image recognition device (30, 130) that recognizes a recognition target from a camera image, The image recognition device includes: an acquisition step of acquiring a camera image; an estimation step of detecting a recognition target from the camera image acquired by the acquisition step and estimating a class of the detected recognition target; a determination step of determining whether or not the recognition targets estimated to be of different classes by the estimation step are the same target.

13. An image recognition method implemented by an image recognition device that recognizes a recognition target from a camera image, an acquisition step of acquiring a camera image; an estimation step of detecting a recognition target from the camera image acquired by the acquisition step and estimating a class of the detected recognition target; a determining step of determining whether or not the recognition targets estimated to be of different classes by the estimating step are the same target.

Images