Recognition processing device and program

Abstract

To detect an object more appropriately in image recognition processing.SOLUTION: A recognition processing device 10 comprises: an image acquisition part 12 for acquiring photographed image; a recognition processing part 24 for extracting candidate areas, where a recognition score indicating the probability that an object is included therein is equal to or higher than a reference value, among detection areas set in the photographed image; a determination processing part 26 which determines that, among the plurality of candidate areas located close to each other within a predetermined distance among the plurality of candidate areas extracted in the photographed image, the candidate area that has a relatively high recognition score includes the object and the candidate area that has a relatively low recognition score does not include the object; and an output control part 16 for outputting object information about the object detected in the candidate area for which it is determined that the object is included by the determination processing part 26.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a recognition processing device and a program. [Background technology]

[0002] There is a known technology that uses image recognition processing such as pattern matching to detect objects such as pedestrians from images captured around a vehicle. A technology has been proposed that detects with high accuracy the range in which an object actually exists when the ground on which the object exists is wet and the reflected image of the object is visible (for example, see Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2022-15696 A Summary of the Invention [Problem to be solved by the invention]

[0004] When the reflected image seen from the ground or other object includes the entire object, the reflected image may be erroneously detected as the object.

[0005] The present invention has been made in consideration of the above circumstances, and has an object to provide a technique for more appropriately detecting an object in image recognition processing. [Means for solving the problem]

[0006] A recognition processing device of one embodiment of the present invention comprises an image acquisition unit that acquires an image captured by the image; a recognition processing unit that extracts candidate areas from among the detection areas set in the image captured by ...

[0007] Another aspect of the present invention is a program configured to cause a processor to execute the following functions: acquiring a captured image, extracting candidate areas having a recognition score indicating a possibility that an object is included therein equal to or greater than a reference value from among detection areas set in the captured image, identifying a plurality of candidate areas that are adjacent to each other within a predetermined distance from among the candidate areas extracted in the captured image, and determining that the candidate area having a relatively large recognition score includes an object among the plurality of candidate areas and that the candidate area having a relatively small recognition score does not include an object if a difference in the recognition scores of the plurality of candidate areas is equal to or greater than a predetermined value, and outputting object information related to the object detected in the candidate area determined to include the object. Effect of the Invention

[0008] According to an embodiment of the present invention, a technique for more appropriately detecting an object in image recognition processing can be provided. [Brief description of the drawings]

[0009] [Figure 1] 1 is a block diagram illustrating a functional configuration of a recognition processing device according to a first embodiment. [Diagram 2] FIG. 2 is a diagram illustrating an example of a captured image including a target object. [Diagram 3]FIG. 10 is a diagram illustrating an example of a candidate region extracted from a captured image. [Figure 4] 5 is a flowchart showing an example of the flow of a recognition processing method according to the first embodiment. [Diagram 5] FIG. 11 is a block diagram illustrating a functional configuration of a recognition processing device according to a second embodiment. [Figure 6] FIG. 2 is a diagram illustrating an example of a candidate region extracted in a first frame of a captured video. [Figure 7] 13 is a diagram showing an example of a candidate region extracted in a second frame of a captured video. FIG. [Figure 8] 10 is a flowchart showing an example of the flow of a recognition processing method according to a second embodiment. [Figure 9] FIG. 11 is a block diagram illustrating a functional configuration of a recognition processing device according to a third embodiment. [Figure 10] 4 is a diagram illustrating an example of a first detection area detected in a captured image. FIG. [Figure 11] 10 is a diagram illustrating an example of a second detection area detected in a captured image. FIG. [Figure 12] 13 is a flowchart showing an example of the flow of a recognition processing method according to the third embodiment. [Figure 13] FIG. 13 is a block diagram illustrating a functional configuration of a recognition processing device according to a fourth embodiment. [Figure 14] 3A and 3B are diagrams illustrating an example of a first detection area and a second detection area detected in a captured image. [Figure 15] 13 is a flowchart showing an example of the flow of a recognition processing method according to a fourth embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. Specific numerical values ​​and the like shown in the embodiment are merely examples for facilitating understanding of the invention, and do not limit the present invention unless otherwise specified. In the drawings, elements that are not directly related to the present invention are omitted.

[0011] (First embodiment) 1 is a block diagram showing a schematic functional configuration of a recognition processing device 10 according to a first embodiment. The recognition processing device 10 includes an image acquisition unit 12, an object detection unit 14, and an output control unit 16. The recognition processing device 10 acquires captured images that may include objects such as pedestrians present in the surrounding area, and detects the objects included in the captured images.

[0012] In this embodiment, the recognition processing device 10 is installed on a smart pole. The smart pole is installed on a street, for example, and includes an antenna and a communication device for providing wireless communication functions, lighting equipment for illuminating the street, and a camera for photographing vehicles and pedestrians passing on the road. The recognition processing device 10 is fixed at a predetermined location. The recognition processing device 10 may be mounted on a moving object, or on an air vehicle such as a vehicle or a drone.

[0013] The recognition processing device 10 can detect any object. In the embodiment of the present invention, the object is a person such as a pedestrian.

[0014] Each functional block shown in this embodiment can be realized, for example, by cooperation of hardware and software. The hardware of the recognition processing device 10 is realized by elements and mechanical devices such as a processor such as a central processing unit (CPU) or a graphics processing unit (GPU) of a computer, and memories such as a read only memory (ROM) or a random access memory (RAM). The software of the recognition processing device 10 is realized by a computer program or the like.

[0015] The image acquisition unit 12 acquires an image captured by the camera 20 (also referred to as a captured image). The camera 20 is provided on the smart pole and captures an image of the surroundings of the smart pole. The camera 20 is provided, for example, above the smart pole and captures an image with an angle of view looking down on the ground on which the smart pole is installed. The camera 20 captures visible light to generate a color image or a monochrome image. The camera 20 may be an infrared camera, and may capture infrared light to generate a thermal image. The image captured by the camera 20 is a moving image, for example, at 30 frames per second or 60 frames per second.

[0016] The object detection unit 14 detects an object from the video acquired by the video acquisition unit 12. In other words, the object detection unit 14 detects an area including an object in the video acquired by the video acquisition unit 12. The object detection unit 14 includes a recognition processing unit 24 and a determination processing unit 26.

[0017] The recognition processing unit 24 scans the detection window for each frame of the image acquired by the image acquisition unit 12 while referring to one or more detection models for detecting the object, and calculates a recognition score indicating the possibility that the object is included in the detection window. The recognition processing unit 24 calculates the recognition score using a detection model that is machine-learned from the entire image of the object. The recognition score is calculated, for example, in the range of 0.0 to 1.0, and the recognition score becomes a larger value (i.e., a value closer to 1.0) as the possibility that the object is included in the image in the detection window increases, and the recognition score becomes a smaller value (i.e., a value closer to 0.0) as the possibility that a person is included decreases. If the recognition score is equal to or greater than a predetermined value, such as 0.7, the recognition processing unit 24 extracts the detection window as a candidate region that is likely to include the object.

[0018] The model used for machine learning may include an input corresponding to the image size (number of pixels) of the input image, an output that outputs a recognition score, and an intermediate layer that connects the input and the output. The intermediate layer may include a convolutional layer, a pooling layer, a fully connected layer, etc. The intermediate layer may have a multi-layer structure, and may be configured to enable so-called deep learning. The model used for machine learning may be constructed using a convolutional neural network (CNN). Note that the model used for machine learning is not limited to the above, and any machine learning model may be used.

[0019] When multiple candidate regions are extracted from the captured image, the determination processing unit 26 uses the relative positions of the multiple candidate regions and the relative values ​​of the recognition scores of the multiple candidate regions to determine whether or not an object is included in each of the multiple candidate regions. Specifically, the determination processing unit 26 identifies multiple candidate regions that are close to each other within a predetermined distance from among the candidate regions extracted from the captured image, and when the difference in the recognition scores of the identified multiple candidate regions is equal to or greater than a predetermined value, determines that the candidate region with the relatively large recognition score among the multiple candidate regions includes an object, and determines that the candidate region with the relatively small recognition score does not include an object. Close to each other within a predetermined distance means, for example, that the extracted multiple candidate regions are close to each other vertically or horizontally at a distance of about 0 to 10 pixels in the captured image. In other words, the determination processing unit 26 identifies multiple candidate regions whose relative positions are in the positional relationship of a direct image and a reflected image. In addition, the determination processing unit 26 uses the relative values ​​of the recognition scores in the identified multiple candidate regions to determine, for example, whether or not an object included in a candidate region can be a reflected image rather than a direct image. Here, the reflected image means an object that is reflected by a reflecting surface included in the captured image. The reflecting surface is, for example, a wet road surface, but is not limited to this, and any location where the entire object may be reflected by specular reflection can be a reflecting surface. For example, the determination processing unit 26 determines that the object is not an object if it corresponds to a reflected image, and determines that the object is an object if it corresponds to a direct image rather than a reflected image.

[0020] When multiple candidate areas are extracted from the captured image, the determination processing unit 26 may identify multiple candidate areas that are adjacent to each other in the vertical direction within a predetermined distance in the captured image. In this case, when the difference in the recognition scores of multiple candidate areas that are adjacent to each other in the vertical direction is equal to or greater than a predetermined value, the recognition score of the upper candidate area is relatively large, and the recognition score of the lower candidate area is relatively small, the determination processing unit 26 determines that the upper candidate area includes an object and that the lower candidate area does not include an object. For example, when a reflected image of an object reflected on the road surface is erroneously detected as an object, it is determined that the object exists at a position closer to the camera 20 than the actual object. By determining that such a reflected image is not an object, it is possible to prevent erroneous output indicating that an object exists at a distance different from the actual object.

[0021] When multiple candidate areas are extracted from the captured image, the determination processing unit 26 may specify multiple candidate areas that are adjacent to each other within a predetermined distance and have the same positional relationship in a direction intersecting the adjacent direction. For example, when multiple candidate areas that are adjacent to each other in the vertical direction are extracted from the captured image, this means that the positions and widths of the multiple candidate areas in the left-right direction (i.e., the horizontal direction) intersecting the vertical direction in which they are adjacent to each other match. In this case, the match may be a difference of about 0 to 10 pixels in the captured image. The approaching direction is the direction in which the multiple adjacent candidate areas move away from each other, for example, the direction connecting specific points (e.g., center points) in the multiple candidate areas.

[0022] FIG. 2 is a diagram showing an example of a captured image 50 including objects 56a to 56c. A specific area 52 on the right side of the captured image 50 is an area where a wet road surface exists. Reflected images 58a and 58b are reflected below objects 56a and 56b located in the specific area 52. A non-specific area 54 on the left side of the captured image 50 is an area where a wet road surface does not exist. No reflected image is reflected below object 56c located in the non-specific area 54. The objects 56a to 56c can be said to be direct images captured by the camera 20 without passing through a reflecting surface. On the other hand, the reflected images 58a and 58b are captured by the camera 20 after being reflected by a wet road surface or the like.

[0023] Fig. 3 is a diagram showing an example of candidate regions 60a-60d extracted from the captured image 50 shown in Fig. 2. The candidate regions 60a-60d are extracted by the recognition processing unit 24. In the example of Fig. 3, the recognition scores of the detection windows including the objects 56a-56c are equal to or greater than a reference value, so they are extracted as candidate regions 60a-60c. In addition, the recognition score of the detection window including the reflected image 58a is equal to or greater than a reference value, so it is extracted as candidate region 60d.

[0024] As an example, the reference value of the recognition score that the recognition processing unit 24 extracts as a candidate region is 0.7. In the example of FIG. 3, the recognition score for the detection window including the object 56a is 0.95, so it is extracted as a candidate region 60a. The recognition score for the detection window including the object 56b is 0.85, so it is extracted as a candidate region 60b. The recognition score for the detection window including the object 56c is 0.90, so it is extracted as a candidate region 60c. The object 56b is located farther from the camera 20 than the objects 56a and 56c, and is small in the captured image 50, so the recognition score is relatively low. The recognition score for the detection window including the reflected image 58a is 0.75, which is equal to or greater than the reference value, so it is extracted as a candidate region 60d. On the other hand, the recognition score for the detection window including the reflected image 58b is 0.60, which is less than the reference value, so it is not extracted as a candidate region.

[0025] The determination processor 26 determines whether an object is included in each of the multiple candidate regions 60a-60d extracted from the captured image 50 in Fig. 3. Of the multiple candidate regions, the determination processor 26 identifies multiple candidate regions that are adjacent to each other within a predetermined distance. In the example of Fig. 3, the candidate region 60a and the candidate region 60d that are adjacent to each other within the predetermined distance are the candidate regions that are adjacent to each other within the predetermined distance.

[0026] The determination processing unit 26 compares the recognition scores of the candidate area 60a and the candidate area 60d that are determined to be close to each other within a predetermined distance, and determines whether or not the candidate area 60a and the candidate area 60d each contain an object based on the difference in the recognition scores. When the difference in the recognition scores of the multiple candidate areas is equal to or greater than a predetermined value, the determination processing unit 26 determines that the candidate area with the relatively large recognition score contains an object, and determines that the candidate area with the relatively small recognition score does not contain an object. Specifically, the determination processing unit 26 determines whether or not the difference in the recognition scores of the multiple candidate areas is equal to or greater than a predetermined value of 0.15. In the example of FIG. 3, the difference in the recognition scores of the candidate area 60a and the candidate area 60d is 0.95-0.75=0.20, which is determined to be equal to or greater than the predetermined value.

[0027] The determination processing unit 26 determines that the candidate area 60d with a relatively small recognition score does not include an object. This makes it possible to determine that the candidate area 60d that is erroneously detected in the reflected image 58a is not an object. The determination processing unit 26 determines that the candidate area 60a with a relatively large recognition score includes an object. The determination processing unit 26 determines that the candidate area 60b and the candidate area 60c include objects because there are no candidate areas that are close within a predetermined distance. This makes it possible to properly detect the objects 56a to 56c that are not reflected images as objects. When the difference in recognition scores between the multiple candidate areas identified as being close within a predetermined distance is less than a predetermined value, the determination processing unit 26 determines that each of the multiple candidate areas includes an object.

[0028] The output control unit 16 causes the output device 22 to output object information related to the object detected by the object detection unit 14. The object information may include, for example, information regarding whether an object has been detected by the object detection unit 14, the number of objects detected by the object detection unit 14, and the position and distance of the detected object. The output device 22 may be a communication device, or a wireless communication device that outputs object information such as the position and distance of the object by road-to-vehicle communication or vehicle-to-vehicle communication.

[0029] In the example of Fig. 3, the output control unit 16 can output object information about objects 56a-56c included in a candidate area determined by the determination processing unit 26 to include an object. In the example of Fig. 3, the output control unit 16 does not output object information about a reflection image 58a included in a candidate area determined by the determination processing unit 26 to not include an object.

[0030] 4 is a flowchart showing an example of the flow of the recognition processing method according to the first embodiment. The image acquisition unit 12 acquires a captured image captured by the camera 20 (step S10). The recognition processing unit 24 starts scanning the captured image using a detection window, calculates a recognition score for the range of the detection window using a detection model that has been machine-learned from the entire image of the target, and extracts candidate areas whose recognition score is equal to or greater than a reference value (step S12).

[0031] The determination processing unit 26 determines whether or not a candidate area has been extracted in step S12 (step S14). If it is determined in step S14 that a candidate area has been extracted (Yes in step S14), it determines whether or not there are multiple adjacent candidate areas that are adjacent within a predetermined distance (step S16). If it is determined in step S16 that there are multiple adjacent candidate areas that are adjacent within a predetermined distance (Yes in step S16), it determines whether or not the difference in recognition scores between the multiple adjacent candidate areas that are adjacent within a predetermined distance is equal to or greater than a predetermined value (step S18).

[0032] If the determination processing unit 26 determines in step S18 that the difference in recognition scores between adjacent candidate areas within a predetermined distance is equal to or greater than a predetermined value (Yes in step S18), it determines that the candidate area with the relatively large recognition score contains the target object. In other words, it determines that the candidate area with the relatively small recognition score does not contain the target object (step S20) and proceeds to step S24. That is, in step S20, the candidate area with the relatively large recognition score is determined to be a direct image of the target object, and the candidate area with the relatively small recognition score is determined to be a reflected image of the target object.

[0033] If it is not determined that there are multiple candidate regions adjacent within a predetermined distance (No in step S16), and if it is determined that the difference in recognition scores between multiple candidate regions adjacent within a predetermined distance is not a predetermined value or more (No in step S18), the determination processing unit 26 determines that all candidate regions include the object (step S22) and proceeds to step S24. In other words, a candidate region that has no adjacent candidate regions within a predetermined distance, and a candidate region that has multiple adjacent candidate regions within a predetermined distance but whose difference in recognition scores is not a predetermined value or more, are determined to be a direct image of the object.

[0034] The output control unit 16 outputs object information related to the detected object (step S24). If it is not determined in step S14 that the recognition processing unit 24 has extracted a candidate area (No in step S14), the processes in steps S16 to S24 can be skipped.

[0035] According to this embodiment, among the candidate areas extracted by the recognition processing unit 24, candidate areas that include a reflected image of an object can be excluded by the determination processing unit 26. As a result, it is possible to prevent erroneous detection of a reflected image reflected on the road surface, etc., and to properly detect an object included in the captured video.

[0036] In the above embodiment, the reflected image reflected on a wet road surface or the like has been described, but the reflected image reflected on a glass window of a building or the like may also be used. In this case, if the object is close to the glass window, the reflected image is reflected on the glass window, so the direct image of the object and the reflected image close to the object can be identified as a pair of proximity candidate areas. In addition, the reflected image reflected on the glass window has a lower visibility than the direct image of the object, so the recognition score is relatively small. In this case, the determination processing unit 26 can exclude the reflected image reflected on the glass window with a relatively small recognition score, and can prevent erroneous detection of the reflected image reflected on the glass window.

[0037] Second embodiment 5 is a block diagram showing a functional configuration of a recognition processing device 10A according to a second embodiment. The second embodiment differs from the first embodiment in that the object detection unit 14A further includes a candidate region tracking unit 28A. The second embodiment will be described below, focusing on the differences from the first embodiment, and descriptions of commonalities will be omitted as appropriate.

[0038] The recognition processing device 10A includes an image acquisition unit 12, an object detection unit 14A, and an output control unit 16. The image acquisition unit 12 and the output control unit 16 can be configured similarly to the first embodiment. The object detection unit 14A includes a recognition processing unit 24, a candidate region tracking unit 28A, and a judgment processing unit 26A. The recognition processing unit 24 can be configured similarly to the first embodiment.

[0039] The candidate area tracking unit 28A tracks the candidate area extracted by the recognition processing unit 24. The candidate area tracking unit 28A tracks the candidate area across multiple frames constituting the captured video. When multiple candidate areas are extracted by the recognition processing unit 24, the candidate area tracking unit 28A distinguishes between the multiple candidate areas and tracks them across multiple frames.

[0040] When identifying multiple candidate areas that are adjacent to each other within a predetermined distance, the determination processing unit 26A identifies multiple candidate areas that are adjacent to each other within a predetermined distance across multiple frames based on the tracking results of the candidate area tracking unit 28A. In other words, multiple candidate areas that are not adjacent to each other within a predetermined distance across multiple frames are not subject to determination based on the difference in recognition scores of the multiple candidate areas.

[0041] When the difference in recognition scores between multiple candidate areas that are close to each other within a predetermined distance over multiple frames is equal to or greater than a predetermined value, the determination processing unit 26A determines that the candidate area with a relatively large recognition score contains an object, and determines that the candidate area with a relatively small recognition score does not contain an object. Here, multiple frames are included within a short time period of less than 0.5 seconds, for example, within 5 frames. For example, when the video captured by the camera 20 is a video image of 30 frames per second, it is determined that the two candidate areas are close to each other within a predetermined distance for 0.1 seconds or 3 frames. The determination processing unit 26A uses the relative positions of the multiple candidate areas over multiple frames and the relative values ​​of the recognition scores to determine whether or not the multiple candidate areas each contain an object.

[0042] For example, in a first frame, when multiple candidate areas are adjacent to each other within a predetermined distance, the judgment processing unit 26A judges that one candidate area does not contain an object because the recognition score of the one candidate area is relatively smaller than the recognition score of the other candidate area, and if, in a second frame after the first frame, the two candidate areas are adjacent to each other within a predetermined distance and the relative value of the recognition score is also a similar judgment result, it judges that the other candidate area does not contain an object.

[0043] For example, in an arbitrary first frame, when a plurality of candidate regions are adjacent to each other within a predetermined distance, the judgment processing unit 26A judges one candidate region as a candidate region that does not contain an object because the recognition score of the one candidate region is relatively smaller than the recognition score of the other candidate region, and if it is determined in a second frame two frames after the first frame that the two candidate regions are not adjacent to each other within the predetermined distance, it judges the other candidate region to contain an object.

[0044] FIG. 6 is a diagram showing an example of candidate regions 60e-60h extracted from the captured image 50a in an arbitrary first frame. The specific region 52 is a location where a wet road surface exists, and a reflected image 658b is reflected below the object 56e located in the specific region 52. The non-specific region 54 is a location where a wet road surface does not exist, and objects 56f and 56g exist. The recognition processing unit 24 extracts candidate regions 60e-60g for the detection window including the objects 56e-56g, and a candidate region 60h for the detection window including the reflected image 58b. As an example, the recognition score of the candidate region 60e is 0.95, the recognition score of the candidate region 60f is 0.95, the recognition score of the candidate region 60g is 0.80, and the recognition score of the candidate region 60h is 0.75.

[0045] The determination processing unit 26A identifies multiple candidate areas that are adjacent to each other within a predetermined distance from the multiple candidate areas extracted from the captured image 50a in FIG. 6. In the example of FIG. 6, the candidate areas 60e and 60h that are adjacent to each other in the vertical direction correspond to multiple candidate areas that are adjacent to each other within a predetermined distance. Furthermore, the candidate areas 60f and 60g that are adjacent to each other in the vertical direction correspond to multiple candidate areas that are adjacent to each other within a predetermined distance. Since the difference in recognition score between the candidate areas 60e and 60h in the captured image 50a is 0.95-0.75=0.20, which is equal to or greater than the predetermined value, the determination processing unit 26A determines that the candidate area 60e with the relatively large recognition score includes an object, and determines that the candidate area 60h with the relatively small recognition score does not include an object. In addition, the difference in recognition score between candidate area 60f and candidate area 60g in the captured image 50a is 0.95-0.80=0.15, which is greater than the predetermined value (0.15), so the judgment processing unit 26A determines that candidate area 60f, which has a relatively large recognition score, contains an object, and determines that candidate area 60g, which has a relatively small recognition score, does not contain an object.

[0046] Since the determination by the determination processing unit 26A for the captured image 50a in the first frame shown in Fig. 6 is a stage before tracking of candidate areas across multiple frames, candidate areas with relatively small recognition scores may be stored as reflection image candidates. In the captured image 50a in the first frame shown in Fig. 6, candidate areas 60g and 60h are stored as reflection image candidates.

[0047] FIG. 7 is a diagram showing an example of candidate regions 60e-60h extracted from the captured image 50b in the second frame. The captured image 50b in the second frame shown in FIG. 7 shows a captured image two frames after the captured image 50a in the first frame shown in FIG. 6. The positions of the objects 56e-56g included in the captured image 50b have moved slightly with respect to the positions of the objects 56e-56g included in the captured image 50a. In the captured image 50b, the object 56e is located in a specific area 52 where a wet road surface exists, so that the reflected image 58b is still reflected below the object 56e. The recognition processing unit 24 extracts candidate regions 60e-60g for the detection window including the objects 56e-56g and a candidate region 60h for the detection window including the reflected image 58b from the captured image 50b. The candidate regions 60e-60h are tracked by the candidate region tracking unit 28A while being distinguished from one another.

[0048] The determination processing unit 26A identifies a plurality of candidate areas that are adjacent to each other within a predetermined distance from among the plurality of candidate areas extracted from the captured image 50b in FIG. 7. In the example of FIG. 7, the candidate area 60e and the candidate area 60h that are adjacent to each other in the vertical direction correspond to the plurality of candidate areas that are adjacent to each other within a predetermined distance. Moreover, the candidate area 60f and the candidate area 60g are not the plurality of candidate areas that are adjacent to each other within a predetermined distance. Since the difference in the recognition score between the candidate area 60e and the candidate area 60h in the captured image 50b is 0.95-0.75=0.20, which is equal to or greater than a predetermined value, the determination processing unit 26A determines that the candidate area 60e with a relatively large recognition score includes an object, and determines that the candidate area 60h with a relatively small recognition score does not include an object. Therefore, since the candidate area 60e and the candidate area 60h are adjacent to each other within a predetermined distance over a plurality of frames, the candidate area 60h is determined to be a reflection image from the reflection image candidate. In other words, the candidate area 60h is determined not to include an object. Also, since candidate area 60f and candidate area 60g in captured image 50b are not adjacent within a predetermined distance, determination processing unit 26A determines that candidate area 60f and candidate area 60g include an object. Therefore, since candidate area 60f and candidate area 60g are not adjacent within a predetermined distance across multiple frames, candidate area 60g is determined to be a direct image from a reflected image candidate. In other words, candidate area 60g is determined to include an object.

[0049] FIG. 8 is a flowchart showing an example of the flow of the recognition processing method according to the second embodiment. The processes from step S30 to step S38 and step S48 shown in FIG. 8 are common to the processes from step S10 to step S18 and step S24 shown in FIG. 4, and therefore the description will be omitted. If the determination processing unit 26A determines in step S38 that the difference in the recognition scores of the multiple candidate areas adjacent to each other within a predetermined distance is equal to or greater than a predetermined value (Yes in step S38), the determination processing unit 26A determines that the candidate areas having relatively small recognition scores are reflection image candidate areas (step S40) and proceeds to step S42. The determination processing unit 26A tracks the candidate areas extracted in step S34 and determines whether the candidate areas identified in step S36 as the multiple candidate areas adjacent to each other within a predetermined distance are adjacent to each other within a predetermined distance across multiple frames (step S42). If the determination in step S42 is that the candidate areas are adjacent to each other within a predetermined distance across multiple frames (Yes in step S42), the determination unit 26A determines that the candidate areas identified as reflection image candidates in step S40 do not include an object (step S44).

[0050] If it is not determined that there are multiple candidate areas that are adjacent within a predetermined distance (No in step S36), if it is determined that the difference in recognition scores between multiple candidate areas that are adjacent within a predetermined distance is not greater than or equal to a predetermined value (No in step S38), or if it is not determined that the candidate areas are adjacent within a predetermined distance across multiple frames (No in step S42), the determination processing unit 26A determines that all of the candidate areas contain the object (step S46).

[0051] According to this embodiment, by using the relative positions of multiple candidate regions across multiple frames and the relative values ​​of the recognition scores, it is possible to exclude candidate regions that include a reflected image of an object. In particular, by utilizing the characteristic that the relative positions of the direct image and the reflected image of an object are similar across multiple frames, it is possible to detect the reflected image with high accuracy. As a result, it is possible to prevent erroneous detection of a reflected image reflected on the road surface, etc., and to properly detect an object included in the captured video.

[0052] Third embodiment 9 is a block diagram showing a schematic functional configuration of a recognition processing device 10B according to a third embodiment. The third embodiment differs from the above-mentioned embodiments in that a first detection unit for detecting an overall image of an object and a second detection unit for detecting a range including a direct image and a reflected image of the object are used in combination. Hereinafter, the third embodiment will be described with a focus on the differences from the first and second embodiments, and descriptions of commonalities will be omitted as appropriate.

[0053] The recognition processing device 10B includes an image acquisition unit 12, an object detection unit 14B, and an output control unit 16. The image acquisition unit 12 and the output control unit 16 can be configured similarly to the first or second embodiment. The object detection unit 14B includes a first detection unit 30, a second detection unit 32, and a validity determination unit 34.

[0054] The first detection unit 30 can be configured in the same manner as the recognition processing unit 24 according to the above-described embodiment. The first detection unit 30 detects an object by calculating a recognition score using a first detection model that is machine-learned from an entire image of the object. The area that contains the object detected by the first detection unit 30 is defined as the first detection area. The first detection unit 30 scans the detection window for each frame of the captured video while referring to the first detection model, and calculates a recognition score that indicates the possibility that the object is included in each detection window. If the recognition score is equal to or greater than a predetermined reference value, such as 0.7, the first detection unit 30 detects the object by determining that the object is included in the detection window.

[0055] FIG. 10 is a diagram showing an example of first detection regions 80h-80k, which are the result of the first detection unit 30 detecting an object in the captured image 50c. The specific region 52 is a location where a wet road surface exists, and a reflected image 58c is reflected below the object 56h located in the specific region 52. The non-specific region 54 is a location where a wet road surface does not exist, and objects 56i and 56j exist. The first detection unit 30 detects the object as the first detection region 80h-80j because the recognition score for the detection window including the objects 56h-56j is equal to or greater than a predetermined value. The first detection unit 30 also detects the object as the first detection region 80k because the recognition score for the detection window including the reflected image 58c is equal to or greater than a predetermined value. As an example, the recognition score of the first detection area 80h is 0.95, the recognition score of the first detection area 80i is 0.95, the recognition score of the first detection area 80j is 0.80, and the recognition score of the first detection area 80k is 0.75.

[0056] The second detection unit 32 detects an object included in the captured image by calculating a recognition score using a second detection model that is machine-learned from an image including a direct image and a reflected image of the object. The area including the direct image and the reflected image of the object detected by the second detection unit 32 is defined as the second detection area. For example, an image used for machine learning of the second detection model is an image in which a direct image portion and a reflected image portion are adjacent to each other vertically and a reflected image portion is included below the direct image portion. The second detection unit 32 scans the detection window while referring to the second detection model for each frame of the captured image, and calculates a recognition score indicating the possibility that a direct image and a reflected image of the object are included for each detection window. If the recognition score is equal to or greater than a predetermined reference value, such as 0.7, the second detection unit 32 detects the object by determining that the object is included in the detection window.

[0057] FIG. 11 is a diagram showing an example of a second detection area 82h that is a result of the second detection unit 32 detecting an object in the captured image 50c. The captured image 70 shown in FIG. 11 is the same as that in FIG. 10. The second detection unit 32 detects the second detection area 82h. The second detection area 82h includes a direct image portion 84h and a reflected image portion 86h that are adjacent to each other vertically, and has a vertically longer size than the first detection area. For example, when the aspect ratio of the first detection area is about 2:1, the aspect ratio of the second detection area is about 4:1. The second detection unit 32 detects the second detection area 82h for a range that includes the object 56h and the reflected image 58c. The object 56i and the object 56j are adjacent to each other vertically in the same manner as the positional relationship between the direct image and the reflected image, but are not detected as the second detection area by the second detection unit 32. As an example, the recognition score of the second detection region 82h is 0.90, and the recognition score of the second detection unit 32 for the detection window including the object 56i and the object 56j is 0.60. The object 56j is located below the object 56i, but since the object 56j is a direct image rather than a reflected image, the recognition score by the second detection unit 32 is calculated to be low.

[0058] 11, when the second detection area 82h is detected by the second detection unit 32, the output control unit 16 may output object information for a direct image portion 84h constituting the second detection area 82h, that is, a region corresponding to a direct image. In this case, the position and distance of the object are calculated based on the direct image portion 84h of the second detection area 82h, and the object information is output.

[0059] When a first detection area in which an object is detected by the first detection unit 30 overlaps with a second detection area in which a direct image and a reflected image of the object are detected by the second detection unit 32, the validity determination unit 34 determines whether detection of the first detection area or the second detection area should be valid. For example, when the second detection area overlaps with the first detection area, the validity determination unit 34 validates the detection area with a relatively high recognition score and invalidates the detection area with a relatively low recognition score.

[0060] In the example shown in FIG. 10 and FIG. 11, the second detection area 82h overlaps with the two first detection areas 80h, 80k. Focusing on the first detection area 80h, the recognition score of 0.95 of the first detection area 80h is greater than the recognition score of 0.90 of the second detection area 82h. Therefore, the second detection area 82h with a relatively low recognition score is invalidated. Focusing on the first detection area 80k, the recognition score of 0.75 of the first detection area 80k is less than the recognition score of 0.90 of the second detection area 82h. Therefore, the first detection area 80k with a relatively low recognition score is invalidated. As a result, the first detection area 80h with a relatively high recognition score is valid. In this case, the first detection area 80h is detected as an object, and the first detection area 80k is not detected as an object.

[0061] When the second detection area does not overlap with the first detection area, the validity determination unit 34 validates the direct image portion of the second detection area and invalidates the reflected image portion. For example, when the second detection area 82h shown in FIG. 11 does not overlap with the first detection area, the upper direct image portion 84h is validated and the lower reflected image portion 86h is invalidated. As a result, the object 56h located in a range of approximately 2:1 corresponding to the direct image portion 84h is detected.

[0062] The output control unit 16 outputs object information related to the object detected by the object detection unit 14B. For example, in the case of the example shown in Fig. 10 and Fig. 11, the output control unit 16 outputs object information related to the objects 56h to 56j, but does not output object information related to the reflected image 58c.

[0063] 12 is a flowchart showing an example of the flow of the recognition processing method according to the third embodiment. The image acquisition unit 12 acquires the captured image captured by the camera 20 (step S50). The first detection unit 30 starts scanning the captured image using a detection window corresponding to a first detection model, and detects a first detection area having a recognition score equal to or greater than a predetermined value by calculating a recognition score using the first detection model (step S52). The second detection unit 32 starts scanning the captured image using a detection window corresponding to a second detection model, and detects a second detection area having a recognition score equal to or greater than a predetermined value by calculating a recognition score using the second detection model (step S54). The processing of steps S52 and S54 may be performed in the opposite order, or may be performed simultaneously on the same frames constituting the captured image.

[0064] The validity determination unit 34 determines whether or not a detection area was detected in step S52 or step S54 (step S56). If it is determined in step S56 that a detection area is detected by the first detection unit 30 or the second detection unit 32 (Yes in step S56), it determines whether or not the first detection area detected by the first detection unit 30 and the second detection area detected by the second detection unit 32 overlap (step S58). If it is determined in step S58 that the first detection area and the second detection area overlap (Yes in step S58), the validity determination unit 34 invalidates the detection area with a relatively small recognition score among the overlapping first detection area and second detection area (step S60).

[0065] If it is determined in step S58 that there is no overlap between the first detection area and the second detection area (No in step S58), the process proceeds to step S66, where object information based on the detection results in steps S52 and S54 is output. Specifically, object information related to the object included in the first detection area and the direct image portion in the second detection area is output.

[0066] If it is determined in step S62 that the second detection area is valid (Yes in step S62), the validity determination unit 34 excludes the reflected image portion of the second detection area and validates the direct image portion (step S64), proceeds to step S66, and outputs object information regarding the object included in the direct image portion of the second detection area. If it is determined in step S62 that the second detection area is not valid (No in step S62), proceeds to step S66, and outputs object information regarding the object included in the first detection area. If it is not determined in step S56 that the detection area has been detected by the first detection unit 30 or the second detection unit 32 (No in step S56), the processes of steps S58 to S66 can be skipped.

[0067] According to this embodiment, by using the second detection model that detects the second detection area including the direct image portion and the reflected image portion of the object, the reflected image of the object can be properly detected and excluded from the object. As a result, it is possible to prevent erroneous detection of the reflected image reflected on the road surface, etc., and to properly detect the object included in the captured video.

[0068] (Fourth embodiment) 13 is a block diagram showing a schematic functional configuration of a recognition processing device 10C according to a fourth embodiment. The fourth embodiment differs from the above-mentioned third embodiment in that an area determination unit 36C is used to determine a specific area in a captured image that is likely to include a reflected image of an object. The fourth embodiment will be described below, focusing on the differences from the third embodiment, and descriptions of commonalities will be omitted as appropriate.

[0069] The recognition processing device 10C includes an image acquisition unit 12, an object detection unit 14C, and an output control unit 16. The image acquisition unit 12 and the output control unit 16 can be configured similarly to the first, second, or third embodiment. The object detection unit 14C includes a first detection unit 30, a second detection unit 32, an area determination unit 36C, and a validity determination unit 34C. The first detection unit 30 and the second detection unit 32 can be configured similarly to the third embodiment.

[0070] The area determination unit 36C determines a specific area that is estimated to be likely to include a reflected image of an object in the captured image. For example, when a flat surface such as a paved road is included in the angle of view of the camera 20 and the road surface is wet during rainfall or immediately after rainfall, the area determination unit 36C determines the range as a specific area. The area determination unit 36C can determine the specific area using information acquired from the external device 38.

[0071] The external device 38 is a device that provides information to the area determination unit 36C for determining that a specific area is included in the angle of view of the camera 20. The external device 38 may be, for example, a rain detection sensor that detects rainfall around the recognition processing device 10C, a communication device that acquires information from an external server device via a network, a storage device, or the like.

[0072] When the recognition processing device 10C has a fixed imaging position and imaging range like a smart pole, the area determination unit 36C acquires information from an external device 38 consisting of a storage device that stores in advance an area in the imaging range that is likely to include a reflected image of an object, and a rain detection sensor. When the recognition processing device 10C is mounted on a moving object such as a vehicle, the area determination unit 36C acquires map information from the storage device or an external server device, and acquires rainfall information from the rain detection sensor or an external server device.

[0073] The area determination unit 36C may determine whether each area of ​​the captured image is a specific area. For example, the angle of view of the camera 20 may be divided into a plurality of partial areas according to the type of ground or building included in the angle of view of the camera 20, and each of the plurality of partial areas may be determined to be a specific area or not. For example, a partial area including a part that may be a reflective surface such as a paved road with poor drainage is determined to be a specific area when the road surface is wet during rainfall or immediately after rainfall. Also, a partial area including a part that is unlikely to be a reflective surface such as a paved road with good drainage or grass is always determined to be not a specific area. The area determination unit 36C can determine an area that is not a specific area as a non-specific area.

[0074] When the region determination unit 36C determines that a specific region exists, the validity determination unit 34C validates the detection result of either the first detection unit 30 or the second detection unit 32, or uses the first detection unit 30 and the second detection unit 32 depending on the region. For a region determined not to be a specific region, the validity determination unit 34C invalidates the detection result of the object by the second detection unit 32, or performs object detection by the first detection unit 30 without detecting the object by the second detection unit 32. For a region determined to be a specific region, the validity determination unit 34C invalidates the detection result of the object by the first detection unit 30, or performs object detection by the second detection unit 32 without detecting the object by the first detection unit 30.

[0075] When the detection result by second detection unit 32 is determined to be valid, validity determination unit 34C determines the direct image portion of the second detection region by second detection unit 32 as valid and determines the reflected image portion as invalid.

[0076] Fig. 14 is a diagram showing an example of first detection regions 80i, 80j and second detection region 82h that are the result of detecting objects by the first detection section 30 and the second detection section 32 in the captured image 50c. The captured image 50c shown in Fig. 14 is similar to the captured image 50c shown in Fig. 10 and Fig. 11. A reflected image 58c is reflected below an object 56h located in the specific area 52, and no reflected image is reflected below objects 56i, 56j located in the non-specific area 54.

[0077] In the captured image 50c shown in FIG. 14, the region determination unit 36C determines the area on the right side where the wet road surface exists as the specific region 52, and determines the area on the left side where the wet road surface does not exist as the non-specific region 54. When the detection window is in the non-specific region 54, the object detection unit 14C performs object detection by the first detection unit 30, and detects the first detection regions 80i and 80j including the objects 56i and 56j in FIG. 14. When the detection window is in the specific region 52, the object detection unit 14C performs object detection by the second detection unit 32, and detects the second detection region 82h including the object 56h and the reflected image 58c of the object 56h in FIG. 14. The validity determination unit 34C validates the direct image portion 84h of the second detection region 82h and invalidates the reflected image portion 86h. As a result, the object 56h is detected in the specific region 52, but the reflected image 58c is not detected as an object. The output control unit 16 outputs object information regarding the detected objects 56h to 56j.

[0078] FIG. 15 is a flowchart showing an example of the flow of the recognition processing method according to the fourth embodiment. The image acquisition unit 12 acquires the captured image captured by the camera 20 (step S70). Next, the area determination unit 36C determines whether or not the captured image acquired in step S70 has a specific area (step S72). If it is determined in step S72 that a specific area exists (Yes in step S72), the first detection unit 30 detects an object in a non-specific area of ​​the captured image, and the second detection unit 32 detects an object in the specific area (step S74), and the process proceeds to step S78. If it is determined in step S72 that a specific area does not exist (No in step S72), the first detection unit 30 detects an object in the entire captured image (step S76), and the process proceeds to step S86.

[0079] In step S78, it is determined whether or not an object was detected in step S74. In step S78, if it is determined that an object was detected by the first detection unit 30 or the second detection unit 32, in other words, if it is determined that the first detection area or the second detection area was detected (Yes in step S78), the process proceeds to step S80. In step S80, it is determined whether or not the second detection unit is included in the detection area detected in step S74. In step S80, if it is determined that the second detection area is included (Yes in step S80), the validity determination unit 34C excludes the reflected image portion of the second detection area and validates the direct image portion (step S82). In step S80, if it is not determined that the second detection area is included (No in step S80), the process skips step S82. The output control unit 16 outputs object information regarding the object included in the valid detection area (step S84). In step S78, if it is not determined that an object has been detected by the first detection unit 30 or the second detection unit 32, in other words, if it is not determined that either the first detection area or the second detection area has been detected (No in step S78), the processing of steps S80 to S84 can be skipped.

[0080] In step S86, it is determined whether or not an object was detected in step S76. If it is determined in step S86 that an object was detected, in other words, if it is determined that the first detection area was detected (Yes in step S86), the process proceeds to step S84, where the output control unit 16 outputs object information regarding the object included in the first detection area (step S84). If it is determined in step S86 that an object was not detected, in other words, if it is determined that the first detection area was not detected (No in step S86), the process of step S84 can be skipped.

[0081] According to this embodiment, by using the second detection model in a specific area where a reflected image is likely to be included, the reflected image of the object can be properly detected and excluded from the object. As a result, it is possible to prevent erroneous detection of a reflected image reflected on the road surface, etc., and to properly detect the object included in the captured video.

[0082] The present invention has been described above with reference to the above-mentioned embodiment, but the present invention is not limited to the above-mentioned embodiment, and appropriate combinations or substitutions of the respective configurations shown in the embodiment are also included in the present invention.

[0083] Several aspects of the disclosure are described below.

[0084] A first aspect of the present disclosure is a recognition processing device that includes an image acquisition unit that acquires an image, a recognition processing unit that extracts candidate areas from detection areas set in the image, the candidate areas having a recognition score indicating a possibility that an object is included therein that is equal to or greater than a reference value, a determination processing unit that identifies a plurality of candidate areas that are adjacent to each other within a predetermined distance from the candidate areas extracted in the image, and, if a difference in recognition scores between the plurality of candidate areas is equal to or greater than a predetermined value, determines that the candidate area with a relatively large recognition score among the plurality of candidate areas contains an object, and determines that the candidate area with a relatively small recognition score does not contain an object, and an output control unit that outputs object information related to the object detected in the candidate area determined by the determination processing unit to contain the object.

[0085] A second aspect of the present disclosure is a recognition processing method performed by a recognition processing device, comprising the steps of: acquiring an image; extracting candidate areas from among detection areas set in the image, the candidate areas having a recognition score indicating a possibility that an object is included therein that is equal to or greater than a reference value; identifying a plurality of candidate areas that are adjacent to each other within a predetermined distance from among the candidate areas extracted in the image, and, if a difference in recognition scores between the plurality of candidate areas is equal to or greater than a predetermined value, determining that the candidate area with a relatively large recognition score among the plurality of candidate areas contains an object and determining that the candidate area with a relatively small recognition score does not contain an object; and outputting object information regarding the object detected in the candidate area determined to contain the object.

[0086] A third aspect of the present disclosure is a program configured to cause a processor to execute the following functions: acquire an image; extract candidate areas from among the detection areas set in the image, the candidate areas having a recognition score indicating a possibility that an object is included therein that is equal to or greater than a reference value; identify multiple candidate areas that are adjacent to each other within a predetermined distance from among the candidate areas extracted in the image, and, if the difference in recognition scores between the multiple candidate areas is equal to or greater than a predetermined value, determine that the candidate area with the relatively large recognition score among the multiple candidate areas contains an object, and determine that the candidate area with the relatively small recognition score does not contain an object; and output object information regarding the object detected in the candidate area determined to contain the object.

[0087] A fourth aspect of the present disclosure is a recognition processing device including an object detection unit that includes an image acquisition unit that acquires an image, a first detection unit that detects an object included in the image using a first detection model that has been machine-learned on an entire image of the object, and a second detection unit that detects an object included in the image using a second detection model that has been machine-learned on an image including a direct image and a reflected image of the object, and an output control unit that outputs object information regarding the object detected using the first detection model or the second detection model.

[0088] A fifth aspect of the present disclosure is a recognition processing method performed by a recognition processing device, comprising the steps of acquiring an image, detecting an object included in the image, using a first detection model that has been machine-learned on an overall image of the object, detecting an object included in the image, using a second detection model that has been machine-learned on an image including an overall image and a reflected image of the object, and outputting object information regarding the object detected using the first detection model or the second detection model.

[0089] A sixth aspect of the present disclosure is a program configured to cause a processor to execute the following functions: acquiring an image captured by the image capture; detecting an object contained in the image captured by the image capture using a first detection model that has been machine-learned on an overall image of the object; detecting an object contained in the image captured by the image capture using a second detection model that has been machine-learned on an image including an overall image and a reflected image of the object; and outputting object information regarding the object detected by the first detection model or the second detection model. [Explanation of symbols]

[0090] 10...recognition processing device, 12...image acquisition unit, 14...object detection unit, 16...output control unit, 24...recognition processing unit, 26...determination processing unit.

Claims

1. an image acquisition unit that acquires captured images; a recognition processing unit that extracts candidate areas from the detection areas set in the captured image, the candidate areas having a recognition score that indicates a possibility that an object is included therein equal to or greater than a reference value; a determination processing unit that identifies a plurality of candidate areas that are adjacent to each other within a predetermined distance from among the candidate areas extracted from the captured video, and when a difference in recognition scores between the plurality of candidate areas is equal to or greater than a predetermined value, determines that the candidate area with a relatively large recognition score among the plurality of candidate areas contains an object, and determines that the candidate area with a relatively small recognition score does not contain an object; an output control unit that outputs object information regarding the object detected in the candidate area determined by the determination processing unit to include the object; A recognition processing device comprising:

2. When a plurality of candidate areas adjacent to each other in the vertical direction are identified in the captured image, and the difference in recognition scores between the plurality of candidate areas is equal to or greater than a predetermined value, and the recognition score of an upper candidate area among the plurality of candidate areas is relatively large and the recognition score of a lower candidate area is relatively small, the determination processing unit determines that the upper candidate area contains an object and determines that the lower candidate area does not contain an object. The recognition processing device according to claim 1 .

3. the determination processing unit identifies a plurality of candidate areas that are adjacent to each other within a predetermined distance and have the same positional relationship in a direction intersecting the direction of proximity; The recognition processing device according to claim 1 .

4. a candidate area tracking unit that tracks each of the plurality of candidate areas across a plurality of frames that make up the captured video; the determination processing unit identifies a plurality of candidate regions that are adjacent to each other within a predetermined distance across a plurality of frames; The recognition processing device according to claim 1 .

5. A function for acquiring captured images; a function of extracting candidate areas from the detection areas set in the captured image, the candidate areas having a recognition score that indicates a possibility that an object is included therein equal to or greater than a reference value; a function of identifying a plurality of candidate areas that are adjacent to each other within a predetermined distance from among the candidate areas extracted from the captured image, and determining that the candidate area with the relatively large recognition score contains an object if the difference in recognition scores between the plurality of candidate areas is equal to or greater than a predetermined value, and determining that the candidate area with the relatively small recognition score does not contain an object; a function of outputting object information regarding the object detected in the candidate area determined to include the object; A program configured to cause a processor to execute the following.

6. An image acquisition unit that acquires captured images; an object detection unit that detects an object included in the captured video using a second detection model that has been machine-learned using an image including a direct image and a reflected image of the object; an output control unit that outputs object information regarding the object detected using the second detection model; A recognition processing device comprising:

7. The object detection unit further detects an object included in the captured image using a first detection model obtained by machine learning an entire image of the object; further comprising a validity determination unit that, when a first detection area in which an object is detected using the first detection model and a second detection area in which an object is detected using the second detection model overlap, determines whether detection in the first detection area or the second detection area is valid; the output control unit outputs object information regarding the object included in the detection area determined to be valid by the validity determination unit. The recognition processing device according to claim 6 .

Images