Recognition processing device, recognition processing method, and program

Abstract

This recognition processing device (10A) comprises: a video acquisition unit (12) that acquires a captured video; an object detection unit (14A) that uses a detection model obtained by machine-learning an image of an object to detect an object included in the captured video; a lower end estimation unit (30) that, when an object included in a range overlapping the lower edge of the captured video is detected by the object detection unit (14A), estimates a lower end position of the object that can be positioned below the lower edge of the captured video; and a distance calculation unit (16A) that calculates the distance information of the object using the lower end position estimated by the lower end estimation unit (30).

Description

Recognition processing device, recognition processing method, and program

[0001] The present invention relates to a recognition processing device, a recognition processing method, and a program.

[0002] 2. Description of the Related Art There is known a technique for detecting an object such as a pedestrian from an image captured around a vehicle using image recognition processing such as pattern matching (see, for example, Patent Document 1).

[0003] Japanese Patent Application Laid-Open No. 2022-139374

[0004] When an object is located near the outer edge of an image captured by a camera, the entire image of the object is not included in the image, and therefore the object may not be detected properly.

[0005] The present invention has been made in view of the above circumstances, and aims to provide a technique for more appropriately detecting an object in image recognition processing.

[0006] A recognition processing device according to one aspect of the present invention includes an image acquisition unit that acquires captured imagery, an object detection unit that detects objects included in the captured imagery using a detection model that has been machine-learned from images of the objects, a bottom edge estimation unit that, when the object detection unit detects an object included in a range that overlaps the bottom edge of the captured image, estimates the bottom edge position of the object that may be located below the bottom edge of the captured image, and a distance calculation unit that calculates distance information of the object using the bottom edge position estimated by the bottom edge estimation unit.

[0007] Another aspect of the present invention is a recognition processing method in which a recognition processing device performs the following steps: acquiring an image of the object; detecting an object included in the image of the object using a detection model that has been machine-learned from an image of the object; if an object included in an area that overlaps the lower edge of the image of the object is detected, estimating the lower end position of the object that may be located below the lower edge of the image of the object; and calculating distance information of the object using the estimated lower end position.

[0008] Yet another aspect of the present invention is a program that causes a processor to realize the following functions: a function of acquiring captured video; a function of detecting an object included in the captured video using a detection model that has been machine-learned from an image of the object; a function of estimating the bottom end position of the object that may be located below the bottom edge of the captured video if an object included in an area that overlaps with the bottom edge of the captured video is detected; and a function of calculating distance information of the object using the estimated bottom end position.

[0009] According to an aspect of the present invention, a technique for more appropriately detecting an object in image recognition processing can be provided.

[0010] 1 is a block diagram schematically showing the functional configuration of a recognition processing device according to a first embodiment. FIG. 4 is a diagram schematically showing an example of a captured image including an object. FIG. 4 is a diagram schematically showing an example of a detection area set for the captured image. FIGS. 4(a) to (e) are diagrams schematically showing an example of an input image used for machine learning of a detection model. FIG. 4 is a flowchart showing an example of the flow of a recognition processing method according to the first embodiment. FIG. 5 is a flowchart showing an example of the flow of the processing of step S16 in FIG. 5. FIG. 8 is a block diagram schematically showing the functional configuration of a recognition processing device according to a second embodiment. FIGS. 8(a) to (c) are diagrams schematically showing an example of an object tracked across multiple frames constituting the captured image. FIG. 8 is a flowchart showing an example of the flow of a recognition processing method according to the second embodiment. FIG. 11 is a block diagram schematically showing the functional configuration of a recognition processing device according to a third embodiment. FIGS. 11(a) and 11(b) are diagrams schematically showing an example of a method for detecting an object located at the bottom edge of the captured image. FIG. 11 is a flowchart showing an example of the flow of a recognition processing method according to the third embodiment.

[0011] Hereinafter, embodiments of the present invention will be described with reference to the drawings. Specific numerical values ​​and the like shown in the embodiments are merely examples for facilitating understanding of the invention, and do not limit the present invention unless otherwise specified. Elements not directly related to the present invention are omitted from the drawings.

[0012] 1 is a block diagram schematically illustrating the functional configuration of a recognition processing device 10 according to a first embodiment. The recognition processing device 10 includes an image acquisition unit 12 and an object detection unit 14. The recognition processing device 10 may further include a distance calculation unit 16 and an output control unit 18. The recognition processing device 10 acquires captured imagery that may include objects such as pedestrians present in the surrounding area, and detects the objects included in the captured imagery.

[0013] In this embodiment, a case where the recognition processing device 10 is installed on a smart pole is illustrated. The smart pole is installed on a street, for example, and includes an antenna and communication equipment for providing wireless communication functions, lighting equipment for illuminating the street, and a camera for photographing vehicles and pedestrians passing on the road. The recognition processing device 10 is fixed at a predetermined location. The recognition processing device 10 may be mounted on a moving object, or on an air vehicle such as a vehicle or a drone.

[0014] The recognition processing device 10 can detect any object. In the embodiment of the present invention, the object is a person such as a pedestrian.

[0015] Each functional block shown in this embodiment can be realized, for example, by a combination of hardware and software. The hardware of the recognition processing device 10 is realized by elements and mechanical devices such as processors such as a central processing unit (CPU) and a graphics processing unit (GPU) of a computer, and memories such as read-only memory (ROM) and random access memory (RAM). The software of the recognition processing device 10 is realized by a computer program or the like.

[0016] The image acquisition unit 12 acquires images captured by the camera 20 (also referred to as captured images). The camera 20 is provided on the smart pole and captures images of the area around the smart pole. The camera 20 is provided, for example, above the smart pole and captures images with an angle of view looking down on the ground on which the smart pole is installed. The camera 20 captures visible light to generate color images or monochrome images. The camera 20 may be an infrared camera, and may capture infrared light to generate thermal images. The images captured by the camera 20 are moving images at, for example, 30 frames per second or 60 frames per second.

[0017] The object detection unit 14 detects an object from the video acquired by the video acquisition unit 12. In other words, the object detection unit 14 detects an area (hereinafter referred to as a detection area) containing an object in the video acquired by the video acquisition unit 12. For each frame of the video acquired by the video acquisition unit 12, the object detection unit 14 scans a detection window while referencing one or more detection models for detecting the object, and calculates a recognition score indicating the likelihood that the object is included in each detection window. The recognition score is calculated, for example, in the range of 0.0 to 1.0. The higher the likelihood that the object is included in the video within the detection window, the larger the recognition score (i.e., a value closer to 1.0), and the lower the likelihood that the object is included, the smaller the recognition score (i.e., a value closer to 0.0). If the recognition score is equal to or greater than a predetermined threshold, such as 0.8, the object detection unit 14 determines that the object is included in the detection window, thereby detecting the object.

[0018] The object detection unit 14 includes a first detection unit 24 and a second detection unit 26 .

[0019] The first detection unit 24 detects an object using a first detection model that is machine-learned from an entire image of a person, which is the object. An entire image of a person is an image that includes the person's entire body. The first detection unit 24 detects an object that is included in an area that does not overlap with the outer edge of the captured image, such as near the center of the captured image. For example, the first detection unit 24 detects an object that is included in its entirety in the captured image.

[0020] The second detection unit 26 detects the object using a second detection model that has been machine-learned from a partial image of a person, which is the object. A partial image of a person is an image that includes approximately half of the person's entire body. The second detection unit 26 detects an object that is included in a range that includes the inside of the outer edge of the captured image, such as near the periphery of the captured image, and overlaps with the outer edge. The second detection unit 26 detects an object that is included in part of the captured image, but is not included in the captured image because the remaining part of the object is outside the angle of view.

[0021] For this reason, the object detection unit 14 uses the first detection unit 24 to detect objects within a range that does not overlap with the outer edge of the captured image using a first detection model that has been machine-learned from the entire image of the object, and uses the second detection unit 26 to detect objects within a range that includes the inside of the outer edge of the captured image and overlaps with the outer edge using a second detection model that has been machine-learned from a partial image of the object.

[0022] The model used for machine learning may include an input corresponding to the image size (number of pixels) of the input image, an output that outputs a recognition score, and an intermediate layer that connects the input and the output. The intermediate layer may include a convolutional layer, a pooling layer, a fully connected layer, etc. The intermediate layer may have a multi-layer structure and may be configured to enable so-called deep learning. The model used for machine learning may be constructed using a convolutional neural network (CNN). Note that the model used for machine learning is not limited to the above, and any machine learning model may be used.

[0023] 2 is a diagram schematically illustrating an example of a captured image 50 including objects 54 (54a to 54e). An outer edge 52 of the captured image 50 corresponds to the angle of view of the camera 20. The outer edge 52 has a left edge 52a, a right edge 52b, an upper edge 52c, and a lower edge 52d. In this specification, the directions of up, down, left, and right are set based on the angle of view of the camera 20, and refer to the upper side, lower side, left side, and right side from the viewpoint of the camera 20.

[0024] The captured image 50 includes an object 54e that does not overlap with the outer edge 52 but is located away from the outer edge 52, and objects 54a, 54b, 54c, and 54d that are located in an area that overlaps with the outer edge 52. The entire image of object 54e is included in the captured image 50 and is included in an area that does not overlap with the outer edge. On the other hand, portions of objects 54a to 54d are included in the captured image 50, and the remaining portions are outside the angle of view and are not included in the captured image 50. In other words, objects 54a to 54d are included in an area that overlaps with the outer edge.

[0025] In the captured image 50, the left portion of object 54a is outside the angle of view and is not included in the captured image 50, while the right portion is included in the captured image 50. The right portion of object 54b is outside the angle of view and is not included in the captured image 50, while the left portion is included in the captured image 50. The upper portion of object 54c is outside the angle of view and is not included in the captured image 50, while the lower portion is included in the captured image 50. The lower portion of object 54d is outside the angle of view and is not included in the captured image 50, while the upper portion is included in the captured image 50.

[0026] 3 is a diagram schematically illustrating an example of detection regions 60a-60e when an object 54 is detected in the captured video 50. In FIG. 3, the detection regions 60a-60e are indicated by rectangular frames with dashed lines. The shape of the detection regions 60a-60e, for example, the aspect ratio, corresponds to the input size of the detection model used by the object detection unit 14, and is, for example, approximately 2:1.

[0027] The first detection unit 24 detects an object (e.g., object 54e) included in the captured image using a first detection model for a detection window that scans inside the outer edge 52 of the captured image 50. The first detection unit 24 detects an object included in a range that includes the inside of the outer edge 52 of the captured image 50 but does not include the outside of the outer edge 52. When scanning the entire captured image 50 while changing the position and size of the detection window, for example, the first detection unit 24 detects an object using the first detection model for a detection window that scans inside the outer edge 52 of the captured image 50.

[0028] The second detection unit 26 detects objects (e.g., objects 54a to 54d) included in the captured image using a second detection model for a detection window that scans a range overlapping with the outer edge 52 of the captured image 50. The second detection unit 26 detects objects included in a range that includes both the inside and outside of the outer edge 52 of the captured image 50. For example, when scanning the entire captured image 50 while changing the position and size of the detection window, the second detection unit 26 detects objects using the second detection model for the detection window that scans a range that includes the outer edge 52 of the captured image 50.

[0029] The second detection unit 26 detects objects using the second detection model in a range overlapping with at least one of the left edge 52a, right edge 52b, top edge 52c, and bottom edge 52d of the captured image 50. For example, when an object 54a is detected, the detection area 60a includes an inner area 62a adjacent to the right side of the left edge 52a and an outer area 64a adjacent to the left side of the left edge 52a. For example, when an object 54 is detected, the detection area 60b includes an inner area 62b adjacent to the left side of the right edge 52b and an outer area 64b adjacent to the right side of the right edge 52b. For example, when an object 54c is detected, the detection area 60c includes an inner area 62c adjacent to the bottom side of the top edge 52c and an outer area 64c adjacent to the top side of the top edge 52c. For example, when an object 54d is detected, the detection region 60d includes an inner region 62d adjacent to the upper side of the lower edge 52d and an outer region 64d adjacent to the lower side of the lower edge 52d.

[0030] The second detection unit 26 can use a plurality of detection models for detecting objects that exist at positions overlapping the left edge 52 a, the right edge 52 b, the upper edge 52 c, and the lower edge 52 d of the captured image 50. The second detection unit 26 can use, for example, at least one of a left edge detection model, a right edge detection model, an upper edge detection model, and a lower edge detection model as the second detection model.

[0031] 4(a) to 4(e) are diagrams schematically illustrating an example of input images used in machine learning of a detection model for detecting a person, as an example. FIG. 4(a) is a right partial image 66a of an object, which is used in machine learning of a left edge detection model. The right partial image 66a is an image that includes the right part of the object but not the left part. FIG. 4(b) is a left partial image 66b of an object, which is used in machine learning of a right edge detection model. The left partial image 66b is an image that includes the left part of the object but not the right part. FIG. 4(c) is a bottom partial image 66c of an object, which is used in machine learning of an upper edge detection model. The bottom partial image 66c is an image that includes the bottom part of the object but not the top part. FIG. 4(d) is an top partial image 66d of an object, which is used in machine learning of a lower edge detection model. The top partial image 66d is an image that includes the top part of the object but not the bottom part. 4E is an entire image 66e of the object, which is used for machine learning of the first detection model. The entire image 66e includes an entire image of the object.

[0032] 4A to 4D include marginal portions 68a to 68d that do not include any object. The marginal portions 68a to 68d are set so that the image size (e.g., aspect ratio) of the partial images 66a to 66d matches the image size (e.g., aspect ratio) of the entire image 66e. The brightness values ​​of the marginal portions 68a to 68d are set to a brightness value different from that of the object, for example, a brightness value equivalent to that of the background of the object. Setting the marginal portions 68a to 68d can improve the recognition accuracy in the second detection area that is set to include the area outside the outer edge 52 of the captured image 50.

[0033] Instead of using both the left edge detection model and the right edge detection model, the second detection unit 26 may use either one of the detection models to detect the left edge object 56a and the right edge object 56b. For example, the second detection unit 26 may use the left edge detection model to detect the right edge object 56b. The second detection unit 26 can detect the right edge object 56b by left-right inverting the image cut out in the right edge detection region 60b and then inputting it into the left edge detection model. Conversely, the second detection unit 26 may use the right edge detection model to detect the left edge object 56a. The second detection unit 26 can detect the left edge object 56a by left-right inverting the image cut out in the left edge detection region 60a and then inputting it into the right edge detection model.

[0034] Returning to FIG. 1 , the distance calculation unit 16 calculates distance information of the object detected by the object detection unit 14. The distance calculation unit 16 calculates the distance to the object using, for example, the bottom edge position of the object included in the captured image 50. The bottom edge position of the object corresponds to the ground contact position of the object and corresponds to the bottom edge positions 70a to 70e (see FIG. 3 ) of the detection areas 60a to 60e in which the object is detected. The distance calculation unit 16 may calculate the direction of the object using the bottom edge position of the object included in the captured image 50. The distance calculation unit 16 may calculate the distance and direction from the camera 20 as the distance information of the object. The distance calculation unit 16 may calculate the position coordinates of the object using, for example, a coordinate system based on the position of the smart pole on which the camera 20 is installed.

[0035] The distance calculation unit 16 can calculate the distance to the object by using, for example, the correlation between the distance from the camera 20 to the object and the bottom position of the object in the captured video 50. The correlation between the distance and the bottom position may be calculated based on, for example, the angle of view of the camera 20, or may be actually measured around the smart pole on which the recognition processing device 10 is installed. The distance calculation unit 16 can calculate the distance using a table or formula that indicates the correlation between the distance and the bottom position.

[0036] When an object 54 is detected at a position overlapping the lower edge 52d of the captured image 50, the distance calculation unit 16 calculates the distance to the object 54 using the lower end position of the detection area in which the object 54 was detected using the second detection model. For example, when the object 54d is detected using the second detection model, the distance calculation unit 16 calculates the distance to the object 54d using the lower end position 70d of the detection area 60d in which the object 54 was detected. The lower end position 70d of the detection area 60d is located below the lower edge 52d of the captured image 50 and is therefore outside the vertical range of the captured image 50, i.e., outside the range of the angle of view of the camera 20. By using such a lower end position located outside the range of the captured image 50, the distance to an object, such as the object 54d, that is included in the area overlapping the lower edge 52d of the captured image 50 can be more appropriately calculated. In addition, objects that are not included in the position overlapping the lower edge 52d, i.e., the lower end positions 70a, 70b, 70c, and 70e of objects 54a, 54b, 54c, and 54e, are within the vertical range of the captured image 50, i.e., within the range of the angle of view of camera 20.

[0037] The output control unit 18 causes the output device 22 to output object information related to objects detected by the object detection unit 14. The object information may include, for example, information regarding whether or not an object has been detected by the object detection unit 14, the number of objects detected by the object detection unit 14, and the positions and distances of the detected objects. The output device 22 may be a communication device, or may be a wireless communication device that outputs object information such as the positions and distances of objects via road-to-vehicle communication or vehicle-to-vehicle communication.

[0038] 5 is a flowchart showing an example of the flow of the recognition processing method according to the first embodiment. The image acquisition unit 12 acquires captured image captured by the camera 20 (step S10). The object detection unit 14 starts scanning the captured image using a detection window and determines whether the detection window is located inside the outer edge of the captured image, in other words, whether the detection window is located in a range that includes the inside of the outer edge of the captured image but does not overlap the outer edge (step S12). The recognition processing in this embodiment is premised on the detection window scanning the captured image over an area wider than the outer edge of the captured image.

[0039] If it is determined that the detection window is located inside the outer edge of the captured image (Yes in step S12), the object detection unit 14 detects the object using the first detection model for the detection window (step S14).If the detection window is not located inside the outer edge of the captured image, that is, if the detection window is located in a range that overlaps the outside of the captured image (No in step S12), the object detection unit 14 detects the object using the second detection model (step S16).

[0040] Next, the object detection unit 14 determines whether an object has been detected (step S18). In step S14, the object detection unit 14 determines that an object has been detected if the recognition score using the first detection model is equal to or greater than a predetermined threshold. In step S16, the object detection unit 14 determines that an object has been detected if the recognition score using the second detection model is equal to or greater than a predetermined threshold.

[0041] If the object detection unit 14 detects an object (Yes in step S18), the distance calculation unit 16 calculates distance information of the object using the bottom edge position of the detection area in which the object is detected (step S20). The output control unit 18 outputs object information regarding the detected object (step S22). If the object detection unit 14 does not detect an object (No in step S18), the processes of steps S20 and S22 can be skipped.

[0042] FIG. 6 is a flowchart showing an example of the processing flow of step S16 in FIG. 5 . FIG. 6 illustrates an example in which the second detection unit 26 uses different second detection models for detecting an object depending on the position of the detection window. As described above, the second detection unit 26 uses a left edge detection model, a right edge detection model, an upper edge detection model, and a lower edge detection model as examples of the second detection model. If the object detection unit 14 determines that the detection window is located at a position overlapping the left edge of the captured image (Yes in step S30), it detects the object using the left edge detection model (step S32). If the object detection unit 14 determines that the detection area is not located at a position overlapping the left edge of the captured image (No in step S30) but is located at a position overlapping the right edge of the captured image (Yes in step S34), it detects the object using the right edge detection model (step S36). If the object detection unit 14 determines that the detection area is not located at a position overlapping either the left or right edge of the captured image (No in step S34) but is located at a position overlapping the upper edge of the captured image (Yes in step S38), it detects the object using the upper edge detection model (step S40).If the object detection unit 14 determines that the detection area is not located at a position overlapping either the left, right, or upper edge of the captured image (No in step S38), it detects the object using the lower edge detection model (step S42).

[0043] According to this embodiment, by using the second detection model, it is possible to improve the detection accuracy of an object that is located on the outer edge of the captured image and therefore not included in the captured image in its entirety. For example, when an object moves in a direction approaching the camera 20, it moves from the upper side to the lower side of the lower edge of the captured image, and as it approaches the camera 20, it may become difficult to capture the entire image of the object. If the angle of view of the camera 20 is reduced, it is possible to capture the entire image of an object located near the camera 20, but it becomes impossible to capture an image of an object located far from the camera 20. According to this embodiment, the detection accuracy of an object located on the outer edge of the captured image is improved, and therefore the range in which objects can be detected using a single camera 20 can be expanded.

[0044] According to this embodiment, even if the bottom edge of the object is not included in the angle of view of the captured image because it is located on the bottom edge of the captured image, the bottom edge of the object can be identified, and as a result, the distance to the object located near the camera 20 can be calculated more appropriately.

[0045] Second Embodiment Fig. 7 is a block diagram schematically showing the functional configuration of a recognition processing device 10A according to a second embodiment. The recognition processing device 10A according to the second embodiment differs from the recognition processing device 10 according to the first embodiment in that it further includes an object tracking unit 28 and a bottom edge estimation unit 30. The following description of the second embodiment will focus on the differences from the first embodiment, and will omit descriptions of commonalities as appropriate.

[0046] The recognition processing device 10A includes an image acquisition unit 12, an object detection unit 14A, an object tracking unit 28, a lower end estimation unit 30, a distance calculation unit 16A, and an output control unit 18. The image acquisition unit 12 and the output control unit 18 may be configured similarly to those in the first embodiment. The object detection unit 14A includes a first detection unit 24A, but differs from the first embodiment in that it does not include a second detection unit 26.

[0047] The first detection unit 24A detects an object using a first detection model that has been machine-learned from an entire image of the object. The first detection unit 24A detects an object included in the captured video using the first detection model. The first detection unit 24A detects an object located near the center of the captured video using the first detection model, and also detects an object located on the outer edge of the captured video using the first detection model.

[0048] The object tracking unit 28 tracks the object detected by the object detection unit 14A. The object tracking unit 28 tracks the object across multiple frames that make up the captured video and identifies the movement of the object across the multiple frames. The object tracking unit 28 identifies, for example, the amount and direction of movement of the object.

[0049] Figures 8(a) to (c) are schematic diagrams showing an example of an object being tracked across multiple frames that make up the captured image, and show the object 54f moving in a direction toward the lower edge 52d of the captured image, in other words, the object 54f approaching the camera 20.

[0050] 8A shows the captured image 50a when the object 54f is located above the lower edge 52d. Also shown is a detection area 60f when the object 54f is detected in the captured image 50a. It is determined that the bottom edge position of the object 54f at this time coincides with the bottom edge position 70f of the detection area 60f. The object tracking unit 28 is tracking the object 54f.

[0051] FIG. 8(b) shows a captured image 50b one or several frames after FIG. 8(a). In the captured image 50b, the lower end position 70f of the detection area 60f when the object 54f is detected coincides with the lower edge 52d. In the example of FIG. 8(b), the position of the object 54f in FIG. 8(a) is indicated by a dashed line for ease of explanation. However, the object 54f indicated by the dashed line is not actually captured in the captured image 50b. The object tracking unit 28 determines the amount and direction of movement, as indicated by the arrow 78b, based on the difference between the position of the object 54f in a previous frame and the position of the object 54f in the current frame. The object tracking unit 28 can determine the movement of the object 54f based on the change in the position of a specific part (e.g., the head) of the object 54f.

[0052] In Figure 8(b), the object tracking unit 28 detects that the object 54f is moving in a direction toward the lower edge 52d of the captured image. Therefore, the object 54f detected in Figure 8(a) and the object 54f detected in Figure 8(b) are determined to be the same object. Furthermore, because the arrangement of the detection region 60f for the object 54f detected in Figure 8(a) is the same as the arrangement of the detection region 60f for the object 54f detected in Figure 8(b), it is determined that the entire object 54f detected in Figure 8(b) has been detected. Therefore, it is determined that the bottom edge position of the object 54f detected in Figure 8(b) coincides with the bottom edge position 70f of the detection region 60f shown in Figure 8(b).

[0053] FIG. 8(c) shows a captured image 50c one or several frames after FIG. 8(b). In the captured image 50c, the bottom end of the object 54f is located below the lower edge 52d, and the lower portion of the object 54f is outside the field of view of the camera 20. In the example of FIG. 8(c), the position of the object 54f in FIG. 8(b) is indicated by a dashed line for ease of explanation, but the object 54f indicated by the dashed line is not actually captured in the captured image 50c. The object tracking unit 28 determines the amount and direction of movement, as indicated by the arrow 78c, based on the difference between the position of the object 54f in the previous frame and the position of the object 54f in the current frame.

[0054] The bottom edge estimation unit 30 estimates the bottom edge position of the object. When an object located at the bottom edge of the captured image is detected, the bottom edge estimation unit 30 estimates the bottom edge position of the object that may be located below the bottom edge of the captured image. The bottom edge estimation unit 30 also estimates the bottom edge position of the object based on the size of the object above the bottom edge of the captured image.

[0055] In FIG. 8(c), the object tracking unit 28 detects that the object 54f is moving in a direction toward the lower edge 52d of the captured image. In FIG. 8(c), the object 54f is not entirely contained in the captured image 50c, so the recognition score based on the first detection model is low. However, because the object tracking unit 28 estimates that the object 54f is located at a position overlapping the lower edge 52d of the captured image, the first detection unit 24A determines that the detection area 60f shown in FIG. 8(c) is the detection area for the object 54f. Therefore, it is determined that the lower end position of the object 54f detected in FIG. 8(c) coincides with the lower end position 70f of the detection area 60f shown in FIG. 8(c).

[0056] The bottom end estimation unit 30 may estimate the bottom end position 70f at the time of Fig. 8(c) based on, for example, the bottom end position 70f at the time of Fig. 8(b) and the movement indicated by the arrow 78c of the object 54f tracked from the time of Fig. 8(b) to the time of Fig. 8(c). For example, as indicated by the arrow 80c, the bottom end position 70f at the time of Fig. 8(c) can be estimated by adding the amount of movement (movement vector) indicated by the arrow 78c to the bottom end position 70f at the time of Fig. 8(b).

[0057] In FIG. 8( c), because the entire image of the object 54f is not included in the captured image 50c, the detection area in the captured image 50c in which the object 54f is detected may be smaller in size in the vertical direction than the detection area 60f of the object 54f detected in FIG. 8( b). In this case, the detection area 60f of the object 54f may be estimated to be positioned overlapping the lower edge 52d of the captured image, as in the detection area 60f shown in FIG. 8( c). Specifically, the bottom edge estimation unit 30 estimates the bottom position of the object 54f based on the vertical size in which the object 54f shown in FIG. 8( c) is detected, i.e., the size above the lower edge 52d. For example, based on the position and size of the detection area 60f of the object 54f detected in Figure 8(a) or 8(b), the object tracking unit 28 tracks and detects the object 54f in Figure 8(c), and from the vertical size of the object 54f, the detection area 60f that includes the entire image of the object 54f in Figure 8(c) is estimated, thereby estimating the bottom position of the object 54f.

[0058] The distance calculation unit 16A calculates distance information of the object using the bottom end position of the object estimated by the bottom end estimation unit 30. The distance calculation unit 16A can calculate the distance information of the object using a method similar to that of the distance calculation unit 16 according to the first embodiment described above.

[0059] 9 is a flowchart showing an example of the flow of the recognition processing method according to the second embodiment. The image acquisition unit 12 acquires captured image captured by the camera 20 (step S50). The object detection unit 14A starts scanning the captured image using a detection window and starts detecting an object using a first detection model (step S52).

[0060] Next, the object detection unit 14A determines whether an object has been detected from the captured image captured by the camera 20 (step S54). If it is determined that an object has been detected in step S54 (Yes in step S54), the object tracking unit 28 tracks the object across multiple frames and identifies the object's movement (step S56). If the detection area when the tracked object was detected overlaps the bottom edge of the captured image (Yes in step S58), the bottom edge estimation unit 30 estimates the bottom edge position of the object based on the object's movement (step S60). If the detection area when the tracked object was detected does not overlap the bottom edge of the captured image (No in step S58), the bottom edge estimation unit 30 estimates the bottom edge position of the object based on the bottom edge position of the detection area (step S62).

[0061] After steps S60 and S62, distance calculation unit 16A calculates distance information of the object using the estimated lower end position of the object (step S64). Output control unit 18 outputs object information regarding the detected object (step S66). If it is not determined in step S54 that the object has been detected (No in step S54), steps S56 to S66 can be skipped.

[0062] According to this embodiment, even if the bottom position of the object is not included in the angle of view of the captured image because it is located at the bottom edge of the captured image, it is possible to estimate the bottom position of the object, and as a result, it is possible to more appropriately calculate the distance to the object located near the camera 20.

[0063] 10 is a block diagram schematically illustrating the functional configuration of a recognition processing device 10B according to a third embodiment. The recognition processing device 10B according to the third embodiment differs from the above-described embodiments in that, when an object is present at a position overlapping the lower edge of a captured image, the recognition processing device 10B detects the object by switching from the first detection model to the second detection model. The following description of the third embodiment will focus on the differences from the first and second embodiments, and will omit descriptions of common features as appropriate.

[0064] The recognition processing device 10B includes an image acquisition unit 12, an object detection unit 14B, a lower end estimation unit 30B, a distance calculation unit 16B, and an output control unit 18. The image acquisition unit 12 and the output control unit 18 can be configured similarly to the first or second embodiment. The object detection unit 14B includes a first detection unit 24B and a second detection unit 26B.

[0065] The first detection unit 24B can be configured in the same manner as the first detection unit 24A according to the second embodiment. The first detection unit 24B detects an object using a first detection model that is obtained by machine learning of an entire image of the object. The first detection unit 24B detects an object included in the captured image using the first detection model. The first detection unit 24B detects an object located near the center of the captured image using the first detection model, and also detects an object located on the outer edge of the captured image using the first detection model. The first detection unit 24B detects an object located on the lower edge of the captured image using the first detection model.

[0066] When the first detection unit 24B detects an object that is included in a position overlapping the lower edge of the captured image, the second detection unit 26B detects the object that is included in a position overlapping the lower edge of the captured image using a second detection model that has been machine-learned from an image of an upper part of the object. The second detection unit 26B uses the second detection model to detect an object that is included in a position overlapping the lower edge of the captured image, so that a part of the upper side of the object is included in the captured image, but whose lower side is outside the angle of view and is not included in the captured image.

[0067] 11(a) and 11(b) are diagrams schematically illustrating an example of a method for detecting an object included in a position overlapping the lower edge of a captured image. FIGS. 11(a) and 11(b) show a captured image 50c similar to FIG. 8(c) according to the second embodiment described above. FIG. 11(a) schematically illustrates a state in which an object 54g has been detected by the first detection unit 24B. The first detection unit 24B scans a detection window over the captured image 50c to detect the object using a first detection model. In FIG. 11(a), the captured image 50c does not include the entire object 54g, so the bottom edge position 70g of the detection area 60g when the object 54g is detected is different from the actual bottom edge position of the object 54g. FIG. 11(b) is a diagram schematically illustrating a state in which the object 54g has been detected by the second detection unit 26B. The second detection unit 26B scans the detection window of the captured image 50c over an area including the detection area in which the object was detected using the first detection model and its surrounding area, and detects the object using the second detection model. Figure 11(b) shows a state in which the object 54g is included in an area overlapping the lower edge of the captured image 50c, and a lower end position 70g of the detection area 60g is estimated when the object 54g is detected using the second detection model.

[0068] The bottom edge estimation unit 30B estimates the bottom edge position of the object. When the first detection unit 24B detects an object that is included in a range overlapping the lower edge of the captured image, the bottom edge estimation unit 30B determines the bottom edge position of the detection area detected by the second detection unit 26B for that object as the bottom edge position of the object. For example, in the case shown in FIG. 11( b), the bottom edge estimation unit 30B determines the bottom edge position 70g of the detection area 60g of the object 54g detected by the second detection unit 26B as the bottom edge position of the object 54g.

[0069] The distance calculation unit 16B calculates distance information of the object using the bottom end position of the object estimated by the bottom end estimation unit 30B. The distance calculation unit 16B can calculate the distance information of the object using a method similar to that of the distance calculation unit 16 according to the first embodiment described above.

[0070] 12 is a flowchart showing an example of the flow of the recognition processing method according to the third embodiment. The image acquisition unit 12 acquires captured image captured by the camera 20 (step S70). The object detection unit 14B starts scanning the captured image using a detection window and starts detecting an object using a first detection model (step S72).

[0071] Next, the object detection unit 14B determines whether an object has been detected from the captured image captured by the camera 20 (step S74). If it is determined in step S74 that an object has been detected (Yes in step S74), the object detection unit 14B determines whether the detection area of ​​the object detected in step S74 is included in a range overlapping the lower edge of the captured image (step S76). If it is determined in step S76 that the detection area of ​​the detected object is included in a range overlapping the lower edge of the captured image (Yes in step S76), the object detection unit 14B detects the object using a second detection model (step S78). Furthermore, the bottom edge estimation unit 30B estimates the bottom edge position of the object based on the bottom edge position of the detection area of ​​the object detected by the second detection model in step S78 (step S80). In step S76, if it is not determined that the detection area of ​​the detected object is included in the range overlapping the lower edge of the captured image (No in step S76), the lower end estimation unit 30B estimates the lower end position of the object based on the lower end position of the detection area of ​​the object detected by the first detection model (step S82).

[0072] After steps S80 and S82, distance calculation unit 16B calculates distance information of the object using the estimated lower end position of the object (step S84). Output control unit 18 outputs object information regarding the detected object (step S86). If it is determined in step S74 that the object is not detected (No in step S74), steps S76 to S86 can be skipped.

[0073] According to this embodiment, when the bottom position of an object is not included in the angle of view of the captured image because it is located at the bottom edge of the captured image, the second detection model is used to detect the object, thereby estimating the bottom position of the object. As a result, distance information of objects located near the camera 20 can be more appropriately calculated.

[0074] The present invention has been described above with reference to the above-mentioned embodiments, but the present invention is not limited to the above-mentioned embodiments, and appropriate combinations or substitutions of the configurations shown in the embodiments are also included in the present invention.

[0075] Several aspects of the present disclosure are described below.

[0076] A first aspect of the present disclosure is a recognition processing device including an image acquisition unit that acquires captured image, and an object detection unit that detects an object located away from the outer edge of the captured image using a first detection model that has been machine-learned on an entire image of the object, and that detects an object located at the outer edge of the captured image using a second detection model that has been machine-learned on a partial image of the object.

[0077] A second aspect of the present disclosure is a recognition processing method in which a recognition processing device executes the steps of acquiring an image of an object, detecting an object located away from the outer edge of the image of the object using a first detection model that has been machine-learned on an entire image of the object, and detecting an object located at the outer edge of the image of the object using a second detection model that has been machine-learned on a partial image of the object.

[0078] A third aspect of the present disclosure is a program configured to cause a processor to execute the following functions: acquiring an image; detecting an object located away from the outer edge of the image using a first detection model that has been machine-learned on an entire image of the object; and detecting an object located at the outer edge of the image using a second detection model that has been machine-learned on a partial image of the object. The third aspect of the present disclosure is a program configured to cause a processor to execute the following functions; acquiring an image; detecting an object located away from the outer edge of the image using a first detection model that has been machine-learned on an entire image of the object; and detecting an object located at the outer edge of the image.

[0079] A fourth aspect of the present disclosure is a recognition processing device including: an image acquisition unit that acquires captured video; an object detection unit that detects an object included in the captured video using a detection model that has been machine-learned from an image of the object; a bottom edge estimation unit that, when the object detection unit detects an object included in a range that overlaps the bottom edge of the captured video, estimates a bottom edge position of the object that may be located below the bottom edge of the captured video; and a distance calculation unit that calculates distance information of the object using the bottom edge position estimated by the bottom edge estimation unit.

[0080] A fifth aspect of the present disclosure is a recognition processing method in which a recognition processing device executes the following steps: acquiring an image of the object; detecting an object included in the image of the object using a detection model that has been machine-learned from an image of the object; if an object included in a range overlapping the lower edge of the image of the object is detected, estimating a lower end position of the object that may be located below the lower edge of the image of the object; and calculating distance information of the object using the estimated lower end position.

[0081] A sixth aspect of the present disclosure is a program configured to cause a processor to execute the following functions: acquiring an image of the object; detecting an object included in the image of the object using a detection model that has been machine-learned from an image of the object; if an object included in a range that overlaps the lower edge of the image of the object is detected, estimating the lower end position of the object, which may be located below the lower edge of the image of the object; and calculating distance information of the object using the estimated lower end position.

[0082] According to an aspect of the present invention, a technique for more appropriately detecting an object in image recognition processing can be provided.

[0083] 10...recognition processing device, 12...image acquisition unit, 14...object detection unit, 16...distance calculation unit, 50...captured image, 52...outer edge, 52d...lower edge.

Claims

1. A recognition processing device comprising: an image acquisition unit that acquires an image; an object detection unit that detects an object included in the image using a detection model that has been machine-learned from an image of the object; a bottom edge estimation unit that, when an object included in an area overlapping the bottom edge of the image is detected by the object detection unit, estimates a bottom edge position of the object that may be located below the bottom edge of the image; and a distance calculation unit that calculates distance information of the object using the bottom edge position estimated by the bottom edge estimation unit.

2. The recognition processing device according to claim 1, further comprising an object tracking unit that tracks the object detected by the object detection unit, wherein the bottom end estimation unit estimates the bottom end position of the object that may be located below the bottom edge of the captured image based on the movement of the object tracked by the object tracking unit when the bottom end position of the object moves toward a position below the bottom edge of the captured image.

3. The recognition processing device according to claim 2, wherein the lower end estimation unit estimates a lower end position of the object based on a size of the object detected by the object detection unit above a lower edge of the captured image.

4. The recognition processing device described in claim 1, wherein the object detection unit detects an object included in the captured image using a first detection model that has been machine-learned from an entire image of the object to be detected, and when the object detection unit detects an object included in an area overlapping the lower edge of the captured image using the first detection model, the object detection unit detects the object included in an area overlapping the lower edge of the captured image using a second detection model that has been machine-learned from an upper partial image of the object, and the bottom edge estimation unit estimates the bottom edge position of the object using the bottom edge position of the detection area of ​​the object detected using the second detection model.

5. The recognition processing device described in claim 4, wherein when the object detection unit detects, using the first detection model, an object that is included in a range overlapping the lower edge of the captured image, such that a portion of the upper side of the object is included in the captured image and a portion of the lower side of the object is outside the angle of view and is not included in the captured image, the object detection unit detects, using the second detection model, an object that is included in a range overlapping the lower edge of the captured image.

6. A recognition processing method in which a recognition processing device executes the following steps: acquiring an image of the captured video; detecting an object included in the captured video using a detection model based on machine learning of an image of the object; when an object included in an area overlapping the lower edge of the captured video is detected, estimating a bottom end position of the object, which may be located below the lower edge of the captured video; and calculating distance information of the object using the estimated bottom end position.

7. A program configured to cause a processor to execute the following functions: a function for acquiring an image of the object; a function for detecting an object included in the image of the object using a detection model based on machine learning of an image of the object; a function for estimating a bottom end position of the object, which may be located below the bottom edge of the image of the object, when an object included in an area overlapping the bottom edge of the image of the object is detected; and a function for calculating distance information of the object using the estimated bottom end position.

Images