Recognition processing device and recognition processing method

Abstract

To more appropriately detect a detection object in image recognition processing.SOLUTION: A recognition processing device 10A includes: a video image acquisition part 12 for acquiring a picked-up video image; an object detection part 14A for detecting an object included in the picked-up video image, using a detection model obtained by machine learning the image of the object; a lower end estimation part 30 for estimating a lower end position of the object which may be positioned on the side lower than the lower edge of the picked-up video image, when the object included in a region overlapped with the lower edge of the picked-up video image is detected by the objection detection part 14A; and a distance calculation part 16A for calculating distance information on the object, using a lower end position estimated by the lower end estimation part 30.SELECTED DRAWING: Figure 7

Description

[Technical field]

[0001] The present invention relates to a recognition processing device and a recognition processing method. [Background technology]

[0002] There is known a technique for detecting objects such as pedestrians from an image captured around a vehicle by using an image recognition process such as pattern matching (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2022-139374 A Summary of the Invention [Problem to be solved by the invention]

[0004] When an object is located near the outer edge of an image captured by a camera, the object may not be detected properly because the entire image of the object is not included in the image.

[0005] The present invention has been made in consideration of the above circumstances, and has an object to provide a technique for more appropriately detecting an object in image recognition processing. [Means for solving the problem]

[0006] A recognition processing device of one embodiment of the present invention includes an image acquisition unit that acquires an image of the object, an object detection unit that detects an object included in the image of the object using a detection model that has been machine-learned based on an image of the object, a bottom end estimation unit that, when an object included in an area overlapping the bottom edge of the image of the object is detected by the object detection unit, estimates a bottom end position of the object that may be located below the bottom edge of the image of the object, and a distance calculation unit that calculates distance information of the object using the bottom end position estimated by the bottom end estimation unit.

[0007] Another aspect of the present invention is a recognition processing method, in which a recognition processing device executes the steps of acquiring an image of the object, detecting an object contained in the image of the object using a detection model that has been machine-learned from an image of the object, and, if an object contained in an area overlapping the lower edge of the image of the object is detected, estimating a lower end position of the object that may be located below the lower edge of the image of the object, and calculating distance information of the object using the estimated lower end position. Effect of the Invention

[0008] According to an aspect of the present invention, a technique for more appropriately detecting an object in image recognition processing can be provided. [Brief description of the drawings]

[0009] [Figure 1] 1 is a block diagram illustrating a functional configuration of a recognition processing device according to a first embodiment. [Diagram 2] FIG. 2 is a diagram illustrating an example of a captured image including a target object. [Diagram 3] 10 is a diagram illustrating an example of a detection area set in a captured image. FIG. [Figure 4] 4(a) to (e) are diagrams each showing a schematic example of an input image used in machine learning of a detection model. [Diagram 5] 5 is a flowchart showing an example of the flow of a recognition processing method according to the first embodiment. [Figure 6] 6 is a flowchart showing an example of the process flow of step S16 in FIG. 5. [Figure 7] FIG. 11 is a block diagram illustrating a functional configuration of a recognition processing device according to a second embodiment. [Figure 8] 8(a) to 8(c) are diagrams showing schematic diagrams of an example of an object being tracked across a plurality of frames constituting a captured video. [Figure 9] 10 is a flowchart showing an example of the flow of a recognition processing method according to a second embodiment. [Figure 10] FIG. 11 is a block diagram illustrating a functional configuration of a recognition processing device according to a third embodiment. [Figure 11] 11(a) and (b) are diagrams illustrating an example of a method for detecting an object located at the lower edge of a captured image. [Figure 12] 13 is a flowchart showing an example of the flow of a recognition processing method according to the third embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. Specific numerical values ​​and the like shown in the embodiment are merely examples for facilitating understanding of the invention, and do not limit the present invention unless otherwise specified. In the drawings, elements that are not directly related to the present invention are omitted.

[0011] (First embodiment) 1 is a block diagram showing a schematic functional configuration of a recognition processing device 10 according to a first embodiment. The recognition processing device 10 includes an image acquisition unit 12 and an object detection unit 14. The recognition processing device 10 may further include a distance calculation unit 16 and an output control unit 18. The recognition processing device 10 acquires captured images that may include objects such as pedestrians present in the surrounding area, and detects the objects included in the captured images.

[0012] In this embodiment, the recognition processing device 10 is installed on a smart pole. The smart pole is installed on a street, for example, and includes an antenna and a communication device for providing wireless communication functions, lighting equipment for illuminating the street, and a camera for photographing vehicles and pedestrians passing on the road. The recognition processing device 10 is fixed at a predetermined location. The recognition processing device 10 may be mounted on a moving object, or on an air vehicle such as a vehicle or a drone.

[0013] The recognition processing device 10 can detect any object. In the embodiment of the present invention, the object is a person such as a pedestrian.

[0014] Each functional block shown in this embodiment can be realized, for example, by cooperation of hardware and software. The hardware of the recognition processing device 10 is realized by elements and mechanical devices such as a processor such as a central processing unit (CPU) or a graphics processing unit (GPU) of a computer, and memories such as a read only memory (ROM) or a random access memory (RAM). The software of the recognition processing device 10 is realized by a computer program or the like.

[0015] The image acquisition unit 12 acquires an image captured by the camera 20 (also referred to as a captured image). The camera 20 is provided on the smart pole and captures an image of the surroundings of the smart pole. The camera 20 is provided, for example, above the smart pole and captures an image with an angle of view looking down on the ground on which the smart pole is installed. The camera 20 captures visible light to generate a color image or a monochrome image. The camera 20 may be an infrared camera, and may capture infrared light to generate a thermal image. The image captured by the camera 20 is a moving image, for example, at 30 frames per second or 60 frames per second.

[0016] The object detection unit 14 detects an object from the image acquired by the image acquisition unit 12. In other words, the object detection unit 14 detects an area (hereinafter, referred to as a detection area) in which an object is included in the image acquired by the image acquisition unit 12. The object detection unit 14 scans the detection window while referring to one or more detection models for detecting an object for each frame of the image acquired by the image acquisition unit 12, and calculates a recognition score indicating the possibility that an object is included for each detection window. The recognition score is calculated, for example, in the range of 0.0 to 1.0, and the recognition score becomes a larger value (i.e., a value closer to 1.0) as the possibility that an object is included in the image in the detection window increases, and the recognition score becomes a smaller value (i.e., a value closer to 0.0) as the possibility that an object is included in the image in the detection window decreases. When the recognition score is equal to or greater than a predetermined threshold value such as 0.8, the object detection unit 14 detects the object by determining that an object is included in the detection window.

[0017] The object detection unit 14 includes a first detection unit 24 and a second detection unit .

[0018] The first detection unit 24 detects the object using a first detection model that is machine-learned from an entire image of a person, which is the object. An entire image of a person is an image that includes the entire body of the person. The first detection unit 24 detects an object that is included in an area that does not overlap with the outer edge inside the captured image, such as near the center of the captured image. For example, the first detection unit 24 detects an object that is included in its entirety in the captured image.

[0019] The second detection unit 26 detects the object using a second detection model that is machine-learned from a partial image of a person, which is the object. A partial image of a person is an image that includes about half of the whole body of a person. The second detection unit 26 detects an object that is included in a range that includes the inside of the outer edge of the captured image, such as near the outer periphery of the captured image, and overlaps with the outer edge. The second detection unit 26 detects an object that is included in the captured image when part of the object is included, and the remaining part of the object is outside the angle of view and is not included in the captured image.

[0020] For this reason, the object detection unit 14 uses the first detection unit 24 to detect objects included in a range that does not overlap the outer edge of the captured image using a first detection model that has been machine-learned from the entire image of the object, and uses the second detection unit 26 to detect objects included in a range that includes the inside of the outer edge of the captured image and overlaps with the outer edge using a second detection model that has been machine-learned from a partial image of the object.

[0021] The model used for machine learning may include an input corresponding to the image size (number of pixels) of the input image, an output that outputs a recognition score, and an intermediate layer that connects the input and the output. The intermediate layer may include a convolutional layer, a pooling layer, a fully connected layer, etc. The intermediate layer may have a multi-layer structure, and may be configured to enable so-called deep learning. The model used for machine learning may be constructed using a convolutional neural network (CNN). Note that the model used for machine learning is not limited to the above, and any machine learning model may be used.

[0022] 2 is a diagram showing a schematic diagram of an example of a captured image 50 including objects 54 (54a to 54e). An outer edge 52 of the captured image 50 corresponds to the angle of view of the camera 20. The outer edge 52 has a left edge 52a, a right edge 52b, an upper edge 52c, and a lower edge 52d. In this specification, the directions of up, down, left, and right are set based on the angle of view of the camera 20, and refer to the upper side, lower side, left side, and right side from the viewpoint of the camera 20.

[0023] The captured image 50 includes an object 54e that does not overlap with the outer edge 52 and is located away from the outer edge 52, and objects 54a, 54b, 54c, and 54d that are located in a range that overlaps with the outer edge 52. The entire image of the object 54e is included in the captured image 50 and is included in a range that does not overlap with the outer edge. On the other hand, parts of the objects 54a to 54d are included in the captured image 50, and the remaining parts are outside the angle of view and are not included in the captured image 50. In other words, the objects 54a to 54d are included in a range that overlaps with the outer edge.

[0024] In the captured image 50, the left side of the object 54a is outside the angle of view and is not included in the captured image 50, while the right side is included in the captured image 50. The right side of the object 54b is outside the angle of view and is not included in the captured image 50, while the left side is included in the captured image 50. The upper side of the object 54c is outside the angle of view and is not included in the captured image 50, while the lower side is included in the captured image 50. The lower side of the object 54d is outside the angle of view and is not included in the captured image 50, while the upper side is included in the captured image 50.

[0025] Fig. 3 is a diagram showing an example of detection regions 60a-60e when an object 54 is detected in a captured image 50. In Fig. 3, the detection regions 60a-60e are indicated by rectangular frames drawn with dashed lines. The shape of the detection regions 60a-60e, for example, the aspect ratio, corresponds to the input size of the detection model used by the object detection unit 14, and is, for example, about 2:1.

[0026] The first detection unit 24 detects an object (e.g., object 54e) included in the captured image using a first detection model for a detection window that scans inside the outer edge 52 of the captured image 50. The first detection unit 24 detects an object included in a range that includes the inside of the outer edge 52 of the captured image 50 but does not include the outside of the outer edge 52. When scanning the entire captured image 50 while changing the position and size of the detection window, for example, the first detection unit 24 detects an object using the first detection model for a detection window that scans inside the outer edge 52 of the captured image 50.

[0027] The second detection unit 26 detects objects (e.g., objects 54a to 54d) included in the captured image using the second detection model for a detection window that scans a range overlapping with the outer edge 52 of the captured image 50. The second detection unit 26 detects objects included in a range including the inside and outside of the outer edge 52 of the captured image 50. When scanning the entire captured image 50 while changing the position and size of the detection window, for example, the second detection unit 26 detects objects using the second detection model for a detection window that scans a range including the outer edge 52 of the captured image 50.

[0028] The second detection unit 26 detects an object using the second detection model in a range overlapping at least one of the left edge 52a, the right edge 52b, the upper edge 52c, and the lower edge 52d of the captured image 50. For example, the detection area 60a when the object 54a is detected includes an inner area 62a adjacent to the right side of the left edge 52a and an outer area 64a adjacent to the left side of the left edge 52a. For example, the detection area 60b when the object 54 is detected includes an inner area 62b adjacent to the left side of the right edge 52b and an outer area 64b adjacent to the right side of the right edge 52b. For example, the detection area 60c when the object 54c is detected includes an inner area 62c adjacent to the lower side of the upper edge 52c and an outer area 64c adjacent to the upper side of the upper edge 52c. For example, when an object 54d is detected, the detection region 60d includes an inner region 62d adjacent to the upper side of the lower edge 52d and an outer region 64d adjacent to the lower side of the lower edge 52d.

[0029] The second detection unit 26 can use a plurality of detection models for detecting objects existing at positions overlapping the left edge 52a, the right edge 52b, the upper edge 52c, and the lower edge 52d of the captured image 50. The second detection unit 26 can use, for example, at least one of a left edge detection model, a right edge detection model, an upper edge detection model, and a lower edge detection model as the second detection model.

[0030] 4(a) to 4(e) are diagrams that show an example of an input image used for machine learning of a detection model for detecting a person as an example. FIG. 4(a) is a right partial image 66a of an object, which is used for machine learning of a left edge detection model. The right partial image 66a is an image that includes the right part of the object but does not include the left part. FIG. 4(b) is a left partial image 66b of an object, which is used for machine learning of a right edge detection model. The left partial image 66b is an image that includes the left part of the object but does not include the right part. FIG. 4(c) is a lower partial image 66c of an object, which is used for machine learning of an upper edge detection model. The lower partial image 66c is an image that includes the lower part of the object but does not include the upper part. FIG. 4(d) is an upper partial image 66d of an object, which is used for machine learning of a lower edge detection model. The upper partial image 66d is an image that includes the upper part of the object but does not include the lower part. 4(e) is an entire image 66e of the object, which is used for machine learning of the first detection model. The entire image 66e includes an entire image of the object.

[0031] The partial images 66a to 66d shown in Fig. 4(a) to (d) include blank areas 68a to 68d that do not include an object. The blank areas 68a to 68d are set so that the image size (e.g., aspect ratio) of the partial images 66a to 66d matches the image size (e.g., aspect ratio) of the entire image 66e. The brightness value of the blank areas 68a to 68d is set to a brightness value different from that of the object, for example, a brightness value equivalent to that of the background of the object. Setting the blank areas 68a to 68d can improve the recognition accuracy in the second detection area that is set to include the outside of the outer edge 52 of the captured image 50.

[0032] The second detection unit 26 may detect the left edge object 56a and the right edge object 56b using either one of the detection models instead of using both the left edge detection model and the right edge detection model. For example, the second detection unit 26 may use the left edge detection model to detect the right edge object 56b. The second detection unit 26 can detect the right edge object 56b by inverting the image cut out in the right edge detection region 60b left and right and then inputting it to the left edge detection model. Conversely, the second detection unit 26 may use the right edge detection model to detect the left edge object 56a. The second detection unit 26 can detect the left edge object 56a by inverting the image cut out in the left edge detection region 60a left and right and then inputting it to the right edge detection model.

[0033] Returning to FIG. 1, the distance calculation unit 16 calculates distance information of the object detected by the object detection unit 14. The distance calculation unit 16 calculates the distance to the object using, for example, the bottom end position of the object included in the captured image 50. The bottom end position of the object corresponds to the grounding position of the object and corresponds to the bottom end positions 70a to 70e (see FIG. 3) of the detection areas 60a to 60e in which the object is detected. The distance calculation unit 16 may calculate the direction of the object using the bottom end position of the object included in the captured image 50. The distance calculation unit 16 may calculate the distance and direction from the camera 20 as the distance information of the object. The distance calculation unit 16 may calculate the position coordinates of the object using, for example, a coordinate system based on the position of the smart pole on which the camera 20 is installed.

[0034] The distance calculation unit 16 can calculate the distance to the object by using, for example, the correlation between the distance from the camera 20 to the object and the bottom end position of the object in the captured image 50. The correlation between the distance and the bottom end position may be calculated based on, for example, the angle of view of the camera 20, or may be actually measured around the smart pole on which the recognition processing device 10 is installed. The distance calculation unit 16 can calculate the distance using a table or formula indicating the correlation between the distance and the bottom end position.

[0035] When an object 54 is detected at a position overlapping the lower edge 52d of the captured image 50, the distance calculation unit 16 calculates the distance to the object 54 using the lower end position of the detection area in which the object 54 is detected using the second detection model. For example, when the object 54d is detected using the second detection model, the distance calculation unit 16 calculates the distance to the object 54d using the lower end position 70d of the detection area 60d in which the object 54 is detected. The lower end position 70d of the detection area 60d is located below the lower edge 52d of the captured image 50, and is therefore outside the vertical range of the captured image 50, that is, outside the range of the angle of view of the camera 20. By using such a lower end position located outside the range of the captured image 50, the distance to an object included in the range overlapping the lower edge 52d of the captured image 50, such as the object 54d, can be more appropriately calculated. In addition, objects that are not overlapped with lower edge 52d, i.e., lower end positions 70a, 70b, 70c, and 70e of objects 54a, 54b, 54c, and 54e, are within the vertical range of captured image 50, i.e., within the range of the angle of view of camera 20.

[0036] The output control unit 18 causes the output device 22 to output object information related to the object detected by the object detection unit 14. The object information may include, for example, information regarding whether an object has been detected by the object detection unit 14, the number of objects detected by the object detection unit 14, and the position and distance of the detected object. The output device 22 may be a communication device, or a wireless communication device that outputs object information such as the position and distance of the object by road-to-vehicle communication or vehicle-to-vehicle communication.

[0037] 5 is a flowchart showing an example of the flow of the recognition processing method according to the first embodiment. The image acquisition unit 12 acquires a captured image captured by the camera 20 (step S10). The object detection unit 14 starts scanning the captured image using a detection window, and determines whether the detection window is inside the outer edge of the captured image, in other words, whether the detection window is located in a range that includes the inside of the outer edge of the captured image and does not overlap with the outer edge (step S12). The premise of the recognition processing in this embodiment is that the detection window is scanned over a range of the captured image that is wider than the outer edge of the captured image.

[0038] If it is determined that the detection window is located inside the outer edge of the captured image (Yes in step S12), the object detection unit 14 detects the object using the first detection model for the detection window (step S14). If the detection window is not located inside the outer edge of the captured image, that is, if the detection window is located in a range overlapping the outside of the captured image (No in step S12), the object detection unit 14 detects the object using the second detection model (step S16).

[0039] Next, the object detection unit 14 determines whether or not an object has been detected (step S18). In step S14, the object detection unit 14 determines that an object has been detected when the recognition score using the first detection model is equal to or greater than a predetermined threshold. In step S16, the object detection unit 14 determines that an object has been detected when the recognition score using the second detection model is equal to or greater than a predetermined threshold.

[0040] If an object is detected by the object detection unit 14 (Yes in step S18), the distance calculation unit 16 calculates distance information of the object using the bottom end position of the detection area in which the object is detected (step S20). The output control unit 18 outputs object information related to the detected object (step S22). If an object is not detected by the object detection unit 14 (No in step S18), the processes of steps S20 and S22 can be skipped.

[0041] FIG. 6 is a flowchart showing an example of the process flow of step S16 in FIG. 5. FIG. 6 shows an example in which the second detection unit 26 uses different second detection models for detecting an object depending on the position of the detection window. As described above, the second detection unit 26 uses a left edge detection model, a right edge detection model, an upper edge detection model, and a lower edge detection model as examples of the second detection model. When the object detection unit 14 determines that the detection window exists at a position overlapping the left edge of the captured image (Yes in step S30), it detects the object using the left edge detection model (step S32). When the object detection unit 14 determines that the detection area does not exist at a position overlapping the left edge of the captured image (No in step S30) and that the detection area exists at a position overlapping the right edge of the captured image (Yes in step S34), it detects the object using the right edge detection model (step S36). If the object detection unit 14 determines that the detection area is not located at a position overlapping either the left edge or the right edge of the captured image (No in step S34) but is located at a position overlapping the upper edge of the captured image (Yes in step S38), it detects the object using the upper edge detection model (step S40).If the object detection unit 14 determines that the detection area is not located at a position overlapping either the left edge, the right edge, or the upper edge of the captured image (No in step S38), it detects the object using the lower edge detection model (step S42).

[0042] According to this embodiment, by using the second detection model, it is possible to improve the detection accuracy of an object that is located on the outer edge of the captured image and therefore is not included in the captured image in its entirety. For example, when an object moves in a direction approaching the camera 20, it moves from the upper side to the lower side of the lower edge of the captured image, so it may become difficult to capture the entire image of the object as it approaches the camera 20. If the angle of view of the camera 20 is lowered, it is possible to capture the entire image of an object located near the camera 20, but it becomes impossible to capture an image of an object located far from the camera 20. According to this embodiment, the detection accuracy of an object located on the outer edge of the captured image is improved, so that the range in which an object can be detected using one camera 20 can be expanded.

[0043] According to this embodiment, even if the bottom end position of the object is not included in the angle of view of the captured image because it is located on the bottom edge of the captured image, the bottom end position of the object can be identified. As a result, the distance to the object located near the camera 20 can be calculated more appropriately.

[0044] Second embodiment 7 is a block diagram showing a schematic functional configuration of a recognition processing device 10A according to the second embodiment. The recognition processing device 10A according to the second embodiment differs from the recognition processing device 10 according to the first embodiment in that it further includes an object tracking unit 28 and a bottom end estimation unit 30. The following description of the second embodiment will focus on the differences from the first embodiment, and descriptions of commonalities will be omitted as appropriate.

[0045] The recognition processing device 10A includes an image acquisition unit 12, an object detection unit 14A, an object tracking unit 28, a lower end estimation unit 30, a distance calculation unit 16A, and an output control unit 18. The image acquisition unit 12 and the output control unit 18 can be configured similarly to the first embodiment. The object detection unit 14A includes a first detection unit 24A, but differs from the first embodiment in that it does not include a second detection unit 26.

[0046] The first detection unit 24A detects an object using a first detection model that is obtained by machine learning an entire image of the object. The first detection unit 24A detects an object included in a captured image using the first detection model. The first detection unit 24A detects an object located near the center of the captured image using the first detection model, and also detects an object located on the outer edge of the captured image using the first detection model.

[0047] The object tracking unit 28 tracks the object detected by the object detection unit 14A. The object tracking unit 28 tracks the object across multiple frames constituting the captured video, and identifies the movement of the object across the multiple frames. The object tracking unit 28 identifies, for example, the amount and direction of movement of the object.

[0048] 8(a) to (c) are schematic diagrams showing an example of an object being tracked across multiple frames constituting the captured image, and show the state in which object 54f is moving in a direction toward the lower edge 52d of the captured image, in other words, the state in which object 54f is approaching camera 20.

[0049] 8(a) shows the captured image 50a when the object 54f is present at a position above the lower edge 52d. Also shown is a detection area 60f when the object 54f is detected in the captured image 50a. It is determined that the bottom end position of the object 54f at this time coincides with the bottom end position 70f of the detection area 60f. The object tracking unit 28 is tracking the object 54f.

[0050] FIG. 8(b) shows a captured image 50b one frame or several frames after FIG. 8(a). In the captured image 50b, the lower end position 70f of the detection area 60f when the object 54f is detected coincides with the lower edge 52d. In the example of FIG. 8(b), the position of the object 54f in FIG. 8(a) is shown by a dashed line for ease of explanation, but the object 54f shown by the dashed line is not actually captured in the captured image 50b. The object tracking unit 28 specifies the amount of movement and the direction of movement as shown by the arrow 78b based on the difference between the position of the object 54f in the past frame and the position of the object 54f in the current frame. The object tracking unit 28 can specify the movement of the object 54f based on the transition of the position of a specific part (e.g., the head) of the object 54f.

[0051] In FIG. 8(b), the object tracking unit 28 detects that the object 54f is moving in a direction toward the lower edge 52d of the captured image. Therefore, the object 54f detected in FIG. 8(a) and the object 54f detected in FIG. 8(b) are determined to be the same object. Furthermore, since the arrangement of the detection area 60f for the object 54f detected in FIG. 8(a) is the same as the arrangement of the detection area 60f for the object 54f detected in FIG. 8(b), it is determined that the entire object 54f detected in FIG. 8(b) has been detected. Therefore, it is determined that the bottom end position of the object 54f detected in FIG. 8(b) coincides with the bottom end position 70f of the detection area 60f shown in FIG. 8(b).

[0052] FIG. 8(c) shows a captured image 50c one or several frames after FIG. 8(b). In the captured image 50c, the lower end of the object 54f is located below the lower edge 52d, and the lower part of the object 54f is outside the angle of view of the camera 20. In the example of FIG. 8(c), the position of the object 54f in FIG. 8(b) is shown by a dashed line for ease of explanation, but the object 54f shown by the dashed line is not actually captured in the captured image 50c. The object tracking unit 28 specifies the amount of movement and the direction of movement as indicated by the arrow 78c based on the difference between the position of the object 54f in the past frame and the position of the object 54f in the current frame.

[0053] The bottom end estimation unit 30 estimates the bottom end position of the object. When an object located at the bottom edge of the captured image is detected, the bottom end estimation unit 30 estimates the bottom end position of the object that may be located below the bottom edge of the captured image. The bottom end estimation unit 30 also estimates the bottom end position of the object based on the size of the object above the bottom edge of the captured image.

[0054] In FIG. 8(c), the object tracking unit 28 detects that the object 54f is moving in a direction toward the lower edge 52d of the captured image. In FIG. 8(c), the object 54f is not entirely included in the captured image 50c, so the recognition score by the first detection model is a low value. However, since the object tracking unit 28 estimates that the object 54f is present at a position overlapping the lower edge 52d of the captured image, the first detection unit 24A determines that the detection area 60f shown in FIG. 8(c) is the detection area of ​​the object 54f. Therefore, it is determined that the lower end position of the object 54f detected in FIG. 8(c) coincides with the lower end position 70f of the detection area 60f shown in FIG. 8(c).

[0055] The bottom end estimation unit 30 may estimate the bottom end position 70f at the time of Fig. 8(c) based on, for example, the bottom end position 70f at the time of Fig. 8(b) and the movement of the target object 54f tracked from the time of Fig. 8(b) to the time of Fig. 8(c) as indicated by the arrow 78c. For example, as indicated by the arrow 80c, the bottom end position 70f at the time of Fig. 8(c) can be estimated by adding the amount of movement (movement vector) indicated by the arrow 78c to the bottom end position 70f at the time of Fig. 8(b).

[0056] In FIG. 8(c), since the entire image of the object 54f is not included in the captured image 50c, the detection area in the captured image 50c in which the object 54f is detected may be smaller in size in the vertical direction than the detection area 60f of the object 54f detected in FIG. 8(b). In this case, the detection area 60f of the object 54f may be estimated to be a position overlapping the lower edge 52d of the captured image, as in the detection area 60f shown in FIG. 8(c). Specifically, the lower edge estimation unit 30 estimates the lower end position of the object 54f based on the vertical size in which the object 54f shown in FIG. 8(c) is detected, that is, the size above the lower edge 52d. For example, based on the position and size of detection area 60f of object 54f detected in FIG. 8(a) or 8(b), the bottom position of object 54f is estimated by estimating detection area 60f that includes the entire image of object 54f in FIG. 8(c) from the vertical size of object 54f tracked and detected by object tracking unit 28.

[0057] The distance calculation unit 16A calculates distance information of the object by using the bottom end position of the object estimated by the bottom end estimation unit 30. The distance calculation unit 16A can calculate the distance information of the object by using a method similar to that of the distance calculation unit 16 according to the above-described first embodiment.

[0058] 9 is a flowchart showing an example of the flow of the recognition processing method according to the second embodiment. The image acquisition unit 12 acquires a captured image captured by the camera 20 (step S50). The object detection unit 14A starts scanning the captured image using a detection window, and starts detecting an object using a first detection model (step S52).

[0059] Next, the object detection unit 14A judges whether or not an object has been detected from the captured image captured by the camera 20 (step S54). If it is judged in step S54 that an object has been detected (Yes in step S54), the object tracking unit 28 tracks the object over multiple frames and identifies the movement of the object (step S56). If the detection area when the tracked object is detected is a position overlapping the lower edge of the captured image (Yes in step S58), the bottom end estimation unit 30 estimates the bottom end position of the object based on the movement of the object (step S60). If the detection area when the tracked object is detected is not a position overlapping the lower edge of the captured image (No in step S58), the bottom end estimation unit 30 estimates the bottom end position of the object based on the bottom end position of the detection area (step S62).

[0060] After steps S60 and S62, distance calculation unit 16A calculates distance information of the object using the estimated bottom end position of the object (step S64). Output control unit 18 outputs object information related to the detected object (step S66). If it is not determined that the object is detected in step S54 (No in step S54), the processes of steps S56 to S66 can be skipped.

[0061] According to this embodiment, even if the bottom end position of the object is not included in the angle of view of the captured image because it is located at the bottom edge of the captured image, the bottom end position of the object can be estimated. As a result, the distance to the object located near the camera 20 can be calculated more appropriately.

[0062] Third embodiment 10 is a block diagram showing a functional configuration of a recognition processing device 10B according to a third embodiment. The recognition processing device 10B according to the third embodiment differs from the above-mentioned embodiments in that, when an object is present at a position overlapping the lower edge of a captured image, the recognition processing device 10B detects the object by switching from the first detection model to the second detection model. Hereinafter, the third embodiment will be described with a focus on the differences from the first and second embodiments, and descriptions of commonalities will be omitted as appropriate.

[0063] The recognition processing device 10B includes an image acquisition unit 12, an object detection unit 14B, a bottom end estimation unit 30B, a distance calculation unit 16B, and an output control unit 18. The image acquisition unit 12 and the output control unit 18 can be configured similarly to the first or second embodiment. The object detection unit 14B includes a first detection unit 24B and a second detection unit 26B.

[0064] The first detection unit 24B can be configured in the same manner as the first detection unit 24A according to the second embodiment. The first detection unit 24B detects an object using a first detection model that is machine-learned from an entire image of the object. The first detection unit 24B detects an object included in the captured image using the first detection model. The first detection unit 24B detects an object located near the center of the captured image using the first detection model, and also detects an object located on the outer edge of the captured image using the first detection model. The first detection unit 24B detects an object located on the lower edge of the captured image using the first detection model.

[0065] When the first detection unit 24B detects an object included in a position overlapping the lower edge of the captured image, the second detection unit 26B detects the object included in the position overlapping the lower edge of the captured image using a second detection model that has been machine-learned from an upper part image of the object. The second detection unit 26B detects, using the second detection model, an object that is included in a position overlapping the lower edge of the captured image, so that a part of the upper side of the object is included in the captured image, but the lower side of the object is outside the angle of view and is not included in the captured image.

[0066] 11(a) and (b) are diagrams showing an example of a method for detecting an object included in a position overlapping the lower edge of a captured image. FIG. 11(a) and (b) show a captured image 50c similar to FIG. 8(c) according to the second embodiment described above. FIG. 11(a) shows a state in which an object 54g is detected by the first detection unit 24B. The first detection unit 24B scans a detection window for the captured image 50c and detects the object using a first detection model. FIG. 11(a) shows a state in which the bottom end position 70g of the detection area 60g when the object 54g is detected is different from the original bottom end position of the object 54g because the whole of the object 54g is not included in the captured image 50c. FIG. 11(b) shows a state in which the object 54g is detected by the second detection unit 26B. The second detection unit 26B scans the detection window of the captured image 50c, including the detection area in which the object was detected by the first detection model and the area nearby, to detect the object using the second detection model. Fig. 11(b) shows a state in which the object 54g is included in the area overlapping the lower edge of the captured image 50c, and the lower end position 70g of the detection area 60g is estimated when the object 54g is detected using the second detection model.

[0067] The bottom end estimation unit 30B estimates the bottom end position of the object. When the first detection unit 24B detects an object included in a range overlapping the bottom edge of the captured image, the bottom end estimation unit 30B determines the bottom end position of the detection area detected by the second detection unit 26B for the object as the bottom end position of the object. For example, in the case shown in FIG. 11(b), the bottom end estimation unit 30B determines the bottom end position 70g of the detection area 60g of the object 54g detected by the second detection unit 26B as the bottom end position of the object 54g.

[0068] The distance calculation unit 16B calculates distance information of the object using the bottom end position of the object estimated by the bottom end estimation unit 30B. The distance calculation unit 16B can calculate the distance information of the object using a method similar to that of the distance calculation unit 16 according to the above-described first embodiment.

[0069] 12 is a flowchart showing an example of the flow of the recognition processing method according to the third embodiment. The image acquisition unit 12 acquires a captured image captured by the camera 20 (step S70). The object detection unit 14B starts scanning the captured image using a detection window, and starts detecting an object using a first detection model (step S72).

[0070] Next, the object detection unit 14B judges whether or not an object has been detected from the captured image captured by the camera 20 (step S74). If it is judged in step S74 that an object has been detected (Yes in step S74), the object detection unit 14B judges whether or not the detection area of ​​the object detected in step S74 is included in a range overlapping the lower edge of the captured image (step S76). If it is judged in step S76 that the detection area of ​​the detected object is included in a range overlapping the lower edge of the captured image (Yes in step S76), the object detection unit 14B detects the object using the second detection model (step S78). In addition, the bottom end estimation unit 30B estimates the bottom end position of the object based on the bottom end position of the detection area of ​​the object detected by the second detection model in step S78 (step S80). In step S76, if it is not determined that the detection area of ​​the detected object is included in the range overlapping the lower edge of the captured image (No in step S76), the bottom end estimation unit 30B estimates the bottom end position of the object based on the bottom end position of the detection area of ​​the object detected by the first detection model (step S82).

[0071] After steps S80 and S82, distance calculation unit 16B calculates distance information of the object using the estimated bottom end position of the object (step S84). Output control unit 18 outputs object information related to the detected object (step S86). If it is not determined in step S74 that the object is detected (No in step S74), the processes of steps S76 to S86 can be skipped.

[0072] According to this embodiment, when the bottom end position of an object is not included in the angle of view of a captured image because it is located at the bottom edge of the captured image, the bottom end position of the object can be estimated by detecting the object using the second detection model. As a result, distance information of an object located near the camera 20 can be calculated more appropriately.

[0073] The present invention has been described above with reference to the above-mentioned embodiment, but the present invention is not limited to the above-mentioned embodiment, and appropriate combinations or substitutions of the respective configurations shown in the embodiment are also included in the present invention.

[0074] Several aspects of the disclosure are described below.

[0075] A first aspect of the present disclosure is a recognition processing device including an image acquisition unit that acquires an image captured by the image capture device, and an object detection unit that detects an object located away from the outer edge of the image captured by the image capture device using a first detection model that has been machine-learned on an entire image of the object, and detects an object located at the outer edge of the image captured by the image capture device using a second detection model that has been machine-learned on a partial image of the object.

[0076] A second aspect of the present disclosure is a recognition processing method performed by a recognition processing device, comprising the steps of acquiring an image captured by the camera, detecting an object located away from the outer edge of the image captured by the camera using a first detection model that has been machine-learned on an entire image of the object, and detecting an object located at the outer edge of the image captured by the camera using a second detection model that has been machine-learned on a partial image of the object.

[0077] A third aspect of the present disclosure is a program configured to cause a processor to execute the following functions: acquiring an image captured by a computer; detecting an object located away from the outer edge of the image captured by a first detection model that has been machine-learned on an entire image of the object; and detecting an object located at the outer edge of the image captured by a second detection model that has been machine-learned on a partial image of the object. The third aspect of the present disclosure is a program configured to cause a processor to execute the following functions: acquiring an image captured by a computer;

[0078] A fourth aspect of the present disclosure is a recognition processing device including: an image acquisition unit that acquires an image of an object; an object detection unit that detects an object included in the image of the object using a detection model that has been machine-learned from an image of the object; a bottom end estimation unit that, when an object included in a range overlapping with the bottom edge of the image of the object is detected by the object detection unit, estimates a bottom end position of the object that may be located below the bottom edge of the image of the object; and a distance calculation unit that calculates distance information of the object using the bottom end position estimated by the bottom end estimation unit.

[0079] A fifth aspect of the present disclosure is a recognition processing method in which a recognition processing device executes the steps of: acquiring an image of the object; detecting an object included in the image of the object using a detection model that has been machine-learned from an image of the object; if an object included in an area overlapping the lower edge of the image of the object is detected, estimating a lower end position of the object that may be located below the lower edge of the image of the object; and calculating distance information of the object using the estimated lower end position.

[0080] A sixth aspect of the present disclosure is a program configured to cause a processor to execute the following functions: acquiring an image of the captured image; detecting an object contained in the captured image using a detection model based on machine learning of an image of the object; if an object contained in an area overlapping the lower edge of the captured image is detected, estimating a bottom end position of the object, which may be located below the lower edge of the captured image; and calculating distance information of the object using the estimated bottom end position. [Explanation of symbols]

[0081] 10...recognition processing device, 12...image acquisition unit, 14...object detection unit, 16...distance calculation unit, 50...captured image, 52...outer edge, 52d...lower edge.

Claims

1. an image acquisition unit that acquires a captured image; an object detection unit that detects an object included in the captured video by using a detection model that has been machine-learned to learn an image of the object; a bottom edge estimation unit that, when an object included in a range overlapping a bottom edge of the captured image is detected by the object detection unit, estimates a bottom edge position of the object that may be located below the bottom edge of the captured image; a distance calculation unit that calculates distance information of the object using the bottom end position estimated by the bottom end estimation unit; A recognition processing device comprising:

2. further comprising an object tracking unit that tracks the object detected by the object detection unit, the lower end estimation unit estimates a lower end position of the object that may be located below a lower edge of the captured image, based on a movement of the object tracked by the object tracking unit when the lower end position of the object moves downward below a lower edge of the captured image. The recognition processing device according to claim 1 .

3. The lower end estimation unit estimates a lower end position of the object based on a size of the object detected by the object detection unit above a lower edge of the captured image. The recognition processing device according to claim 2 .

4. The object detection unit detects an object included in the captured video by using a first detection model that has been machine-learned to learn an entire image of the detection object; when detecting an object included in a range overlapping a lower edge of the captured image using the first detection model, the object detection unit detects the object included in a range overlapping a lower edge of the captured image using a second detection model that has been machine-learned on an upper portion image of the object, The lower end estimation unit estimates a lower end position of the object by using a lower end position of a detection area of ​​the object detected by using the second detection model. The recognition processing device according to claim 1 .

5. acquiring a captured image; Detecting an object included in the captured video using a detection model that has been machine-learned from an image of the object; When an object is detected that is included in a range overlapping with a lower edge of the captured image, estimating a lower end position of the object that may be located below the lower edge of the captured image; calculating distance information of the object using the estimated bottom end position; The recognition processing method is executed by a recognition processing device.

Images