Identification processing device, identification processing method, and computer program

Abstract

This recognition processing device (10A) is provided with: an image acquisition unit (12) that acquires a captured image; an object detection unit (14A) that detects an object included in the captured image using a detection model obtained by machine learning of an image of the object; a lower end estimation unit (30) that estimates the lower end position of an object that may be located below the lower edge of the captured image when the object detection unit (14A) detects the object included in a range overlapping the lower edge of the captured image; and a distance calculation unit (16A) that calculates distance information of the object using the lower end position estimated by the lower end estimation unit (30).

Description

TECHNICAL FIELD

[0001] The present application relates to an identification processing device, an identification processing method, and a computer program. BACKGROUND

[0002] There is known a technique of detecting an object such as a pedestrian from an image obtained by photographing a periphery of a vehicle using image recognition processing such as pattern matching (for example, refer to Patent Literature 1).

[0003] PRIOR ART DOCUMENT PATENT LITERATURE Patent Literature 1: Japanese Patent Literature Laid-Open No. 2022-139374 SUMMARY

[0004] In a case where an object exists near an edge of a video photographed by a camera, since an entire image of the object is not included in the video, the object can not be appropriately detected at times.

[0005] The present application has been achieved in view of the above-described circumstances, and an object thereof is to provide a technique of more appropriately detecting an object in image recognition processing.

[0006] An identification processing device of one embodiment of the present application includes: an image acquisition section that acquires a photographed image; an object detection section that detects an object included in the photographed image using a detection model that has been machine-learned on an image of the object; a lower end estimation section that estimates a lower end position of the object that is likely to be positioned lower than a lower edge of the photographed image in a case where the object included in a range overlapping with the lower edge of the photographed image is detected by the object detection section; and a distance calculation section that calculates distance information of the object using the lower end position estimated by the lower end estimation section.

[0007] Another embodiment of the present application is an identification processing method in which the identification processing device executes the following steps: a step of acquiring a photographed image; a step of detecting an object included in the photographed image using a detection model that has been machine-learned on an image of the object; a step of estimating a lower end position of the object that is likely to be positioned lower than a lower edge of the photographed image in a case where the object included in a range overlapping with the lower edge of the photographed image is detected; and a step of calculating distance information of the object using the estimated lower end position.

[0008] Still another aspect of the present application is a computer program that causes a processor to function as: an acquisition unit that acquires a captured image; a detection unit that detects an object included in the captured image using a detection model that has been subjected to machine learning with respect to images of objects; a presumption unit that, in a case where the object included in a range overlapping with a lower edge of the captured image is detected, presumes a lower end position of the object that is likely to be positioned lower than the lower edge of the captured image; and a calculation unit that calculates distance information of the object using the presumed lower end position.

[0009] According to one embodiment of the present application, a technique can be provided that more appropriately detects an object in image recognition processing. BRIEF DESCRIPTION OF DRAWINGS

[0010] Figure 1 is a block diagram schematically showing a functional configuration of an identification processing apparatus according to the first embodiment.

[0011] Figure 2 is a diagram schematically showing an example of a captured image including an object.

[0012] Figure 3 is a diagram schematically showing an example of a detection region set with respect to a captured image.

[0013] Figure 4 (a) to (e) of is a diagram schematically showing an example of an input image for machine learning of a detection model.

[0014] Figure 5 is a flowchart showing an example of a flow of a recognition processing method according to the first embodiment.

[0015] Figure 6 is a flowchart showing an example of a processing flow of step S16 of Figure 5

[0016] Figure 7 is a block diagram schematically showing a functional configuration of an identification processing apparatus according to the second embodiment.

[0017] Figure 8 (a) to (c) of is a diagram schematically showing an example of an object that is tracked in a plurality of frames constituting a captured image.

[0018] Figure 9 is a flowchart showing an example of a flow of a recognition processing method according to the second embodiment.

[0019] Figure 10 is a block diagram schematically showing a functional configuration of an identification processing apparatus according to the third embodiment.

[0020] Figure 11 ​Fig. 1 is a diagram schematically showing an example of a method of detecting an object located at a lower edge of a captured image.

[0021] Figure 12 Fig. 2 is a flowchart showing an example of a flow of a recognition processing method according to the third embodiment. DETAILED DESCRIPTION

[0022] Embodiments of the present application will be described below with reference to the accompanying drawings. Specific numerical values and the like shown in these embodiments are merely examples for easy understanding of the application and do not limit the present application unless otherwise specified. In the drawings, the illustration of elements not directly related to the present application is omitted.

[0023] (First Embodiment) Figure 1 Fig. 1 is a block diagram schematically showing a functional configuration of a recognition processing device 10 according to the first embodiment. The recognition processing device 10 includes an image acquisition unit 12 and an object detection unit 14. The recognition processing device 10 can further include a distance calculation unit 16 and an output control unit 18. The recognition processing device 10, for example, acquires a captured image that can contain an object such as a pedestrian present in the surroundings, and detects the object contained in the captured image.

[0024] In the present embodiment, a case where the recognition processing device 10 is provided to a smart pole is exemplified. The smart pole is provided, for example, to a street, and includes an antenna and a communication device for providing a wireless communication function, a lighting device for illuminating the street, and a camera for capturing a vehicle or a pedestrian passing through the road. The recognition processing device 10 is fixed to a prescribed site. The recognition processing device 10 can also be mounted to a mobile body, and can be mounted to a vehicle or a flying body such as a drone.

[0025] The object detected by the recognition processing device 10 can be applied to any object. In the embodiments of the present application, a case where the object is a human body such as a pedestrian is described as an example.

[0026] Each functional block shown in the present embodiment can be realized, for example, by cooperation of hardware and software. The hardware of the recognition processing device 10 is realized by elements or mechanical devices including a processor such as a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit) of a computer, and a memory such as a ROM (Read Only Memory) or a RAM (Random Access Memory). The software of the recognition processing device 10 is realized by a computer program or the like.

[0027] The image acquisition unit 12 acquires an image (also referred to as a captured image) captured by the camera 20. The camera 20 is provided to the intelligent stick and captures an image of the surroundings of the intelligent stick. The camera 20 is provided, for example, above the intelligent stick and captures an image of a visual angle overlooking the ground on which the intelligent stick is provided. The camera 20 captures visible light and generates a color image or a monochrome image. The camera 20 can also be an infrared camera that captures infrared rays and generates a thermal image. The image captured by the camera 20 is, for example, a moving image of 30 frames per second or 60 frames per second.

[0028] The object detection unit 14 detects an object from the image acquired by the image acquisition unit 12. In other words, the object detection unit 14 detects a region (hereinafter referred to as a detection window) in which an object is included in the image acquired by the image acquisition unit 12. The object detection unit 14 calculates, for each frame of the image acquired by the image acquisition unit 12, a recognition score indicating a possibility of including an object while scanning the detection window with reference to one or more detection models for detecting an object. The recognition score is calculated, for example, in a range of 0.0 to 1.0, and the higher the possibility of including an object is for the image in the detection window, the larger the value of the recognition score (i.e., a value close to 1.0) becomes, and the lower the possibility of including an object is, the smaller the value of the recognition score (i.e., a value close to 0.0) becomes. The object detection unit 14 determines that the detection window includes an object when the recognition score is equal to or higher than a predetermined threshold value, for example, 0.8, and thereby detects an object.

[0029] The object detection unit 14 includes a first detection unit 24 and a second detection unit 26.

[0030] The first detection unit 24 detects an object using a first detection model that has been subjected to machine learning on an overall image of a human body as an object. The overall image of a human body is an image including the entire human body. The first detection unit 24 detects an object included in a range that does not overlap with an outer edge, for example, in a range included in an inner side of a captured image near the center of the captured image or the like. The first detection unit 24 detects an object, for example, that includes the entire object in the captured image.

[0031] The second detection unit 26 detects an object using a second detection model that has been subjected to machine learning on a partial image of a human body as an object. The partial image of a human body is an image including about half of the entire human body. The second detection unit 26 detects an object included in a range that overlaps with an outer edge of a captured image, for example, in a range included in an inner side of the outer edge of the captured image near the periphery of the captured image or the like. The second detection unit 26 detects an object that includes a part of the object in the captured image and the remaining part of the object is outside the visual angle and not included in the captured image.

[0032] Therefore, the object detection unit 14 uses the first detection unit 24 to detect objects contained in a range that does not overlap with the outer edge of the captured image using a first detection model that performs machine learning on the overall image of the object, and uses the second detection unit 26 to detect objects contained in a range that overlaps with the outer edge of the captured image using a second detection model that performs machine learning on a partial image of the object.

[0033] A machine learning model can include: an input corresponding to the image size (number of pixels) of the input image, an output of a recognition score, and intermediate layers connecting the input and output. These intermediate layers can contain convolutional layers, pooling layers, fully connected layers, etc. The intermediate layers can be multi-layered and can be configured to perform so-called deep learning. The machine learning model can also be built using a convolutional neural network (CNN). Furthermore, the machine learning model is not limited to the above and can use any machine learning model.

[0034] Figure 2 This diagram schematically illustrates an example of a captured image 50 containing object 54 (54a-54e). The outer edge 52 of the captured image 50 corresponds to the viewing angle of the camera 20. The outer edge 52 has a left edge 52a, a right edge 52b, a top edge 52c, and a bottom edge 52d. In this specification, the directions up, down, left, and right are set based on the viewing angle of the camera 20, meaning the top, bottom, left, and right sides from the viewpoint of the camera 20.

[0035] The captured image 50 includes an object 54e that does not overlap with and is located far from the outer edge 52, and objects 54a, 54b, 54c, and 54d located within the range that overlaps with the outer edge 52. Object 54e is included entirely in the captured image 50, within the range that does not overlap with the outer edge. On the other hand, objects 54a to 54d are partially included in the captured image 50, while the remaining parts are outside the viewing angle and not included in the captured image 50. That is, objects 54a to 54d are included within the range that overlaps with the outer edge.

[0036] In the captured image 50, the left side of object 54a is out of view and not included in the captured image 50, while the right side is included in the captured image 50. The right side of object 54b is out of view and not included in the captured image 50, while the left side is included in the captured image 50. The upper part of object 54c is out of view and not included in the captured image 50, while the lower part is included in the captured image 50. The lower part of object 54d is out of view and not included in the captured image 50, while the upper part is included in the captured image 50.

[0037] Figure 3is a drawing schematically showing an example of the detection regions 60a to 60e when the object 54 is detected for the captured image 50. In Figure 3 The detection regions 60a to 60e are represented by single-dotted line rectangular frames. The shape, such as the aspect ratio, of the detection regions 60a to 60e corresponds to the input size of the detection model used by the object detection section 14, for example, and is approximately 2: 1.

[0038] The first detection section 24 detects an object (for example, the object 54e) included in the captured image, using the first detection model, for a detection window that scans the inside of the outer edge 52 of the captured image 50. The first detection section 24 detects an object included in a range that includes the inside of the outer edge 52 of the captured image 50 and does not include the outside of the outer edge 52. The first detection section 24, for example, performs object detection using the first detection model for a detection window that scans the inside of the outer edge 52 of the captured image 50, while changing the position and size of the detection window and scanning the entire captured image 50.

[0039] The second detection section 26 detects an object (for example, the objects 54a to 54d) included in the captured image, using the second detection model, for a detection window that scans a range overlapping the outer edge 52 of the captured image 50. The second detection section 26 detects an object included in a range that includes the inside and outside of the outer edge 52 of the captured image 50. The second detection section 26, for example, performs object detection using the second detection model for a detection window that scans a range including the outer edge 52 of the captured image 50, while changing the position and size of the detection window and scanning the entire captured image 50.

[0040] The second detection section 26 detects an object using the second detection model for a range overlapping at least any one of the left edge 52a, the right edge 52b, the upper edge 52c, and the lower edge 52d of the captured image 50. For example, the detection region 60a when the object 54a is detected includes an inside region 62a adjacent to the right side of the left edge 52a and an outside region 64a adjacent to the left side of the left edge 52a. For example, the detection region 60b when the object 54 is detected includes an inside region 62b adjacent to the left side of the right edge 52b and an outside region 64b adjacent to the right side of the right edge 52b. For example, the detection region 60c when the object 54c is detected includes an inside region 62c adjacent to the lower side of the upper edge 52c and an outside region 64c adjacent to the upper side of the upper edge 52c. For example, the detection region 60d when the object 54d is detected includes an inside region 62d adjacent to the upper side of the lower edge 52d and an outside region 64d adjacent to the lower side of the lower edge 52d.

[0041] The second detection section 26 can use a plurality of detection models for detecting the object present in a position overlapping each of the left edge 52a, the right edge 52b, the upper edge 52c, and the lower edge 52d of the captured image 50. The second detection section 26 can use, as the second detection model, at least any one of a left edge detection model, a right edge detection model, an upper edge detection model, and a lower edge detection model, for example.

[0042] Figure 4 (a) to (e) of FIG. 10 are diagrams schematically showing an example of an input image for machine learning of a detection model for detecting a human body as an example. Figure 4 (a) of FIG. 10 is a right partial image 66a of the object, which is used for machine learning of the left edge detection model. The right partial image 66a is an image including the right side portion of the object but not including the left side portion. Figure 4 (b) of FIG. 10 is a left partial image 66b of the object, which is used for machine learning of the right edge detection model. The left partial image 66b is an image including the left side portion of the object but not including the right side portion. Figure 4 (c) of FIG. 10 is a lower partial image 66c of the object, which is used for machine learning of the upper edge detection model. The lower partial image 66c is an image including the lower side portion of the object but not including the upper side portion. Figure 4 (d) of FIG. 10 is an upper partial image 66d of the object, which is used for machine learning of the lower edge detection model. The upper partial image 66d is an image including the upper side portion of the object but not including the lower side portion. Figure 4 (e) of FIG. 10 is an entire image 66e of the object, which is used for machine learning of the first detection model. The entire image 66e includes the entire image of the object.

[0043] Figure 4 The partial images 66a to 66d shown in (a) to (d) of FIG. 10 include margin portions 68a to 68d, which do not include the object. The margin portions 68a to 68d are set so that the image size (e.g., aspect ratio) of the partial images 66a to 66d coincides with the image size (e.g., aspect ratio) of the entire image 66e. The luminance value of the margin portions 68a to 68d is set to a luminance value different from that of the object, for example, to a luminance value equivalent to that of the background of the object. By setting the margin portions 68a to 68d, it is possible to improve the recognition accuracy in the second detection region set to include a position further outside than the outer edge 52 of the captured image 50.

[0044] The second detection unit 26 can also use either the left edge detection model or the right edge detection model instead of simultaneously using both. For example, to detect the right edge object 56b, the second detection unit 26 can also use the left edge detection model. The second detection unit 26 can detect the right edge object 56b by flipping the image cut out from the right edge detection region 60b horizontally and then inputting it into the left edge detection model. Conversely, to detect the left edge object 56a, the second detection unit 26 can also use the right edge detection model. The second detection unit 26 can detect the left edge object 56a by flipping the image cut out from the left edge detection region 60a horizontally and then inputting it into the right edge detection model.

[0045] return Figure 1 The distance calculation unit 16 calculates the distance information of the object detected by the object detection unit 14. For example, the distance calculation unit 16 uses the lower end position of the object included in the captured image 50 to calculate the distance to the object. The lower end position of the object corresponds to the grounding position of the object, and to the lower end positions 70a-70e of the detection areas 60a-60e where the object was detected (see reference). Figure 3 The distance calculation unit 16 can also use the lower position of the object included in the captured image 50 to calculate the direction of the object. The distance calculation unit 16 can also use the distance information of the object to calculate the distance and direction from the camera 20. For example, the distance calculation unit 16 can also use a coordinate system based on the position of the smart stick on which the camera 20 is set to calculate the position coordinates of the object.

[0046] The distance calculation unit 16 can calculate the distance to the object, for example, by using the correlation between the distance from the camera 20 to the object and the lower position of the object in the captured image 50. The correlation between the distance and the lower position can be calculated, for example, based on the viewing angle of the camera 20, or by actual measurement around the smart pole where the recognition processing device 10 is installed. The distance calculation unit 16 can calculate the distance using a table or formula representing the correlation between the distance and the lower position.

[0047] The distance calculating section 16 calculates the distance to the object 54 when the object 54 is detected in a position overlapping the lower edge 52d of the captured image 50, using the lower end position in the detection region in which the object 54 is detected by the second detection model. For example, the distance calculating section 16 calculates the distance to the object 54d when the object 54d is detected by the second detection model, using the lower end position 70d of the detection region 60d in which the object 54 is detected. The lower end position 70d of the detection region 60d is located lower than the lower edge 52d of the captured image 50, and thus is located outside the range in the up-down direction of the captured image 50, i.e., outside the range of the angle of view of the camera 20. By using the lower end position thus located outside the range of the captured image 50, the distance to the object such as the object 54d included in a range overlapping the lower edge 52d of the captured image 50 can be more appropriately calculated. In addition, the lower end positions 70a, 70b, 70c, 70e of the objects 54a, 54b, 54c, 54e not included in a position overlapping the lower edge 52d are located within the range in the up-down direction of the captured image 50, i.e., within the range of the angle of view of the camera 20.

[0048] The output control section 18 causes the output device 22 to output object information related to the object detected by the object detection section 14. The object information may, for example, include the presence or absence of detection of the object by the object detection section 14, the number of detections of the object by the object detection section 14, the position or distance related information of the detected object. The output device 22 can be a communication device, or a wireless communication device that outputs the position or distance of the object and the like object information through inter-vehicle or inter-road-vehicle communication.

[0049] Figure 5 is a flowchart showing an example of the flow of the recognition processing method according to the first embodiment. The image acquisition section 12 acquires a captured image captured by the camera 20 (step S10). The object detection section 14 starts scanning using a detection window with respect to the captured image, and determines whether the detection window is located inside the outer edge of the captured image, in other words, whether the detection window is located inside a range including the outer edge of the captured image and does not overlap the outer edge (step S12). As a prerequisite of the recognition processing according to the present embodiment, the detection window scans a wider range than the outer edge of the captured image with respect to the captured image.

[0050] The object detection unit 14 detects the object with respect to the detection window using the first detection model when it is determined that the detection window is located inside the outer edge of the captured image (YES in step S12). The object detection unit 14 detects the object using the second detection model when the detection window is not located at a position inside the outer edge of the captured image, that is, when the detection window is located in a range overlapping the outer side of the captured image (NO in step S12).

[0051] Next, the object detection unit 14 determines whether or not the object is detected (step S18). The object detection unit 14 determines that the object is detected when the recognition score using the first detection model is equal to or higher than a predetermined threshold in step S14. In addition, the object detection unit 14 determines that the object is detected when the recognition score using the second detection model is equal to or higher than a predetermined threshold in step S16.

[0052] When the object is detected by the object detection unit 14 (YES in step S18), the distance calculation unit 16 calculates the distance information of the object using the lower end position of the detection region in which the object is detected (step S20). The output control unit 18 outputs the object information related to the detected object (step S22). When the object is not detected by the object detection unit 14 (NO in step S18), the processes of steps S20 and S22 can be skipped.

[0053] Figure 6 is an example of the process flow of step S16 of the process flow of Figure 5 is an example of the process flow of step S16 of the process flow of Figure 6The second detection section 26 distinguishes the second detection model used for the object detection according to the position of the detection window. As an example of the second detection model, the second detection section 26 uses the left edge detection model, the right edge detection model, the upper edge detection model, and the lower edge detection model as described above. The object detection section 14 detects the object using the left edge detection model when it is determined that the detection window is positioned at a position overlapping the left edge of the captured image (YES in step S30) (step S32). The object detection section 14 detects the object using the right edge detection model when the detection region is not present at a position overlapping the left edge of the captured image (NO in step S30) and it is determined that the detection region is present at a position overlapping the right edge of the captured image (YES in step S34) (step S36). The object detection section 14 detects the object using the upper edge detection model when the detection region is not present at a position overlapping either of the left edge and the right edge of the captured image (NO in step S34) and it is determined that the detection region is present at a position overlapping the upper edge of the captured image (YES in step S38) (step S40). The object detection section 14 detects the object using the lower edge detection model when it is determined that the detection region is not present at a position overlapping either of the left edge, the right edge, and the upper edge of the captured image (NO in step S38) (step S42).

[0054] According to the present embodiment, by using the second detection model, it is possible to improve the detection accuracy of the object positioned at a position overlapping the outer edge of the captured image and thus not containing the entire image in the captured image. For example, in a case where the object moves in a direction approaching the camera 20, since it moves from the upper side to the lower side of the lower edge of the captured image, it can be difficult to capture the entire image of the object as it approaches the camera 20. If the angle of view of the camera 20 is reduced, it is possible to capture the entire image of the object positioned near the camera 20, but it is not possible to capture the object positioned far from the camera 20. According to the present embodiment, since the detection accuracy of the object positioned at the outer edge of the captured image is improved, it is possible to expand the range in which the object is detected using one camera 20.

[0055] According to the present embodiment, even if the lower end position of the object is not contained within the angle of view of the captured image due to the lower end position of the object being positioned at a position overlapping the lower edge of the captured image, it is possible to determine the lower end position of the object. As a result, it is possible to more appropriately calculate the distance to the object positioned near the camera 20.

[0056] (Second Embodiment) Figure 7is a block diagram schematically showing a functional configuration of the recognition processing apparatus 10A according to the second embodiment. The recognition processing apparatus 10A according to the second embodiment differs from the recognition processing apparatus 10 according to the first embodiment in that the object tracking section 28 and the lower end estimation section 30 are further provided. Hereinafter, the second embodiment will be described focusing on the difference from the first embodiment, and the commonalities will be appropriately omitted.

[0057] The recognition processing apparatus 10A includes the image acquisition section 12, the object detection section 14A, the object tracking section 28, the lower end estimation section 30, the distance calculation section 16A, and the output control section 18. The image acquisition section 12 and the output control section 18 can be configured as in the first embodiment. The object detection section 14A differs from the first embodiment in that the first detection section 24A is provided, but the second detection section 26 is not provided.

[0058] The first detection section 24A detects an object using the first detection model that has been subjected to machine learning on an entire image of the object. The first detection section 24A detects an object included in the captured image using the first detection model. The first detection section 24A detects an object located near the center of the captured image using the first detection model, and also detects an object located overlapping the outer edge of the captured image using the first detection model.

[0059] The object tracking section 28 tracks the object detected by the object detection section 14A. The object tracking section 28 tracks the object in a plurality of frames constituting the captured image, and determines the movement of the object in the plurality of frames. The object tracking section 28 determines, for example, the amount of movement or the direction of movement of the object.

[0060] Figure 8 (a) to (c) of FIG. 54 are diagrams schematically showing an example of an object that is tracked in a plurality of frames constituting a captured image, and show a state in which the object 54f is moving in the direction of the lower edge 52d of the captured image, in other words, a state in which the object 54f is approaching the camera 20.

[0061] Figure 8 (a) of FIG. 54 shows the captured image 50a when the object 54f is present at a position further up than the lower edge 52d. In addition, a detection region 60f at the time when the object 54f is detected with respect to the captured image 50a is shown. The lower end position of the object 54f at this time is judged to coincide with the lower end position 70f of the detection region 60f. The object tracking section 28 is tracking the object 54f.

[0062] Figure 8 (b) of FIG. 54 shows the captured image 50b when the object 54f has moved to a position overlapping the lower edge 52d of the captured image 50a. In addition, a detection region 60g at the time when the object 54f is detected with respect to the captured image 50b is shown. The lower end position of the object 54f at this time is judged to coincide with the lower end position 70g of the detection region 60g. The object tracking section 28 is tracking the object 54f. Figure 8the (a) after 1 frame or several frames. In the captured image 50b, the lower end position 70f of the detection region 60f at the time of detection of the object 54f coincides with the lower edge 52d. In the (b) of FIG. 7, the object 54f is detected in the captured image 50b after 1 frame or several frames from the (a). In the captured image 50b, the lower end position 70f of the detection region 60f at the time of detection of the object 54f coincides with the lower edge 52d. In Figure 8 In the example of the (b), for the sake of explanation, the position of the object 54f in the (a) is indicated by a broken line, but the object 54f indicated by the broken line is not captured in the actual captured image 50b. The object tracking section 28 determines the amount of movement or the direction of movement as indicated by an arrow 78b based on the difference from the position of the object 54f in the past frame to the position of the object 54f in the current frame. The object tracking section 28 can determine the motion of the object 54f based on the change in the position of a specific part (for example, the head) of the object 54f. Figure 8

[0063] In the (b) of FIG. 7, it is detected by the object tracking section 28 that the object 54f is moving in the direction of the lower edge 52d of the captured image. Therefore, it is determined that the object 54f detected in the (a) of FIG. 7 is the same object as the object 54f detected in the (b) of FIG. 7. Further, since the configuration of the detection region 60f of the object 54f detected in the (a) of FIG. 7 is the same as the configuration of the detection region 60f of the object 54f detected in the (b) of FIG. 7, it is determined that the object 54f detected in the (b) of FIG. 7 is detected as a whole. Therefore, it is determined that the lower end position of the object 54f detected in the (b) of FIG. 7 coincides with the lower end position 70f of the detection region 60f indicated in the (b) of FIG. 7. Figure 8 Figure 8 Figure 8 Figure 8 Figure 8 Figure 8 Figure 8 Figure 8

[0064] Figure 8 The (c) of FIG. 8 indicates a captured image 50c after 1 frame or several frames from the (b) of FIG. 7. In the captured image 50c, the lower end of the object 54f is positioned on the lower side than the lower edge 52d, and the lower side portion of the object 54f is outside the angle of view of the camera 20. In the example of the (c) of FIG. 8, for the sake of explanation, the position of the object 54f in the (b) of FIG. 7 is also indicated by a broken line, but the object 54f indicated by the broken line is not captured in the actual captured image 50c. The object tracking section 28 determines the amount of movement or the direction of movement as indicated by an arrow 78c based on the difference from the position of the object 54f in the past frame to the position of the object 54f in the current frame. Figure 8 Figure 8 Figure 8

[0065] ​​​​​​​​​​​​The lower end estimation section 30 estimates the lower end position of the object. The lower end estimation section 30 estimates the lower end position of the object that is likely to be positioned lower than the lower edge of the captured image, in a case where the object positioned at the lower edge of the captured image is detected. Further, the lower end estimation section 30 estimates the lower end position of the object on the basis of the size of the object that is positioned above the lower edge of the captured image.

[0066] In Figure 8 (c) of FIG. 8, the object 54f is detected by the object tracking section 28 to be moving in the direction of the lower edge 52d of the captured image. In Figure 8 (c) of FIG. 8, since the entirety of the object 54f is not included in the captured image 50c, the recognition score of the first detection model becomes a low value. However, since the object 54f is estimated by the object tracking section 28 to be present at a position overlapping the lower edge 52d of the captured image, the first detection section 24A sets the detection region 60f shown in Figure 8 (c) of FIG. 8 as the detection region of the object 54f. Thus, it is determined that the lower end position of the object 54f detected in Figure 8 (c) of FIG. 8 coincides with the lower end position 70f of the detection region 60f shown in Figure 8 (c) of FIG. 8.

[0067] The lower end estimation section 30 estimates the lower end position 70f of the time point of Figure 8 (b), for example, on the basis of the lower end position 70f of the time point of Figure 8 (b) and the movement of the object 54f indicated by the arrow 78c that is tracked from the time point of Figure 8 (b) to the time point of Figure 8 (c). For example, as indicated by the arrow 80c, the lower end position 70f of the time point of Figure 8 (c) can be estimated by adding the amount of movement (movement vector) indicated by the arrow 78c to the lower end position 70f of the time point of Figure 8 (b). The lower end estimation section 30 estimates the lower end position 70f of the time point of

[0068] In Figure 8 (c) of FIG. 8, since the entirety of the object 54f is not included in the captured image 50c, the detection region of the object 54f within the captured image 50c is likely to be detected with a smaller up-and-down direction size than the detection region 60f of the object 54f detected in Figure 8 (b). In this case, the detection region 60f of the object 54f can also be estimated to be positioned at a position overlapping the lower edge 52d of the captured image, as the detection region 60f shown in Figure 8 (c) of FIG. 8. Specifically, the lower end estimation section 30 estimates the lower end position 70f of the time point of Figure 9the object 54f in (c) is detected, the lower end position of the object 54f is estimated based on the size of the object 54f in the up-down direction, i.e., the size of the object 54f located on the upper side than the lower edge portion 52d, detected in (a) or (b). For example, based on the position of the detection region 60f of the object 54f detected in (a) or (b) and the size of the detection region 60f, the size of the object 54f in the up-down direction detected by the object tracking section 28 is used to estimate the lower end position of the object 54f. Figure 10 the object 54f in (c) is detected, the lower end position of the object 54f is estimated based on the size of the object 54f in the up-down direction, i.e., the size of the object 54f located on the upper side than the lower edge portion 52d, detected in (a) or (b). For example, based on the position of the detection region 60f of the object 54f detected in (a) or (b) and the size of the detection region 60f, the size of the object 54f in the up-down direction detected by the object tracking section 28 is used to estimate the lower end position of the object 54f. Figure 11 the object 54f in (c) is detected, the lower end position of the object 54f is estimated based on the size of the object 54f in the up-down direction, i.e., the size of the object 54f located on the upper side than the lower edge portion 52d, detected in (a) or (b). For example, based on the position of the detection region 60f of the object 54f detected in (a) or (b) and the size of the detection region 60f, the size of the object 54f in the up-down direction detected by the object tracking section 28 is used to estimate the lower end position of the object 54f. Figure 11 the object 54f in (c) is detected, the lower end position of the object 54f is estimated based on the size of the object 54f in the up-down direction, i.e., the size of the object 54f located on the upper side than the lower edge portion 52d, detected in (a) or (b). For example, based on the position of the detection region 60f of the object 54f detected in (a) or (b) and the size of the detection region 60f, the size of the object 54f in the up-down direction detected by the object tracking section 28 is used to estimate the lower end position of the object 54f. Figure 8 the object 54f in (c) is detected, the lower end position of the object 54f is estimated based on the size of the object 54f in the up-down direction, i.e., the size of the object 54f located on the upper side than the lower edge portion 52d, detected in (a) or (b). For example, based on the position of the detection region 60f of the object 54f detected in (a) or (b) and the size of the detection region 60f, the size of the object 54f in the up-down direction detected by the object tracking section 28 is used to estimate the lower end position of the object 54f.

[0069] The distance calculating section 16A calculates the distance information of the object using the lower end position of the object estimated by the lower end estimating section 30. The distance calculating section 16A can calculate the distance information of the object using the same method as the distance calculating section 16 involved in the first embodiment described above.

[0070] Figure 11 is a flowchart showing an example of the flow of the recognition processing method involved in the second embodiment. The image acquiring section 12 acquires the captured image captured by the camera 20 (step S50). The object detecting section 14A starts the scanning using the detection window with respect to the captured image and starts detecting the object using the first detection model (step S52).

[0071] Next, the object detecting section 14A determines whether the object is detected from the captured image captured by the camera 20 (step S54). In a case where it is determined that the object is detected in step S54 (YES in step S54), the object tracking section 28 tracks the object in a plurality of frames and determines the movement of the object (step S56). In a case where the detection region of the tracked object is located at a position overlapping the lower edge of the captured image (YES in step S58), the lower end estimating section 30 estimates the lower end position of the object based on the movement of the object (step S60). In a case where the detection region of the tracked object is not located at a position overlapping the lower edge of the captured image (NO in step S58), the lower end estimating section 30 estimates the lower end position of the object based on the lower end position of the detection region (step S62).

[0072] After step S60 and step S62, the distance calculating section 16A calculates the distance information of the object using the estimated lower end position of the object (step S64). The output control section 18 outputs the object information related to the detected object (step S66). In a case where it is determined that the object is not detected in step S54 (NO in step S54), the processing of steps S56 to S66 can be skipped.

[0073] According to the present embodiment, even if the lower end position of the object is not included in the angle of view of the captured image due to being located at the lower edge of the captured image, the lower end position of the object can be estimated. As a result, the distance to the object located near the camera 20 can be more appropriately calculated.

[0074] (Third Embodiment) Figure 11 (a) and (b) of FIG. 10 are block diagrams that schematically represent the functional configuration of the recognition processing apparatus 10B according to the third embodiment. The recognition processing apparatus 10B according to the third embodiment differs from the above-described embodiments in that, in a case where the object exists at a position overlapping the lower edge of the captured image, the object is detected by switching from the first detection model to the second detection model. Hereinafter, the third embodiment will be described focusing on the differences from the first and second embodiments, and the commonalities will be appropriately omitted.

[0075] The recognition processing apparatus 10B includes the image acquisition section 12, the object detection section 14B, the lower end estimation section 30B, the distance calculation section 16B, and the output control section 18. The image acquisition section 12 and the output control section 18 can be configured similarly to the first or second embodiment. The object detection section 14B includes the first detection section 24B and the second detection section 26B.

[0076] The first detection section 24B can be configured similarly to the first detection section 24A according to the second embodiment. The first detection section 24B detects the object using the first detection model that has been machine-learned on the entire image of the object. The first detection section 24B detects the object included in the captured image using the first detection model. The first detection section 24B detects the object located near the center of the captured image using the first detection model, and also detects the object located at the outer edge of the captured image using the first detection model. The first detection section 24B detects the object located at the lower edge of the captured image using the first detection model.

[0077] In a case where the object included at a position overlapping the lower edge of the captured image is detected by the first detection section 24B, the second detection section 26B detects the object included at a position overlapping the lower edge of the captured image using the second detection model that has been machine-learned on the upper partial image of the object. The second detection section 26B detects the object that is not included in the captured image due to a part of the upper side of the object included at a position overlapping the lower edge of the captured image being included in the captured image and the lower side of the object being outside the angle of view, using the second detection model.

[0078] Figure 11 (a) and (b) of FIG. 10 are block diagrams that schematically represent the functional configuration of the recognition processing apparatus 10B according to the third embodiment. The recognition processing apparatus 10B according to the third embodiment differs from the above-described embodiments in that, in a case where the object exists at a position overlapping the lower edge of the captured image, the object is detected by switching from the first detection model to the second detection model. Hereinafter, the third embodiment will be described focusing on the differences from the first and second embodiments, and the commonalities will be appropriately omitted.Figure 11 (a) and (b) represent the embodiments described above, as per the second embodiment. Figure 11 (c) The same image 50c. Figure 12 (a) schematically illustrates the state in which the object 54g is detected by the first detection unit 24B. The first detection unit 24B scans the detection window against the captured image 50c and detects the object using a first detection model. ​ In (a), since the entire object 54g is not included in the captured image 50c, it indicates that the lower position 70g of the detection area 60g when the object 54g is detected is different from the original lower position of the object 54g. ​ (b) schematically illustrates the state in which the object 54g is detected by the second detection unit 26B. The second detection unit 26B scans a detection window on the captured image 50c, targeting the area containing the detection region where the object was detected by the first detection model and its surrounding area, and detects the object using the second detection model. ​ In (b), it indicates that the object 54g is contained within the area overlapping with the lower edge of the captured image 50c, and it is presumed that there is a lower end position 70g of the detection area 60g when the object 54g is detected by using the second detection model.

[0079] The lower end estimation unit 30B estimates the lower end position of the object. When the first detection unit 24B detects that the object is contained within a range overlapping with the lower edge of the captured image, the lower end estimation unit 30B takes the lower end position of the detection area detected by the second detection unit 26B as the lower end position of the object. For example, in... ​ In the case shown in (b), the lower end position 70g of the detection area 60g of the object 54g detected by the second detection unit 26B is taken as the lower end position of the object 54g.

[0080] The distance calculation unit 16B calculates the distance information of the object using the lower end position of the object estimated by the lower end estimation unit 30B. The distance calculation unit 16B can calculate the distance information of the object using the same method as the distance calculation unit 16 according to the first embodiment described above.

[0081] ​ This is a flowchart illustrating an example of the flow of the identification processing method according to the third embodiment. The image acquisition unit 12 acquires the captured image taken by the camera 20 (step S70). The object detection unit 14B begins scanning the captured image using a detection window and begins detecting objects using the first detection model (step S72).

[0082] Next, the object detection section 14B determines whether an object has been detected from the captured image captured by the camera 20 (step S74). In a case where it is determined in step S74 that an object has been detected ("Yes" in step S74), the object detection section 14B determines whether the detection region of the object detected in step S74 is included in a range overlapping with the lower edge of the captured image (step S76). In a case where it is determined in step S76 that the detection region of the detected object is included in the range overlapping with the lower edge of the captured image ("Yes" in step S76), the object detection section 14B detects the object using the second detection model (step S78). In addition, the lower end estimation section 30B estimates the lower end position of the object based on the lower end position of the detection region of the object detected in step S78 by the second detection model (step S80). In a case where it is determined in step S76 that the detection region of the detected object is not included in the range overlapping with the lower edge of the captured image ("No" in step S76), the lower end estimation section 30B estimates the lower end position of the object based on the lower end position of the detection region of the object detected by the first detection model (step S82).

[0083] After step S80 and step S82, the distance calculation section 16B calculates the distance information of the object using the estimated lower end position of the object (step S84). The output control section 18 outputs the object information related to the detected object (step S86). In a case where it is determined in step S74 that no object has been detected ("No" in step S74), the processing of steps S76 to S86 can be skipped.

[0084] According to the present embodiment, in a case where the lower end position of the object is not included in the angle of view of the captured image due to being located at the lower edge of the captured image, the lower end position of the object can be estimated by detecting the object using the second detection model. As a result, the distance information of the object located near the camera 20 can be more appropriately calculated.

[0085] The present application has been described above with reference to the above-described embodiments, but the present application is not limited to the above-described embodiments, and includes contents obtained by appropriately combining or replacing each configuration shown in the embodiments.

[0086] Hereinafter, several modes of the present disclosure will be described.

[0087] A first mode of the present disclosure is an identification processing apparatus including: an image acquisition section that acquires a captured image; and an object detection section that detects an object located at a position away from an outer edge of the captured image using a first detection model that has been machine-learned on an entire image of the object, and detects an object located at the outer edge of the captured image using a second detection model that has been machine-learned on a partial image of the object.

[0088] A second aspect of the present disclosure is an identification processing method in which an identification processing device executes the steps of: acquiring a captured image; detecting an object located at a position away from an outer edge of the captured image using a first detection model that has been subjected to machine learning on an entire image of an object, and detecting an object located at the outer edge of the captured image using a second detection model that has been subjected to machine learning on a partial image of an object.

[0089] A third aspect of the present disclosure is a computer program, or a non-transitory storage medium storing the computer program, the computer program being configured to cause a processor to execute the functions of: acquiring a captured image; detecting an object located at a position away from an outer edge of the captured image using a first detection model that has been subjected to machine learning on an entire image of an object, and detecting an object located at the outer edge of the captured image using a second detection model that has been subjected to machine learning on a partial image of an object.

[0090] A fourth aspect of the present disclosure is an identification processing device including: an image acquisition unit that acquires a captured image; an object detection unit that detects an object included in the captured image using a detection model that has been subjected to machine learning on an image of an object; a lower end estimation unit that estimates a lower end position of the object that is likely to be located on a lower side than a lower edge of the captured image, in a case where an object included in a range overlapping with the lower edge of the captured image is detected by the object detection unit; and a distance calculation unit that calculates distance information of the object using the lower end position estimated by the lower end estimation unit.

[0091] A fifth aspect of the present disclosure is an identification processing method in which the identification processing device executes the steps of: acquiring a captured image; detecting an object included in the captured image using a detection model that has been subjected to machine learning on an image of an object; estimating a lower end position of the object that is likely to be located on a lower side than a lower edge of the captured image, in a case where an object included in a range overlapping with the lower edge of the captured image is detected; and calculating distance information of the object using the estimated lower end position.

[0092] A sixth aspect of the present disclosure is a computer program, or a non-transitory storage medium storing the computer program, the computer program being configured to cause a processor to execute: a function of acquiring a captured image; a function of detecting an object included in the captured image using a detection model that has been machine-learned with respect to images of objects; a function of, in a case where an object included in a range overlapping with a lower edge of the captured image is detected, estimating a lower end position of the object that is likely to be positioned lower than the lower edge of the captured image; and a function of calculating distance information of the object using the estimated lower end position.

[0093] Industrial applicability According to one embodiment of the present application, a technique can be provided that more appropriately detects an object in image recognition processing.

[0094] Symbol explanation 10…recognition processing device, 12…image acquisition unit, 14…object detection unit, 16…distance calculation unit, 50…captured image, 52…outer edge, 52d…lower edge.

Claims

1. An identification processing device, comprising: The image acquisition department acquires captured images; The object detection unit uses a detection model that has undergone machine learning to analyze images of objects to detect objects contained in the captured image. The lower estimation unit, when the object detection unit detects an object within a range overlapping the lower edge of the captured image, estimates the lower end position of the object that may be located below the lower edge of the captured image; and The distance calculation unit calculates the distance information of the object using the lower end position estimated by the lower end estimation unit.

2. The identification processing device as described in claim 1, further comprising: The object tracking unit tracks the objects detected by the object detection unit. The lower end estimation unit estimates the lower end position of the object that may be located below the lower edge of the captured image, based on the movement of the object as it moves from the lower end position of the object tracked by the object tracking unit toward the lower edge of the captured image.

3. The identification processing device as described in claim 2, wherein, The lower end estimation unit estimates the lower end position of the object based on the size of the object detected by the object detection unit, which is above the lower edge of the captured image.

4. The identification processing device as described in claim 1, wherein, The object detection unit uses a first detection model that performs machine learning on the overall image of the object to detect objects, and detects objects contained in the captured image. When the object detection unit detects an object within the range overlapping with the lower edge of the captured image using the first detection model, it uses a second detection model that has performed machine learning on the upper part of the object image to detect the object within the range overlapping with the lower edge of the captured image. The lower end estimation unit uses the lower end position of the detection area of ​​the object detected by the second detection model to estimate the lower end position of the object.

5. The identification processing device as described in claim 4, wherein, When the object detection unit detects an object whose upper part is included in the captured image but whose lower part is outside the view and not included in the captured image using the first detection model, and considers it as an object included in the range overlapping with the lower edge of the captured image, the second detection model is used to detect objects included in the range overlapping with the lower edge of the captured image.

6. An identification processing method, wherein the identification processing device performs the following steps: Acquire captured images; The detection model, which uses machine learning to detect images of objects, detects objects contained in the captured images; When an object is detected that is contained within a range overlapping the lower edge of the captured image, it is presumed that the object is located at a lower end position, possibly below the lower edge of the captured image; and The distance information of the object is calculated using the estimated lower position.

7. A computer program configured to cause a processor to perform the following functions: Acquire captured images; The detection model, which uses machine learning to detect images of objects, detects objects contained in the captured images; When an object is detected that is contained within a range overlapping the lower edge of the captured image, it is presumed that the object is located at a lower end position, possibly below the lower edge of the captured image; and The distance information of the object is calculated using the estimated lower position.

Images