Movable body information calculation method and movable body information calculation program

Abstract

To prevent an erroneous distance from being calculated.SOLUTION: A processing part 1a of an information processing device 1 acquires photographic images 11, 12 photographed by a camera 2 installed on a first movable body. The processing part 1a detects a second movable body (a vehicle 13 in the example of Fig. 1) from the photographic images 11, 12, and detects a shield region 14 reflected at a lower side in the photographic images 11, 12 to shield a portion of the road region. The processing part 1a determines whether a prescribed condition showing that at least a portion of lower sides 15a, 16a of circumscribed rectangles 15, 16 that circumscribe a second movable body overlaps or is adjacent to the shield region 14 is satisfied, on the basis of detection results of the second movable body and the shield region 14. The processing part 1a calculates a distance between the first movable body and the second movable body on the basis of coordinates of reference points 15b, 16b on the lower sides when the prescribed condition is not satisfied, and skips distance calculation processing when the prescribed condition is satisfied.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a mobile object information calculation method and a mobile object information calculation program. [Background technology]

[0002] The distance between a vehicle and another vehicle may be calculated from an image captured by a camera installed on the vehicle. For example, distance is calculated when calculating the position and speed of the other vehicle involved in an accident from images captured in front of and behind the vehicle by the vehicle's onboard camera. The distance may be calculated based on the positional relationship between the position of the vehicle's camera and the reference point of the bottom side of a circumscribing rectangle (sometimes called a bounding box) that circumscribes the other vehicle. When the circumscribing rectangle surrounds the entire vehicle, the bottom side of the circumscribing rectangle is in contact with the road surface, so the height can be considered to be 0, and therefore the reference point of the bottom side of the circumscribing rectangle is used to calculate the distance.

[0003] In addition, a technology has been proposed in the past in which a tracking target contained in a captured image is recognized as a tracking area, and based on the presence of an area including movement within the tracking area, it is determined whether or not to stop tracking when the target becomes obscured by an obstacle, for example. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2016-85487 A Summary of the Invention [Problem to be solved by the invention]

[0005] When the distance is calculated using the reference point of the lower side of the circumscribing rectangle as described above, if the lower side is not detected properly, an error may occur in the calculated distance. For example, when a target vehicle is close to the subject vehicle, the lower part of the target vehicle may be occluded by a part of the subject vehicle (such as the bonnet or dashboard) in the captured image. In that case, the circumscribing rectangle that circumscribes the part detected as the vehicle does not include the occluded lower part of the target vehicle, so the lower side of the circumscribing rectangle may not be in contact with the road surface. An error may occur in the distance calculated using such a reference point of the lower side.

[0006] According to one aspect, the present invention has an object to provide a moving object information calculation method and a moving object information calculation program capable of preventing an erroneous distance calculation. [Means for solving the problem]

[0007] In one proposal, a moving body information calculation method is provided in which a computer acquires a first captured image taken by a camera installed on a first moving body, detects a second moving body from the first captured image, detects an occluded area that appears below a road area in the first captured image and occludes part of the road area, determines whether a predetermined condition is satisfied indicating that at least a portion of the bottom edge of a circumscribing rectangle circumscribing the detected second moving body overlaps or is close to the occluded area based on the detection results of the second moving body and the occluded area, calculates a distance between the first moving body and the second moving body based on the coordinates of a reference point on the bottom edge if the predetermined condition is satisfied, and skips the distance calculation process if the predetermined condition is satisfied.

[0008] In addition, in one proposal, a mobile object information calculation program is provided that causes a computer to execute a process similar to the mobile object information calculation method described above. Effect of the Invention

[0009] In one aspect, it is possible to prevent an erroneous distance from being calculated. [Brief description of the drawings]

[0010] [Figure 1] 1 illustrates a configuration example and a processing example of an image processing system according to a first embodiment; [Diagram 2] FIG. 13 illustrates an example of a configuration of an image processing system according to a second embodiment. [Diagram 3] FIG. 11 is a diagram illustrating an example of detection of a target vehicle. [Figure 4] FIG. 1 is a diagram showing an example of the geometric relationship between a reference point and a position on a road surface when a camera position is vertically projected onto the road surface (part 1). [Diagram 5] FIG. 2 is a diagram showing an example of the geometric relationship between a position on the road surface and a reference point when the camera position is vertically projected onto the road surface (part 2). [Figure 6] FIG. 2 is a diagram illustrating an angle of view and a focal length. [Figure 7] FIG. 13 is a diagram showing an example of a captured image in which a part of a target vehicle is obstructed. [Figure 8] FIG. 2 is a diagram illustrating an example of the configuration of processing functions included in an image processing apparatus. [Figure 9] FIG. 4 is a diagram illustrating an example of vehicle information storage. [Figure 10] FIG. 11 is a diagram illustrating an example of storage of a target detection result. [Figure 11] FIG. 13 is a diagram showing an example of detection of a road area and an obstructed area. [Figure 12] 4 is a flowchart showing an example of an overall process of the image processing device. [Figure 13] 10 is a flowchart illustrating an example of a process for detecting an occluded area. [Figure 14] 11 is a flowchart showing a flow of an example of a distance measurement feasibility determination process. [Figure 15] 13 is a flowchart illustrating an example of a target speed calculation process. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, an embodiment of the invention will be described with reference to the drawings. First Embodiment 1 is a diagram showing an example of the configuration and processing of an image processing system according to a first embodiment. The image processing system shown in FIG.

[0012] The information processing device 1 includes a processing unit 1a. The information processing device 1 is realized, for example, as a personal computer or a server computer. The processing unit 1a is, for example, a processor. In this case, the processing of the processing unit 1a described below can be realized by the processor executing a predetermined program.

[0013] The information processing device 1 calculates the distance between a first moving body and a second moving body that appears in an image captured by a camera 2 installed on the first moving body. However, if the bottom side of a circumscribing rectangle that circumscribes the second moving body is not properly detected, and the reference point of the bottom side is used to calculate the distance, an incorrect distance may be calculated (a calculation error may occur).

[0014] The processing unit 1a of the information processing device 1 performs a process to prevent an erroneous calculation of the distance in the following procedure. Note that, hereinafter, calculating the distance may be referred to as distance measurement. The processing unit 1a acquires a photographed image captured by a camera 2 installed on the first moving body. Then, the processing unit 1a detects a second moving body from the photographed image. FIG. 1 shows examples of the acquired photographed images 11 and 12. A vehicle 13, which is an example of the second moving body, is captured in the photographed images 11 and 12. The second moving body is detected, for example, using a machine learning model that has previously learned about a moving body (such as a vehicle) to be detected. Even when the vehicle 13 is not captured in its entirety because it is blocked by a blocked area described below, as in the photographed image 12, it may be detected as the second moving body.

[0015] The processing unit 1a also detects an occluded area that appears below the road area in the acquired photographed image and occludes a part of the road area. For example, the processing unit 1a can detect the road area from the acquired photographed image by a semantic segmentation technique using a machine learning model that has previously learned the road area. Then, the processing unit 1a can detect the non-road area below the detected road area as an occluded area.

[0016] FIG. 1 shows an example of a blocked area 14 detected from the captured images 11 and 12. Such a blocked area 14 is an area in which a part of the first moving body on which the camera 2 is mounted is captured in the captured images 11 and 12. When the first moving body is an automobile and the camera 2 captures an image in front of the automobile, the blocked area 14 is a part of the automobile, such as a bonnet or a dashboard. For example, when the second moving body is in close proximity to the first moving body, the blocked area 14 may block a part of the second moving body in the captured image. In the captured image 11, the blocked area 14 blocks a part of the road area, but in the captured image 12, the blocked area 14 also blocks a part of a vehicle 13, which is an example of the second moving body.

[0017] When the camera 2 captures the rear of a vehicle, the rear of the vehicle may appear below the road area. In this case, the rear of the vehicle can be detected as an obstructed area that appears below the road area in the captured image.

[0018] The order of the above-mentioned second moving object detection process and the blocking area detection process may be interchanged. Next, the processing unit 1a determines whether a predetermined condition indicating that at least a part of the bottom side of a circumscribing rectangle circumscribing the detected second moving object overlaps or is close to the occluded area is satisfied based on the detection result of the second moving object and the occluded area. The circumscribing rectangle is set, for example, when the second moving object is detected. The circumscribing rectangle is also called a bounding box.

[0019] If the above-mentioned predetermined condition is not satisfied, the processing unit 1a calculates the distance between the first moving body and the second moving body based on the coordinates of the reference point on the bottom side of the circumscribing rectangle. If the above-mentioned predetermined condition is satisfied, the processing unit 1a skips the distance calculation process.

[0020] In the example of FIG. 1, the predetermined condition is that a part of the bottom side of the circumscribing rectangle overlaps the shielded area 14. In the example of the captured image 11 in FIG. 1, the bottom side 15a of the circumscribing rectangle 15 circumscribing the vehicle 13 is away from the shielded area 14 and does not overlap with the shielded area 14. In this case, a distance calculation process (distance measurement) is performed based on the coordinates of the reference point 15b on the bottom side 15a. In contrast, in the example of the captured image 12 in FIG. 1, a part of the bottom side 16a of the circumscribing rectangle 16 circumscribing a part of the vehicle 13 (the part detected as the second moving body) overlaps with the shielded area 14. For this reason, the distance measurement is not performed.

[0021] 1, bottom side 16a is located above bottom side 16c (which is in contact with the road surface) of an ideal circumscribing rectangle that encloses the entire vehicle 13. In this case, since bottom side 16a is not in contact with the road surface, an error will occur if distance calculation processing is performed using reference point 16b on bottom side 16a.

[0022] In this embodiment, when a predetermined condition is satisfied indicating that at least a part of the bottom side of the circumscribing rectangle overlaps or is close to the occluded area, distance measurement is skipped. In other words, since distance measurement is not performed, it is possible to prevent an erroneous distance from being calculated. In addition, by preventing an erroneous distance from being calculated, it is possible to suppress the occurrence of an error when calculating the position or speed of the second moving body using the distance.

[0023] Furthermore, as described above, in the captured image, the non-road area below the detected road area is detected as the blocked area. Therefore, it is not necessary to set the blocked area in advance, and the blocked area can be detected regardless of the type of the first moving object (such as the type of vehicle) or the type of the camera 2.

[0024] The above-mentioned distance calculation process by the processing unit 1a may be executed, for example, inside the camera 2 or by a device (for example, a drive recorder) in which the camera 2 is mounted. Furthermore, the first moving body and the second moving body are not limited to automobiles, but may be other vehicles such as motorcycles and bicycles, or may be moving bodies other than vehicles, such as people.

[0025] Second Embodiment Next, as a second embodiment, an image processing system will be described that calculates information on a vehicle, which is an example of a second moving body, included in an image captured by a camera installed in a vehicle, which is an example of a first moving body (hereinafter, sometimes referred to as a host vehicle). Note that, hereinafter, the vehicle, which is an example of the second moving body, will be referred to as a target vehicle.

[0026] 2 is a diagram showing an example of the configuration of an image processing system according to the second embodiment. As shown in FIG. Drive recorder 210 is mounted on vehicle 200, which is an example of a first moving body, and includes camera 211 and flash memory 212. Camera 211 is a monocular camera that captures road conditions in the traveling direction of vehicle 200 or the opposite direction to the traveling direction. If the image signal output by camera 211 is an analog signal, it is converted into a digital signal. Also, if the captured image is a color image, it may be converted into a monochrome image. Data of the captured image is coded using a predetermined coding method and stored in flash memory 212 as video image data.

[0027] Furthermore, the drive recorder 210 receives vehicle information for calculating the travel distance of the vehicle 200 from a vehicle information output device 220 mounted on the vehicle 200. Such vehicle information includes, for example, a measurement value of the position of the vehicle 200 (for example, position information by GNSS (GNSS: Global Navigation Satellite System)), a measurement value of the vehicle speed, a vehicle speed pulse according to the vehicle speed, and the like.

[0028] The vehicle information output device 220 is realized, for example, as an ECU (Electronic Control Unit). The drive recorder 210 outputs to the image processing device 100 moving image data and vehicle information for each frame of the moving image data.

[0029] The image processing device 100 acquires video image data and vehicle information from the drive recorder 210. In this embodiment, as an example, the image processing device 100 receives this information from the drive recorder 210 by communication, but for example, the image processing device 100 may acquire this information via a portable recording medium. Using this acquired information, the image processing device 100 calculates the distance between the target vehicle and the vehicle 200 (own vehicle), and the position and speed of the target vehicle.

[0030] The image processing device 100 is realized as, for example, a personal computer or a server computer. In this case, the image processing device 100 includes a processor 101, a RAM (Random Access Memory) 102, a HDD (Hard Disk Drive) 103, a GPU (Graphics Processing Unit) 104, an input interface (I / F) 105, a reader 106, and a communication interface (I / F) 107, as shown in FIG.

[0031] The processor 101 performs overall control of the entire image processing device 100. The processor 101 is, for example, a central processing unit (CPU), a micro processing unit (MPU), a digital signal processor (DSP), an application specific integrated circuit (ASIC), or a programmable logic device (PLD). The processor 101 may also be a combination of two or more elements of a CPU, an MPU, a DSP, an ASIC, or a PLD.

[0032] The RAM 102 is used as a main storage device of the image processing device 100. The RAM 102 temporarily stores at least a part of an OS (Operating System) program and application programs to be executed by the processor 101. The RAM 102 also stores various data necessary for processing by the processor 101.

[0033] The HDD 103 is used as an auxiliary storage device for the image processing device 100. An OS program, an application program, and various data are stored in the HDD 103. Note that other types of non-volatile storage devices such as an SSD (Solid State Drive) can also be used as the auxiliary storage device.

[0034] A display device 104a is connected to the GPU 104. The GPU 104 displays an image on the display device 104a in accordance with an instruction from the processor 101. The display device may be a liquid crystal display or an organic EL (ElectroLuminescence) display.

[0035] An input device 105a is connected to the input interface 105. The input interface 105 transmits a signal output from the input device 105a to the processor 101. The input device 105a may be a keyboard or a pointing device. The pointing device may be a mouse, a touch panel, a tablet, a touch pad, a trackball, or the like.

[0036] A portable recording medium 106a is detachably attached to the reading device 106. The reading device 106 reads data recorded on the portable recording medium 106a and transmits it to the processor 101. The portable recording medium 106a may be an optical disk, a magneto-optical disk, a semiconductor memory, or the like.

[0037] The communication interface 107 transmits and receives data to and from other devices via, for example, a network (not shown). In this embodiment, the video data and vehicle information transmitted from the drive recorder 210 are received by the communication interface 107.

[0038] The processing functions of the image processing device 100 can be realized by the above hardware configuration. The image processing device 100 detects a target vehicle from an image captured by a camera mounted on the drive recorder 210 of the vehicle 200, and calculates the distance between the detected target vehicle and the vehicle 200. The detection result of the target vehicle is output as the position of a circumscribing rectangle (bounding box) that circumscribes the range of the target vehicle in the captured image. The image processing device 100 calculates the position and speed of the target vehicle in real space based on the distance between the target vehicle and the vehicle 200.

[0039] Fig. 3 is a diagram showing an example of detection of a target vehicle, in which road areas are omitted from the example of Fig. 3. 3 shows an example in which a target vehicle 301 is detected from a captured image 300 having an image width w, an image height h, and center coordinates (cx, cy). The detected target vehicle 301 is surrounded by a circumscribing rectangle 302. A reference point 302b on a bottom side 302a of the circumscribing rectangle 302 is used to calculate the distance. In the example of this embodiment, the reference point 302b is the left end of the bottom side 302a.

[0040] It should be noted that the reference point 302b is not limited to the left end of the bottom side 302a of the circumscribing rectangle 302. For example, it may be the right end or the midpoint of the bottom side 302a of the circumscribing rectangle 302. In the following, the X-axis and Y-axis coordinates of the reference point 302b are represented as (x, y).

[0041] Furthermore, a blocked area 200a is captured in the captured image 300. The blocked area 200a is an area that captures a part of the vehicle 200. When the camera 211 captures an image of the area in front of the vehicle 200, the part of the vehicle 200 that is the blocked area 200a includes the hood or dashboard.

[0042] Hereinafter, a method for calculating the distance between the vehicle 200 and the target vehicle 301 from a captured image 300 such as that shown in FIG. 3 will be described. The distance between vehicle 200 and target vehicle 301 can be expressed, for example, as the distance between the position on the road surface when the installation position of camera 211 (hereinafter referred to as the camera position) is projected vertically onto the road surface, and reference point 302b.

[0043] 4 and 5 are diagrams showing examples of the geometric relationship between the position on the road surface when the camera position is vertically projected onto the road surface and the reference point. Fig. 4 shows an example of the geometric relationship between the position 211a on the road surface and the reference point 302b when viewed from above. Fig. 5 shows an example of the geometric relationship between the position 211a on the road surface and the reference point 302b when viewed from the side with respect to the traveling direction of the vehicle 200 and the target vehicle 301.

[0044] In Fig. 4, the distance between position 211a on the road surface and reference point 302b is indicated as D2. D1 shown in Fig. 4 and Fig. 5 represents the distance from position 211a to the target vehicle in the traveling direction of vehicle 200. Specifically, D1 represents the distance between position 211a and a straight line that is perpendicular to the shooting direction of camera 211 in the left-right direction and passes through reference point 302b.

[0045] 4 is the angle between the traveling direction of the vehicle 200 and the direction from the position 211a to the reference point 302b, and is the angle in the lateral direction with respect to the center of the captured image 300.

[0046] H shown in FIG. 5 is the height of the camera 211 from the road surface. H may be measured in advance. H may also be calculated from the captured image 300 (for a method of calculating the camera height from a captured image, see, for example, JP 2021-174288 A). θ1 shown in FIG. 5 is the angle between the optical axis direction of the camera 211 and the direction from the camera 211 to the reference point 302b, and is a vertical angle with respect to the center of the captured image 300. The blocked area 200a of the captured image 300 is, for example, an image of the bonnet 201 of the vehicle 200 in FIG. 5.

[0047] D1 can be calculated by the following formula (1) using H and θ1.

[0048]

number

[0049] D2 can be calculated using D1 and θ2 according to the following formula (2).

[0050]

number

[0051] θ1 and θ2 can be calculated from the angle of view (horizontal angle of view and vertical angle of view) of the camera 211, the focal length, and the image width w and image height h of the captured image 300. FIG. 6 is a diagram illustrating the angle of view and the focal length.

[0052] The focal length is the distance between the center of lens 211b and projection plane 211c on which captured image 300 is obtained. The angle of view represents the range captured in captured image 300. Information on the focal length and angle of view is stored in advance as information on camera 211 in, for example, HDD 103.

[0053] θ1 can be calculated by the following formula (3).

[0054]

number

[0055] y is the Y-axis coordinate of the reference point 302b in the photographed image 300, and cy is the Y-axis coordinate of the center of the photographed image 300. h is the image height of the photographed image 300. θ2 can be calculated by the following equation (4).

[0056]

number

[0057] x is the X-axis coordinate of the reference point 302b in the photographed image 300, and cx is the X-axis coordinate of the center of the photographed image 300. w is the image width of the photographed image 300. The vertical angle of view in equation (3) can be calculated by the following equation (5).

[0058]

number

[0059] The horizontal angle of view in equation (4) can be calculated by the following equation (6).

[0060]

number

[0061] By using the above formulas (1) to (6), it is possible to calculate D1 or D2 as the distance to the target vehicle from the coordinates of the reference point 302b on the captured image. In the example of the captured image 300 in Figure 3, the entire target vehicle 301 is included, but there are cases where a portion of the target vehicle 301 is blocked by the occlusion area 200a, resulting in an image in which the entire target vehicle 301 is not captured.

[0062] Fig. 7 is a diagram showing an example of a captured image in which a part of a target vehicle is occluded. In the example of a captured image 320 shown in Fig. 7, the lower part of the target vehicle 301 is occluded by an occluded area 200a. In such a case, a circumscribing rectangle 321 that surrounds the unoccluded part of the target vehicle 301 may be obtained as a detection result of the target vehicle 301.

[0063] The bottom side 321a of this circumscribing rectangle 321 is located above the captured image 320 with respect to the bottom side 321c (which is in contact with the road surface) of an ideal circumscribing rectangle that encloses the entire target vehicle 301. In this case, the bottom side 321a is not in contact with the road surface. If distance calculation processing is performed using a reference point 321b on such a bottom side 321a, an error will occur. Because the reference point 321b is located higher than the road surface, θ1 shown in FIG. 5 and formula (3) cannot be calculated correctly, and therefore D1 and D2 cannot be calculated correctly either.

[0064] In order to prevent an erroneous distance calculation, the image processing device 100 of this embodiment skips distance measurement when a predetermined condition is satisfied indicating that at least a portion of the bottom side 321a of the circumscribing rectangle 321 overlaps or is close to the occluded area 200a, as shown in Figure 7.

[0065] 8 is a diagram showing an example of the configuration of processing functions of an image processing device. As shown in FIG. 8, the image processing device 100 includes a storage unit 110, an image input unit 121, an object detection unit 122, an obstructed area detection unit 123, a distance measurement feasibility determination unit 124, and an object speed calculation unit 125.

[0066] The storage unit 110 is realized as a storage area of ​​a storage device included in the image processing device 100, such as the RAM 102 or the HDD 103. The storage unit 110 stores, for example, video data 111, camera information 112, vehicle information 113, object detection results 114, and object speed calculation results 115.

[0067] The video data 111 is data of captured images generated by the camera 211 and transmitted from the drive recorder 210. Camera information 112 includes the vertical and horizontal angles of view of camera 211 used to calculate θ1 and θ2 in equations (3) and (4), and the height H of camera 211 from the road surface used to calculate D1 in equation (1).

[0068] The vehicle information 113 includes a measurement value of the position of the vehicle 200 when the captured image of each frame is captured, measured by, for example, GNSS. The vehicle information 113 may also include a measurement value of the speed of the vehicle 200 when the captured image of each frame is captured, a vehicle speed pulse corresponding to the speed, and the like.

[0069] The target detection result 114 includes the detection result of the vehicle to be detected (target vehicle). The detection result is expressed by the coordinates of the four corners of the circumscribing rectangle of the target vehicle. If there are multiple target vehicles, an identification number may be assigned to the detection result of each target vehicle so that each target vehicle can be distinguished and tracked, or the attributes of each target vehicle can be distinguished.

[0070] Target speed calculations 115 include speed calculations calculated for each target vehicle. The processes of the image input unit 121, the target detection unit 122, the blocked area detection unit 123, the distance measurement feasibility determination unit 124, and the target speed calculation unit 125 are realized by, for example, the processor 101 executing a predetermined program.

[0071] The image input unit 121 acquires video data 111 including captured images of each frame and vehicle information 113 from the drive recorder 210. The image input unit 121 stores the video data 111 and the vehicle information 113 in the storage unit 110. The image input unit 121 also inputs the captured images to the object detection unit 122 for each frame or at intervals of a predetermined number of frames.

[0072] The object detection unit 122 detects an object vehicle from the input captured image. The object detection unit 122 outputs an object detection result 114 represented by the coordinates of the four corners of a circumscribing rectangle of the detected object vehicle. The object detection result 114 is stored in the storage unit 110.

[0073] The detection of the target vehicle by the target detection unit 122 is performed, for example, using a trained model for vehicle detection. Such a trained model is generated, for example, by machine learning (for example, deep learning) using a large number of images in which the vehicle to be detected is captured as training data. The training data includes position information of a circumscribing rectangle corresponding to the position of the vehicle, and the position information is used as ground truth data during machine learning.

[0074] In addition, when there are multiple target vehicles to be detected, the target detection unit 122 may output an identification number of the detection result of each target vehicle so that each target vehicle can be distinguished and tracked, or so that the attributes of each target vehicle can be distinguished.

[0075] The occluded area detection unit 123 detects a road area from a captured image, and detects a non-road area below the road area as an occluded area. The detection of the road area by the occluded area detection unit 123 is performed, for example, by semantic segmentation using a trained model. Such a trained model is generated, for example, by machine learning (for example, deep learning) using as training data a large number of images classified into road areas and other areas for each pixel.

[0076] The distance measurement feasibility determination unit 124 determines whether or not the distance between the vehicle 200 and the detected target vehicle can be calculated (distance measurement) based on the target detection result 114 and the detection result of the blocked area. The distance measurement feasibility determination unit 124 determines that distance measurement is not possible when a part of the bottom side of the circumscribing rectangle of the target vehicle is included in the blocked area. For example, the distance measurement feasibility determination unit 124 determines that distance measurement is possible when the bottom side of the circumscribing rectangle of the target vehicle is not included in the blocked area. Note that even if the bottom side of the circumscribing rectangle of the target vehicle is not included in the blocked area, the distance measurement feasibility determination unit 124 may determine that distance measurement is not possible when at least a part of the bottom side is close to the blocked area. For example, when the distance on the captured image between at least a part of the bottom side of the circumscribing rectangle and the blocked area is within a predetermined value, the distance measurement feasibility determination unit 124 can determine that at least a part of the bottom side of the circumscribing rectangle is close to the blocked area.

[0077] The target speed calculation unit 125 calculates the distance, position, and speed of a target vehicle that has been determined to be distance-measurable. FIG. 9 is a diagram illustrating an example of how vehicle information is stored.

[0078] 9, the measured value of the position of the vehicle 200 (host vehicle position) at each time is expressed as a two-dimensional coordinate system of, for example, latitude and longitude. The host vehicle position at time 1 is expressed as (X2, Y2), and the host vehicle position at time n is expressed as (X1, Y1).

[0079] FIG. 10 is a diagram illustrating an example of storage of the object detection result. In the storage example of the object detection result 114 in FIG. 10, the detection result of the object vehicle detected from the captured image of the frame at each time is represented by the coordinates of the four corners of an outer rectangle. Each of the four corner coordinates is represented by the X-axis coordinate and the Y-axis coordinate in the captured image. The four corner coordinates at time 1 are represented as (xp1, yp1, xp2, yp2, xp3, yp3, xp4, yp4), and the four corner coordinates at time n are represented as (x1, y1, x2, y2, x3, y3, x4, y4). When multiple object vehicles are detected from the same frame, multiple sets of the coordinates of the four corners of the outer rectangle are stored in the record of the same frame.

[0080] FIG. 11 is a diagram showing an example of detection of a road area and an obstructed area. 11 shows a circumscribing rectangle 331 obtained by detecting a target vehicle, a road area 332, and a blocked area 200a. The road area 332 includes a roadway area 332a, a roadside strip area 332b, and a sidewalk area 332c. The blocked area 200a is the bonnet or dashboard of the vehicle 200, and in the example of Fig. 11, it blocks part of the road area 332. The way parts of the vehicle 200, such as the bonnet or dashboard, are captured by the camera 211 varies widely depending on the drive recorder 210 and the vehicle model.

[0081] In contrast, there is little variation in how the road area 332 is captured by the camera 211. For this reason, for example, by using a semantic segmentation technique using a machine learning model that has previously learned road areas, the road area 332 can be accurately detected from the captured image 330. Then, in the captured image 330, the non-road area below the detected road area 332 can be detected as the blocked area 200a.

[0082] Next, the processing of the image processing device 100 will be described with reference to a flowchart. FIG. 12 is a flowchart showing an example of the overall process of the image processing device. [Step S11] The image input unit 121 acquires one frame of a captured image to be processed from the video data 111 captured by the drive recorder 210, and inputs the acquired captured image to the object detection unit 122. The image input unit 121 also acquires information corresponding to the acquired captured image from the vehicle information 113 (for example, position information of the vehicle 200).

[0083] In step S11, for example, captured images are acquired one frame at a time from the beginning of the video data 111. Alternatively, captured images may be acquired at intervals of a predetermined number of frames from the beginning of the video data 111.

[0084] [Step S12] The object detection unit 122 performs a process of detecting an object vehicle from the input captured image. If an object vehicle is detected, an object detection result 114 including the coordinates of the four corners of a circumscribing rectangle corresponding to the object vehicle is output. Note that if no object vehicle is detected, steps S13 to S15 are skipped.

[0085] [Step S13] The masked region detection unit 123 detects a masked region from the captured image. An example of the procedure for detecting a masked region will be described later (see FIG. 13). [Step S14] The distance measurement feasibility determination unit 124 determines whether or not it is possible to calculate (measure) the distance between the vehicle 200 and the detected target vehicle, based on the target detection result 114 output in step S12 and the detection result of the blocked area. An example of the procedure for the distance measurement feasibility determination process will be described later (see FIG. 14).

[0086] [Step S15] The target speed calculation unit 125 calculates the distance, the position, and the speed of the target vehicle for which distance measurement is possible. An example of the procedure of these processes (target speed calculation process) by the target speed calculation unit 125 will be described later (see FIG. 15).

[0087] [Step S16] The image processing device 100 determines whether to end the process. For example, the image processing device 100 determines to end the process when the processing of all frames of the video data 111 captured by the drive recorder 210 is completed, or when the end of the speed calculation process is instructed by a user operation. If the image processing device 100 determines not to end the process, the process from step S11 is repeated for the next frame.

[0088] With the above, the processing of the image processing device 100 ends. In addition, when step S13 is executed regardless of the processing result of step S12, the order of the processing of steps S12 and S13 can be changed. If steps S13 and S12 are executed in this order and the target vehicle is not detected, steps S14 and S15 are skipped.

[0089] FIG. 13 is a flowchart showing an example of the flow of the detection process of the obstructed area. [Step S21] The masked area detection unit 123 detects a road area from the captured image of the detection target of the target vehicle. For example, the masked area detection unit 123 sets a flag value for each pixel of the captured image indicating whether or not the pixel belongs to a road area. For example, the flag value is set to 1 if the pixel belongs to a road area, and 0 if the pixel belongs to a non-road area.

[0090] [Step S22] The masked area detection unit 123 starts searching the captured image in the horizontal direction to detect a masked area. For example, the masked area detection unit 123 first selects a pixel in the upper left corner of the captured image.

[0091] [Step S23] The masked area detection unit 123 starts searching the captured image in the vertical direction of the image. For example, when the masked area detection unit 123 selects a pixel in the upper left corner of the captured image in the process of step S22, the masked area detection unit 123 selects pixels included in the vertical line including the selected pixel in the downward direction in order, and performs the following process.

[0092] [Step S24] The masked area detection unit 123 determines whether or not a road area is present. In the process of step S24, if the selected vertical line includes a pixel that belongs to a road area, it is determined that a road area is present. For example, if a pixel has a flag value of 1 set for that pixel, the masked area detection unit 123 determines that the pixel belongs to a road area. If the masked area detection unit 123 determines that a road area is present, it performs the process of step S25. If the masked area detection unit 123 determines that a road area is not present, it performs the process of step S27.

[0093] [Step S25] If the masked area detection unit 123 determines that there is a road area, it determines whether or not there is a non-road area below the road area. In the process of step S25, the masked area detection unit 123 identifies the bottom pixel belonging to the road area from the pixels of the vertical line, and searches whether or not there is a pixel belonging to the non-road area below that (i.e., whether or not the road area reaches below the line). If the masked area detection unit 123 determines that there is a pixel belonging to the non-road area, it determines that there is a non-road area. For example, if 0 is set for a certain pixel as the above-mentioned flag value, the masked area detection unit 123 determines that the pixel belongs to a non-road area. If the masked area detection unit 123 determines that there is a non-road area, it performs the process of step S26. If the masked area detection unit 123 determines that there is no non-road area, it performs the process of step S27.

[0094] [Step S26] The occluded area detection unit 123 detects the non-road area below the road area as an occluded area. For example, the occluded area detection unit 123 sets a detection result (for example, 1) indicating that the pixel is an occluded area to the pixel determined to belong to the non-road area.

[0095] [Step S27] The masked region detection unit 123 determines whether or not the horizontal search of the captured image is complete. If the processing of all pixels in the horizontal direction of the image is complete, the masked region detection unit 123 determines that the horizontal search of the image is complete, and ends the masked region detection process. If the masked region detection unit 123 determines that the horizontal search of the image is not complete, it selects, for example, the top pixel of one line to the right, and repeats the process from step S23.

[0096] The detection result of the obstructed area obtained by the above process is input to the distance measurement feasibility determination unit 124. The detection result of the obstructed area may be stored in the storage unit 110. If the orientation and position of the camera 211 are the same, there is a high possibility that the same detection result of the occluded area is obtained for the captured image of each frame. Therefore, the detection of the occluded area does not necessarily have to be performed for the captured image of all frames in which the detection of the target vehicle is performed. For example, the detection of the occluded area may be performed when the speed of the host vehicle exceeds a predetermined value, and the detection result may be used thereafter while the orientation and position of the camera 211 are the same.

[0097] FIG. 14 is a flowchart showing an example of the process of determining whether or not distance measurement is possible. [Step S31] The distance measurement feasibility determination unit 124 determines whether or not at least a part of the bottom side of the circumscribing rectangle overlaps with the occluded region based on the target detection result 114 output in step S12 and the detection result of the occluded region in step S13. The distance measurement feasibility determination unit 124 performs the determination process of step S31 by detecting whether or not a value (e.g., 1) indicating that the pixel belongs to the occluded region is set to a part of the pixels of the bottom side of the circumscribing rectangle. If the distance measurement feasibility determination unit 124 determines that at least a part of the bottom side of the circumscribing rectangle overlaps with the occluded region, it performs the process of step S32. If the distance measurement feasibility determination unit 124 determines that at least a part of the bottom side of the circumscribing rectangle does not overlap with the occluded region, it performs the process of step S33.

[0098] [Step S32] The distance measurement feasibility determination unit 124 sets to 0 the result of the determination as to whether distance measurement is possible for the target vehicle corresponding to the circumscribing rectangle. [Step S33] The distance measurement feasibility determination unit 124 sets to 1 the result of the determination as to whether distance measurement is possible for the target vehicle corresponding to the circumscribing rectangle.

[0099] In the example of FIG. 14, a result of the determination as to whether or not distance measurement is possible indicates that distance measurement is not possible, and a result of the determination as to whether or not distance measurement is possible indicates that distance measurement is possible. After steps S32 and S33, the distance measurement feasibility determination process ends.

[0100] When a captured image includes multiple circumscribing rectangles and it is to be determined whether or not distance measurement is possible for the target vehicle corresponding to each circumscribing rectangle, the above process is performed for each circumscribing rectangle. Furthermore, when n% or more (e.g., 40% or more) of the bottom side of the circumscribing rectangle is included in the obstructed area, the distance measurement feasibility determination unit 124 may set the distance measurement feasibility result to 0. Furthermore, when the distance between the bottom side of the circumscribing rectangle and the obstructed area on the captured image is within a predetermined value, the distance measurement feasibility determination unit 124 may set the distance measurement feasibility result to 0 even if the bottom side of the circumscribing rectangle is not included in the obstructed area.

[0101] 15 is a flowchart showing an example of the target speed calculation process, which is executed for each target vehicle detected in step S12 of FIG. The speed of the target vehicle is calculated based on the change in the position of the target vehicle over time, and is therefore calculated using the detection results at different first and second times. In the following description, the first time is the shooting time of the captured image currently being processed, acquired in step S11 of Fig. 11. The second time is the past time at which the distance and position of the same target vehicle were detected.

[0102] The first time and the second time may be times in adjacent frames. However, if the time between the first time and the second time is too short, the time change in the position of the target vehicle during that time is small, so it may be difficult to properly calculate the speed. Therefore, it is desirable to properly determine the first time and the second time so that the time change in the position is large enough to properly calculate the speed.

[0103] [Step S41] The target speed calculation unit 125 judges whether the result of the distance measurement feasibility judgment of the target vehicle whose speed is to be calculated is 1 or not. As described above, the distance measurement feasibility judgment result = 1 indicates that distance measurement is possible. If the target speed calculation unit 125 judges that the distance measurement feasibility judgment result is 1, it performs the process of step S42. If the target speed calculation unit 125 judges that the distance measurement feasibility judgment result is not 1, it skips the execution of the distance, position and speed calculation process. In this case, the process of FIG. 15 is terminated.

[0104] [Step S42] The target speed calculation unit 125 calculates the distance to the target vehicle for which the distance measurement feasibility determination result is determined to be 1, using equations (1) to (4). This calculates the distance to the target vehicle at the first time instant.

[0105] [Step S43] The target speed calculation unit 125 acquires the host vehicle position at the first time included in the vehicle information 113. In the example of the vehicle information 113 shown in Fig. 9, for example, the host vehicle position (X2, Y2) at time 1 is acquired as the host vehicle position at the first time.

[0106] [Step S44] The target speed calculation unit 125 calculates the position of the target vehicle (target position) at the first time. The target speed calculation unit 125 calculates the position of the target vehicle at the first time based on the host vehicle position at the first time and the calculation result of the distance at the first time.

[0107] [Step S45] The target speed calculation unit 125 acquires the host vehicle position at the second time included in the vehicle information 113. In the example of the vehicle information 113 shown in Fig. 9, for example, the host vehicle position (X1, Y1) at the time n is acquired as the host vehicle position at the second time.

[0108] [Step S46] The target speed calculation unit 125 calculates the target position at the second time. The target speed calculation unit 125 acquires the calculation result of the distance to the target vehicle calculated from the image captured at the second time, and calculates the position of the target vehicle at the second time based on the acquired calculation result of the distance and the vehicle position at the second time. Note that instead of calculating the position, the position of the target vehicle calculated from the image captured at the second time may be acquired.

[0109] [Step S47] The object speed calculation unit 125 calculates the speed of the object vehicle based on the positions of the object vehicle at the first and second times. The speed calculation result is stored in the storage unit 110 as the object speed calculation result 115.

[0110] When the process of step S47 ends, the target speed calculation process ends. The object speed calculation unit 125 may display the calculation results of the distance, the position, and the speed on the display device 104a in Fig. 2. The object speed calculation unit 125 may transmit the calculation results of the distance, the position, and the speed to an information processing device external to the image processing device 100 via a network (not shown).

[0111] Furthermore, the procedure for calculating the speed of the target vehicle is not limited to the procedure shown in Fig. 15. For example, the speed of the target vehicle at the first time may be calculated based on the speed of the host vehicle at the first time and the second time, and the distance to the target vehicle at the first time and the second time.

[0112] Furthermore, the above processing order is just an example, and the order may be changed as appropriate. As shown in Fig. 14, if at least a part of the bottom side of the circumscribing rectangle of the target vehicle overlaps with the obstructed area (if there is a possibility that an erroneous distance will be calculated), the distance measurement feasibility determination result will be 0. Then, as shown in Fig. 15, for a target vehicle whose distance measurement feasibility determination result is 0, distance measurement is skipped. This makes it possible to prevent an erroneous distance from being calculated.

[0113] Furthermore, as shown in FIG. 15, when the distance measurement feasibility determination result is 0, the target speed calculation process is skipped, so that calculation of an erroneous position or speed can be prevented. Images captured by the drive recorder 210 are used for various purposes. For example, in order for an automobile insurance company to grasp the situation of an accident as accurately as possible, the company acquires and analyzes images captured by the drive recorder of the customer. According to the process of this embodiment, it is possible to prevent the calculation of an erroneous distance, position, and speed, and therefore it is expected to provide accurate moving object information that indicates the situation of the accident.

[0114] Furthermore, in the captured image as described above, the non-road area below the detected road area is detected as the blocked area. Therefore, it is not necessary to set the blocked area in advance, and the blocked area can be detected regardless of the type of the vehicle (such as the type of car) or the type of the camera 211.

[0115] At least a part of the processing functions of the image processing device 100 shown in FIG. The processing functions of the devices (for example, the information processing device 1 and the image processing device 100) shown in each of the above embodiments can be realized by a computer. In this case, a program describing the processing contents of the functions that each device should have is provided, and the above processing functions are realized on the computer by executing the program on a computer. The program describing the processing contents can be recorded on a computer-readable recording medium. Examples of computer-readable recording media include magnetic storage devices, optical discs, and semiconductor memories. Examples of magnetic storage devices include HDDs and magnetic tapes. Examples of optical discs include CDs (Compact Discs), DVDs (Digital Versatile Discs), and Blu-ray Discs (BD, registered trademark).

[0116] When distributing a program, for example, the program is recorded on a portable recording medium such as a DVD or CD and then sold. The program can also be stored in a storage device of a server computer and transferred from the server computer to other computers via a network.

[0117] A computer that executes a program stores, for example, a program recorded on a portable recording medium or a program transferred from a server computer in its own storage device. The computer then reads the program from its own storage device and executes processing according to the program. The computer can also read the program directly from a portable recording medium and execute processing according to the program. The computer can also execute processing according to the received program each time a program is transferred from a server computer connected via a network.

[0118] While one aspect of the mobile object information calculation method and the mobile object information calculation program of the present invention has been described above based on the embodiment, these are merely examples and the present invention is not limited to the above description. [Explanation of symbols]

[0119] 1. Information processing device 1a Processing section 2 Camera 11,12 Photographed images 13 Vehicles 14 Shield area 15,16 circumscribed rectangle 15a, 16a, 16c Bottom 15b,16b Reference point

Claims

1. The computer A first captured image is acquired by a camera installed on the first moving object; Detecting a second moving object from the first captured image; detecting an occluding area that is captured below a road area in the first captured image and occludes a part of the road area; determining whether a predetermined condition is satisfied based on a detection result of the second moving object and the occlusion area, the predetermined condition indicating that at least a part of a bottom side of a circumscribing rectangle circumscribing the detected second moving object overlaps with or is close to the occlusion area; if the predetermined condition is not satisfied, a distance between the first moving body and the second moving body is calculated based on a coordinate of a reference point on the lower side, and if the predetermined condition is satisfied, the distance calculation process is skipped. A mobile object information calculation method for performing processing.

2. The moving object information calculation method according to claim 1 , wherein the obstructed area is an area in the first captured image in which a part of the first moving object is captured.

3. The mobile object information calculation method according to claim 2 , wherein when the first mobile object is a vehicle, the part of the first mobile object includes a bonnet or a dashboard.

4. Detecting the road area from the first captured image; a non-road area below the detected road area in the first captured image is detected as the occluded area; The moving object information calculation method according to claim 1 .

5. if the predetermined condition is not satisfied, a speed calculation process is executed to calculate the distance, calculate a first position of the second moving object based on the distance, and calculate a speed of the second moving object based on the first position and a second position of the second moving object calculated based on a second captured image previously captured by the camera; If the predetermined condition is satisfied, the speed calculation process is skipped. The moving object calculation method according to claim 1 .

6. On the computer, A first captured image is acquired by a camera installed on the first moving object; Detecting a second moving object from the first captured image; detecting an occluding area that is captured below a road area in the first captured image and occludes a part of the road area; determining whether a predetermined condition is satisfied based on a detection result of the second moving object and the occlusion area, the predetermined condition indicating that at least a part of a bottom side of a circumscribing rectangle circumscribing the detected second moving object overlaps with or is close to the occlusion area; if the predetermined condition is not satisfied, a distance between the first moving body and the second moving body is calculated based on a coordinate of a reference point on the lower side, and if the predetermined condition is satisfied, the distance calculation process is skipped. A mobile object information calculation program for executing the processing.

Images