Pause detection device and pause detection method

Abstract

To provide a technique by which a driving at a halt position can accurately be detected.SOLUTION: A halt detection device comprises: an acquisition unit for acquiring multiple captured images captured in time-sequential order using a camera mounted on a vehicle; an image processing unit for performing image processing on the acquired multiple captured images; and a detection unit for detecting a vehicle speed of the vehicle at a halt position based on a result of the image processing. The image processing unit includes: a halt position identification unit for identifying the halt position by detecting an indication that indicates the halt position from the captured images; and a vehicle speed calculation unit for calculating the vehicle speed by comparing the captured images in time-sequential order. The vehicle speed calculation unit compares the captured images excluding specific regions from the multiple captured images.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a technique for detecting driving at a stop sign based on an image captured by an in-vehicle camera. [Background technology]

[0002] Patent Document 1 discloses a driving safety evaluation device that determines whether a vehicle has stopped temporarily at a stop point where a stop is specified. This driving safety evaluation device determines whether a vehicle has passed a stop point based on positioning data and map data, and determines whether a vehicle has stopped temporarily at the stop point based on detection data from a vehicle speed sensor and an acceleration sensor. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent Publication No. 2021-125142 Summary of the Invention [Problem to be solved by the invention]

[0004] In the technology described in Patent Document 1, the passing of a stop point is determined using positioning data, and the driving situation at the stop point is determined using detection data from a vehicle speed sensor and an acceleration sensor, so there is a risk of the data becoming out of sync. If a synchronization error occurs, the accuracy of determining whether or not a vehicle has stopped at a stop point decreases.

[0005] An object of the present invention is to provide a technique for accurately detecting driving at a temporary stop position. [Means for solving the problem]

[0006] In order to solve the above problem, a temporary stop detection device according to one aspect of the present invention includes an acquisition unit that acquires a plurality of captured images taken in chronological order by a camera mounted on a vehicle, an image processing unit that performs image processing on the acquired plurality of captured images, and a detection unit that detects the vehicle speed at a temporary stop position based on the image processing results. The image processing unit has a stop position identification unit that identifies the temporary stop position based on a sign indicating the temporary stop position detected from the captured images, and a vehicle speed calculation unit that compares the captured images in chronological order to calculate the vehicle speed. The vehicle speed calculation unit compares the captured images excluding a specific area from the plurality of captured images. ,The size of a specific area is set based on road information. .

[0007] Another aspect of the present invention is a stop sign detection method. This method is a stop sign position detection method executed by a computer, and includes the steps of acquiring a plurality of captured images taken in chronological order by a camera mounted on a vehicle, performing image processing on the acquired plurality of captured images, and detecting the vehicle speed of the vehicle at the stop sign position based on the image processing results. The image processing step includes a step of identifying the stop sign position based on a sign indicating the stop sign detected from the captured images, and a step of comparing the captured images in chronological order to calculate the vehicle speed, and the step of calculating the vehicle speed includes a step of comparing the captured images excluding a specific area from the captured images. ,The size of a specific area is set based on road information. . Effect of the Invention

[0008] According to the present invention, a technique for accurately detecting driving at a temporary stop position can be provided. [Brief description of the drawings]

[0009] [Figure 1] FIG. 2 is a diagram for explaining an overview of the temporary stop detection device of the embodiment, showing an image captured by an in-vehicle camera. FIG. [Diagram 2] FIG. 2 is a diagram illustrating a functional configuration of a temporary stop detection device. [Diagram 3] FIG. 13 is a diagram showing a captured image in which a central region has been removed by a vehicle speed calculation unit. [Figure 4]13 is a diagram showing a captured image in which a moving object area has been removed by a vehicle speed calculation unit; FIG. [Diagram 5] 11 is a diagram for explaining a process for detecting a vehicle speed at a temporary stop position. FIG. [Figure 6] 1 shows a flowchart of a pause detection process. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] 1 is a diagram for explaining an overview of a temporary stop detection device according to an embodiment, and shows a captured image 20 captured by an in-vehicle camera. The temporary stop detection device analyzes the captured image 20 stored in a drive recorder of the vehicle to detect whether the vehicle has been driven appropriately at a temporary stop position.

[0011] For example, when a vehicle is brought to an automobile dealer, an operator retrieves a series of captured images 20 from the vehicle's drive recorder and inputs the captured images 20 to the stop sign detection device. The stop sign detection device analyzes the input series of captured images 20 to detect whether the vehicle was driven appropriately at the stop sign position and calculates a driving score. The calculated driving score is provided to the driver of the vehicle, so that the driver can know his or her own driving ability.

[0012] 1 shows a vehicle front 22, a stop line 24, a stop sign 26, a pedestrian crossing marking 28, and a vending machine 29. The stop detection device can detect traffic signs such as the stop line 24, the stop sign 26, and the pedestrian crossing marking 28 by image processing, and identify the stop position. The stop detection device also detects predetermined objects such as the vending machine 29 as feature points, and calculates the vehicle speed by tracking the feature points included in the chronologically arranged captured images 20. In this way, the stop detection device can derive the stop position and the vehicle speed, and detect whether the vehicle was driven appropriately at the stop position.

[0013] In this way, the temporary stop detection device can detect whether appropriate driving was performed at the temporary stop position using only the captured image 20, and therefore data can be extracted more easily than when sensor values ​​other than the captured image are used. The captured image 20 can be extracted by an operator simply removing the memory from the drive recorder and inserting it into the input slot of the temporary stop detection device. Furthermore, when multiple types of sensor values ​​are used, there is a risk that detection accuracy will decrease due to missynchronization of the sensor values, but by performing detection using only the captured image 20, it is possible to prevent problems with missynchronization.

[0014] 2 is a diagram showing the functional configuration of the pause detection device 10. In terms of hardware, each function of the pause detection device 10 can be configured with a circuit block, memory, or other LSI, and in terms of software, it is realized by system software or application programs loaded into memory. Therefore, it will be understood by those skilled in the art that each function of the pause detection device 10 can be realized in various forms by hardware alone, software alone, or a combination of both, and is not limited to any one of them.

[0015] The image storage unit 12 stores captured images of the surroundings of the vehicle captured by an on-board camera. The captured images stored in the image storage unit 12 are arranged in chronological order and are moving images. The temporary stop detection device 10 acquires the moving images stored in the drive recorder from the image storage unit 12. The image storage unit 12 is hardware such as a flash memory, and may be manually removed from the vehicle by an operator and connected to the temporary stop detection device 10. The image storage unit 12 may also be connected to the temporary stop detection device 10 via a network such as the Internet using a communication device of the vehicle.

[0016] The output unit 14 is a display, a speaker, or the like, and outputs to the driver the detection result by the temporary stop detection device 10. The output unit 14 may be provided integrally with the temporary stop detection device 10, or may be a separate unit.

[0017] The temporary stop detection device 10 includes an acquisition unit 30, an image processing unit 32, a detection unit 34, and a driving score calculation unit 36. The acquisition unit 30 acquires from the image storage unit 12 a plurality of captured images captured in chronological order by the vehicle-mounted camera.

[0018] The image processing unit 32 performs image processing on the acquired multiple captured images 20. The image processing unit 32 has an object detection unit 37, a stop position identification unit 38, and a vehicle speed calculation unit 40. The image processing unit 32 may perform image processing on all data stored in the image storage unit 12, but may also perform image processing on data during driving. In other words, before processing is performed by the object detection unit 37, processing may be performed to remove data not during driving.

[0019] The object detection unit 37 detects an object included in the captured image 20 using an object detection model such as YOLO. The object detection unit 37 executes object detection processing on all input captured images 20. The multiple captured images 20, for example, form a video of 30 frames per second and are arranged in chronological order. The object detection unit 37 derives the position coordinates of the detected object on the captured image and the type of the object. The position coordinates of the object may be the position coordinates of the four corners of a rectangle surrounding the object.

[0020] The stop position identification unit 38 detects a marking indicating a stop position from the captured image 20 and identifies the stop position. The marking indicating a stop position is a specific traffic marking such as the stop line 24 or the stop sign 26 shown in FIG.

[0021] The stop position identification unit 38 extracts a marking indicating a stop position from among the objects detected by the object detection unit 37, calculates the distance between the extracted stop line and the vehicle, and identifies the stop position. The stop position identification unit 38 may identify the stop position only when both the stop line 24 and the stop sign 26 are detected from one frame of the captured image 20. This makes it possible to suppress erroneous detection of the stop position.

[0022] The vehicle speed calculation unit 40 calculates the vehicle speed by comparing the captured images 20 in chronological order. The vehicle speed calculation unit 40 compares the captured images 20 by excluding a specific region from the multiple captured images 20. This specific region will be described with reference to a new drawing.

[0023] FIG. 3 shows the captured image 20 with the central region 42 removed by the vehicle speed calculation unit 40. The captured image 20 shown in FIG. 3 also shows a plurality of objects 44a, 44b, 44c, and 44d that are characteristic points. The plurality of objects 44a, 44b, 44c, and 44d that are characteristic points are detected by the object detection unit 37. The vehicle speed calculation unit 40 calculates the positional deviation amounts of the plurality of objects 44a, 44b, 44c, and 44d in chronological order, and detects the vehicle speed based on the positional deviation amounts. In this way, the vehicle speed calculation unit 40 tracks the characteristic points included in the captured image 20 arranged in chronological order to calculate the vehicle speed.

[0024] The feature points may be the center points of the objects 44a, 44b, 44c, and 44d, or may be points at the four corners. The feature points are mainly signs, trees, and buildings located outside the roadway, and few are located above the roadway. The objects 44a and 44c are trees, and the object 44b is a distribution box, which are fixed to the ground.

[0025] Here, since the object 44d is a moving vehicle, it becomes noise in the vehicle speed calculation process, and tracking the object 44d to calculate the vehicle speed leads to a decrease in calculation accuracy. Therefore, the vehicle speed calculation unit 40 performs a process of removing the central region 42 as a specific region from the captured image 20, and tracks the objects 44a, 44b, 44c, and 44d, excluding objects overlapping the central region 42. By removing the specific region (for example, the central region 42) from the captured image 20, it is possible to remove noise factors in vehicle speed detection. In other words, the specific region refers to a region that masks a part of the captured image 20. As shown in FIG. 3, the central region 42 is an area that includes the center point of the captured image 20 and is located in the center, but the left / right / up / down positions of the central region 42 in the captured image 20 may be adjusted by experiments, etc.

[0026] By setting the specific area as a predetermined central area 42 located in the center of the captured image 20, it is possible to easily exclude areas where a vehicle traveling ahead or an oncoming vehicle is likely to be present, and image processing can be simplified. Also, signs and buildings outside the roadway are often located in areas on both ends of the central area 42, so it is possible to ensure that objects to be tracked are available even if the central area 42 is excluded.

[0027] The vehicle speed calculation unit 40 sets the size of the specific area, the central area 42, based on the road information. For example, the vehicle speed calculation unit 40 may set the width of the central area 42 according to the road width indicated in the road information, and may set the width of the central area 42 to be larger as the road width increases. The width of the central area 42 to be set is set to three levels, for example, "large," "medium," and "small," and is set to "small" for one lane on one side, "medium" for two lanes on one side, and "large" for three or more lanes on one side.

[0028] Furthermore, if the road information indicates a slope, the height of the central region 42 in the captured image 20 may be adjusted up or down according to the inclination angle of the slope, and the vertical width of the central region 42 may be adjusted. For example, if the road is an uphill road, the vehicle speed calculation unit 40 sets the vertical width of the central region 42 to be longer upward than on flat ground, and if the road is a downhill road, the vehicle speed calculation unit 40 sets the vertical width of the central region 42 to be longer downward than on flat ground. In this way, the size of the central region 42 is set based on the road information, so that it is possible to eliminate areas where a moving object is likely to enter. The road information is derived from the captured image 20.

[0029] Fig. 4 shows the captured image 20 from which the moving object region 46 has been removed by the vehicle speed calculation unit 40. In Fig. 4, a moving object region 46 that individually covers the moving object is set as a specific region for the captured image 20, which is the same as that in Fig. 3, and image processing is performed to remove the moving object region 46.

[0030] As shown in Fig. 4, moving object regions 46a, 46b, and 46c are set and masked individually for a plurality of moving objects included in the captured image 20. These moving objects are detected by the object detection unit 37, and are identified as vehicles by the object detection unit 37. The moving objects may include not only vehicles but also people. The moving object region 46 is formed in a rectangular shape based on the maximum vertical length and maximum horizontal length of the moving object in the captured image, and is formed according to the size of the moving object.

[0031] In this way, the vehicle speed calculation unit 40 sets moving objects included in the captured image 20 to specific regions (moving object regions 46a, 46b, 46c), masks the moving object regions 46a, 46b, 46c, tracks the fixed objects 44a, 44b, 44c as feature points, and calculates the vehicle speed. The process of removing the moving object region 46 removes moving objects detected by the object detection model, and therefore can remove noise more accurately than removing the central region 42.

[0032] The vehicle speed calculation unit 40 sets the specific area to either a predetermined central area or a moving object area based on the driving environment information. That is, the vehicle speed calculation unit 40 uses the specific area to be masked, either the central area or the moving object area, depending on the driving environment. For example, when the weather is rainy, the detection accuracy of the object detection unit 37 decreases, so there is a possibility that the moving object cannot be accurately removed. Therefore, when the weather is bad, such as rain or snow, the vehicle speed calculation unit 40 sets the central area 42 as the specific area, and when the weather is not bad, the moving object area 46 as the specific area. This allows the tracking process of the feature points to be performed with high accuracy.

[0033] Returning to Fig. 2, the detection unit 34 detects the vehicle speed at the stop position based on the image processing result. Since the vehicle-mounted camera has a blind spot, the captured image 20 does not capture the moment when the vehicle passes over the stop line 24. Therefore, the processing of the detection unit 34 will be described with reference to Fig. 5.

[0034] Fig. 5 is a diagram for explaining the process of detecting the vehicle speed at a stop position. In Fig. 5, the progress of the vehicle passing through a stop line 24 is indicated by a circle, and the on-board camera captures the image at the position indicated by the circle. Due to the angle of view and mounting position of the on-board camera, the front part 22 of the vehicle is captured in the captured image 20 as shown in Fig. 1, and the stop line 24 cannot be captured as the vehicle approaches the stop line 24. For example, the stop line 24 shown in FIG. 5 is not captured by the vehicle-mounted camera when the vehicle is located in the non-detection range 48.

[0035] Therefore, the detection unit 34 determines the timing when the vehicle has passed the stop position as the timing when the vehicle has approached within a predetermined distance from the stop position, and detects the vehicle speed at that time. The detection unit 34 receives, as the image processing result, distance information between the host vehicle and the stop position from the stop position identification unit 38, and vehicle speed information from the vehicle speed calculation unit 40. The distance information between the host vehicle and the stop position received from the stop position identification unit 38 and the vehicle speed information received from the vehicle speed calculation unit 40 are associated with any one of the captured images 20.

[0036] The non-detection range 48 is determined depending on the vehicle type, and may be, for example, 3 meters. Therefore, when the detection unit 34 detects that the vehicle has passed the stop position based on the captured image 20 captured at a position 50 outside the non-detection range 48, and the vehicle approaches the stop position within a predetermined distance, the vehicle is deemed to have passed the stop position. This predetermined distance is set based on the non-detection range 48, and is set to any value within a range of, for example, 1 meter to 4 meters. The predetermined distance may also be set depending on the vehicle type, and may be set to be shorter for vehicles with a short vehicle front portion 22.

[0037] Returning to FIG. 2, the detection unit 34 detects whether the vehicle speed at the stop position is equal to or lower than a predetermined vehicle speed based on the image processing result. This determines whether the vehicle has stopped. The predetermined vehicle speed is set to 10 kilometers per hour or less. If the vehicle speed at the stop position is equal to or lower than the predetermined vehicle speed, the detection unit 34 determines that the vehicle has been driven appropriately, and if the vehicle speed at the stop position exceeds the predetermined vehicle speed, the detection unit 34 determines that the vehicle has violated the stop sign.

[0038] The driving score calculation unit 36 ​​calculates a driving score that represents the driving skill of the driver based on the detection result by the detection unit 34. The driving score is calculated so that it is higher if the driver drives appropriately at the stop sign, and the lower the vehicle speed at the stop sign, the higher the driving score is. Note that the stop sign detection device 10 may also detect traffic violations other than stop signs, and may calculate an overall driving score. For example, the speed limit may be detected from traffic signs, and it may be determined whether the vehicle is observing the speed limit.

[0039] The driving score calculated by the driving score calculation unit 36 ​​is output from the output unit 14 and shown to the driver by voice and image. In this way, the temporary stop detection device 10 can diagnose the driver's skill only from the captured image 20, so that the diagnosis can be easily made at a car dealership or the like. Therefore, when an elderly person or the like brings their vehicle to a car dealership, it becomes possible to diagnose their driving skill and provide advice.

[0040] 6 shows a flowchart of the temporary stop detection process. The acquisition unit 30 acquires a series of captured images captured by the vehicle-mounted camera from the image storage unit 12 (S10). The object detection unit 37 of the image processing unit 32 executes a process of detecting an object from the acquired series of captured images (S12). The object detection unit 37 derives the type of the detected object and the position of the object on the captured images.

[0041] The stop position identification unit 38 detects a sign indicating a stop position among the objects detected by the object detection unit 37, and identifies the stop position (S14). In addition, the stop position identification unit 38 calculates the distance between the stop position and the vehicle.

[0042] The vehicle speed calculation unit 40 regards objects not included in the specific area among the objects detected by the object detection unit 37 as feature points, tracks the feature points in the captured images in chronological order, and calculates the vehicle speed based on the positional deviation amount of the feature points (S16). An object not included in the specific area may be an object that does not overlap even a part of the specific area, or an object that does not overlap half or more of the area of ​​the specific area. The vehicle speed calculation unit 40 may calculate the vehicle speed for each captured image, or may calculate the vehicle speed only for captured images in which a stop position is detected.

[0043] The detection unit 34 receives the distance between the stop position and the vehicle from the stop position identification unit 38, receives the vehicle speed from the vehicle speed calculation unit 40, and detects the vehicle speed at the stop position (S18). The detection unit 34 may detect whether the vehicle speed at the stop position is equal to or lower than a predetermined vehicle speed.

[0044] The driving score calculation unit 36 ​​calculates a driving score indicating driving skill based on the detection result of the detection unit 34 (S20). The output unit 14 displays the calculated driving score to the dealer's staff or the driver (S22).

[0045] The present disclosure has been described above based on the embodiments. The present disclosure is not limited to the above embodiments, and various modifications such as design changes may be made based on the knowledge of those skilled in the art. [Explanation of symbols]

[0046] 10 stop stop detection device, 12 image memory unit, 14 output unit, 20 captured image, 22 front of vehicle, 24 stop line, 26 stop sign, 28 pedestrian crossing marking, 29 vending machine, 30 acquisition unit, 32 image processing unit, 34 detection unit, 36 driving score calculation unit, 37 object detection unit, 38 stopping position identification unit, 40 vehicle speed calculation unit, 42 central area, 46 moving object area.

Claims

1. an acquisition unit that acquires a plurality of captured images captured in chronological order by a camera mounted on the vehicle; an image processing unit that performs image processing on the acquired plurality of captured images; a detection unit that detects the vehicle speed at a stop position based on the image processing result, The image processing unit includes: a stop position identification unit that identifies a stop position based on a marking indicating a stop position detected from a captured image; a vehicle speed calculation unit that calculates a vehicle speed by comparing the captured images in chronological order, The vehicle speed calculation unit compares a plurality of captured images with a specific region removed from each of the captured images, and sets the size of the specific region based on road information.

2. The temporary stop detection device according to claim 1 , wherein the vehicle speed calculation unit sets the specific area to a predetermined central area located at the center of the captured image.

3. 3. The temporary stop detection device according to claim 1, wherein the vehicle speed calculation unit sets an area that individually covers a moving object included in the captured image as the specific area.

4. 4. The temporary stop detection device according to claim 1, wherein the vehicle speed calculation unit sets the specific area to either a predetermined central area or a moving body based on driving environment information.

5. 5. The temporary stop detection device according to claim 1, wherein the vehicle speed calculation unit calculates the vehicle speed by tracking characteristic points included in the captured images arranged in chronological order.

6. A pause position detection method executed by a computer, comprising the steps of: acquiring a plurality of images captured in chronological order by a camera mounted on a vehicle; A step of performing image processing on the acquired plurality of captured images; detecting a vehicle speed of the vehicle at a stop position based on a result of the image processing; The image processing step includes: identifying a stop position based on a sign indicating a stop position detected from the captured image; and calculating a vehicle speed by comparing the captured images in chronological order. A method for detecting a temporary stop, wherein the step of calculating the vehicle speed includes comparing the captured images with a specific area removed from the captured images, and setting the size of the specific area based on road information.

Images