Image processing system, image processing method, and image processing program
Patent Information
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- NAGOYA ELECTRIC WORKS
- Filing Date
- 2024-12-18
- Publication Date
- 2026-06-30
Smart Images
Figure 2026106550000001_ABST
Abstract
Description
[Technical Field]
[0001] This invention relates to an image processing system, an image processing method, and an image processing program. [Background technology]
[0002] Patent Document 1 describes a method for learning the normal driving area of vehicles in a monitored area from the behavior of each vehicle actually driving in the monitored area, and for detecting vehicles that exhibit behavior deviating from the learned normal behavior as dangerous vehicles. Patent Document 2 describes a method for generating virtual lines connecting vehicle drivers on the road and detecting a restricted area based on the positional relationship between the virtual lines and the camera. [Prior art documents] [Patent Documents]
[0003] [Patent Document 1] Patent No. 6448171 [Patent Document 2] Patent No. 7209785 [Overview of the project] [Problems that the invention aims to solve]
[0004] In the method described in Patent Document 1, the time required for learning may depend on the traffic volume during learning. Also, in the method described in Patent Document 1, if a vehicle that does not behave as expected passes through during learning, that behavior may affect the learning results. In Patent Document 2, an alarm is issued when a vehicle is detected entering the restricted area beyond a virtual line, but it was also desired to issue a warning before the vehicle enters the restricted area. For this reason, it was desired to be able to automatically and quickly set multiple areas in images of the road. This invention has been made in view of the above-mentioned problems, and aims to set multiple areas in an image of a road. [Means for solving the problem]
[0005] To achieve the above object, an image processing system includes an image acquisition unit that acquires an image captured by a camera of the rear side in the traveling direction of a road, and based on the image, a first rear area that is a strip-shaped rear area extending in the direction in which the road extends and including the midpoint in the left-right direction of the lower side of the image and having a distance from the camera less than a threshold distance, a second rear area that is a rear area having a distance from the camera greater than or equal to the threshold distance, and an area setting unit that sets a side area on the side of the rear area.
[0006] According to this image processing system, in an image captured of the rear side in the traveling direction of a road, a side area can be set on the side of a strip-shaped rear area that extends in the direction in which the road extends and includes the midpoint in the left-right direction of the lower side of the image. Further, according to this image processing system, in the rear area, a first rear area having a distance from the camera less than the threshold distance and a second rear area having a distance from the camera greater than or equal to the threshold distance can be set.
Brief Description of the Drawings
[0007] [Figure 1] It is a block diagram of an image processing system. [Figure 2] It is a diagram schematically showing an example of the positional relationship among a road, a work vehicle, a regulating material, and a worker. [Figure 3] It is a diagram schematically showing an example of the positional relationship among a road, a work vehicle, a regulating material, and a worker. [Figure 4] FIG. 4A is a diagram showing an example of an image captured by a camera, and FIG. 4B is a diagram schematically showing each area to be set. [Figure 5] FIGS. 5A and 5B are diagrams schematically showing each area to be set. [Figure 6] FIG. 6A is a diagram showing an example of an image captured by a camera, and FIG. 6B is a diagram schematically showing each area to be set. [Figure 7] FIGS. 7A and 7B are diagrams schematically showing each area to be set. [Figure 8]Figure 8A shows an example of an image captured by the camera, and Figure 8B shows a schematic representation of each area that is set. [Figure 9] This is a flowchart showing the area setting process. [Figure 10] This is a flowchart showing the warning process. [Figure 11] Figure 11A shows an example of an image captured by a camera, and Figure 11B shows a schematic representation of each area that is set. [Modes for carrying out the invention]
[0008] Here, embodiments of the present invention will be described in the following order. (1) Configuration of the image processing system: (2) Area setting process: (3) Warning handling: (4) Other embodiments:
[0009] (1) Configuration of the image processing system: Figure 1 is a block diagram showing the configuration of the image processing system 10. In this embodiment, the image processing system 10 acquires images from the camera 50, sets up multiple areas based on the images, determines whether a warning is necessary based on the multiple areas and the position of the vehicle, and issues an alarm using the display device 60, the first warning device 70, and the second warning device 80 if a warning is necessary. In this embodiment, the image processing system 10 is attached to a work vehicle used at a road construction site.
[0010] Figures 2 and 3 are schematic diagrams illustrating an example of the positional relationship between a work vehicle, a worker, and a safety barrier. In Figure 2, the direction of travel of the vehicle is from top to bottom of the paper. Therefore, in Figure 2, the bottom of the paper is the front and the top of the paper is the rear. As shown in Figure 2, worker WK is positioned in the work vehicle C W In the lane where the vehicle is parked (L1 in the example in Figure 2), work vehicle C W Work will be carried out in front of it. Also, work vehicle C W A regulating barrier (RM) is installed behind it.
[0011] As shown in FIG. 3, the camera 50 is attached to the work vehicle C W so as to photograph the rear side of the work vehicle C W . The camera 50 includes an imaging optical system and an imaging element, and is a device that captures images at a predetermined frame rate. The control unit 20 acquires the images output by the camera 50 via an interface (not shown).
[0012] The display device 60 is attached to the work vehicle C W in a posture visible from a vehicle (for example, C1 etc.) traveling behind the work vehicle C W . The display device 60 displays characters and images for warning toward the driver of the vehicle traveling behind.
[0013] In the present embodiment, the first warning device 70 is assumed to be a speaker. In the present embodiment, the speaker as the first warning device 70 is attached toward the rear side of the work vehicle C W . In the present embodiment, the second warning device 80 is assumed to be a speaker. In the present embodiment, the speaker as the second warning device 80 is attached toward the front side of the work vehicle C W . When mainly warning the driver of a vehicle traveling behind the work vehicle C, the control unit 20 causes the first warning device 70 to output a warning sound or voice. When mainly warning the worker WK, the control unit 20 causes the second warning device 80 to output a warning sound or voice. W
[0014] As shown in FIG. 1, the image processing system 10 includes a control unit 20 and a recording medium 30. The control unit 20 is a computer including a CPU, a RAM, a ROM, etc. The control unit 20 performs area setting processing and warning processing by executing various programs recorded in the recording medium 30 and the ROM. Various programs and various information are stored in the recording medium 30. The various information includes image data 30a indicating the images captured by the camera 50.
[0015] The control unit 20 uses the above configuration to set multiple areas on the road based on the image. Furthermore, the control unit 20 detects vehicles based on the image, determines whether a warning is necessary based on the area where the vehicle is located and the direction of travel, and issues a warning if it determines that a warning is necessary. For this purpose, the control unit 20 executes the image processing program 21. When the image processing program 21 is executed, the control unit 20 functions as an image acquisition unit 21a, an area setting unit 21b, a vehicle position detection unit 21c, and a warning unit 21d.
[0016] The image acquisition unit 21a instructs the control unit 20 to perform the function of acquiring images taken by the camera of the area behind the road in the direction of travel. In other words, the control unit 20 acquires images output from the camera 50 at a predetermined frame rate using the function of the image acquisition unit 21a and records them as image data 30a on the recording medium 30.
[0017] The area setting unit 21b causes the control unit 20 to perform the function of setting a first rear area, which is a band-shaped rear area extending in the direction in which the road extends and includes the midpoint in the left-right direction of the lower edge of the image, and is less than a threshold distance from the camera; a second rear area, which is a rear area and is greater than or equal to the threshold distance from the camera; and a lateral area located to the side of the rear area, based on the image data 30a, using the function of the area setting unit 21b.
[0018] In this embodiment, the rear area is the work vehicle C to which the camera 50 is attached. W Work vehicle C in the lane where vehicle C is stopped W This is the area behind the vehicle C. The first rear area is located behind the second rear area. WThis area is close to the rear area. In this embodiment, the lateral area is a lane where vehicle traffic is not restricted and is adjacent to the rear area. Vehicles traveling in the rear area need to move to the lateral area. The second rear area is used to warn vehicle drivers not to enter the area or to exit the area. The first rear area is used primarily to notify and warn road workers by detecting vehicles located in the area or vehicles attempting to enter the area.
[0019] Specifically, the control unit 20 recognizes objects in the image, such as road markings, vehicles, and traffic control materials. Known pattern matching methods or machine learning models may be used to recognize these objects. If road markings exist on both sides of the midpoint of the bottom edge of the image, the control unit 20 designates the area between the two road markings as the rear area. Figure 4A shows a work vehicle C in the rightmost lane of the three lanes (left side relative to the direction of road travel), as shown in Figure 2. W The vehicle stopped, and work vehicle C W An example of an image taken by camera 50 mounted on point P is shown. This example shows a cone-shaped traffic control RM lying on its side on the road. C is the midpoint of the bottom edge of the image. In the example shown in Figure 4A, the midpoint P is on the bottom edge of the image. C On both sides of the lane, there are intersection points P1 with lane marking Lb1 and P2 with lane marking Lb2. Therefore, the control unit 20 designates the area between the two lane marking lines Lb1 and Lb2 as the rear area. In other words, in the case of Figure 4A, the control unit 20 designates the area corresponding to lane L1 in the image as the rear area. Thus, when the control unit 20 determines, based on the image, that lane marking lines exist on both sides of the camera 50's installation position, it can designate the lane between those lane marking lines as the rear area.
[0020] Furthermore, the control unit 20 designates the area in the rear area where the distance to the camera 50 is less than the threshold distance as the first rear area, and the area where the distance to the camera 50 is greater than or equal to the threshold as the second rear area. The position on the road in the vertical direction of the image can be converted to the distance on the road from the camera 50. The control unit 20 considers the distance to the camera 50 to be the threshold distance D. th The virtual line d in the image corresponds to this. th Identify the area between the boundary line Lb1 and the boundary line Lb2 in the rear area, and the bottom edge of the image and the virtual line d th The trapezoidal area sandwiched between the two is at a threshold distance D in the rear area where the distance to camera 50 is. th This area corresponds to less than [a certain value]. The control unit 20 sets this area as the first rear area. In addition, the control unit 20 sets a virtual line d in the rear area between the boundary line Lb1 and the boundary line Lb2. th The area closer to the top edge of the image is set as the second rear area. Note that the default threshold distance D th The system may be configured to use a value entered by a worker into the image processing system, or it may be configured to detect the distance between the furthest regulating material RM and the camera 50 as shown in Figure 4B and use that distance, or it may be a value preset by the image processing system 10.
[0021] If traffic control materials are placed on the road, the control unit 20 sets a lateral area based on the arrangement of the images of the traffic control materials included in the image. Specifically, the control unit 21b functions such that the intersection point of the imaginary line connecting the images of multiple traffic control materials RM lined up on the road and the bottom edge of the image is the midpoint P. C If it is closer to the first side, a lateral area is set on the first side of the rear area, and the intersection is the midpoint P. C If it is further to the second side, a lateral area is set on the second side of the rear area, and if the intersection is on both sides of the midpoint, lateral areas are set on both sides of the rear area.
[0022] In the example in Figure 4A, a virtual line L connects multiple regulatory materials RM. V The intersection point P3 with the bottom edge of the image is the midpoint P CIt is located to the left of the lane L1. In this case, the control unit 20 sets the rear area, i.e., lane L2 which is adjacent to the left of lane L1 in this case, as the lateral area. Note that the left side in the image is the right side when the direction of travel on the road is used as the reference point. Hereafter, when simply referring to right and left, it refers to right and left on the page of the image or drawing.
[0023] Figure 4B schematically shows the areas set in the example shown in Figure 4A. The threshold distance D from camera 50 in lane L1 where camera 50 is located. th The first rear area A1 is set in the area less than the threshold distance D from camera 50 in lane L1. th This indicates that the second rear area A2 is set in the area described above, and the lateral area A3 is set in lane L2, which is adjacent to the left side of lane L1.
[0024] Figures 5A and 5B show another example of the lane and corridor arrangement where the camera 50 is located. In the example shown in Figures 5A and 5B, as in the example shown in Figures 4A and 4B, lane markings exist on both sides of the midpoint of the bottom edge of the image captured by the camera 50. Therefore, the control unit 20 designates the area between these lane markings as the rear area. Specifically, in the example shown in Figure 5A, in lane L3 where the camera 50 is located, the distance from the camera 50 is D th The area less than D is the first rear area A1, and the distance from camera 50 is D. th The second rear area A2 is set in the above area. In the example shown in Figure 5B, the distance to camera 50 in lane L2 where camera 50 is located is D th The first rear area A1 is set in the region less than D, and the distance to camera 50 is D th The second rear area A2 is set up in the above-mentioned area.
[0025] In the example shown in Figure 5A, in the image captured by camera 50, the intersection point of the virtual line connecting multiple traffic control members RM with the bottom edge of the image is to the right of the midpoint of the bottom edge of the image. Therefore, the control unit 20 designates the lane adjacent to the right of lane L3 where camera 50 is located as the lateral area. In the example shown in Figure 5B, the control unit 20 sets two virtual lines in the image captured by camera 50: one connecting the traffic control members RM positioned from the central rear of the image toward the left foreground, and another connecting the traffic control members RM positioned from the central rear of the image toward the right foreground. The intersection points of these virtual lines with the bottom edge of the image are located on both sides of the midpoint of the bottom edge of the image. Therefore, in this case, as shown in Figure 5B, the control unit 20 designates the lanes adjacent to both sides of lane L2 where camera 50 is located as the lateral area A3.
[0026] In addition to being placed within the lane as shown in Figures 4 and 5, the traffic control material RM may also be placed on the lane line as shown in Figures 6A and 6B. In the example shown in Figure 6A, the imaginary line connecting multiple traffic control material RMs roughly coincides with the lane line Lb2. The intersection point P2 of the lane line Lb2 and the bottom edge of the image is the midpoint P of the bottom edge of the image. C Because it is located further to the left, the control unit 20 sets a lateral area in lane L2 adjacent to the left of lane L1 where the camera 50 is located. Similarly, in the example shown in Figure 7A, a lateral area A3 is set in lane L3 adjacent to the right of lane L3 where the camera 50 is located, and in the example shown in Figure 7B, a lateral area A3 is set in lanes L1 and L3 adjacent to both sides of lane L2 where the camera 50 is located, respectively. Thus, according to this embodiment, the control unit 20 can set a lateral area, which is an area where vehicle movement is not restricted, based on the arrangement of the restricting material.
[0027] The vehicle position detection unit 21c causes the control unit 20 to perform the function of detecting the position of a vehicle traveling on the road. That is, the control unit 20 detects the position of a vehicle traveling behind the work vehicle based on the image data 30a, using the function of the vehicle position detection unit 21c. Vehicle detection is performed using pattern matching or a machine learning model. The control unit 20 identifies the position of the detected vehicle within the image.
[0028] The warning unit 21d causes the control unit 20 to execute a function that warns the vehicle driver if the vehicle is located in the second rear area A2. In this embodiment, the control unit 20, using the function of the warning unit 21d, displays a warning message on the display device 60, such as prompting the vehicle to move to the side area A3, and outputs a warning sound from the first warning device 70. With this configuration, the control unit 20 determines that the distance between the vehicle traveling in the rear area and the camera 50 is D th The system can warn the vehicle driver before the reading falls below a certain level.
[0029] Furthermore, the warning unit 21d causes the control unit 20 to issue a warning to road workers if the vehicle is located in the first rear area A1. In this embodiment, the control unit 20 outputs a warning sound from the second warning device 80. With this configuration, the control unit 20 determines that the distance between the vehicle and the camera 50 in the rear area is D th If the value falls below a certain level, a warning can be issued to the workers.
[0030] Furthermore, the warning unit 21d causes the control unit 20 to issue a warning to the vehicle when the vehicle is located in the lateral area A3 and its direction of travel changes towards the second rear area A2. Specifically, the control unit 20 obtains the direction of travel of the vehicle from the position of the vehicle in images captured in, for example, two consecutive frames of images. When the direction of travel of the vehicle changes towards the second rear area A2, the control unit 20 causes the display device 60 to display a warning message and the first warning device 70 to output a warning sound. With this configuration, the control unit 20 can issue a warning to the vehicle driver when it is estimated that a vehicle traveling in the lateral area A3 is moving to the rear area and entering the second rear area A2.
[0031] Furthermore, the warning unit 21d causes the control unit 20 to issue a warning to road workers when the vehicle is located in the lateral area A3 and its direction of travel changes towards the first rear area A1. Specifically, when the direction of travel of the vehicle changes towards the first rear area A1, the control unit 20 causes the second warning device 80 to output a warning sound. With this configuration, if it is estimated that a vehicle traveling in the lateral area A3 will move to the rear area and enter the first rear area A1 without passing through the second rear area A2, the control unit 20 can alert the workers.
[0032] The warning unit 21d causes the control unit 20 to terminate the warning if the vehicle's direction of travel changes towards the lateral area A3. That is, if the direction of travel of a vehicle traveling in the second rear area A2 changes towards the lateral area A3, the control unit 20 terminates the display of the warning message on the display device 60 and terminates the output of the warning sound on the first warning device 70. Also, if the direction of travel of a vehicle traveling in the first rear area A1 changes towards the lateral area A3, the control unit 20 terminates the output of the warning sound on the second warning device 80. With this configuration, the control unit 20 can terminate the warning to the vehicle driver or worker when it is estimated that a vehicle traveling in the rear area is moving towards the lateral area.
[0033] Furthermore, due to the function of the area setting unit 21b, the control unit 20 increases the threshold distance when the speed of a vehicle traveling in the second rear area A2 is high compared to when it is slow. In other words, when the speed of a vehicle traveling in the second rear area is high, the length of the first rear area (length in the direction in which the road extends) is increased compared to when it is slow. That is, since the speed of vehicles traveling in the lateral and rear areas can vary from vehicle to vehicle, the length of the first rear area is set to vary each time according to the speed of the vehicle. With this configuration, compared to a case where a fixed length of the first rear area is set according to a predetermined assumed speed regardless of the actual speed of the vehicles traveling, it is possible to warn workers at an earlier stage if there is a vehicle traveling at a speed faster than the assumed speed.
[0034] The vehicle's speed is calculated, for example, as follows: The control unit 20 compares the vehicle's position in images captured in, for example, two consecutive frames, and converts the change in position into distance on the road. Then, the control unit 20 determines the time required to travel that distance from the frame interval and obtains the vehicle's speed by dividing the distance by the time. Various methods can be used to obtain the vehicle's speed. For example, the vehicle's speed may be determined based on images from multiple frames.
[0035] Note that, as shown in Figures 8A and 8B, there are cases where the regulatory material RM is not present in the image. In the example shown in Figure 8A, the regulatory material RM is not present in the image, but the midpoint P on the bottom edge of the image is not present. CThere are intersection points P1 and P2 between the lane markings and the lower edge on both sides. Therefore, the control unit 20 sets the first rear area A1 and the second rear area A2, similar to the examples shown in Figures 4A and 6A. For the lateral area A3, the control unit 20 sets it based on the line type of the lane markings. That is, if one of the lane markings that demarcate the lane where the camera 50 is located is a dashed line and the other is a solid line, the control unit 20 sets the lateral area in the adjacent lane demarcated by the dashed line. In the example shown in Figure 8A, the lane marking Lb2 on the left side of the lane L1 where the camera 50 is located is a dashed line and the lane marking Lb1 on the right side is a solid line, so the control unit 20 sets the lateral area A3 in the lane L2 adjacent to the left. If the lane markings on both sides of the lane where the camera 50 is located are dashed lines, the control unit 20 sets the lateral areas in the lanes adjacent to both sides of the lane where the camera 50 is located.
[0036] As described above, according to this embodiment, in an image taken of the rear side in the direction of travel of a road, a lateral area can be set to the side of a band-shaped rear area that extends in the direction in which the road extends, including the midpoint in the left-right direction of the lower edge of the image. Furthermore, according to this image processing system, a first rear area where the distance from the camera is less than a threshold distance and a second rear area where the distance from the camera is greater than or equal to the threshold distance can be set within the rear area. In addition, according to this image processing system, when multiple traffic control materials are set within a lane as shown in Figures 4 and 5, a warning can be issued to the driver or worker of a vehicle before the vehicle enters the restricted area indicated by the virtual line connecting the multiple traffic control materials. Furthermore, according to this embodiment, it is possible to set each area and to determine whether or not a warning is necessary based on the behavior of the vehicle, without depending on the traffic volume.
[0037] (2) Area setting process: Next, the area setting process will be explained. Figure 9 is a flowchart of the area setting process. The area setting process is started, for example, when the user instructs the start of the area setting process. When the area setting process is started, the control unit 20 acquires an image using the function of the image acquisition unit 21a (step S100). That is, the control unit 20 acquires image data showing the image captured by the camera 50.
[0038] Next, the control unit 20 performs image processing using the functions of the area setting unit 21b (step S105). Specifically, the control unit 20 performs image recognition processing to detect objects such as demarcation lines and regulatory materials contained in the image.
[0039] Next, the control unit 20 detects the position of the regulated material using the function of the area setting unit 21b (step S110). That is, the control unit 20 acquires the position in the image of the object representing the detected regulated material.
[0040] Next, the control unit 20 determines whether or not the regulating material is present in front, based on the function of the area setting unit 21b (step S115). The control unit 20 then determines the midpoint P of the lower edge of the image. C Determine whether or not a traffic control device is placed within the lane containing the vehicle.
[0041] If it is determined in step S115 that the vehicle is within the lane, the control unit 20 determines whether the traffic control material is arranged in a left-right branch configuration (step S120). That is, as shown in Figure 5B, if the traffic control material is arranged to indicate that both the left and right sides of the lane where the camera 50 is located are drivable lanes, the control unit 20 sets two virtual lines connecting the traffic control material in the image captured by the camera 50. Since the intersections of the two virtual lines and the bottom edge of the image are located on both sides of the midpoint PC of the bottom edge of the image, the control unit 20 determines that the traffic control material is arranged in a left-right branch configuration.
[0042] If it is determined in step S120 that the regulating material is arranged in a left-right branch configuration, the control unit 20 sets lateral areas in the left and right lanes (step S125) and sets the lane behind the camera as the rear area (step S130). In other words, the control unit 20 identifies the lane to be designated as the rear area based on the lane markings. After executing step S130, the control unit 20 determines whether time t has elapsed (step S180), and if it is determined that time t has elapsed, it executes step S100. Time t is a predetermined period of time used as the cycle for setting areas.
[0043] If it is not determined in step S120 that the traffic control material is arranged in a left-right branch configuration, the control unit 20 determines whether it is arranged diagonally downward to the right (step S135). As shown in Figure 5A, if the traffic control material RM is arranged so that the closer it is to the camera 50, the closer it is to the right lane, then in the image captured by the camera 50, the intersection point of the virtual line connecting the traffic control material RM and the bottom edge of the image is located to the right of the midpoint of the bottom edge of the image. In such a case, the control unit 20 determines that the traffic control material is arranged diagonally downward to the right. If it is determined in step S135 that the traffic control material is arranged diagonally downward to the right, the control unit 20 sets a lateral area in the right lane (step S140) and executes step S130. That is, in step S140, the control unit 20 sets the lane to the right of the lane where the camera 50 is located as the lateral area, as shown in Figure 5A.
[0044] If it is not determined in step S135 that the traffic control material is positioned diagonally downward to the right, the control unit 20 determines whether or not it is positioned diagonally downward to the left (step S145). As shown in Figures 4A and 4B, if the traffic control material RM is positioned such that the closer it is to the camera 50, the closer it is to the left lane, then in the image captured by the camera 50, the intersection point of the imaginary line connecting the traffic control material RM and the bottom edge of the image is located to the left of the midpoint of the bottom edge of the image. In such a case, the control unit 20 determines that the traffic control material is positioned diagonally downward to the left. If it is determined in step S145 that the traffic control material is positioned diagonally downward to the left, the control unit 20 sets a lateral area in the left lane (step S150) and executes step S130. That is, in step S150, the control unit 20 sets the lane to the left of the lane where the camera 50 is located as the lateral area, as shown in Figure 4B. If it is not determined in step S145 that the traffic control material is positioned diagonally downward to the left, the control unit 20 executes step S175.
[0045] If it is not determined in step S115 that the traffic control material is present in front, the control unit 20 determines whether it is present only on the left side (step S155). As shown in Figures 6A and 6B, if the traffic control material is placed only on the lane marking to the left of the lane where the camera 50 is located, the control unit 20 determines that the traffic control material is present only on the left side. If it is determined in step S155 that the traffic control material is present only on the left side, the control unit 20 sets a lateral area in the left lane (step S160) and executes step S130. That is, in step S160, the control unit 20 sets the lane to the left of the lane where the camera 50 is located as the lateral area, as shown in Figure 6B.
[0046] If it is not determined in step S155 that the restricting material is present only on the left side, the control unit 20 determines whether the restricting material is present only on the right side (step S165). As shown in Figure 7A, if the restricting material is placed only on the lane marking to the right of the lane where the camera 50 is located, the control unit 20 determines that the restricting material is present only on the right side. If it is determined in step S165 that the restricting material is present only on the right side, the control unit 20 sets a lateral area in the right lane (step S170) and executes step S130. That is, in step S170, the control unit 20 sets the lane to the right of the lane where the camera 50 is located as the lateral area, as shown in Figure 7A.
[0047] If it is not determined in step S165 that the restricting material is present only on the right side, the control unit 20 sets a lateral area outside the lane (step S175) and executes step S130. If it is determined to be negative in step S165, this includes the case where the row of restricting material is present on the lane markings on both the left and right sides of the lane where the camera 50 is located, as shown in Figure 7B, and the case where the restricting material is not present in the image. If it is present on both the left and right sides as shown in Figure 7B, the control unit 20 sets the adjacent lanes on both the left and right sides of the lane where the camera 50 is located as the lateral area. If the restricting material is not present in the image as shown in Figure 8A, the control unit 20 sets the lane adjacent to the side where the lane markings are dashed as the lateral area.
[0048] (3) Warning handling: Next, the warning process will be explained. Figure 10 is a flowchart of the warning process. The warning process shown in Figure 10 starts after the lateral area A3, the first rear area A1, and the second rear area A4 have been set at least once by the area setting process shown in Figure 9. Note that steps S200 to S290 of the warning process may be executed multiple times within time t in the area setting process.
[0049] The control unit 20 acquires an image (step S200). That is, the control unit 20 acquires image data representing the image captured by the camera 50 at a predetermined frame rate and records it on the recording medium 30. Subsequently, the control unit 20 performs image processing (step S205). That is, the control unit 20 performs image recognition processing on the image acquired in step S200 and detects objects contained in the image.
[0050] Next, the control unit 20 detects the vehicle (step S210). That is, the control unit 20 obtains the position of the vehicle detected by image recognition processing and records it in association with image data 30a representing the image obtained in step S200. Next, the control unit 20 detects the direction of travel of the vehicle (step S215). That is, the control unit 20 obtains the direction of travel of the vehicle from the change in the position of the vehicle detected in the latest frame image and the previous frame image, respectively, and records it in association with image data 30a representing the latest frame image.
[0051] Next, the control unit 20 detects the vehicle's speed (step S220). Specifically, the control unit 20 converts the change in the vehicle's position between the latest frame's image and the previous frame's image into distance on the road, calculates the speed by dividing it by the time indicated by the frame interval, and records it in association with the image data 30a representing the latest frame's image.
[0052] Next, the control unit 20 sets a first rear area and a second rear area based on the vehicle's speed (step S225). That is, the control unit 20 sets a larger threshold distance when the vehicle traveling in the second rear area is moving at a faster speed than when it is moving at a slower speed. More specifically, if the speed detected in step S220 is faster than the speed corresponding to the default threshold distance, the control unit 20 sets the threshold distance to a value greater than the default value. If the speed detected in step S220 is less than or equal to the speed corresponding to the default threshold distance, the threshold distance is set to the default value. The control unit 20 divides the rear area set in step S130 into a first rear area and a second rear area based on the threshold distance.
[0053] Next, the control unit 20 determines whether or not a vehicle has entered the second rear area (step S230). That is, the control unit 20 determines whether or not the position detected in step S210 is within the second rear area A2. If it is determined in step S230 that a vehicle has entered the second rear area, the control unit 20 determines whether or not this continued for a certain period of time (step S235). That is, the control unit 20 determines whether or not the state of the vehicle being located in the second rear area continued for a certain period of time, based on the position of the vehicle in the image of the latest frame and the position of the vehicle recorded in association with the images of each frame taken within a certain period of time prior to the latest frame.
[0054] If it is determined in step S235 that the situation has continued for a certain period of time, the control unit 20 will issue an alarm to the rear (step S240). That is, the control unit 20 will display a warning message on the display device 60 and output a warning sound to the first warning device 70.
[0055] Next, the control unit 20 determines whether the vehicle is facing the lateral area (step S245). That is, the control unit 20 determines whether there is a lateral area ahead in the direction of travel of the vehicle detected in step S215. If it is determined in step S245 that the vehicle is facing the lateral area, the control unit 20 determines whether a certain period of time has continued (step S250). That is, the control unit 20 determines whether the state in which the vehicle's direction of travel is facing the lateral area has continued for a certain period of time, based on the direction of travel in the image of the latest frame and the direction of travel in the images of each frame over the past certain period of time. If it is determined in step S250 that a certain period of time has continued, the control unit 20 cancels the alarm (step S255) and returns to step S200.
[0056] If it is not determined in step S250 that the condition has continued for a certain period of time, the control unit 20 executes the process in step S265. If it is not determined in step S245 that the vehicle is facing the lateral area, the control unit 20 executes the process in step S265. If it is not determined in step S235 that the condition has continued for a certain period of time, the control unit 20 executes the process in step S265.
[0057] If it is not determined in step S230 that the vehicle has entered the second rear area, the control unit 20 determines whether the vehicle is facing the direction of the second rear area (step S260). That is, the control unit 20 determines whether the second rear area is ahead of the direction of travel of the vehicle detected in step S215. If it is determined in step S260 that the vehicle is facing the direction of the second rear area, the control unit 20 executes step S235. That is, the control unit 20 determines whether the state of the vehicle facing the second rear area has continued for a certain period of time, and if it is determined that it has continued for a certain period of time, it issues a rearward alarm.
[0058] If it is not determined in step S260 that the vehicle is facing the direction of the second rear area, the control unit 20 determines whether or not the vehicle has entered the first rear area (step S265). That is, the control unit 20 determines whether or not the position of the vehicle detected in step S210 is within the first rear area. If it is determined in step S265 that the vehicle has entered the first rear area, the control unit 20 determines whether or not this has continued for a certain period of time (step S270). That is, the control unit 20 determines whether or not the state in which the vehicle is located in the first rear area has continued for a certain period of time. If it is determined in step S270 that this has continued for a certain period of time, the control unit 20 issues a warning to the front (step S275). That is, the control unit 20 causes the second warning device 80 to output a warning sound.
[0059] Next, the control unit 20 determines whether the vehicle is facing the lateral area (step S280). That is, the control unit 20 determines whether there is a lateral area ahead in the direction of travel of the vehicle detected in step S215. If it is determined in step S280 that the vehicle is facing the lateral area, the control unit 20 determines whether this has continued for a certain period of time (step S285). That is, the control unit 20 determines whether the state in which the vehicle's direction of travel is facing the lateral area has continued for a certain period of time. If it is determined in step S285 that this has continued for a certain period of time, the control unit 20 executes step S255. That is, the control unit 20 cancels the alarm.
[0060] If it is not determined in step S285 that the condition has continued for a certain period of time, or if it is not determined in step S280 that the vehicle is facing the lateral area, the control unit 20 executes step S200. If it is not determined in step S270 that the condition has continued for a certain period of time, the control unit 20 returns to step S200.
[0061] If it is not determined in step S265 that the vehicle has entered the first rear area, the control unit 20 determines whether the vehicle is facing the direction of the first rear area (step S290). That is, the control unit 20 determines whether the first rear area is ahead of the direction of travel of the vehicle detected in step S215 (i.e., the first rear area is located without passing through the second rear area). If it is determined in step S290 that the vehicle is facing the direction of the first rear area, the control unit 20 executes step S270. That is, the control unit 20 determines whether the state of the vehicle facing the direction of the first rear area has continued for a certain period of time, and if it is determined that it has continued for a certain period of time, it issues a warning to the front. If it is not determined in step S290 that the vehicle is facing the direction of the first rear area, the control unit 20 returns to step S200.
[0062] (4) Other embodiments: The embodiments described above are merely examples for carrying out the present invention, and various other embodiments can be adopted. For example, the image processing system 10 may be implemented by multiple devices. For instance, the image acquisition unit and the area setting unit may be implemented by devices separate from the vehicle position detection unit and the warning unit. Furthermore, the image processing system may be configured integrally with a camera, a display device, and a warning device.
[0063] The image acquisition unit only needs to be able to acquire images of the area behind the road in the direction of travel using a camera. It is not limited to cameras that capture visible light; cameras that capture infrared light, etc., may also be used. Furthermore, monocular cameras or stereo cameras may be used. For example, if two cameras are mounted at different heights from the road surface, it becomes possible to detect vehicles at long distances and vehicles at short distances, and the detection range can be expanded compared to vehicle detection with a single camera.
[0064] The vehicle position detection unit only needs to be able to detect the position of a vehicle traveling on the road. In addition to a configuration that detects position using images captured by a camera, a configuration may also be adopted in which, for example, millimeter wave or LiDAR is used to determine the distance and direction of the vehicle from the transmitter and receiver, and the vehicle's position is obtained.
[0065] The warning unit may be configured to terminate the warning when the vehicle's direction of travel changes towards the lateral area, or when the vehicle moves into the lateral area. In the latter case, the warning continues until the vehicle moves into the lateral area. When a vehicle is located in the first rear area, the system may be configured to issue an alarm to both the vehicle driver and the worker, or it may be configured to issue an alarm to the worker and cancel the alarm for the driver.
[0066] Furthermore, the warning unit may be configured to notify or warn workers by wirelessly controlling and vibrating a vibration device installed in the worker's helmet. Also, if a speaker that can be switched between forward and backward direction is used, the warning unit may be configured to direct the speaker forward when warning workers and to switch the speaker direction backward when warning vehicle drivers.
[0067] In the above embodiment, the rear and side areas were regions enclosed by lane markings. However, as shown in Figure 11B, the camera 50 may be located outside the lane in which the vehicle is traveling. In such cases, as shown in Figure 11A, the midpoint P of the bottom edge of the image captured by the camera 50 is located. C There are no intersection points between the dividing line and the bottom edge of the image on either side. In such cases, the control unit 20 controls the midpoint P C The rear area is defined as a strip-shaped area with a predetermined width W that extends in the direction in which the road extends, including the camera 50. The predetermined width W is, for example, equivalent to the width of a road lane. With this configuration, even when the camera 50 is located outside the lane in which the vehicle is traveling, the first rear area, second rear area, and lateral area can be set.
[0068] Furthermore, if the road has only one lane, the area setting unit may be configured to set a first rear area and a second rear area, including the areas outside the lane on both sides of the lane, but without setting a lateral area.
[0069] Furthermore, the present invention can also be implemented as a program or a method. Moreover, the system, program, and method described above can be implemented as a single device or by multiple devices, encompassing various embodiments. Furthermore, it can be modified as appropriate, such as being partly software and partly hardware. The invention also functions as a recording medium for a program that controls the system. Of course, the recording medium for the software may be a magnetic recording medium, a semiconductor memory, or any recording medium developed in the future can be considered in exactly the same way. [Explanation of symbols]
[0070] 10...Image processing system, 20...Control unit, 21...Image processing program, 21a...Image acquisition unit, 21b...Area setting unit, 21c...Vehicle position detection unit, 21d...Warning unit, 30...Recording medium, 30a...Image data, 50...Camera, 60...Display device, C W ...work vehicle, D th ...threshold distance, L1...lane, L2...lane, L3...lane, L V ...Imaginary line, Lb1...partition line, Lb2...partition line, P1...intersection, P2...intersection, P3...intersection, P C ...midpoint, RM...regulator, W...standard width, WK...worker
Claims
1. An image acquisition unit that acquires images of the rear side of the road in the direction of travel using a camera, Based on the aforementioned image, an area setting unit sets a first rear area which is a band-shaped rear area extending in the direction in which the road extends, including the midpoint in the left-right direction of the lower edge of the image, and the distance from the camera is less than a threshold distance; a second rear area which is the rear area and the distance from the camera is greater than or equal to the threshold distance; and a lateral area which is located to the side of the rear area. An image processing system equipped with the following features.
2. A vehicle position detection unit for detecting the position of a vehicle traveling on the aforementioned road, A warning unit that issues a warning to the vehicle when the vehicle is located in the second rear area, The image processing system according to claim 1, comprising:
3. The warning unit provides a warning to road workers when the vehicle is located in the first rear area. The image processing system according to claim 2.
4. The warning unit terminates the warning if the direction of travel of the vehicle changes to the direction of the lateral area. The image processing system according to claim 2 or claim 3.
5. The warning unit provides a warning to the vehicle when the vehicle is located in the lateral area and the direction of travel of the vehicle changes to the direction of the second rear area. The image processing system according to claim 2.
6. The warning unit provides a warning to road workers when the vehicle is located in the lateral area and the direction of travel of the vehicle changes to the direction of the first rear area. The image processing system according to claim 2.
7. The area setting unit makes the threshold distance larger when the speed of the vehicle traveling in the second rear area is high compared to when it is slow. The image processing system according to claim 2.
8. The area setting unit is, If the intersection point of the imaginary line connecting the multiple regulatory materials lined up on the road and the bottom edge of the image is on the first side of the midpoint, then the lateral area is set on the first side of the rear area. If the intersection is on the second side of the midpoint, the lateral area is set on the second side of the rear area. When the aforementioned intersection points are on both sides of the aforementioned midpoint, the aforementioned lateral areas are set on both sides of the aforementioned rear area. The image processing system according to claim 1.
9. The area setting unit is, Based on the aforementioned image, the road markings are detected. If the demarcation lines exist on both sides of the midpoint, the area between the two demarcation lines shall be the rear area. If no lane markings exist on either side of the midpoint, the rear area shall be a strip-shaped area of a predetermined width that includes the midpoint and extends in the direction in which the road extends. The image processing system according to claim 1.
10. The camera captures an image of the road behind the direction of travel. Based on the aforementioned image, a first rear area is defined as a strip-shaped rear area extending in the direction in which the road extends, including the midpoint in the left-right direction of the lower edge of the image, and the distance from the camera is less than a threshold distance; a second rear area is defined as the rear area, and the distance from the camera is greater than or equal to the threshold distance; and a lateral area is defined to the side of the rear area. Image processing methods including [specific details omitted].
11. Computers, An image acquisition unit that acquires images of the rear side of the road in the direction of travel using a camera, Based on the aforementioned image, an area setting unit sets a first rear area which is a band-shaped rear area extending in the direction in which the road extends, including the midpoint in the left-right direction of the lower edge of the image, and the distance from the camera is less than a threshold distance; a second rear area which is the rear area and the distance from the camera is greater than or equal to the threshold distance; and a lateral area which is located to the side of the rear area. An image processing program that functions as such.