Processing device, processing method, and recording medium

Abstract

A processing device according to the present disclosure comprises an acquisition unit, a first estimation unit, a second estimation unit, and an output unit. The acquisition unit acquires a moving image generated by means of a fixed point camera. The first estimation unit estimates a lane region by using a first method, on the basis of the moving image. The second estimation unit estimates a lane region by using a second method different from the first method, on the basis of a moving image accumulated further after the lane region is estimated by using the first method. The output unit outputs information indicating the lane region estimated by using the first method, and then outputs information indicating the lane region estimated by using the second method.

Description

Processing device, processing method, and recording medium

[0001] The present disclosure relates to a processing device, a processing method, a program, and a recording medium.

[0002] A technology related to the present disclosure is disclosed in Patent Document 1. The technology disclosed in Patent Document 1 detects white lines on a road by image analysis, and identifies the lane in which a vehicle is traveling based on the detection result of the white lines.

[0003] JP 2009-26164 A

[0004] The applicants have investigated a technology for estimating lane areas based on images generated by a fixed camera and performing various processes based on the estimation results, and as a result have discovered the following new problem.

[0005] First, a sufficient number of images may be required to obtain a desired estimated result (e.g., a highly accurate estimated result). In such cases, it may take some time to obtain the estimated result due to factors such as the time it takes to accumulate a sufficient number of images. This may result in a problem in that various processes based on the estimated result cannot be executed until the estimated result is obtained. Patent Document 1 does not disclose this problem or a means for solving it.

[0006] An example of an objective of the present disclosure is to provide a technology that solves the problem that various processes based on an estimation result cannot be executed until a predetermined estimation result is obtained.

[0007] According to the present disclosure, there is provided a processing device having: an acquisition means for acquiring a moving image generated by a fixed camera; a first estimation means for estimating a lane area using a first method based on the moving image; a second estimation means for estimating a lane area using a second method different from the first method based on the moving image that is further accumulated after the lane area has been estimated using the first method; and an output means for outputting information indicating the lane area estimated using the first method, and then outputting information indicating the lane area estimated using the second method.

[0008] The present disclosure also provides a processing method in which one or more computers acquire video images generated by a fixed camera, estimate lane areas using a first method based on the video images, estimate lane areas using a second method different from the first method based on the video images that are further accumulated after the lane areas have been estimated using the first method, output information indicating the lane areas estimated using the first method, and then output information indicating the lane areas estimated using the second method.

[0009] Furthermore, according to the present disclosure, there is provided a program that causes a computer to function as: an acquisition means that acquires video images generated by a fixed camera; a first estimation means that estimates a lane area using a first method based on the video images; a second estimation means that estimates a lane area using a second method different from the first method based on the video images that are further accumulated after the lane area has been estimated using the first method; and an output means that outputs information indicating the lane area estimated using the first method, and then outputs information indicating the lane area estimated using the second method.

[0010] According to one aspect of the present disclosure, a technology is realized that solves the problem that various processes based on an estimation result cannot be executed until a predetermined estimation result is obtained.

[0011] FIG. 1 is a diagram illustrating an example of a functional block diagram of a processing device according to the present disclosure. FIG. 2 is a flowchart illustrating an example of a processing flow of a processing device according to the present disclosure. FIG. 3 is a diagram illustrating an example of a hardware configuration of a processing device according to the present disclosure. FIG. 4 is a flowchart illustrating another example of a processing flow of a processing device according to the present disclosure. FIG. 5 is a flowchart illustrating another example of a processing flow of a processing device according to the present disclosure. FIG. 6 is a diagram illustrating another example of a processing flow of a processing device according to the present disclosure. FIG. 7 is a diagram illustrating another example of a processing flow of a processing device according to the present disclosure. FIG. 8 is a diagram illustrating another example of a processing flow of a processing device according to the present disclosure. FIG. 9 is a diagram illustrating another example of a processing flow of a processing device according to the present disclosure. FIG. 10 is a diagram illustrating another example of a processing flow of a processing device according to the present disclosure. Fig. 1 is a flowchart showing another example of the processing flow of the processing device according to the present disclosure. Fig. 2 is a diagram showing another example of the functional block diagram of the processing device according to the present disclosure. Fig. 3 is a flowchart showing another example of the processing flow of the processing device according to the present disclosure. Fig. 4 is a diagram for explaining other processing of the processing device according to the present disclosure.

[0012] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. In this disclosure, the drawings relate to one or more embodiments. In all drawings, similar components are designated by similar reference numerals, and descriptions thereof will be omitted as appropriate.

[0013] <<First Embodiment>> Fig. 1 is a functional block diagram showing an overview of a processing device 10. Fig. 2 is a flowchart showing an example of the flow of processing executed by the processing device 10.

[0014] 1, the processing device 10 includes an acquisition unit 11, a first estimation unit 12, a second estimation unit 13, and an output unit 14. These functional units execute the processing of the flowchart in FIG.

[0015] In S10, the acquisition unit 11 acquires video images generated by a fixed camera. In S11, the first estimation unit 12 estimates a lane area using a first method based on the video images. In S12, the output unit 14 outputs information indicating the lane area estimated using the first method. In S13, the second estimation unit 13 estimates the lane area using a second method different from the first method based on video images acquired and stored after the lane area was estimated using the first method. In S14, the output unit 14 outputs information indicating the lane area estimated using the second method. That is, the output unit 14 outputs information indicating the lane area estimated using the first method (S12), and then outputs information indicating the lane area estimated using the second method (S14).

[0016] In this way, the processing device 10 can first estimate lane areas using the first method and quickly output the estimation result, and then perform highly accurate estimation using the second method based on the video images acquired and stored after the lane areas have been estimated using the first method, and output the estimation result.

[0017] According to the processing device 10, various processes can be performed based on the estimation result obtained by the first method until a predetermined estimation result (highly accurate estimation result obtained by the second method) is obtained. As a result, the problem of not being able to perform various processes based on the estimation result until a predetermined estimation result (highly accurate estimation result obtained by the second method) is obtained is solved.

[0018] <<Second Embodiment>> <Overview> A processing apparatus 10 according to a second embodiment is a specific embodiment of the configuration of the processing apparatus 10 according to the first embodiment. The processing apparatus 10 will be described in detail below.

[0019] <Hardware Configuration> First, an example of the hardware configuration of the processing device 10 will be described. Each functional unit of the processing device 10 is realized by any combination of hardware and software. Those skilled in the art will understand that there are various variations in the realization method and device. The software includes programs that are pre-loaded when the device is shipped, and programs downloaded from recording media such as CDs (Compact Discs) or servers on the Internet.

[0020] FIG. 3 is a block diagram illustrating an example of the hardware configuration of a processing device 10. As shown in FIG. 3, the processing device 10 has a processor 1A, a memory 2A, an input / output interface 3A, a peripheral circuit 4A, and a bus 5A. The peripheral circuit 4A includes various modules. The processing device 10 does not necessarily have to have the peripheral circuit 4A. Note that the processing device 10 may be composed of multiple devices that are physically and / or logically separated. In this case, each of the multiple devices may have the above hardware configuration.

[0021] The bus 5A is a data transmission path for the processor 1A, memory 2A, peripheral circuit 4A, and input / output interface 3A to mutually transmit and receive data. The processor 1A is, for example, a central processing unit (CPU) or a graphics processing unit (GPU). The memory 2A is, for example, a random access memory (RAM) or a read-only memory (ROM). The input / output interface 3A includes interfaces for acquiring information from input devices, external devices, external servers, external sensors, cameras, etc., and interfaces for outputting information to output devices, external devices, external servers, etc. The input / output interface 3A also includes an interface for connecting to a communication network such as the Internet. Examples of input devices include a keyboard, mouse, microphone, physical buttons, and touch panel. Examples of output devices include a display, projection device, speaker, printer, and mailer. The processor 1A can issue commands to each module and perform calculations based on the results of those calculations.

[0022] <Functional Configuration> Next, the functional configuration of the processing device 10 will be described in detail. Fig. 1 shows an example of a functional block diagram of the processing device 10. As shown in the figure, the processing device 10 has an acquisition unit 11, a first estimation unit 12, a second estimation unit 13, and an output unit 14. These functional units execute various processes shown in Fig. 4.

[0023] As shown in FIG. 4, the processing device 10 can execute an external auxiliary information collection process, an image acquisition process, a non-target area exclusion process, a first lane estimation process, a vehicle detection process, a second lane estimation process, and an external output process.

[0024] The processing device 10 may not execute one or both of the external auxiliary information collection step and the non-target area excluding step. In this embodiment, the image acquisition step, the first lane step, the vehicle detection step, the second lane estimation step, and the external output step are essential steps, but the external auxiliary information collection step and the non-target area excluding step can be omitted as appropriate.

[0025] 4 is merely an example and is not limited to the illustrated example. The processing order shown in FIG. 4 is the order that allows the most efficient operation, but the same output can be obtained even if the processing order is changed. For example, the processing order can be changed according to the following rules.

[0026] The external auxiliary information collection process can be performed at any timing, but must be performed before the process of using the collected information. The external auxiliary information collection process can acquire multiple types of information, and these may be acquired together at the same time or at different times. Except for the external auxiliary information collection process, the first process performed after the start of processing is the image acquisition process. After the image acquisition process is performed, the non-target area exclusion process, first lane estimation process, vehicle detection process, and second lane estimation process are performed. These processes are generally performed in any order, with some exceptions. The second lane estimation process must be performed after the vehicle detection process. However, they do not need to be performed consecutively. In other words, other processes may be inserted between the vehicle detection process and the second lane estimation process. The external output process is performed depending on the processing results of the first lane estimation process and the second lane estimation process. The external auxiliary information collection process, image acquisition process, non-target area exclusion process, first lane estimation process, vehicle detection process, and second lane estimation process can be repeated as appropriate.

[0027] Each step will be described in detail below.

[0028] "External auxiliary information collection step (FIG. 4)" The acquisition unit 11 executes the external auxiliary information collection step. In the external auxiliary information collection step, the acquisition unit 11 acquires various pieces of information by at least one of acquisition from an external device different from the processing device 10 and user input. For example, the acquisition unit 11 may acquire information published on the web from a web server. Alternatively, the acquisition unit 11 may acquire information stored in an external device or information input by a user to an external device connected to the processing device 10 so as to be able to communicate with the external device. Alternatively, the acquisition unit 11 may acquire information input by a user via an input device provided in the processing device 10.

[0029] The acquisition unit 11 can output the acquired information as is or after performing predetermined processing. The information output in the external auxiliary information collection step is used in other steps shown in FIG.

[0030] The acquisition unit 11 acquires various types of information in the external auxiliary information collection step. For example, the acquisition unit 11 may acquire time information, calendar information, weather information (weather, temperature, humidity, etc.) at the location where the fixed camera is installed, etc.

[0031] Alternatively, the acquisition unit 11 may display a GUI (Graphical User Interface) via a display device and accept input of various information from a user via the GUI. The GUI is displayed via a display device (a display or a projection device) included in the processing device 10 or a display device included in an external device communicably connected to the processing device 10.

[0032] For example, the acquisition unit 11 may accept input of various information used in other steps shown in Fig. 4. Examples of the various information that can be accepted as input from the user will be described in the description of the other steps shown in Fig. 4.

[0033] Alternatively, the acquisition unit 11 may display an image showing the estimated lane area on the GUI based on the information showing the lane area (any of the primary to tertiary lane areas in FIG. 4) output in the external output step in FIG. 4. The acquisition unit 11 may then accept a user input to modify the estimation result on the GUI.

[0034] "Video Acquisition Step (FIG. 4)" The acquisition unit 11 executes the video acquisition step. In the video acquisition step, the acquisition unit 11 acquires video images generated by a fixed camera. Note that in the video acquisition step, the acquisition unit 11 may also acquire other information such as the camera's data transfer method. Then, the acquisition unit 11 outputs the acquired video images. The video images output in the video acquisition step are used in other steps shown in FIG. 4.

[0035] The acquisition unit 11 may acquire the video generated by the fixed camera by real-time processing or by batch processing. For example, the processing device 10 and the fixed camera may be communicably connected. The fixed camera may then transmit the generated video to the processing device 10. Alternatively, the video generated by the fixed camera may be stored in an arbitrary storage device. The storage device may be provided in the fixed camera or in an external device. The video stored in the storage device may then be input to the processing device 10 by any means and at any timing. Alternatively, the video generated by the fixed camera may be made public on a web server. The acquisition unit 11 may then acquire the video that is made public on the web server.

[0036] A fixed camera stays at a certain location for a certain period of time and captures the road on which vehicles are traveling. The fixed camera may detect visible light to create an image, or other light such as infrared light, ultraviolet light, or X-rays to create an image.

[0037] For example, a fixed camera may be a camera that is installed in a position where it can capture images of the road and remains in that position. In this example, the fixed camera may be installed in that position for a relatively long period (e.g., several years) and capture images of the road over a long period of time. Alternatively, the fixed camera may be installed in that position for a relatively short period of time (e.g., several days to several months) and capture images of the road over a short period of time.

[0038] Such fixed cameras may be able to change at least one of the orientation, position, and shooting conditions (zoom, etc.) automatically or in response to an operator's operation. Such a function is preferable because it widens the range that can be shot with a single fixed camera.

[0039] Alternatively, the fixed camera may be a camera attached to a moving object. In this case, the camera attached to the moving object captures the road when the moving object is stopped. When the moving object is stopped, the position of the camera attached to the moving object does not change. Therefore, the camera attached to the moving object functions as a fixed camera. The moving object may be an airborne vehicle such as a drone, or a landborne vehicle such as a motorcycle or automobile. The moving object may also be a robot equipped with a means of locomotion. The means of locomotion may be a mechanism for moving on land or a mechanism for moving in the air. In either case, the robot may employ any well-known locomotion mechanism. For example, the means of locomotion may be a bipedal mechanism or a mechanism for walking on three or more legs. The means of locomotion may be a mechanism equipped with wheels or crawlers, or may be any other special mechanism. The robot may also be an airborne vehicle for moving in the air, such as a drone.

[0040] "Non-target area exclusion process (FIG. 4)" The first estimation unit 12 executes the non-target area exclusion process. In the non-target area exclusion process, the first estimation unit 12 identifies areas (non-target areas) within the moving image (within the frame image) that are not subject to the process of estimating lane areas, and generates information indicating the non-target areas. The information indicating the non-target areas is information that indicates a partial area within the frame image. The first estimation unit 12 can then output the generated information. The information output in the non-target area exclusion process is used in other processes shown in FIG. 4.

[0041] The non-target area is, for example, an area in which an object other than a road (e.g., a building, etc.) is captured. The non-target area may also be an area in which a portion of multiple roads captured in a video image is captured. There are multiple types of roads, such as expressways, general roads, national highways, and private roads, and there is a demand to monitor only a portion (one or more) of these roads. In such cases, the non-target area can be an area in which a road other than the road to be monitored is captured.

[0042] For example, a group managing a highway may install a camera on the highway to monitor the highway. Video images generated by a camera installed on the highway often include not only the highway itself but also ordinary roads in the background. In such cases, designating the ordinary road as an out-of-target area can avoid the inconvenience of estimating lane areas for roads not being monitored. By designating a portion of the frame image as an out-of-target area in this way, the processing load on the processing device 10 can be reduced, and the probability of false detections and malfunctions can also be expected to be reduced.

[0043] The first estimation unit 12 may analyze the video to identify the non-target region. For example, the first estimation unit 12 may identify a region in the video (frame image) in which a road is shown by using a deep learning technique such as depth estimation or segmentation. Then, the first estimation unit 12 may identify a region in the video (frame image) other than the region in which the road is shown as the non-target region.

[0044] Alternatively, the first estimation unit 12 may identify the non-target region based on a user input. In this example, in the external auxiliary information collection step described above, the acquisition unit 11 accepts a user input specifying a non-target region within a moving image (within a frame image). Then, the first estimation unit 12 identifies the non-target region based on the user input.

[0045] Alternatively, the first estimation unit 12 may be capable of performing both a process of analyzing a moving image to identify a non-target region and a process of identifying a non-target region based on user input. In this case, the first estimation unit 12 may prioritize the process of identifying a non-target region based on user input. That is, if the acquisition unit 11 receives user input specifying a non-target region within a moving image (within a frame image) in the external auxiliary information collection step described above, the first estimation unit 12 identifies a non-target region based on the user input. In this case, the first estimation unit 12 does not need to perform a process of analyzing a moving image to identify a non-target region. Furthermore, if the acquisition unit 11 does not receive user input specifying a non-target region within a moving image (within a frame image) in the external auxiliary information collection step described above, the first estimation unit 12 analyzes the moving image to identify a non-target region.

[0046] Furthermore, when an area in which a part of a plurality of roads is captured in a video is determined as a non-target area, the plurality of roads needs to be distinguished from one another. The first estimation unit 12 may distinguish the plurality of roads from one another based on a user input.

[0047] Alternatively, the first estimation unit 12 may distinguish between multiple roads based on differences in the heights of the locations where the multiple roads are located. For example, a highway and a general road may be located at different heights and appear simultaneously in a video image (e.g., when the highway is on an elevated road). This method is effective in such cases. The differences in the heights of the locations can be distinguished from the results of measuring the distance in the depth direction using depth estimation technology, such as LIDAR, ultrasonic sensors, or Deep Learning (AI) technology.

[0048] In addition, when the heights of the positions where the multiple roads are located are different from each other as described above, the sizes of the vehicles traveling on each road will be different from each other in the video. Therefore, the first estimation unit 12 may distinguish the multiple roads from each other based on the differences in the sizes of the vehicles traveling on the roads.

[0049] In addition, the types of objects installed on roads may differ depending on the type of road. For example, pedestrian signals are installed on general roads, but not on expressways. Furthermore, signs installed on general roads and signs installed on expressways may differ from each other. Therefore, the first estimation unit 12 may distinguish between multiple roads based on the objects installed on the roads.

[0050] Alternatively, the first estimation unit 12 may use the installation position and the direction of the camera to distinguish between multiple roads from road information used in a navigation system or the like.

[0051] The process of identifying the non-target area in this step only needs to be executed once after the processing device 10 is started / initialized, and does not need to be executed repeatedly. When the processing device 10 repeatedly executes the multiple steps in Fig. 4, the first estimation unit 12 can output the same information in the second and subsequent loops as in the first loop. The process of identifying the non-target area may be executed every time the second and subsequent loops are executed.

[0052] "First lane estimation process (FIG. 4)" The first estimation unit 12 executes the first lane estimation process. In the first lane estimation process, the first estimation unit 12 estimates a lane area using a first method based on a moving image. The lane area indicated by the estimation result of the first lane estimation process becomes the "primary lane area" in FIG. 4. The first estimation unit 12 can output information indicating the estimated lane area. The information output in the first lane estimation process is used in other processes shown in FIG. 4.

[0053] The lane area estimation result indicates the area within the video (area within the frame image) of each lane that appears in the video. If multiple lanes appear in the video, the lane area estimation result indicates the area within the video (area within the frame image) of each of the multiple lanes.

[0054] The "first method" is a method that can output lane area estimation results earlier than the second method described below. Any method that can achieve this condition can be adopted as the first method.

[0055] The first method may be a method in which a computer analyzes a moving image to estimate a lane area. Alternatively, the first method may be a method in which a computer receives a user input specifying a lane area on a frame image using a GUI generated based on the moving image (frame image), and estimates the lane area based on the user input.

[0056] Here, a method for estimating lane areas by analyzing moving images will be described.

[0057] For example, the first method may be a method of estimating a lane area based on one frame image in a video sequence, which can output an estimated lane area result more quickly than the second method described below.

[0058] For example, the first method may be a method of detecting white lines in a moving image (frame images) and estimating a lane area based on the detected white lines. In this method, an area between two white lines is estimated as one lane area. This method can estimate a lane area based on one frame image.

[0059] For example, the first estimation unit 12 can detect white lines and estimate lane areas in a moving image (frame image) using deep learning techniques such as white line detection and segmentation. Alternatively, the first estimation unit 12 can detect white lines and estimate lane areas in a moving image (frame image) using other image analysis techniques such as edge detection. Such techniques for detecting white lines and estimating lane areas are widely known. In this embodiment, any of these techniques can be adopted.

[0060] In addition, the first estimation unit 12 may use, in the estimation, time information, calendar information, weather information (weather, temperature, humidity, etc.) at the location where the fixed camera is installed, etc., acquired by the acquisition unit 11 in the external auxiliary information collection step. For example, the first estimation unit 12 may classify the environment at the time of shooting based on this information, and perform the estimation using an algorithm according to the environment when the frame image to be processed was captured or a learning model created for that environment.

[0061] The first estimation unit 12 may also identify the non-target area based on the information output in the non-target area exclusion step. Then, the first estimation unit 12 may determine only the area other than the non-target area in the moving image (frame image) as the processing target area, and estimate the lane area by analyzing the image of the processing target area.

[0062] The first lane estimation step only needs to be executed once after the processing device 10 is started / initialized, and does not need to be executed repeatedly. When the processing device 10 repeatedly executes the steps of Fig. 4, the first estimation unit 12 can output the same information as in the first loop in the second and subsequent loops. Note that the process of estimating the lane area using the first method may be executed every time in the second and subsequent loops.

[0063] "Vehicle Detection Process (FIG. 4)" The second estimation unit 13 executes the vehicle detection process. In the vehicle detection process, the second estimation unit 13 detects a vehicle from a moving image (frame image). The second estimation unit 13 then tracks the detected vehicle within the moving image and generates trajectory data indicating the movement trajectory of the vehicle. The second estimation unit 13 can output the generated trajectory data. The trajectory data output in the vehicle detection process is used in other processes shown in FIG. 4.

[0064] A "vehicle" is a vehicle that travels on the road. Examples of vehicles include, but are not limited to, four-wheeled vehicles, large vehicles, and motorcycles.

[0065] "Vehicle detection" can be realized using any well-known technology. For example, the second estimation unit 13 can detect vehicles in a moving image (frame image) by using deep learning technologies such as object detection, object tracking, and classification.

[0066] Note that the second estimation unit 13 may use time information, calendar information, weather information (weather, temperature, humidity, etc.) at the location where the fixed camera is installed, etc. acquired by the acquisition unit 11 in the external auxiliary information collection step for vehicle detection. For example, the second estimation unit 13 may classify the environment at the time of shooting based on this information, and detect vehicles using an algorithm according to the environment at the time the frame image to be processed was captured or a learning model created for that environment.

[0067] The second estimation unit 13 may also identify a non-target area based on the information output in the non-target area exclusion step. Then, the second estimation unit 13 may determine only the area other than the non-target area in the moving image (frame image) as the processing target area, and may detect a vehicle by analyzing the image of the processing target area.

[0068] Alternatively, the second estimation unit 13 may identify a lane area in the moving image (frame image) estimated by the first estimation unit 12 based on the information output in the first lane estimation step. Then, the second estimation unit 13 may set only the identified lane area as the processing target area and detect vehicles by analyzing the image of the processing target area.

[0069] "Vehicle tracking" can be achieved using any well-known technology.

[0070] The "trajectory data" indicates the movement trajectory of the vehicle within the shooting range of the fixed camera (within the video image). In one example, the trajectory data indicates the movement trajectory of the vehicle's reference location. In this embodiment, the reference location of the vehicle is the point obtained by lowering the approximate center of the left-right direction (vehicle width direction) of the vehicle toward the direction of travel of the vehicle to the ground. For example, the reference location of the vehicle may simply be the center of the base of a rectangular area in which the vehicle is detected in the video image. Note that the reference location of the vehicle is not limited to the one shown here. Other locations on the vehicle may also be used as the reference location of the vehicle. Even in such a modified example, the lane area can be estimated by appropriately adjusting the second lane estimation process described below. Details will be described later. Furthermore, a driver may be detected using person detection technology, and the movement trajectory of the detected person (driver) may be used as a pseudo-trajectory of the vehicle's movement.

[0071] The trajectory data is data that indicates the movement trajectory of a vehicle as a set of coordinates in a two-dimensional coordinate system set in moving images (frame images) generated by a fixed camera, for example.

[0072] "Second lane estimation process (FIG. 4)" The second estimation unit 13 executes the second lane estimation process. In the second lane estimation process, the second estimation unit 13 estimates the lane area using a second method different from the first method, based on video images acquired and stored after the lane area has been estimated using the first method. The lane area indicated by the final estimation result of the second lane estimation process becomes the "tertiary lane area" in FIG. 4. The lane area indicated by the estimation result obtained as an intermediate product of the second lane estimation process becomes the "secondary lane area" in FIG. 4. The second estimation unit 13 can output information indicating the estimated lane area. The information output in the second lane estimation process is used in other processes shown in FIG. 4.

[0073] The "second method" is a method for estimating lane areas based on the movement trajectory of a vehicle detected based on a plurality of frame images in a moving image.

[0074] The second estimation unit 13 can execute the processes of S1 to S5 in Fig. 5 as a second lane estimation step. The outline of each process is as follows.

[0075] Data acquisition process S1: The second estimation unit 13 acquires the trajectory data generated in the vehicle detection process. Extraction process S2: The second estimation unit 13 extracts trajectory data suitable for extracting lane areas from the acquired trajectory data. Determination process S3: The second estimation unit 13 determines whether the processes S1 and S2 executed up to that point satisfy predetermined conditions. The second estimation unit 13 then repeats S1 and S2 until the predetermined conditions are satisfied. If it is determined that the predetermined conditions are satisfied, the process proceeds to S4. By performing the process of S3, a sufficient amount of trajectory data can be extracted in S2, after which the process can proceed to S4 and subsequent processes. Lane area estimation process S4: The second estimation unit 13 estimates lane areas based on the trajectory data extracted up to that point in S2. Lane area confirmation process S5: The second estimation unit 13 determines whether the lane area is confirmed by the estimation result of S4. If not confirmed, the process returns to S1 and the same process is repeated. By performing the processing of S5, the estimation result of S4 is not determined as the lane area as it is, but rather the estimation result can be determined as the lane area at the stage when a reliable estimation result is obtained in S4.

[0076] Each process will be described in detail below.

[0077] "Data Acquisition Process S1 (FIG. 5)" The second estimation unit 13 acquires the trajectory data generated in the vehicle detection process.

[0078] "Extraction Process S2 (FIG. 5)" The second estimation unit 13 extracts trajectory data suitable for extracting lane areas from the trajectory data acquired in S1. Specifically, the second estimation unit 13 extracts trajectory data from the trajectory data acquired in S1, the trajectory data having at least one of length and direction that satisfy a predetermined condition. The predetermined condition can be defined based on the "length of the trajectory" or the "angle between the trajectory (e.g., a partial section of the trajectory) and a reference direction," for example. The predetermined condition here may be, for example, one of the following conditions, or a condition in which both conditions are connected by a predetermined logical operator. Note that these predetermined conditions are merely examples and are not limited to these.

[0079] The length of the trajectory is equal to or greater than a threshold value. There is no section where the angle between the trajectory and the reference direction is equal to or greater than a threshold value. The threshold value is a value that is set in advance.

[0080] By using the "length of the trajectory" as a condition, it is possible to extract trajectory data whose trajectory length is equal to or greater than a threshold, and to exclude noise data whose trajectory length is less than the threshold.

[0081] Furthermore, by using the "angle between the trajectory and the reference direction" as a condition, it is possible to extract trajectories in which no lane changes were made and exclude trajectories in which lane changes were made. The reference direction is a direction determined within the shooting range of a fixed camera (within a moving image), and is the same as or close to the direction in which the lane extends (the difference in direction is less than or equal to a threshold). A trajectory that has a section where the angle with the reference direction is greater than or equal to a threshold can be considered to be a trajectory in which a lane change was made.

[0082] The reference direction may be set by an operator. For example, in the external auxiliary information collection step ( FIG. 4 ) described above, the acquisition unit 11 may accept a user input for setting a reference direction on a moving image. Alternatively, the second estimation unit 13 may calculate the reference direction by statistically processing the trajectory data acquired in S1. Typically, the number of vehicles that change lanes is considered to be smaller than the number of vehicles that do not change lanes. For this reason, the second estimation unit 13 may, for example, take a majority vote on the extension direction of each piece of trajectory data acquired in S1 and determine the most common extension direction as the reference direction.

[0083] The second estimation unit 13 may delete the trajectory data that was not extracted in S2 from the trajectory data acquired in S1.

[0084] "Determination process S3 (FIG. 5)" The second estimation unit 13 determines whether the processes S1 and S2 executed up to that point satisfy a predetermined condition. The second estimation unit 13 then repeats S1 and S2 until the predetermined condition is satisfied. If it is determined that the predetermined condition is satisfied, the process proceeds to S4.

[0085] As described above, the predetermined condition here is a condition for advancing the process to S4. In order to improve the accuracy of estimation from S4 onwards, it is preferable to perform the processes from S4 onwards after a certain amount of trajectory data has been extracted in S2. The predetermined condition has contents that make it possible to realize such a preferred form. The predetermined condition here may be any one of the following, or may be a condition in which two or more of the following are connected by a predetermined logical operator:

[0086] The number of trajectory data (the number of vehicle movement trajectories) extracted in S2 up to that point is equal to or greater than a threshold value. The duration of execution of the processes of S1 and S2 is equal to or greater than a threshold value. The number of trajectory data (the number of vehicle movement trajectories) passing through at least one area set within the fixed camera's shooting range (within the video) is equal to or greater than a threshold value. Note that the threshold value is a preset value. The "at least one area set within the fixed camera's shooting range" can be set in advance by, for example, an operator. For example, in the external auxiliary information collection step ( FIG. 4 ) described above, the acquisition unit 11 may accept a user input to set the area in the video image.

[0087] "Lane Area Estimation Process S4 (Fig. 5)" The lane area estimation process S4 includes four processes (S4-1 to S4-4) shown in Fig. 6. By executing these four processes, the second estimation unit 13 estimates a lane area based on the trajectory data extracted in S2 up to that point.

[0088] Specifically, by executing the processes of S4-1 to S4-3, the second estimation unit 13 calculates a line indicating the tendency of the vehicle's movement trajectory as a lane reference line based on the trajectory data extracted in S2 up to that point. As described above, the trajectory data in this embodiment indicates the movement trajectory of a point that is approximately the center of the vehicle's left-right direction in the traveling direction and is lowered to the ground. Therefore, the lane reference line is a line that indicates the tendency of the movement trajectory of a point that is approximately the center of the vehicle's left-right direction in the traveling direction and is lowered to the ground.

[0089] The second estimation unit 13 then executes the process of S4-4 to estimate the lane area from the lane reference lines calculated in S4-1 to S4-3. Each process will be described in detail below.

[0090] "Calculation of lane candidate points S4-1 (FIG. 6)" First, the second estimation unit 13 determines a reference direction within the shooting range of the fixed camera (within the moving image). The definition of the reference direction and the method for setting it have been described above, so a description thereof will be omitted here.

[0091] An example is shown in Figure 7. Figure 7 shows a frame image captured by a fixed camera. Also shown are the movement trajectories VL of multiple vehicles indicated by the trajectory data extracted in S2. The up-down direction of this frame image is defined as the reference direction. In this embodiment, this reference direction is referred to as the y-axis direction. Note that in the frame image of Figure 7, the up-down direction of the frame image is defined as the reference direction (y-axis direction), but depending on the position and orientation of the camera, a diagonal direction or left-right direction of the frame image may also be defined as the reference direction (y-axis direction).

[0092] Then, the second estimation unit 13 determines a plurality of coordinates in the reference direction at predetermined intervals as shown in FIG. 7. ０ , y １ , y ２ ...) are multiple coordinates of the reference direction determined at predetermined intervals.

[0093] The value of the predetermined interval is a design factor. The second estimation unit 13 may determine multiple y coordinates at predetermined intervals across the entire shooting range of the fixed camera (across the entire frame image in the case of FIG. 7 ). Alternatively, the second estimation unit 13 may determine multiple y coordinates at predetermined intervals in a portion of the shooting range of the fixed camera. This portion is, for example, an area where the vehicle movement trajectory VL indicated by the trajectory data extracted in S2 exists. In the example of FIG. 7 , the vehicle movement trajectory VL does not extend toward the upper edge of the frame image. The second estimation unit 13 may exclude such an area where the vehicle movement trajectory VL does not extend from the target for determining multiple y coordinates at predetermined intervals.

[0094] Next, the second estimation unit 13 calculates a plurality of y coordinates (y ０ , y １ , y ２ ...), the second estimation unit 13 calculates the position through which the vehicle movement trajectory VL indicated by the trajectory data extracted in S2 passes. ０ , y １ , y ２In each of the trajectories (...), the intersection of a line where y = each y coordinate and the vehicle movement trajectory VL indicated by the trajectory data extracted in S2 is calculated. When an x-axis perpendicular to the y-axis is defined on the frame image, multiple y-coordinates (y ０ , y １ , y ２ In each of the above, {x １ , x ２ , . . .}, a set of x coordinates indicating positions through which the vehicle movement trajectory VL passes is obtained.

[0095] "Calculation of the number of lanes S4-2 (FIG. 6)" First, the second estimation unit 13 calculates a plurality of y coordinates (y ０ , y １ , y ２ ...) to extract the y coordinate used to calculate the number of lanes.

[0096] As shown in FIG. 8, a plurality of y coordinates (y ０ , y １ , y ２ In each of the above cases, the number of intersections between the line where y = each y coordinate and the vehicle movement trajectory VL indicated by the trajectory data extracted in S2 (the number of positions through which the vehicle movement trajectory VL passes) may be different. In the example of FIG. 8, the coordinate y ０ The number of intersections in the coordinate y １ The number of intersections is 3, and the coordinate y ２ The number of intersections is 1. In S4-1, a plurality of y coordinates (y ０ , y １ , y ２ The number of elements in the set of x coordinates calculated in each of the above steps is the number of intersections.

[0097] The second estimation unit 13 estimates a plurality of y coordinates (y ０ , y １ , y ２ . . ) where the number of intersections is equal to or greater than a threshold is extracted as the y coordinate to be used for calculating the number of lanes.

[0098] The threshold value here may be determined in advance. Alternatively, the threshold value here may be determined based on the number of pieces of trajectory data extracted in S2. For example, the second estimation unit 13 may set the threshold value here to a value obtained by multiplying the number of pieces of trajectory data extracted in S2 by a predetermined coefficient greater than 0 and equal to or less than 1. The predetermined coefficient is a predetermined value.

[0099] By this process, y coordinates that are passed through (intersected with) less by the vehicle movement trajectory VL indicated by the trajectory data extracted in S2 can be excluded from the coordinates used to calculate the number of lanes.

[0100] Next, for each y coordinate extracted in the above process (hereinafter, sometimes simply referred to as an "extracted y coordinate"), the second estimation unit 13 can calculate a histogram of the positions (x coordinates) of the intersections between a line, where y = each y coordinate, and the vehicle movement trajectory VL indicated by the trajectory data extracted in S2, as shown in Figure 9(A). The horizontal axis of Figure 9(A) is the x coordinate, and the vertical axis is the number of intersections.

[0101] Furthermore, the second estimation unit 13 can calculate a probability density function for the position where the vehicle passes for each extracted y coordinate, as shown in FIG. 9B . For example, the second estimation unit 13 can calculate the probability density function using kernel density estimation. The "position where the vehicle passes" is the intersection position (x coordinate) between a line where y = each y coordinate and the vehicle movement trajectory VL indicated by the trajectory data extracted in S2. The horizontal axis of FIG. 9B represents the x coordinate, and the vertical axis represents density. Note that the second estimation unit 13 can treat all values ​​below the threshold r as "0," as shown in FIG. 10 . This can reduce the influence of the vehicle's movement trajectory VL, which rarely occurs due to lane changes, on the passing position (intersection position).

[0102] Next, as shown in FIG. 11A, the second estimation unit 13 calculates the number of peaks and the positions (x coordinates) of the peaks of the probability density function for each extracted y coordinate. A peak is a position where a maximum value is obtained. In FIG. 11A, two peaks P １ and P ２ The probability density function of FIG. 11A is calculated by the coordinate y ４11B is a probability density function relating to the position where the vehicle passes through. In FIG. 11B, two peaks P calculated using the probability density function of FIG. 11A are １ and P ２ This shows the location of.

[0103] After calculating the number of peaks for each extracted y-coordinate, the second estimation unit 13 calculates the number of lanes based on the number of peaks for each extracted y-coordinate. For example, the second estimation unit 13 takes a majority vote and calculates the number of peaks with the largest number as the number of lanes.

[0104] "Combining S4-3 (FIG. 6)" The second estimation unit 13 combines the peak positions for each y coordinate calculated in S4-2 along the y-axis direction, thereby generating lines for the number of lanes calculated in S4-2.

[0105] The second estimation unit 13 may determine the peak position for the y coordinate not extracted in S4-2 by the method described in S4-2. The second estimation unit 13 may also use the peak position calculated for the y coordinate not extracted in S4-2 as a target for the combining process. That is, the second estimation unit 13 may use a plurality of y coordinates (y ０ , y １ , y ２ ...), the peak position may be determined using the method described in S4-2, and the peak positions for each y coordinate may be combined along the y-axis direction to generate lines for the number of lanes calculated in S4-2.

[0106] Hereinafter, the line generated in this manner will be referred to as the “lane reference line.” Furthermore, hereinafter, the peak position determined for each y coordinate by the method described in S4-2 may be simply referred to as the “peak position for each y coordinate.”

[0107] There are various methods for combining the peak positions for each y coordinate along the y-axis direction, but one example will be described below with reference to FIGS.

[0108] First, the second estimation unit 13 determines the y coordinate to be the start position of the bond. In one example, among the multiple y coordinates with the largest number of peaks, the y coordinate that is located in the middle in the arrangement along the reference direction can be determined as the start position of the bond. In the example of FIG. 12, yｎ , y ｎ＋１ , y ｎ＋２ , y ｎ＋８ , y ｎ＋９ Among these five y coordinates, the coordinate y ｎ＋２ In the example of FIG. ｎ＋２ is determined as the bond start position. By determining the bond start position in this manner, the following effects can be obtained.

[0109] Generally, in many angles of view, the actual number of lanes (the most frequent peaks) is distributed mostly near the center of the image. In other words, the y coordinates with the most frequent peaks tend to be concentrated near the center of the image. For example, if multiple y coordinates at a predetermined interval are used, ０ or y ２０ If " is set, y １０ Near, for example, y ８ or y １３ This is because the number of peaks increases in the downward direction of the image (y ０ ) the vehicle will appear larger, and the upward (y ２０ ) the vehicle appears smaller, which reduces the accuracy of vehicle detection.

[0110] In this way, in many cases, the most common peaks are near the center of the image (in the above example, y １０ In rare cases, it may be near the top of the image (for example, y １９ ) or near the bottom (in the above example, for example, y ３ ) may appear isolated in a certain area. If the above-mentioned joining is started from the y-coordinate where such a peak appears isolated, a situation may arise in which it is not possible to find a joining destination for some peaks due to the joining cost. As a result, the number of lines generated by the above-mentioned joining may be less than the actual number of lanes. By determining the starting position of the joining as described above, it is more likely that the y-coordinate in the center of the image, where the peaks are expected to be concentrated, can be used as the starting position. As a result, the above-mentioned inconvenience can be suppressed.

[0111] The y-coordinate of the joining start position may be determined by other methods that can set the y-coordinate of the central part of the image where the most peaks are expected to be concentrated as the starting position. The y-coordinate of the joining start position may also be determined by other methods, although such an effect may not be obtained. Even when other methods are used, the lane area can be estimated with a certain degree of accuracy.

[0112] The second estimation unit 13 determines the y-coordinate as the starting position of the joining, and then joins the peak positions for each y-coordinate along the y-axis direction (a direction parallel to the reference direction) to generate lines for the number of lanes calculated in S4-2.

[0113] Specifically, the second estimation unit 13 can perform the following processes (1) to (8) in this order.

[0114] (1) The second estimation unit 13 selects one lane as a processing target. (2) The second estimation unit 13 selects one peak position as a processing target. (3) The second estimation unit 13 extracts, as a combining candidate, a peak position whose cost for combining with the peak position selected in (2) is within the maximum cost. (4) The second estimation unit 13 narrows down the combining candidates by extracting combining candidates that satisfy predetermined conditions from the combining candidates extracted in (3). (5) The second estimation unit 13 determines one of the combining candidates extracted in (4) as a peak position to be combined with the processing target peak position selected in (2). (6) The second estimation unit 13 combines the peak positions to generate (extend) a line. (7) The second estimation unit 13 returns to (1), selects a new lane as a processing target, and executes (2) to (6). (8) After (7), the second estimation unit 13 returns to (1), selects a new lane as the processing target, and executes (2) to (6).

[0115] Each process will be described in detail below.

[0116] (1) "The second estimation unit 13 selects one lane as the processing target." If the number of lanes calculated in S4-2 is Q, the second estimation unit 13 selects one of the Q lanes as the processing target. Then, the second estimation unit 13 generates a lane reference line corresponding to the selected lane by the following process.

[0117] (2) "The second estimation unit 13 selects one peak position to be processed." When combining peak positions for the first time in correspondence with the lane selected in (1), one of the peak positions present at the y coordinate to be the starting position for combining is selected by any means and made the peak position to be processed.

[0118] As will be described below, the processes from (2) onward are repeatedly executed. Therefore, there are cases where the process of combining peak positions corresponding to the lanes selected in (1) is executed at least once, and a line is generated, and then (2) is executed. In this case, the second estimation unit 13 selects the peak position at the beginning of the line generated up to that point as the processing target.

[0119] The "peak position at the beginning of a line" is the beginning (end) of a line generated by combining the peak positions up to that point. In FIG. 12, when peak positions are combined to extend the line downward in the figure, the lower end of the line becomes the peak position at the beginning of the line. On the other hand, in FIG. 12, when peak positions are combined to extend the line upward in the figure, the upper end of the line becomes the peak position at the beginning of the line. Note that before the combining begins (before the line is generated), the peak position at the y coordinate that is the starting position of the combining is treated as the peak position at the beginning of the line.

[0120] After selecting one peak position to be processed, the second estimation unit 13 calculates the maximum cost that can be combined for the selected peak positions based on the following formulas (1) and (2). Formula (1) is a formula for calculating the cost for combining two peak positions, and formula (2) is a formula for calculating the maximum cost.

[0121]

[0122]

[0123] (3) "The second estimation unit 13 extracts, as a combination candidate, a peak position whose cost for combining with the peak position selected in (2) is within the maximum cost." This process will be described with reference to FIG. 13. １is the peak position selected in (2). Figure 13 shows the line (P ０ and P １ The line connecting the two points is generated.

[0124] In this case, P １ is the peak position at the beginning of the line, which is the peak position selected in (2) as described above. ０ is the position of the peak immediately before that. The maximum combinable cost (max_cost in equation (2)) obtained in (2) above shows the combinable range in FIG. 13. In FIG. 13, P selected in (2) １ When the peak position where the cost to combine with P is within the maximum cost is extracted, ２ and P ３ As shown in equation (2), the peak position P １ and the previous peak position P ０ By adding a constant to the cost of combining and determining the maximum cost, the range of possible combinations can be narrowed as the line extends.

[0125] (4) "The second estimation unit 13 narrows down the bond candidates by extracting bond candidates that satisfy a predetermined condition from the bond candidates extracted in (3)." The predetermined condition here may be one of the following conditions, or may be a condition in which both conditions are connected by a predetermined logical operator. Note that these predetermined conditions are merely examples and are not limiting.

[0126] - The peak position is not already combined with another peak position. - The cost of combining with the peak position at the beginning of the line corresponding to the lane being processed is smaller than the cost of combining with the peak position at the beginning of the line corresponding to another lane.

[0127] (5) "The second estimation unit 13 determines one of the bond candidates extracted in (4) as the peak position to be combined with the peak position of the processing target selected in (2)." The second estimation unit 13, for example, determines the bond candidate that has the smallest cost for combining with the peak position of the processing target selected in (2) from the bond candidates extracted in (4) as the peak position to be combined with the peak position of the processing target selected in (2).

[0128] (6) "The second estimation unit 13 generates (extends) a line by joining the peak positions." The second estimation unit 13 joins the peak position to be processed selected in (2) with the joining candidate determined in (5) to generate (extend) a line. Then, the second estimation unit 13 sets the joining candidate determined in (5) as the leading peak position of the line.

[0129] (7) "The second estimation unit 13 returns to (1), selects a new lane as the processing target, and executes (2) to (6)." If the number of lanes calculated in S4-2 is Q, the second estimation unit 13 selects all of the Q lanes and repeats the process until it executes (2) to (6) for all lanes. The second estimation unit 13 selects a new lane as the processing target from among the lanes that have not yet been selected. In this way, it is possible to proceed with the joining for all lanes in parallel. Then, after executing (2) to (6) for all lanes, the second estimation unit 13 proceeds to (8).

[0130] (8) "After (7), the second estimation unit 13 returns to (1), selects a new lane as a processing target, and executes (2) to (6)." After selecting all lanes as processing targets in (7), the second estimation unit 13 executes the process of (8). In such (8), the second estimation unit 13 resets the lane selection history in (7) immediately before (8) and selects a new lane from scratch. That is, the second estimation unit 13 may newly select a lane that was previously selected in (7). The second estimation unit 13 repeats the processes of (1) to (7) until the leading positions of all lanes are no longer updated.

[0131] Here, the characteristics of the combination in the above processes (1) to (8) will be explained using FIG. 14. As shown in FIG. 14, two or more peak positions are not combined to one peak position. The two peak positions with the smallest combination cost are combined. Furthermore, peak positions that are separated by a distance in the y-axis direction by a predetermined value or more can also be combined. That is, as shown in the figure, ｎ＋１ The peak position and y ｎ＋２ Not only the peak positions of adjacent y coordinates such as the peak positions of ｎ＋１ The peak position and y ｎ＋３ Peak positions on the y coordinate that are separated by a predetermined value or more may be directly combined, such as the peak positions in (a) and (b). Also, even if the number of lanes and the peaks are the same, they will not be combined if the combining cost is high.

[0132] The second estimation unit 13 may perform a process of removing noise data after generating a lane reference line for each lane by the processes (1) to (8) above. For example, the second estimation unit 13 may delete lane reference lines that satisfy conditions such as a length that is equal to or less than a threshold or a length in the y-axis direction that is equal to or less than a threshold.

[0133] As shown in FIG. 15, the three lane reference lines EL １ ~EL ３ By the removal process, the second estimation unit 13 generates the lane reference line EL ３ Then, as shown in FIG. 16, the lane reference line EL after removing the noise data is obtained. １ and EL ２ can be output as the processing result.

[0134] For example, there is a possibility that a larger number of lane reference lines than the actual number of lanes may be generated due to some malfunction. When a larger number of lane reference lines than the actual number of lanes are generated by this process, noise data can be removed.

[0135] "Lane Area Estimation S4-4 (FIG. 6)" The second estimation unit 13 estimates the lane area based on the lane reference lines generated in S4-3. Specifically, the second estimation unit 13 determines the boundary in the width direction of the lane based on the lane reference lines. This processing by the second estimation unit 13 will be explained using FIG. 17. In S4-3, the two lane reference lines EL shown in the figure are used to estimate the lane area. １ and EL ２ is generated.

[0136] When a plurality of lane reference lines are detected, the second estimation unit 13 estimates a position where adjacent lane reference lines are divided at a predetermined ratio as a boundary between adjacent lanes.

[0137] First, the second estimation unit 13 can identify a location where an adjacent lane reference line exists. The location can be identified by various methods. For example, the second estimation unit 13 can identify a location where an adjacent lane reference line exists. １ The second estimation unit 13 then generates a line extending from any point in the line in a direction perpendicular to the reference direction. The second estimation unit 13 then identifies a location where another lane reference line intersects with the line as a location where an adjacent lane reference line exists.

[0138] Next, the second estimation unit 13 estimates the position at the identified location where the adjacent lane reference lines are divided by a predetermined ratio as the boundary of the adjacent lanes. The predetermined ratio is determined according to the setting of the "vehicle reference point" described above. The vehicle reference point in this embodiment is the point where the "approximate center" in the left-right direction of the vehicle is lowered to the ground in the direction of travel of the vehicle. In other words, it is the point where the width direction of the vehicle is divided at a 1:1 ratio. In this embodiment, the predetermined ratio is 1:1. That is, the second estimation unit 13 estimates the midpoint between the adjacent lane reference lines as the boundary of the adjacent lanes, as shown in FIG. 17. By this processing, the boundary BL shown in the figure is １ The reference point of the vehicle is determined as E １ Against E ２ When the predetermined ratio is E ２ Against E １ is.

[0139] In addition, in a location where there is no adjacent lane reference line, the second estimation unit 13 can estimate the boundary by, for example, the following method.

[0140] First, the boundary BL in FIG. ２ and BL ３ The process of estimating the lane reference line EL will be described. ２ From the perspective of the boundary BL ２ In this case, the second estimation unit 13 determines whether there is another lane reference line on the side of the lane reference line EL. ２ The boundary BL on the opposite side １ Based on the distance to the boundary BL ２ can be estimated.

[0141] That is, the second estimation unit 13 estimates the lane reference line EL ２ From the boundary BL １ The second estimation unit 13 then calculates the distance D to the lane reference line EL ２ From the boundary BL １ The position at a distance D in the opposite direction is the boundary BL ２ Note that due to the perspective effect, the distance D may differ for each coordinate in the reference direction on the image. For this reason, the distance D is calculated for each coordinate in the reference direction, and the boundary BL is estimated using the distance D for each coordinate in the reference direction. ２ It is preferable to calculate the boundary BL ３ is the boundary BL ２ It can be estimated using the same method.

[0142] Next, the boundary BL in FIG. ４ and BL ５ The process of estimating the lane reference line EL will be described. ２ From the perspective of the boundary BL ４ There are no other lane reference lines on the side of the lane reference line EL. ２ From the perspective of the boundary BL ４ In such a case, the second estimation unit 13 uses the distance D calculated at another coordinate in the reference direction to determine the boundary BL ４ That is, the second estimation unit 13 estimates the lane reference line EL ２ From the perspective of the boundary BL ４ The position at a distance D in the direction of ４As described above, the distance D may differ for each coordinate in the reference direction. Therefore, the second estimation unit 13 estimates the boundary BL ４ The boundary BL is calculated by using the distance D calculated at the point closest to the point where the boundary BL is to be estimated (the point closest in the y direction). ４ It is preferable to estimate the boundary BL ５ is the boundary BL ４ It can be estimated using the same method.

[0143] "Lane Area Confirmation Process S5 (FIG. 5)" The second estimation unit 13 determines whether to confirm the lane area based on the estimation result of S4. If not, the process returns to S1 and the same process is repeated. By performing the process of S5, the estimation result of S4 is not confirmed as the lane area as it is, but rather, once a reliable estimation result is obtained in S4, the estimation result can be confirmed as the lane area. As a result, it becomes possible to output a highly reliable result.

[0144] The lane area that was not determined in S5, i.e., the lane area before being determined in S5, is the "secondary lane area" in Fig. 4. The lane area that was determined in S5 is the "tertiary lane area" in Fig. 4.

[0145] As described above, the second estimation unit 13 continues to acquire trajectory data and repeatedly estimates lane areas based on the accumulated trajectory data.

[0146] After estimating a lane area in the nth processing, the second estimation unit 13 determines whether each estimated lane area is identical to the lane area estimated in the (n-1)th processing. Specifically, the second estimation unit 13 determines the difference between each lane area estimated in the nth processing and each lane area estimated in the (n-1)th processing. If the difference between the lane area estimated in the (n-1)th processing and the lane area estimated in the nth processing is equal to or smaller than a reference value, the second estimation unit 13 determines that the lane area estimated in the (n-1)th processing and the lane area estimated in the nth processing are identical.

[0147] The second estimation unit 13 performs the above-mentioned identity determination each time a lane area is estimated in each processing. If the lane areas are determined to be identical a predetermined number of times or more in succession, the second estimation unit 13 confirms the lane areas estimated during that time as the estimation result. In other words, the second estimation unit 13 confirms the lane areas that have been determined to be identical a predetermined number of times or more in succession.

[0148] For example, the second estimation unit 13 can execute, as S5, three processes (S5-1 to S5-3) shown in Fig. 18. Note that the second estimation unit 13 may determine the identity of the lane areas described above using a method different from the process described below.

[0149] "Comparison with previous lane area S5-1 (FIG. 18)" The second estimation unit 13 determines whether the lane area estimated in the (n-1)th processing is the same as the lane area estimated in the nth processing. If the difference between the lane area estimated in the (n-1)th processing and the lane area estimated in the nth processing is equal to or less than a predetermined standard, the second estimation unit 13 determines that the lane area estimated in the (n-1)th processing is the same as the lane area estimated in the nth processing. There are various methods for calculating the difference between two lane areas, but one example will be described below. Hereinafter, the lane area estimated in the (n-1)th processing may be referred to as the "(n-1)th lane area," and the lane area estimated in the nth processing may be referred to as the "nth lane area."

[0150] First, the second estimation unit 13 calculates a three-dimensional probability density function based on the constituent points on the lane reference line used to estimate the (n-1)th lane area. For example, the second estimation unit 13 can calculate a three-dimensional probability density function as shown in Fig. 19 by kernel density estimation (KDE).

[0151] The second estimation unit 13 generates data for generating a three-dimensional probability density function from data on the group of constituent points on the lane reference line used to estimate the (n-1)th lane area, and can calculate the three-dimensional probability density function based on the generated data for generating the three-dimensional probability density function.

[0152] The data for generating the three-dimensional probability density function is random data that approximates the data of the group of constituent points on the lane reference line used to estimate the (n-1)th lane area. Such data for generating the three-dimensional probability density function can be generated by various methods. For example, the second estimation unit 13 generates the i-th data (x ｉ ,y i ) as a processing target. Then, the second estimation unit 13 randomly generates data that is approximate to the selected data based on the following equation (3).

[0153]

[0154] b and α are predetermined values. b is the coordinate y when a vehicle is detected by image analysis, for example. i Alternatively, the variance range may be calculated from the average width of the base of a rectangular area (area where a vehicle is present) detected in the vicinity of the vehicle.

[0155] The second estimation unit 13 performs this process for all i and generates a set of randomly generated data corresponding to each piece of data on the constituent points on the lane reference line used to estimate the (n-1)th lane area. This set of data becomes data on the constituent points that approximates the data on the constituent points on the lane reference line used to estimate the (n-1)th lane area. The second estimation unit 13 repeats this process a predetermined number of times (e.g., 100 times) to generate a predetermined number of pieces of data on the constituent points that approximates the data on the lane reference line used to estimate the (n-1)th lane area. The second estimation unit 13 then calculates a three-dimensional probability density function based on the data for the predetermined number of times.

[0156] The second estimation unit 13 generates data for generating a three-dimensional probability density function from data on the group of constituent points on the lane reference line used to estimate the (n-1)th lane area, for example, as described above. Then, the second estimation unit 13 inputs each of the constituent points on the lane reference line used to estimate the nth lane area into the three-dimensional probability density function, and calculates the probability (occurrence rate) of the value input for each constituent point.

[0157] The second estimation unit 13 determines that the (n-1)th lane area and the nth lane area are the same when the number of component points whose calculated probabilities are equal to or greater than the threshold is equal to or greater than a reference value. On the other hand, the second estimation unit 13 determines that the (n-1)th lane area and the nth lane area are different when the number of component points whose calculated probabilities are equal to or greater than the threshold is less than the reference value. The threshold and reference value here are predetermined values.

[0158] "Lane area determination S5-2 (Figure 18)" If the second estimation unit 13 determines that the (n-1)th lane area and the nth lane area are the same for a predetermined number of consecutive times or more, it determines the lane area estimated during that time as the estimation result.

[0159] In the condition "when the (n-1)th lane area and the nth lane area are judged to be the same for a predetermined number of consecutive times or more," "judged to be the same" may also be "judged to be the same in S5-1."

[0160] As another example, the second estimation unit 13 may perform a more detailed determination of identity in addition to the determination of identity in S5-1. In addition, the "determined to be identical" in the condition "when the (n-1)th lane area and the nth lane area are determined to be identical for a predetermined number of consecutive times or more" may be "determined to be identical in this detailed determination of identity" or "determined to be identical in both this detailed determination of identity and in S5-1."

[0161] The second estimation unit 13 can perform a detailed determination of identity based on, for example, the following formula (4): If the condition of the following formula (4) is satisfied, the second estimation unit 13 determines that the (n-1)th lane area and the nth lane area are identical.

[0162]

[0163] NewLane x ｉ is the x-coordinate of the i-th point on the lane reference line used to estimate the n-th lane area. ｉ is the i-th x coordinate in the group of constituent points on the lane reference line used to estimate the (n-1)-th lane area. ｉ and OldLane xｉ are the same y-coordinate (y ｉ ) is the x-coordinate corresponding to

[0164] ImageShape X is the width of the image in the x-axis direction, and M is a predetermined value, such as 0.9.

[0165] "Post-processing S5-3 (FIG. 18)" When the second estimation unit 13 determines a lane area in S5-2, it deletes the trajectory data used to estimate the determined lane area from the trajectory data extracted in S2. The trajectory data used to estimate the lane area is the trajectory data extracted in S2 before the lane area is determined and used to calculate the lane reference line of the lane area in S4.

[0166] When there are multiple lanes in the road, multiple lane areas may be determined simultaneously, or the lane areas may be determined one by one. When the lane areas are determined one by one, steps S1 to S5 in FIG. 5 are repeated after one lane area is determined until another lane area is determined.

[0167] Under these assumptions, if the above deletion is performed and there is no trajectory data related to one lane area (a state in which it is not included in the extraction result of S2), an incorrect result may be calculated when calculating the probability density function of S4-2 based on the trajectory data extracted in S2.

[0168] Therefore, in order to prevent this problem, the second estimation unit 13 can delete trajectory data related to one lane area and then create dummy data related to that lane area. Thereafter, the second estimation unit 13 can calculate the probability density function in S4-2 using the dummy data related to that lane area and the accumulated trajectory data related to other lane areas (trajectory data extracted in S2). The second estimation unit 13 can create the dummy data, for example, using the following method.

[0169] The dummy data is random data that approximates the data of the constituent points on the lane reference line used to estimate the confirmed lane area. Such dummy data can be generated by various methods. For example, the second estimation unit 13 generates the i-th data (x ｉ ,y i ) as a processing target. Then, the second estimation unit 13 randomly generates data that is approximate to the selected data based on the following equation (5).

[0170]

[0171] β is a predetermined value. The second estimation unit 13 performs this process for all i to generate a set of randomly generated data corresponding to each piece of data on the constituent points on the lane reference line used to estimate the confirmed lane area. This set of data becomes data on the constituent points that approximates the data on the constituent points on the lane reference line used to estimate the confirmed lane area. The second estimation unit 13 repeats this process a predetermined number of times (e.g., 10 times) to generate dummy data for the predetermined number of times.

[0172] "External Output Step (FIG. 4)" The output unit 14 executes the external output step. In the external output step, the output unit 14 outputs information indicating the lane areas estimated in the first lane estimation step and the second lane estimation step (information indicating the lane areas in the video).

[0173] First, the output unit 14 outputs information indicating the primary lane area estimated by the first method in the first lane estimation process. Then, the output unit 14 outputs information indicating the secondary lane area estimated by the second method in the second lane estimation process (the lane area indicated by the estimation result until it is confirmed). Then, the output unit 14 outputs information indicating the tertiary lane area estimated by the second method in the second lane estimation process (the lane area indicated by the confirmed estimation result).

[0174] The output unit 14 outputs information indicating the primary lane area, and thereafter, when the secondary lane area is estimated by the second method in the second lane estimation process, the output unit 14 can output information indicating the secondary lane area. Thereafter, when the tertiary lane area is estimated by the second method in the second lane estimation process, that is, when the estimation result is confirmed in the second lane estimation process, the output unit 14 can output information indicating the tertiary lane area.

[0175] The information indicating the lane area may be output to various destinations, for example, an external device that performs various processes based on the estimated lane area.

[0176] The external device performs various processes based on the estimated lane area. For example, the external device may detect a target, such as a vehicle that frequently changes lanes or a vehicle that travels across multiple lanes, based on the estimated lane area and video images generated by a fixed camera. The external device may also measure the traffic conditions (e.g., congestion) and traffic volume for each lane based on the estimated lane area and video images generated by a fixed camera. These measurements can be realized using any well-known technology.

[0177] Alternatively, the output unit 14 may generate and output a processed image in which information indicating the lane area is superimposed on a moving image generated by a fixed camera as information indicating the lane area. In this case, the output unit 14 may input the generated processed image to an output device such as a display or a projection device, and display the generated processed image. Alternatively, the output unit 14 may transmit the generated processed image to another device. The output unit 14 may identify a non-target area based on the information output in the non-target area exclusion step. Then, the output unit 14 may further superimpose information indicating the non-target area on the processed image.

[0178] Next, an example of the processing flow of the processing device 10 will be described with reference to the flowchart of FIG.

[0179] Separately from the process shown in FIG. 20, the video capturing step (see FIG. 4) is executed, and the capturing of moving images generated by the fixed camera continues.

[0180] First, the processing device 10 executes a first lane estimation step (see FIG. 4). That is, the processing device 10 estimates a lane area based on one frame image in a video image generated by a fixed camera (S20). The lane area indicated by this estimation result becomes the primary lane area described with reference to FIG. 4. Next, the processing device 10 executes an external output step (see FIG. 4) and outputs the estimation result (primary lane area) of S20 (S21).

[0181] The processing device 10 may execute at least one of an external auxiliary information collection step (see FIG. 4) and a non-target area exclusion step (see FIG. 4) before the first lane estimation step (see FIG. 4). The output results of these steps may be used in the first lane estimation step (see FIG. 4), a vehicle detection step (see FIG. 4) and a second lane estimation step (see FIG. 4) described later.

[0182] After S21, the processing device 10 executes a vehicle detection step (see FIG. 4) and a second lane estimation step (see FIG. 4). That is, the processing device 10 estimates the lane area based on the moving images that are further acquired and stored after the lane area is estimated in S20.

[0183] As described above, in the vehicle detection step (see FIG. 4) and the second lane estimation step (see FIG. 4), the process of calculating the vehicle's movement trajectory based on the moving image and estimating the lane area based on the vehicle's movement trajectory is repeated. If the same estimation result is obtained a predetermined number of times or more, the lane area indicated by the estimation result is determined to be the estimation result.

[0184] In S22, the processing device 10 determines whether a lane area has been estimated based on the vehicle's movement trajectory. For example, in a process of repeatedly estimating a lane area, if an initial estimation result is obtained in the first loop, the processing device 10 can determine that a lane area has been estimated based on the vehicle's movement trajectory. The lane area indicated by the initial estimation result obtained in this first loop becomes the secondary lane area described with reference to FIG. 4. Note that instead of the initial estimation result of the first loop, the estimation result of the second loop or the estimation result of a subsequent loop may be used as the secondary lane area. However, using the initial estimation result as the secondary lane area is preferable because it allows for earlier output timing of the secondary lane area.

[0185] The processing device 10 repeats this determination until it determines that the lane area has been estimated based on the vehicle movement trajectory (No in S22).

[0186] Then, when it is determined that the lane area has been estimated based on the vehicle's movement trajectory (Yes in S22), the processing device 10 executes an external output process (see Figure 4) and outputs the estimation result of S22 (secondary lane area) (S23).

[0187] After S23, the processing device 10 continues the vehicle detection process (see FIG. 4) and the second lane estimation process (see FIG. 4). Then, in S24, the processing device 10 determines whether the estimation result has been confirmed. As described above, if the processing device 10 obtains the same estimation result a predetermined number of times or more, it confirms the lane area indicated by the estimation result as the estimation result. The lane area indicated by this confirmed estimation result becomes the tertiary lane area described with reference to FIG. 4.

[0188] The processing device 10 repeats this determination until the estimation result is confirmed (No in S24). During this time, the processing device 10 continues the vehicle detection process (see FIG. 4) and the second lane estimation process (see FIG. 4).

[0189] Then, when it is determined that the estimation result has been confirmed (Yes in S24), the processing device 10 executes an external output process (see FIG. 4) and outputs the confirmed estimation result (tertiary lane area) (S25).

[0190] As mentioned above, when multiple lanes are captured in a video image generated by a fixed camera, the lane areas (estimated results) of each of the multiple lanes may be determined at once, or they may be determined one by one.

[0191] When the lane areas are determined one by one, the processing device 10 may execute the external output step (see FIG. 4) each time a lane area is determined, and output the determined estimation result (tertiary lane area) (S25). Then, the processing device 10 may repeat S24 and S25 until the lane areas (estimation results) of all lanes are determined.

[0192] Alternatively, if the lane areas are determined one by one, the processing device 10 may execute an external output process (see Figure 4) after the lane areas (estimation results) of all lanes have been determined, and output all the determined estimation results (tertiary lane areas) together (S25).

[0193] Here, a modified example of the process described using the flowchart in Fig. 20 will be described. In the above process, the processing device 10 repeats the process of estimating a lane area in the second lane estimation step (see Fig. 4) until the estimation result is confirmed. Then, the processing device 10 outputs the confirmed estimation result as the tertiary lane area, and outputs the estimation result obtained in any one of the multiple loops until the confirmation is confirmed as the secondary lane area.

[0194] In a modified example, the processing device 10 outputs the confirmed estimation result as the tertiary lane area, and outputs the estimation results obtained in any "two or more" of the multiple loops until the confirmation as the secondary lane area. That is, the processing device 10 outputs two or more estimation results as the secondary lane area. The processing device 10 may output the estimation results obtained in all of the multiple loops as the secondary lane area. Alternatively, the processing device 10 may output the estimation results obtained in some of the multiple loops as the secondary lane area. In the latter case, the processing device 10 may output the estimation result as the secondary lane area every n loops (n is any integer). The processing device 10 can output the estimation result as the secondary lane area each time an estimation result is obtained in each loop.

[0195] Although the estimation results obtained in each of the multiple loops until a final determination is made are all inconclusive and insufficiently accurate, the accuracy of each result varies. The more loops are performed, the more accumulated data can be used for estimation processing, and therefore the accuracy tends to increase. Therefore, according to this modification, the accuracy of the multiple secondary lane areas output from the processing device 10 gradually improves.

[0196] Furthermore, by outputting the estimation result as the secondary lane area every n loops (n is any integer), the amount of data accumulated between the previous output of the secondary lane area and the next output of the secondary lane area increases. As a result, the difference (degree of accuracy improvement) between the accuracy of the previously output secondary lane area and the accuracy of the next output secondary lane area increases. This processing can prevent the inconvenience of unnecessarily outputting many secondary lane areas, which hardly change the accuracy of the estimation result. As a result, the amount of data transmission and the processing load on the computer can be reduced. Furthermore, effects such as the possibility of using relatively inexpensive equipment (e.g., LAN cables and buses) can be obtained.

[0197] <Effects> The processing device 10 can first estimate lane areas using the first method and quickly output the estimation results (primary lane areas). After that, the processing device 10 can perform highly accurate estimation using the second method based on the video images acquired and stored after the lane areas have been estimated using the first method, and output the estimation results (secondary lane areas and tertiary lane areas).

[0198] According to the processing device 10, various processes can be performed based on the estimation result obtained by the first method until a predetermined estimation result (highly accurate estimation result obtained by the second method) is obtained. As a result, the problem of not being able to perform various processes based on the estimation result until a predetermined estimation result (highly accurate estimation result obtained by the second method) is obtained is solved.

[0199] Furthermore, the processing device 10 can employ, as a first method, a process of estimating a lane area based on one frame image in a moving image. For example, the processing device 10 can employ, as the first method, a process of detecting a white line in one frame image and estimating a lane area based on the detected white line.

[0200] In the case of such a process of estimating a lane area based on a single frame image, the data to be processed (a single frame image) can be acquired more quickly than in a process of estimating a lane area based on a video sequence consisting of multiple frame images. Furthermore, the amount of data to be analyzed is relatively small. Therefore, in the case of a process of estimating a lane area based on a single frame image, the estimation result can be output more quickly.

[0201] According to the processing device 10, which can employ a process for estimating a lane area based on one frame image in a moving image as the first method, the estimation results of the first method can be output quickly.

[0202] However, with the first method, the accuracy of the estimation results may be low if a part of the white line is hidden or unclear in the video. This problem is relatively likely to occur with video images generated by a fixed camera. That is, with a fixed camera located far from the white line, the white line is likely to be hidden by an obstacle. Furthermore, the white line may become unclear in the image due to the influence of light (such as sunlight or vehicle headlights).

[0203] The processing device 10 compensates for the disadvantages of the first method by adopting a process of estimating lane areas based on the vehicle's movement trajectory as the second method. Note that the second method provides relatively high estimation accuracy, but has the problem of taking a long time to obtain estimation results. By adopting the first method, the processing device 10 compensates for the disadvantages of the second method. In this way, by using the first method and the second method in combination, the processing device 10 makes use of the strengths of each method while compensating for the problems of each.

[0204] In the second technique, the processing device 10 estimates lane areas based on the movement trajectory of a vehicle detected by a fixed camera. Vehicles travel along lanes. Therefore, the movement trajectory of a vehicle detected by a fixed camera tends to follow the lane. The processing device 10 estimates lane areas based on the movement trajectory of a vehicle that tends to follow the lane.

[0205] In the case of the second method, it is sufficient to detect the movement trajectory of the vehicle on the road, and there is no need to detect white lines on the road by image analysis, etc. The second method can compensate for the disadvantages of the first method.

[0206] Furthermore, the processing device 10 can automatically or semi-automatically estimate lane areas after a fixed camera is installed based on video images generated by the fixed camera after installation. Furthermore, the processing device 10 can automatically or semi-automatically estimate lane areas after the orientation, position, or measurement conditions (zoom, etc.) of the fixed camera are changed based on video images generated by the fixed camera after the change. This processing device 10 can avoid the tedious task of having workers register lane areas within the capture range of each fixed camera each time a fixed camera is installed or the orientation, position, measurement conditions, etc. are changed. For example, when installing fixed cameras in locations that require temporary monitoring, such as construction sites, the installation and settings of fixed cameras may be frequently changed. In this case, it would be tedious if workers had to register lane areas each time. The processing device 10 can alleviate this inconvenience.

[0207] <<Third Embodiment>> Fig. 21 shows the processing executed by the processing device 10. As shown in Fig. 21, the processing device 10 outputs the primary lane area and the tertiary lane area as the lane area estimation results, but does not output the secondary lane area. The processing device 10 of this embodiment differs from the processing device 10 of the second embodiment (see Fig. 4) in that it does not output the secondary lane area.

[0208] The output unit 14 of this embodiment first outputs information indicating the lane area (primary lane area) estimated by the first method (first lane estimation process). After that, when the estimation result of the second method (second lane estimation process) is confirmed, the output unit 14 outputs information indicating the lane area (tertiary lane area) indicated by the confirmed estimation result.

[0209] Next, an example of the processing flow of the processing device 10 will be described with reference to the flowchart of FIG.

[0210] Separately from the process shown in FIG. 22, the video capturing step (see FIG. 21) is executed, and the capturing of moving images generated by the fixed camera continues.

[0211] First, the processing device 10 executes a first lane estimation step (see FIG. 21). That is, the processing device 10 estimates a lane area based on one frame image in a video image generated by a fixed camera (S30). The lane area indicated by this estimation result becomes the primary lane area in FIG. 21. Next, the processing device 10 executes an external output step (see FIG. 21) and outputs the estimation result (primary lane area) of S30 (S31).

[0212] The processing device 10 may execute at least one of an external auxiliary information collection step (see FIG. 21 ) and a non-target area exclusion step (see FIG. 21 ) before the first lane estimation step (see FIG. 21 ). The output results of these steps may then be used in the first lane estimation step (see FIG. 21 ), a vehicle detection step (see FIG. 21 ) and a second lane estimation step (see FIG. 21 ) described later.

[0213] After S31, the processing device 10 executes a vehicle detection step (see FIG. 21) and a second lane estimation step (see FIG. 21). That is, the processing device 10 estimates the lane area based on the moving images that are further acquired and stored after the lane area is estimated in S30.

[0214] As described above, in the vehicle detection step (see FIG. 21 ) and the second lane estimation step (see FIG. 21 ), the process of calculating the vehicle movement trajectory based on the moving image and estimating the lane area based on the vehicle movement trajectory is repeated. Then, when the same estimation result is obtained a predetermined number of times or more, the lane area indicated by the estimation result is determined as the estimation result.

[0215] In S32, the processing device 10 determines whether the estimation result has been confirmed. The lane area indicated by this confirmed estimation result becomes the tertiary lane area in FIG. 21. The processing device 10 repeats this determination until the estimation result is confirmed (No in S32). During this time, the processing device 10 continues the vehicle detection process (see FIG. 21) and the second lane estimation process (see FIG. 21).

[0216] Then, when it is determined that the estimation result has been confirmed (Yes in S32), the processing device 10 executes an external output process (see FIG. 21) and outputs the confirmed estimation result (tertiary lane area) (S33).

[0217] As mentioned above, when multiple lanes are captured in a video image generated by a fixed camera, the lane areas (estimated results) of each of the multiple lanes may be determined at once, or they may be determined one by one.

[0218] When the lane areas are determined one by one, the processing device 10 may execute the external output step (see FIG. 21 ) each time a lane area is determined, and output the determined estimation result (tertiary lane area) (S33). Then, the processing device 10 may repeat S32 and S33 until the lane areas (estimation results) of all lanes are determined.

[0219] Alternatively, if the lane areas are determined one by one, the processing device 10 may execute an external output process (see FIG. 21) after the lane areas (estimation results) of all lanes have been determined, and output all the determined estimation results (tertiary lane areas) together (S33).

[0220] Other configurations of the processing apparatus 10 of this embodiment are similar to those of the processing apparatus 10 of the second embodiment.

[0221] According to the processing apparatus 10 of this embodiment, the same effects as those of the processing apparatus 10 of the second embodiment are achieved.

[0222] The processing device 10 of this embodiment can first estimate lane areas using the first method and quickly output the estimation result (primary lane areas). After that, the processing device 10 can perform highly accurate estimation using the second method based on the video images acquired and stored after the lane areas have been estimated using the first method, and output the estimation result (tertiary lane areas).

[0223] The processing device 10 of the second embodiment outputs two estimation results (primary lane area and secondary lane area) before outputting a highly accurate estimation result (tertiary lane area). In contrast, the processing device 10 of the present embodiment outputs one estimation result (primary lane area) before outputting a highly accurate estimation result (tertiary lane area).

[0224] Depending on the application of the processing device 10, it may be sufficient to obtain some estimation result until a highly accurate estimation result (tertiary lane area) is obtained. In such a case, it is sufficient to output the primary lane area before the tertiary lane area, and there is no need to output the secondary lane area. The processing device 10 of this embodiment can provide an output that meets such needs.

[0225] <<Fourth Embodiment>> Figure 23 shows the processing executed by the processing device 10. As shown in Figure 23, the processing device 10 does not execute the first lane estimation process. The processing device 10 of this embodiment outputs the secondary lane area and the tertiary lane area as lane area estimation results, but does not output the primary lane area. The processing device 10 of this embodiment differs from the processing device 10 of the second embodiment (see Figure 4) in that it does not execute the first lane estimation process and does not output the primary lane area.

[0226] The "first method" executed by the first estimation unit 12 of this embodiment is a method of outputting the lane area indicated by the pre-confirmation estimation result obtained in the second lane estimation process (see Figure 23) as the estimation result.

[0227] The "second method" executed by the second estimation unit 13 of this embodiment is a method of outputting the lane area indicated by the confirmed estimation result obtained in the second lane estimation process (see Figure 23) as the estimation result.

[0228] The output unit 14 of this embodiment first outputs information indicating the lane area estimated by the first method (the lane area indicated by the estimation result before it is confirmed: the secondary lane area), and then outputs information indicating the lane area estimated by the second method (the lane area indicated by the confirmed estimation result: the tertiary lane area).

[0229] Next, an example of the processing flow of the processing device 10 will be described with reference to the flowchart of FIG.

[0230] Separately from the process shown in FIG. 24, the video capturing step (see FIG. 23) is executed, and the capturing of moving images generated by the fixed camera continues.

[0231] The processing device 10 then executes a vehicle detection step (see FIG. 23 ) and a second lane estimation step (see FIG. 23 ). As described above, in the vehicle detection step (see FIG. 23 ) and the second lane estimation step (see FIG. 23 ), a vehicle movement trajectory is calculated based on the moving image, and a process of estimating a lane area based on the vehicle movement trajectory is repeated. Then, when a similar estimation result is obtained a predetermined number of times or more, the lane area indicated by the estimation result is determined as the estimation result.

[0232] In S40, the processing device 10 determines whether a lane area has been estimated based on the vehicle's movement trajectory. For example, in a process of repeatedly estimating a lane area, if an initial estimation result is obtained in the first loop, the processing device 10 can determine that a lane area has been estimated based on the vehicle's movement trajectory. The lane area indicated by the initial estimation result obtained in this first loop becomes the secondary lane area in FIG. 23. Note that instead of the initial estimation result of the first loop, the estimation result of the second loop or the estimation result of a subsequent loop may be used as the secondary lane area. However, using the initial estimation result as the secondary lane area is preferable because it allows for earlier output of the secondary lane area.

[0233] The processing device 10 repeats this determination until it determines that the lane area has been estimated based on the vehicle movement trajectory (No in S40).

[0234] Then, when it is determined that the lane area has been estimated based on the vehicle's movement trajectory (Yes in S40), the processing device 10 executes an external output process (see Figure 23) and outputs the estimation result of S40 (secondary lane area) (S41).

[0235] After S41, the processing device 10 continues the vehicle detection process (see FIG. 23) and the second lane estimation process (see FIG. 23). Then, in S42, the processing device 10 determines whether the estimation result has been confirmed. As described above, if the processing device 10 obtains the same estimation result a predetermined number of times or more, it confirms the lane area indicated by the estimation result as the estimation result. The lane area indicated by this confirmed estimation result becomes the tertiary lane area in FIG. 23.

[0236] The processing device 10 repeats this determination until the estimation result is confirmed (No in S42). During this time, the processing device 10 continues the vehicle detection process (see FIG. 23) and the second lane estimation process (see FIG. 23).

[0237] Then, when it is determined that the estimation result has been confirmed (Yes in S43), the processing device 10 executes an external output process (see FIG. 23) and outputs the confirmed estimation result (tertiary lane area) (S43).

[0238] As mentioned above, when multiple lanes are captured in a video image generated by a fixed camera, the lane areas (estimated results) of each of the multiple lanes may be determined at once, or they may be determined one by one.

[0239] When the lane areas are determined one by one, the processing device 10 may execute the external output step (see FIG. 23 ) each time a lane area is determined, and output the determined estimation result (tertiary lane area) (S43). Then, the processing device 10 may repeat S42 and S43 until the lane areas (estimation results) of all lanes are determined.

[0240] Alternatively, if the lane areas are determined one by one, the processing device 10 may execute an external output process (see FIG. 23) after the lane areas (estimation results) of all lanes have been determined, and output all the determined estimation results (tertiary lane areas) together (S43).

[0241] The processing device 10 may execute at least one of an external auxiliary information collection step (see FIG. 23 ) and a non-target area exclusion step (see FIG. 23 ) before the vehicle detection step (see FIG. 23 ) and the second lane estimation step (see FIG. 23 ). Then, the output results of these steps may be used in the vehicle detection step (see FIG. 23 ) and the second lane estimation step (see FIG. 23 ).

[0242] Here, a modified example of the process described using the flowchart in Fig. 24 will be described. In the above process, the processing device 10 repeats the process of estimating a lane area in the second lane estimation step (see Fig. 4) until the estimation result is confirmed. Then, the processing device 10 outputs the confirmed estimation result as the tertiary lane area, and outputs the estimation result obtained in any one of the multiple loops until the confirmation is confirmed as the secondary lane area.

[0243] In a modified example, the processing device 10 outputs the confirmed estimation result as the tertiary lane area, and outputs the estimation results obtained in any "two or more" of the multiple loops until the confirmation as the secondary lane area. That is, the processing device 10 outputs two or more estimation results as the secondary lane area. The processing device 10 may output the estimation results obtained in all of the multiple loops as the secondary lane area. Alternatively, the processing device 10 may output the estimation results obtained in some of the multiple loops as the secondary lane area. In the latter case, the processing device 10 may output the estimation result as the secondary lane area every n loops (n is any integer). The processing device 10 can output the estimation result as the secondary lane area each time an estimation result is obtained in each loop.

[0244] Other configurations of the processing apparatus 10 of this embodiment are similar to those of the processing apparatus 10 of the second embodiment.

[0245] According to the processing apparatus 10 of this embodiment, the same effects as those of the processing apparatus 10 of the second embodiment are achieved.

[0246] Furthermore, the processing device 10 of this embodiment can first estimate lane areas using the first method and quickly output the estimation result (secondary lane areas). After that, the processing device 10 can perform highly accurate estimation using the second method based on the video images acquired and stored after the lane areas have been estimated using the first method, and output the estimation result (tertiary lane areas).

[0247] The processing device 10 of the second embodiment outputs two estimation results (primary lane area and secondary lane area) before outputting a highly accurate estimation result (tertiary lane area). In contrast, the processing device 10 of the present embodiment outputs one estimation result (secondary lane area) before outputting a highly accurate estimation result (tertiary lane area).

[0248] Depending on the application of the processing device 10, it may be sufficient to obtain some estimation result until a highly accurate estimation result (tertiary lane area) is obtained. In such a case, it is sufficient to output the secondary lane area before the tertiary lane area, and there is no need to output the primary lane area. The processing device 10 of this embodiment can provide an output that meets such needs.

[0249] <<Fifth Embodiment>> As described in the second embodiment, the fixed camera can change at least one of the orientation, position, and shooting conditions (zoom, etc.) automatically or in response to the operator's operation. In response to such changes, the area captured in the video generated by the fixed camera changes. When the area captured in the video generated by the fixed camera changes, the lane area in the video also changes, making it necessary to re-estimate the lane area. The processing device 10 of this embodiment detects a change in the area captured in the video generated by the fixed camera and re-estimates the lane area in response to the detection. This will be described in detail below.

[0250] 25 shows an example of a functional block diagram of the processing device 10. As shown in the figure, the processing device 10 includes an acquisition unit 11, a first estimation unit 12, a second estimation unit 13, an output unit 14, and a change detection unit 16.

[0251] The change detection unit 16 detects a change in the area captured in the moving image generated by the fixed camera.

[0252] The change detection unit 16 can analyze video images generated by a fixed camera and detect changes in the area captured in the video images generated by the fixed camera. Detection of changes in the area captured in the video images through image analysis can be achieved using any technology. For example, deep learning technology for determining the identity of two images, template matching, ORB (Oriented Fast and Rotated BRIEF) feature matching, etc. can be used. An example will be described below, but the present invention is not limited to this.

[0253] For example, when a fixed camera continues to capture images of the same area, an object that continues to exist in that area will continue to be captured. An object that continues to exist in that area is an object whose position does not change, such as a building or a tree. Therefore, the change detection unit 16 may analyze the images and detect an object that is detected in the same position across multiple frame images as an object that continues to exist in that area. Then, when the change detection unit 16 detects that the position of the detected object within the frame image has changed or that the detected object is no longer detected in the frame image, it may determine that the area captured in the moving image generated by the fixed camera has changed.

[0254] Additionally, the change detection unit 16 may detect a change in the area captured in the video generated by the fixed camera based on additional information transmitted from the fixed camera.

[0255] In this example, when the fixed camera changes at least one of the orientation, position, and shooting conditions (zoom, etc.) automatically under its own control or in response to the operation of a worker, it can transmit information indicating this to the processing device 10. Then, when the change detection unit 16 receives this information from the fixed camera, it can determine that the area captured in the video generated by the fixed camera has changed. The fixed camera can detect changes in response to the operation of the worker based on change instructions input by the worker and sensing data from various sensors (attitude sensors) equipped in the fixed camera.

[0256] Each time a change in an area captured in a video generated by a fixed camera is detected, the processing device 10 redoes the steps shown in Figures 4, 21, and 23. In the non-target area exclusion step, first lane estimation step, vehicle detection step, and second lane estimation step, the processing device 10 does not use the video before the change, but instead uses the video after the change to perform various processes.

[0257] Each time a change in the area captured in the video generated by the fixed camera is detected, the first estimation unit 12 estimates the lane area using the first method based on the video after the change.

[0258] Furthermore, when a change in the area captured in the video generated by the fixed camera is detected, the second estimation unit 13 estimates the lane area using the second method based on the video after the change. Note that the second estimation unit 13 does not use the video before the change.

[0259] Then, each time a change in the area captured in the video generated by the fixed camera is detected, the output unit 14 outputs information indicating the lane area estimated by the first technique based on the video after the change. Thereafter, the output unit 14 outputs information indicating the lane area estimated by the second technique based on the video after the change.

[0260] Other configurations of the processing apparatus 10 of this embodiment are similar to those of the processing apparatus 10 of the first to fourth embodiments.

[0261] According to the processing apparatus 10 of this embodiment, the same effects as those of the processing apparatus 10 of the first to fourth embodiments are achieved.

[0262] Furthermore, the processing device 10 of this embodiment can automatically detect changes in the area captured in the video images generated by the fixed camera and redo the estimation of the lane area in response to the detection. The processing device 10 of this embodiment is advantageous in that it can automate or semi-automate the redo of the lane area in response to changes in the orientation, position, shooting conditions (zoom, etc.) of the fixed camera, etc.

[0263] <<Sixth Embodiment>> As described in the first to fifth embodiments, the processing device 10 can output multiple types of estimation results (at least two of the primary to tertiary lane areas). The processing device 10 of this embodiment sets which of the multiple types of estimation results to output based on user input. Then, the processing device 10 outputs the estimation result that has been set to be output. This will be described in detail below.

[0264] 26 shows an example of a functional block diagram of the processing device 10. As shown in the figure, the processing device 10 includes an acquisition unit 11, a first estimation unit 12, a second estimation unit 13, an output unit 14, and a setting unit 15. The processing device 10 may further include a change detection unit 16.

[0265] The setting unit 15 sets, based on a user input, which of the multiple types of lane areas estimated by the first method and the second method is to be output.

[0266] When the configuration of the second embodiment is adopted, the "plural types of lane areas" that can be set to be output or not are the primary to tertiary lane areas.

[0267] When the configuration of the third embodiment is adopted, the "plural types of lane areas" that can be set to be output or not are the primary and tertiary lane areas.

[0268] When the configuration of the fourth embodiment is adopted, the "plural types of lane areas" that can be set to be output or not are secondary and tertiary lane areas.

[0269] The output unit 14 outputs information indicating a lane area of ​​a type that is set to be output. The output unit 14 does not output information indicating a lane area of ​​a type that is set not to be output.

[0270] Other configurations of the processing apparatus 10 of this embodiment are similar to those of the processing apparatus 10 of the first to fifth embodiments.

[0271] According to the processing apparatus 10 of this embodiment, the same effects as those of the processing apparatuses 10 of the first to fifth embodiments are achieved.

[0272] Furthermore, the processing device 10 of the present embodiment sets which of a plurality of types of estimation results to output based on user input, and outputs the estimation result that has been set to be output. Such a processing device 10 is advantageous in that the user can customize the type of estimation result to be output.

[0273] Seventh Embodiment The processing device 10 of this embodiment differs from the processing device 10 of the second embodiment in the vehicle detection process (FIGS. 4, 21, and 23) and the second lane estimation process (FIGS. 4, 21, and 23). In the second embodiment, the process of estimating a lane area based on a vehicle movement trajectory was described. In this embodiment, the process of estimating a lane area based on a vehicle position will be described.

[0274] Vehicles travel along lanes. Therefore, the positions of vehicles detected by fixed cameras tend to be scattered along the lanes. The processing device 10 estimates lane areas based on the vehicle positions that tend to be scattered along the lanes. This will be described in detail below.

[0275] "Vehicle detection process (FIGS. 4, 21, 23)" The second estimation unit 13 executes the vehicle detection process. In the vehicle detection process, the second estimation unit 13 detects a vehicle from a moving image (frame image). Then, the second estimation unit 13 generates position data indicating the position of the detected vehicle. The second estimation unit 13 can output the generated position data. The position data output in the vehicle detection process is used in other processes shown in FIGS. 4, 21, and 23.

[0276] The "position data" indicates the position of the vehicle within the range of the fixed camera. More specifically, the position data indicates the position of the reference point of the vehicle. In this embodiment, the reference point of the vehicle is the point where the approximate center of the left-right direction of the vehicle is lowered to the ground in the direction of travel of the vehicle. For example, the center of the base of a rectangular area in which the vehicle detected in the video is captured may be used as the reference point of the vehicle. As explained in the second embodiment, other locations on the vehicle may also be used as the reference point.

[0277] The position data is data that indicates the position of the vehicle using coordinates in a two-dimensional coordinate system set in moving images (frame images) generated by a fixed camera, for example. Note that it is not necessary to link the position data of the same vehicle that exists across multiple frame images; it is sufficient to simply detect the position of the reference point of the vehicle for each frame image and accumulate the results as position data.

[0278] "Second lane estimation step (FIGS. 4, 21, 23)" The second estimation unit 13 executes the second lane estimation step. In this step, the second estimation unit 13 estimates a lane area based on the position data generated in the vehicle detection step (FIGS. 4, 21, 23).

[0279] The second estimation unit 13 can execute the processes of S1, S4, and S5 in Fig. 27 as the second lane estimation step (Figs. 4, 21, and 23). The outline of each process is as follows.

[0280] Data acquisition process S1: The second estimation unit 13 acquires position data. Lane area estimation process S4: The second estimation unit 13 estimates a lane area based on the position data acquired in S1 up to that point. Lane area confirmation process S5: The second estimation unit 13 determines whether the lane area is confirmed by the estimation result of S4. If not confirmed, the process returns to S1 and the same process is repeated. By performing the process of S5, the estimation result of S4 is not confirmed as the lane area as it is, but rather the estimation result can be confirmed as the lane area once a reliable estimation result is obtained in S4. As a result, it is possible to output a highly reliable result.

[0281] Each process will be described in detail below.

[0282] "Data Acquisition Process S1 (FIG. 27)" The second estimation unit 13 acquires the position data generated in the vehicle detection process (FIGS. 4, 21, 23).

[0283] "Lane Area Estimation Process S4 (FIG. 27)" The second estimation unit 13 estimates the lane area based on the vehicle position data. The second estimation unit 13 estimates the lane area based on a plurality of y coordinates (y ０ , y １ , y ２The method of generating the probability density function for each of the points (...) differs from that of the second embodiment. The processing after generating the probability density function is the same as that of the second embodiment. The method of generating the probability density function of this embodiment will be described below.

[0284] The second estimation unit 13 calculates a three-dimensional probability density function based on the vehicle position data as shown in Fig. 28(A) . For example, the second estimation unit 13 can calculate the three-dimensional probability density function as shown in Fig. 28(A) by kernel density estimation.

[0285] This three-dimensional probability density function is used to calculate multiple y coordinates (y ０ , y １ , y ２ The second estimation unit 13 estimates the probability of occurrence of vehicle position data at each position (x coordinate) in the width direction of the lane for each of the y coordinates (y ０ , y １ , y ２ . . ), the peak positions and the number of peaks of such a three-dimensional probability density function are calculated (FIG. 28B).

[0286] Other configurations of the processing apparatus 10 of this embodiment are similar to those of the processing apparatus 10 of the first to sixth embodiments.

[0287] According to the processing apparatus 10 of this embodiment, the same effects as those of the processing apparatuses 10 of the first to sixth embodiments are achieved.

[0288] Furthermore, the processing device 10 of this embodiment can estimate lane areas using vehicle position data, without using vehicle trajectory data. In this way, the processing device 10 of this embodiment can estimate lane areas based on video images generated by a fixed camera using a new method that is partially different from that of the second embodiment.

[0289] <<Modifications>> <Modification 1>> The second estimation unit 13 estimates the “lane reference line” by using the methods of the first to seventh embodiments (multiple y coordinates at predetermined intervals (y ０ , y １ , y ２The calculation may be performed using a method different from the method using a probability density function for each lane (...). For example, the second estimation unit 13 may calculate the lane reference line as a line indicated by data obtained by averaging the trajectory data. If it is expected that the trajectory data to be processed includes trajectory data for multiple lanes, the second estimation unit 13 may group the trajectory data based on positions (x-coordinate positions) close to each other and calculate the lane reference line by averaging the data for each group. The grouping is achieved using a clustering technique or the like.

[0290] <Modification 2> The second estimation unit 13 may identify the traveling direction of the vehicle for each estimated lane area. For example, the second estimation unit 13 identifies the direction in which the vehicle detected in each estimated lane area moves over time based on video images generated by a fixed camera. Then, the second estimation unit 13 sets the identified direction as the traveling direction of the vehicle in each lane area.

[0291] <Variation 3> In the second lane estimation step, the processing device 10 may perform a process of removing motorcycle data (trajectory data / position data) from the acquired vehicle data (trajectory data / position data). The processing device 10 may then estimate the lane area based on the vehicle data (trajectory data / position data) from which the motorcycle data (trajectory data / position data) has been removed. The motorcycle may be a motorcycle, electric kick scooter, bicycle, etc.

[0292] As described above, in the second lane estimation process, the processing device 10 estimates the lane area based on the vehicle's travel trajectory and position while traveling. However, the travel trajectory and position while traveling of a motorcycle tend to differ from the travel trajectory and position while traveling of a four-wheeled vehicle or a large vehicle. Specifically, four-wheeled vehicles and large vehicles, which are wider than motorcycles, tend to travel approximately in the center of the lane. However, because motorcycles have a wider range of travelable positions, they tend to travel at the edge of the lane or approximately in the center of the lane. Removing such motorcycle trajectory data and position data improves the accuracy of lane area estimation. Whether acquired vehicle data (trajectory data / position data) is motorcycle data can be determined by analyzing images generated by a fixed camera. This determination process may be performed by the processing device 10 or another device.

[0293] <Modification 4> In the second lane estimation step, the processing device 10 may extract data (trajectory data / position data) of a predetermined vehicle from the acquired vehicle data (trajectory data / position data). Then, the processing device 10 may estimate a lane area based on the extracted vehicle data (trajectory data / position data).

[0294] The predetermined vehicle may be a vehicle for which a dedicated lane or a priority lane exists. For example, the predetermined vehicle may be a bus, a large vehicle, a taxi, a bicycle, etc. Data (trajectory data / position data) of such a predetermined vehicle is suitable for estimating the dedicated lane or priority lane for each vehicle. By extracting data (trajectory data / position data) of such a predetermined vehicle and estimating lane areas based on the extracted data (trajectory data / position data) of the predetermined vehicle, the dedicated lane or priority lane for each vehicle can be estimated with high accuracy.

[0295] Whether the acquired vehicle data (trajectory data / position data) is data of a specific vehicle can be determined by analyzing images generated by a fixed camera. This determination process may be performed by the processing device 10 or another device.

[0296] <Modification 5> The processing device 10 may output only one of the lane area estimated by the first method and the lane area estimated by the second method.

[0297] In addition, the processing device 10 may select whether to output the lane area estimated using the first method or the lane area estimated using the second method in response to receiving a request to complete the estimation of the lane area early.

[0298] For example, the request may further indicate a request for the estimation result. The request may be "prioritize speed" or "prioritize accuracy," etc. The processing device 10 may present a plurality of selectable request contents to the user and accept an input to select from among them. For example, when the request content is "prioritize speed," the processing device 10 may select to output the lane area estimated using the first method. When the request content is "prioritize accuracy," the processing device 10 may select to output the lane area estimated using the second method.

[0299] Additionally, the request may further indicate a desired time until an estimation result is obtained. If the desired time is less than a threshold, the processing device 10 may select to output the lane area estimated by the first method. If the desired time is equal to or greater than the threshold, the processing device 10 may select to output the lane area estimated by the second method. For example, the threshold is set to an estimate of the time required for the tertiary lane area to be output in the second lane estimation process.

[0300] <<Usage Scenarios>> The processing device 10 described in the first to seventh embodiments and the modified examples can be used in various situations. An example will be described below. Note that the example here is merely an example, and the usage scenario of the processing device 10 is not limited to the example here.

[0301] In one example, the processing device 10 can notify a vehicle of the lane area estimation result. The processing device 10 can notify a vehicle traveling on a road of the lane area estimation result via any communication means, such as road-to-vehicle communication. The vehicle can use the lane area estimation result received from the processing device 10 to control autonomous driving.

[0302] In another example, the processing device 10 can estimate each parking space (lane area) based on vehicle trajectory data or position data generated based on images captured by a fixed camera installed in the parking lot. The processing device 10 can then notify vehicles located in the parking lot of the estimated results of each parking space (lane area) via any communication means, such as road-to-vehicle communication. The vehicles can use the estimated results of each parking space (lane area) received from the processing device 10 to control automatic parking. Alternatively, the vehicle may notify the driver of the estimated results of each parking space (lane area) via an output device such as a display. In parking lots, the white lines separating parking spaces may fade and become difficult to see due to aging or other reasons. By using the processing device 10 of this embodiment, the inconvenience in such cases can be alleviated.

[0303] In another example, when the processing device 10 detects an accident or the like within the range of a fixed camera, the processing device 10 can notify a predetermined notification destination, including information about the lane in which the accident occurred. Also, in a modified example in which the direction of the traveling lane is additionally detected, the processing device 10 can detect a wrong-way driving vehicle and a lane in which there is a risk of the vehicle being driven wrong-way, and notify the detected vehicle and its location information.

[0304] Although the present disclosure has been described above with reference to the embodiments, the present disclosure is not limited to the above-described embodiments. Various modifications that can be understood by those skilled in the art can be made to the configuration and details of the present disclosure within the scope of the present disclosure. Furthermore, each embodiment can be combined with other embodiments as appropriate.

[0305] In addition, in the flowcharts used in the above description, multiple steps (processes) are described in order. However, the order of the steps performed in each embodiment is not limited to the order described. In each embodiment, the order of the steps shown in the drawings can be changed as long as it does not cause any problems in terms of the content.

[0306] Some or all of the above embodiments can be described as, but are not limited to, the following supplementary notes: 1. A processing device comprising: an acquisition means for acquiring video images generated by a fixed camera; a first estimation means for estimating lane areas using a first method based on the video images; a second estimation means for estimating lane areas using a second method different from the first method based on the video images that are further accumulated after the lane areas have been estimated using the first method; and an output means for outputting information indicating the lane areas estimated using the first method, and thereafter outputting information indicating the lane areas estimated using the second method. 2. The processing device according to 1, wherein the first estimation means estimates lane areas based on one frame image in the video images, and the second estimation means estimates lane areas based on multiple frame images in the video images. 3. 3. The processing device according to claim 1 or 2, wherein the first estimation means detects white lines from the moving image and estimates a lane area based on the detected white lines, and the second estimation means detects a vehicle movement trajectory or a vehicle position based on the moving image and estimates a lane area based on the detected vehicle movement trajectory or vehicle position. 3. The processing device described in 3, wherein the second estimation means continues to detect a vehicle movement trajectory or a vehicle position, repeatedly estimates a lane area based on the accumulated vehicle movement trajectory or vehicle position, and determines that the lane area estimated in the (n-1)th processing and the lane area estimated in the nth processing are identical if a difference between the lane area estimated in the (n-1)th processing and the lane area estimated in the nth processing is equal to or smaller than a reference value, and when they are determined to be identical a predetermined number of times or more in succession, confirms the lane area estimated during that time as the estimation result, and the output means outputs information indicating the lane area estimated by the first method, and then outputs information indicating the lane area indicated by the confirmed estimation result in response to the estimation result being confirmed. 5. The processing device described in 4, wherein the output means outputs information indicating the lane area estimated by the first method, and then outputs information indicating the lane area indicated by the estimation result before it was confirmed, and then outputs information indicating the lane area indicated by the confirmed estimation result.6. The processing device according to any one of 1 to 5, wherein the second estimation means detects a vehicle movement trajectory or a vehicle position based on the moving image, and repeatedly executes a process of estimating a lane area based on the detected vehicle movement trajectory or vehicle position, and determines that the lane area estimated in the (n-1)th process and the lane area estimated in the nth process are the same if a difference between the lane area estimated in the (n-1)th process and the lane area estimated in the nth process is equal to or less than a reference value, and when they are determined to be the same a predetermined number of times or more in succession, confirms the lane area estimated during that time as an estimation result, and regards the lane area indicated by the confirmed estimation result as the lane area estimated by the second method, and the first estimation means regards the lane area indicated in the estimation result before being confirmed as the lane area estimated by the first method. 7. The processing device according to any one of 1 to 6, further comprising change detection means for detecting a change in an area captured in a video generated by the fixed camera, wherein the first estimation means, each time the change is detected, estimates a lane area using the first method based on the video after the change, and the second estimation means, when the change is detected, estimates a lane area using the second method based on the video after the change, and the output means, each time the change is detected, outputs information indicating the lane area estimated using the first method based on the video after the change, and thereafter outputs information indicating the lane area estimated using the second method based on the video after the change. 8. The processing device according to any one of 1 to 7, further comprising setting means for setting, based on a user input, which of multiple types of lane areas estimated by the first method and the second method to output, and the output means outputs information indicating the type of lane area that is set to be output. A processing method in which one or more computers acquire video images generated by a fixed camera, estimate lane areas using a first method based on the video images, estimate lane areas using a second method different from the first method based on the video images that are further accumulated after the lane areas have been estimated using the first method, output information indicating the lane areas estimated using the first method, and then output information indicating the lane areas estimated using the second method.10. A program that causes a computer to function as: an acquisition means that acquires moving images generated by a fixed camera; a first estimation means that estimates lane areas using a first method based on the moving images; a second estimation means that estimates lane areas using a second method different from the first method based on the moving images that are further accumulated after the lane areas have been estimated using the first method; and an output means that outputs information indicating the lane areas estimated using the first method, and then outputs information indicating the lane areas estimated using the second method.

[0307] Some or all of Supplements 2 to 8 that are dependent on the processing device of Supplement 1 described above may also be dependent on the processing method of Supplement 9 and the program of Supplement 10 in the same dependent relationship as Supplement 1 and Supplements 2 to 8. Furthermore, within the scope of each of the above-described embodiments, some or all of the configurations described as Supplements can be realized in various hardware, software, various recording means for recording software, or systems.

[0308] This application claims priority based on Japanese Patent Application No. 2024-010839, filed January 29, 2024, the disclosure of which is incorporated herein by reference in its entirety.

[0309] REFERENCE SIGNS LIST 10 Processing device 11 Acquisition unit 12 First estimation unit 13 Second estimation unit 14 Output unit 15 Setting unit 16 Change detection unit 1A Processor 2A Memory 3A Input / output I / F 4A Peripheral circuit 5A Bus

Claims

1. A processing device having: an acquisition means for acquiring video images generated by a fixed camera; a first estimation means for estimating lane areas using a first method based on the video images; a second estimation means for estimating lane areas using a second method different from the first method based on the video images that are further accumulated after the lane areas have been estimated using the first method; and an output means for outputting information indicating the lane areas estimated using the first method, and then outputting information indicating the lane areas estimated using the second method.

2. The processing device according to claim 1, wherein the first estimation means estimates a lane area based on one frame image in the video, and the second estimation means estimates a lane area based on multiple frame images in the video.

3. A processing device as described in claim 1 or 2, wherein the first estimation means detects white lines from the moving image and estimates a lane area based on the detected white lines, and the second estimation means detects a vehicle movement trajectory or a vehicle position based on the moving image and estimates a lane area based on the detected vehicle movement trajectory or a vehicle position.

4. The processing device described in claim 3, wherein the second estimation means continues to detect the vehicle's movement trajectory or vehicle position, repeatedly estimates the lane area based on the accumulated vehicle movement trajectory or vehicle position, and if the difference between the lane area estimated in the (n-1)th processing and the lane area estimated in the nth processing is below a reference value, determines that the lane area estimated in the (n-1)th processing and the lane area estimated in the nth processing are the same, and if they are determined to be the same for a predetermined number of consecutive times or more, confirms the lane area estimated during that time as the estimation result, and the output means outputs information indicating the lane area estimated by the first method, and then, in response to the estimation result being confirmed, outputs information indicating the lane area indicated by the confirmed estimation result.

5. The processing device described in claim 4, wherein the output means outputs information indicating the lane area estimated by the first method, then outputs information indicating the lane area indicated by the estimation result before it is confirmed, and then outputs information indicating the lane area indicated by the confirmed estimation result.

6. The second estimation means detects a vehicle movement trajectory or vehicle position based on the moving image, and repeatedly executes a process of estimating a lane area based on the detected vehicle movement trajectory or vehicle position, and if the difference between the lane area estimated in the (n-1)th processing and the lane area estimated in the nth processing is below a reference value, determines that the lane area estimated in the (n-1)th processing and the lane area estimated in the nth processing are the same, and if they are determined to be the same for a predetermined number of consecutive times or more, confirms the lane area estimated during that time as the estimation result, and regards the lane area indicated by the confirmed estimation result as the lane area estimated by the second method, and the first estimation means regards the lane area indicated in the estimation result before being confirmed as the lane area estimated by the first method.

7. A processing device as described in any one of claims 1 to 6, further comprising a change detection means for detecting a change in the area captured in the video generated by the fixed camera, wherein the first estimation means, each time a change is detected, estimates a lane area using the first method based on the video after the change, and the second estimation means, when a change is detected, estimates a lane area using the second method based on the video after the change, and the output means, each time a change is detected, outputs information indicating the lane area estimated using the first method based on the video after the change, and thereafter outputs information indicating the lane area estimated using the second method based on the video after the change.

8. A processing device as described in any one of claims 1 to 7, further comprising a setting means for setting, based on user input, which of the multiple types of lane areas estimated by the first method and the second method to output, wherein the output means outputs information indicating the type of lane area that is set to be output.

9. A processing method in which one or more computers acquire video images generated by a fixed camera, estimate lane areas using a first method based on the video images, estimate lane areas using a second method different from the first method based on the video images that are further accumulated after the lane areas have been estimated using the first method, output information indicating the lane areas estimated using the first method, and then output information indicating the lane areas estimated using the second method.

10. A processing method as described in claim 9, wherein in the process of estimating lane areas using the first method, lane areas are estimated based on one frame image in the video, and in the process of estimating lane areas using the second method, lane areas are estimated based on multiple frame images in the video.

11. A processing method as described in claim 9 or 10, wherein in the process of estimating a lane area using the first method, white lines are detected from the moving image and the lane area is estimated based on the detected white lines, and in the process of estimating a lane area using the second method, a vehicle movement trajectory or vehicle position is detected based on the moving image and the lane area is estimated based on the detected vehicle movement trajectory or vehicle position.

12. A processing method according to claim 11, wherein the processing for estimating a lane area using the second method comprises: continuing to detect a vehicle movement trajectory or vehicle position; repeatedly estimating a lane area based on the accumulated vehicle movement trajectory or vehicle position; if the difference between the lane area estimated in the (n-1)th processing and the lane area estimated in the nth processing is equal to or less than a reference value, determining that the lane area estimated in the (n-1)th processing and the lane area estimated in the nth processing are the same; if they are determined to be the same for a predetermined number of consecutive times or more, confirming the lane area estimated during that time as the estimation result; and the output processing comprises outputting information indicating the lane area estimated using the first method, and then, in response to the estimation result being confirmed, outputting information indicating the lane area indicated by the confirmed estimation result.

13. A processing method as described in claim 12, wherein the output process outputs information indicating the lane area estimated by the first method, then outputs information indicating the lane area indicated by the estimation result before it is confirmed, and then outputs information indicating the lane area indicated by the confirmed estimation result.

14. A processing method according to any one of claims 9 to 13, wherein the process of estimating a lane area using the second method detects a vehicle movement trajectory or vehicle position based on the moving image, and repeatedly executes a process of estimating a lane area based on the detected vehicle movement trajectory or vehicle position, and if the difference between the lane area estimated in the (n-1)th process and the lane area estimated in the nth process is equal to or less than a reference value, determines that the lane area estimated in the (n-1)th process and the lane area estimated in the nth process are the same, and if they are determined to be the same for a predetermined number of consecutive times or more, confirms the lane area estimated during that time as the estimation result, and the lane area indicated by the confirmed estimation result is taken as the lane area estimated by the second method, and in the process of estimating a lane area using the first method, the lane area indicated in the estimation result before being confirmed is taken as the lane area estimated by the first method.

15. A recording medium having recorded thereon a program that causes a computer to function as: an acquisition means that acquires video images generated by a fixed camera; a first estimation means that estimates lane areas using a first method based on the video images; a second estimation means that estimates lane areas using a second method different from the first method based on the video images that are further accumulated after the lane areas have been estimated using the first method; and an output means that outputs information indicating the lane areas estimated using the first method, and then outputs information indicating the lane areas estimated using the second method.

16. A recording medium as described in claim 15, wherein the first estimation means estimates a lane area based on one frame image in the video, and the second estimation means estimates a lane area based on multiple frame images in the video.

17. A recording medium as described in claim 15 or 16, wherein the first estimation means detects white lines from the moving image and estimates a lane area based on the detected white lines, and the second estimation means detects a vehicle movement trajectory or a vehicle position based on the moving image and estimates a lane area based on the detected vehicle movement trajectory or a vehicle position.

18. The recording medium described in claim 17, wherein the second estimation means continues to detect the vehicle's movement trajectory or vehicle position, repeatedly estimates lane areas based on the accumulated vehicle movement trajectory or vehicle position, and if the difference between the lane area estimated in the (n-1)th processing and the lane area estimated in the nth processing is below a reference value, determines that the lane area estimated in the (n-1)th processing and the lane area estimated in the nth processing are the same, and if they are determined to be the same for a predetermined number of consecutive times or more, confirms the lane area estimated during that time as the estimation result, and the output means outputs information indicating the lane area estimated by the first method, and then, in response to the estimation result being confirmed, outputs information indicating the lane area indicated by the confirmed estimation result.

19. A recording medium as described in claim 18, wherein the output means outputs information indicating the lane area estimated by the first method, then outputs information indicating the lane area indicated by the estimation result before it is confirmed, and then outputs information indicating the lane area indicated by the confirmed estimation result.

20. The second estimation means detects a vehicle movement trajectory or vehicle position based on the moving image, and repeatedly executes a process of estimating a lane area based on the detected vehicle movement trajectory or vehicle position, and if the difference between the lane area estimated in the (n-1)th processing and the lane area estimated in the nth processing is below a reference value, determines that the lane area estimated in the (n-1)th processing and the lane area estimated in the nth processing are the same, and if they are determined to be the same for a predetermined number of consecutive times or more, confirms the lane area estimated during that time as the estimation result, and regards the lane area indicated by the confirmed estimation result as the lane area estimated by the second method, and the first estimation means regards the lane area indicated in the estimation result before it is confirmed as the lane area estimated by the first method.

Images