Computer program, image processing device, image processing system, and image processing method

Abstract

To provide a computer program, image processing device, image processing system, and image processing method.SOLUTION: A computer program makes a computer perform processing for: acquiring captured images from an image capturing device configured to chronologically capture images of a vehicle tracking area; using a learning model for object detection to perform object detection on each of the captured images captured at multiple points in time by the image capturing device; extracting a detection area including a vehicle to be detected from a captured image of each point in time; deriving a travel trajectory of the vehicle from the extracted detection areas; and determining entry or exit to and from a given area on the basis of the derived travel trajectory.SELECTED DRAWING: Figure 10

Description

[Technical Field]

[0001] The present invention relates to a computer program, an image processing device, an image processing system, and an image processing method. [Background technology]

[0002] In recent years, traffic counters that count the number of vehicles traveling on a road have come into use. Traffic counters that use, for example, magnetic sensors, surveillance cameras, ultrasonic sensors, etc. For example, Patent Document 1 discloses a method for detecting traveling vehicles from images obtained by a fixed camera using a learning model obtained by deep learning. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2019-154027 Summary of the Invention [Problem to be solved by the invention]

[0004] Patent Document 1 discloses a method for detecting a moving vehicle, but does not disclose a configuration for determining whether the vehicle is entering or exiting a predetermined area.

[0005] An object of the present invention is to provide a computer program, an image processing device, an image processing system, and an image processing method that can determine entry and exit to a predetermined area. [Means for solving the problem]

[0006] A computer program according to one aspect of the present invention is a computer program for causing a computer to execute a process of acquiring captured images from an imaging device that captures images of a vehicle tracking area in time series, using a learning model for object detection to perform object detection on each of the captured images taken by the imaging device at each point in time, extracting a detection area containing the vehicle to be detected from the captured images at each point in time, deriving the movement trajectory of the vehicle from the extracted multiple detection areas, and determining entry and exit to a specified area based on the derived movement trajectory.

[0007] An image processing device according to one aspect of the present invention includes an acquisition unit that acquires captured images from an imaging device that captures images of a vehicle tracking area in time series; an extraction unit that uses a learning model for object detection to perform object detection on each of the captured images taken by the imaging device at each point in time and extracts a detection area containing the vehicle to be detected from the captured image at each point in time; a derivation unit that derives the movement trajectory of the vehicle from the extracted multiple detection areas; and a determination unit that determines entry and exit to a specified area based on the derived movement trajectory.

[0008] An image processing system according to one aspect of the present invention includes an image processing device that includes an imaging device that captures images of a vehicle tracking area in chronological order, an acquisition unit that acquires the captured images from the imaging device, an extraction unit that uses a learning model for object detection to perform object detection on each of the captured images acquired by the acquisition unit at each point in time and extracts a detection area containing the vehicle to be detected from the captured image at each point in time, a derivation unit that derives the movement trajectory of the vehicle from the extracted multiple detection areas, and a determination unit that determines entry and exit to a specified area based on the derived movement trajectory.

[0009] An image processing method according to one aspect of the present invention acquires captured images from an imaging device that captures images of a vehicle tracking area in time series, uses a learning model for object detection to perform object detection on each of the captured images taken by the imaging device at each point in time, extracts a detection area containing the vehicle to be detected from the captured images at each point in time, derives the movement trajectory of the vehicle from the extracted multiple detection areas, and executes a process by a computer to determine entry and exit to a specified area based on the derived movement trajectory. [Effects of the Invention]

[0010] According to the present invention, it is possible to determine whether a person has entered or left a predetermined area. [Brief explanation of the drawings]

[0011] [Figure 1] 1 is a schematic diagram showing the overall configuration of a vehicle tracking system according to a first embodiment. [Figure 2] FIG. 2 is a block diagram illustrating the internal configuration of the image processing device. [Figure 3] 1 is a schematic diagram illustrating an example of a captured image acquired by an image processing device from an imaging device. [Figure 4] FIG. 1 is a schematic diagram illustrating an example of the configuration of a learning model. [Figure 5] FIG. 10 is a schematic diagram illustrating an example of object detection using a learning model. [Figure 6] FIG. 2 is an explanatory diagram illustrating a method for tracking a vehicle position. [Figure 7] FIG. 10 is a schematic diagram showing an example of derivation of a vehicle movement trajectory. [Figure 8] FIG. 10 is an explanatory diagram illustrating an example of setting a reference line. [Figure 9] FIG. 10 is an explanatory diagram illustrating a method for determining entry and exit into a parking area. [Figure 10] 10 is a flowchart illustrating a procedure for an entrance determination process executed by the image processing device. [Figure 11] 10 is a flowchart illustrating a procedure for an exit determination process executed by the image processing device. [Figure 12] FIG. 10 is a schematic diagram showing an example of displaying the number of parked vehicles. [Figure 13] 10A and 10B are schematic diagrams showing an example of detection when the detection position of the vehicle vibrates; [Figure 14] 10 is a flowchart illustrating a processing procedure for determining admission in the second embodiment. [Figure 15] 10 is a flowchart illustrating a procedure for determining whether to leave the venue in the second embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0012] The present invention will now be described in detail with reference to the drawings showing embodiments thereof. (Embodiment 1) 1 is a schematic diagram showing the overall configuration of a vehicle tracking system according to embodiment 1. In this embodiment, a vehicle tracking system 1 for tracking vehicles such as refuse collection vehicles at a waste treatment facility will be described. In the following description, when there is no need to distinguish between refuse collection vehicles and general vehicles other than refuse collection vehicles, they will also be simply referred to as vehicles.

[0013] A waste treatment facility generally comprises a platform with a parking area for vehicles, a waste pit for temporarily storing waste discharged from vehicles parked in the parking area, a transport crane for transporting the waste in the waste pit to the incinerator, and an incinerator for incinerating the waste transported by the transport crane. The vehicle tracking method according to this embodiment is applicable to such waste treatment facilities equipped with incinerators. Furthermore, the vehicle tracking method according to this embodiment is not limited to waste treatment facilities equipped with incinerators, but can also be applied to transfer facilities and sorting facilities equipped with platforms.

[0014] FIG. 1 shows a waste treatment facility's platform PF and a waste pit GP adjacent to the platform PF. The platform PF has a parking area PA with two parking spaces PS1 and PS2 for refuse collection trucks and one parking space PS3 for general vehicles. The platform PF also has a slow-motion area SA adjacent to the parking area PA. Refuse collection trucks and general vehicles that enter the waste treatment facility proceed slowly through the slow-motion area SA to the parking area PA, where they park in parking spaces PS1 to PS3 provided for each vehicle and discharge the collected waste into the waste pit GP. The number of parking spaces provided on the platform PF is not limited to that shown in FIG. 1 and can be designed appropriately depending on the waste treatment facility.

[0015] The vehicle tracking system 1 includes an imaging device 120 that captures images of the platform PF from a bird's-eye view, and an image processing device 100 that tracks the position of the vehicle by analyzing the captured images obtained in time series by the imaging device 120. That is, in this embodiment, the platform PF is set as a vehicle tracking area, and the position of each vehicle within the platform PF is tracked by the image processing device 100.

[0016] Imaging device 120 is installed in the waste treatment facility so as to capture images of platform PF from diagonally above. The imaging range of imaging device 120 does not need to include the entire platform PF, but only needs to include a certain area including the boundary between parking area PA and slow-down area SA.

[0017] The imaging device 120 includes a solid-state imaging element such as a CMOS (Complementary Metal Oxide Semiconductor), and a driver circuit incorporating a timing generator (TG) and an analog signal processing circuit (AFE). The driver circuit of the imaging device 120 receives RGB color signals output from the solid-state imaging element in synchronization with a clock signal output from the TG, and performs necessary processing such as noise removal, amplification, and AD conversion in the AFE to generate digital image data (captured image) in a time series. The driver circuit of the imaging device 120 transmits the generated image data to the image processing device 100. The image data may be transmitted via a wired or wireless method.

[0018] The image processing device 100 analyzes the captured image acquired from the imaging device 120 and tracks the positions of the vehicles within the platform PF. By tracking the positions of the vehicles within the platform PF, the image processing device 100 can count the number of vehicles parked in the parking area PA.

[0019] 2 is a block diagram illustrating the internal configuration of the image processing device 100. The image processing device 100 is a dedicated or general-purpose computer, and includes a control unit 101, a storage unit 102, an operation unit 103, an input unit 104, an output unit 105, and a communication unit 106.

[0020] The control unit 101 includes, for example, a CPU (Central Processing Unit), a ROM (Read Only Memory), and a RAM (Random Access Memory). The ROM included in the control unit 101 stores a control program that controls the operation of each hardware unit included in the image processing device 100. The CPU in the control unit 101 executes the control program stored in the ROM and various computer programs stored in a storage unit 102 (described later) to control the operation of each hardware unit, thereby realizing the functions of the image processing device 100 in this embodiment. The RAM included in the control unit 101 temporarily stores data used during execution of calculations, etc.

[0021] The control unit 101 is configured to include a CPU, a ROM, and a RAM, but may alternatively be one or more arithmetic circuits or control circuits including a GPU (Graphics Processing Unit), an FPGA (Field Programmable Gate Array), a DSP (Digital Signal Processor), a quantum processor, volatile or non-volatile memory, etc. The control unit 101 may also include functions such as a clock that outputs date and time information, a timer that measures the elapsed time from when an instruction to start measurement is given until when an instruction to end measurement is given, and a counter that counts numbers.

[0022] The storage unit 102 includes a storage device using a hard disk, a flash memory, etc. The storage unit 102 stores computer programs executed by the control unit 101, various data acquired from the outside, various data generated inside the image processing device 100, etc.

[0023] The computer programs stored in the storage unit 102 include a vehicle tracking program PG that causes a computer to execute a process of detecting a vehicle to be tracked from each of images captured at multiple points in time by the imaging device 120 and tracking the position of the vehicle within a vehicle tracking area based on the detection results. The computer program including the vehicle tracking program PG is provided by a non-transitory recording medium RM on which the computer program is readably recorded. The recording medium RM is, for example, a portable memory such as a CD-ROM, a USB memory, or an SD (Secure Digital) card. The control unit 101 reads the various programs from the recording medium RM using a reading device (not shown) and stores the read programs in the storage unit 102. Alternatively, the computer program including the vehicle tracking program PG may be provided via communication.

[0024] The storage unit 102 also stores a learning model MD that has been trained to output information about a vehicle included in a captured image when the captured image is input. The learning model MD may be a learning model for object detection, such as YOLO (You Only Look Once) or SSD (Single Shot Multi-Box Detector). Alternatively, the learning model MD may be a learning model that performs image segmentation, such as SegNet, FCN (Fully Convolutional Network), U-Net (U-Shaped Network), or PSPNet (Pyramid Scene Parsing Network). The storage unit 102 stores information about the layer structure of the learning model MD and the nodes included in each layer.

[0025] The operation unit 103 is configured with input devices such as a keyboard and a mouse, and accepts various operations. The control unit 101 executes appropriate processing based on the operations accepted through the operation unit 103. Note that, in this embodiment, the image processing device 100 is configured to include the operation unit 103, but the operation unit 103 is not essential, and the operation may be accepted via an externally connected device or the communication unit 106.

[0026] The input unit 104 includes an input interface for connecting the imaging device 120. The input interface included in the input unit 104 may be a wired interface or a wireless interface. As a wired interface, for example, an interface such as DVI (Digital Visual Interface) or HDMI (High-Definition Multimedia Interface, registered trademark) can be used. Furthermore, as a wireless interface, an interface such as Wireless HDMI can be used. Image data (image capture device 120) input through the input unit 104 is stored in the storage unit 102.

[0027] The output unit 105 includes an output interface for connecting a display device 140 such as a liquid crystal monitor. The output interface included in the output unit 105 may be an output interface that outputs an analog video signal, or an output interface that outputs a digital video signal. For example, the output unit 105 outputs display data to the display device 140 so that the number of parked vehicles in the parking area PA is displayed on the display device 140. In this embodiment, the display device 140 is connected to an external device of the image processing device 100, but the image processing device 100 may also include the display device 140.

[0028] The communication unit 106 includes a communication interface for transmitting and receiving various data to and from an external device. The external device may be, for example, an external server device or a mobile terminal carried by a worker at a waste disposal facility. To communicate with the external device, the communication unit 106 includes a communication interface conforming to a wireless communication standard such as WiFi (registered trademark), 3G, 4G, 5G, or LTE (Long Term Evolution).

[0029] FIG. 3 is a schematic diagram showing an example of a captured image acquired by the image processing device 100 from the imaging device 120. As described above, the imaging device 120 is installed diagonally above the platform PF and captures an image of the platform PF from a bird's-eye view. The captured image thus captured includes, for example, as shown in FIG. 3, substantially the entire platform PF including the parking area PA and the slow-down area SA, and a part of the garbage pit GP. The captured image shown in FIG. 3 shows that of the three parking spaces PS1 to PS3 provided in the parking area PA, a garbage collection truck GT1 is parked in the front parking space PS1, and a general vehicle C1 is parked in the back parking space PS3.

[0030] If the number of vehicles parked on the platform PF is simply to be counted, it is possible to use an image captured from the front of the vehicle. However, if an image captured from the front of the vehicle is used, it becomes difficult to distinguish between general vehicles and garbage collection trucks. In order to detect garbage collection trucks and distinguish them from general vehicles, both the cab and garbage collection box, which are characteristic of garbage collection trucks, must be included in the captured image. For this reason, in this embodiment, vehicle detection is performed using an image captured from diagonally above the vehicle.

[0031] The image processing device 100 acquires captured images from the imaging device 120 and analyzes the acquired captured images to detect a vehicle to be tracked. In this embodiment, a learning model MD for object detection is used to analyze the captured images.

[0032] The detection method using the learning model MD will be explained below. FIG. 4 is a schematic diagram showing an example configuration of the learning model MD. The learning model MD is a learning model for object detection, such as YOLO or SSD, and includes an input layer, an intermediate layer, and an output layer. In this embodiment, the objects to be detected are garbage collection trucks and ordinary vehicles. Images captured in time series by the imaging device 120 are sequentially input to the input layer of the learning model MD. The intermediate layer of the learning model MD includes a convolutional layer, a pooling layer, a fully connected layer, and the like, and performs calculations such as selecting candidate object-like regions included in the input images and identifying objects based on features extracted from the input images. The output layer of the learning model MD outputs object detection results based on the calculation results of the intermediate layer. The detection results include, for example, information on the bounding box surrounding the object, the class name of the object enclosed by the bounding box, and a score indicating the detection reliability. The detection reliability is expressed as the product of the probability (a value between 0 and 1) that the bounding box contains some object and the overlap rate (a value between 0 and 1) between the bounding box and the correct region.

[0033] The learning model MD is generated by performing machine learning using a predetermined algorithm, by providing as training data a large number of images containing garbage collection vehicles or general vehicles as objects, the class names of the objects included in each image (garbage collection vehicle or general vehicle), and information about the bounding boxes surrounding the objects (e.g., center coordinates, width, height). The learning model MD may be generated inside the image processing device 100 or on an external server. The generated learning model MD is stored in the storage unit 102 of the image processing device 100. In this case, the control unit 101 of the image processing device 100 inputs the captured image to be analyzed into the learning model MD and performs calculations using the learning model MD to obtain object detection results.

[0034] Alternatively, the learning model MD may be stored in an external server. In this case, the image processing device 100 uploads the captured image acquired by the imaging device 120 to the external server, requests the external server to perform a calculation using the learning model MD, and acquires the calculation result (object detection result) using the learning model MD from the external server.

[0035] FIG. 5 is a schematic diagram showing an example of object detection using the learning model MD. The image processing device 100 inputs captured images acquired by the imaging device 120 into the learning model MD and executes calculations using the learning model MD to obtain information on bounding boxes surrounding these objects, the class names of the objects surrounded by the bounding boxes, and scores indicating the reliability of detection. The captured image in FIG. 5 includes a sewage collection vehicle GT1 and a general vehicle C1 as objects to be detected. By inputting such captured images into the learning model MD, the image processing device 100 obtains information on the bounding box BOX1 surrounding the sewage collection vehicle GT1 and information on the bounding box BOX2 surrounding the general vehicle C1.

[0036] Here, the bounding box information is information for defining the rectangular area indicated by the bounding box, and is defined, for example, by the center coordinates, width, and height of the rectangular area. When the center coordinates of bounding box BOX1 are (x1, y1), the width is w1, and the height is h1, bounding box BOX1 is expressed as (x1, y1, w1, h1). Similarly, when the center coordinates of bounding box BOX2 are (x2, y2), the width is w2, and the height is h2, bounding box BOX2 is expressed as (x2, y2, w2, h2). In this embodiment, the bounding box is defined by the center coordinates, width, and height of the rectangular area, but the coordinates of the upper left corner of the rectangular area may be used instead of the center coordinates of the rectangular area.

[0037] The detection results shown in Figure 5 indicate that the class name of the object enclosed by bounding box BOX1 is "Garbage Truck" and that the detection reliability is "0.98." Also, the class name of the object enclosed by bounding box BOX2 is "Car (general vehicle)" and that the detection reliability is "0.97."

[0038] The image processing device 100 sequentially inputs captured images obtained at each time point from the imaging device 120 into the learning model MD, and tracks the position of the vehicle within the vehicle tracking area by detecting the target vehicle from each captured image. Information such as the bounding box obtained from the captured image at each time point is stored in the memory unit 102 as needed.

[0039] 6 is an explanatory diagram illustrating a method for tracking a vehicle position. In this embodiment, a vehicle movement trajectory is derived by searching a predetermined number of detection areas, going back from the most recent detection area. Specifically, the image processing device 100 performs the following processes.

[0040] The image processing device 100 sequentially inputs captured images (frames) captured at each time t=t1, t2, ..., tn to the learning model MD, and stores information on bounding boxes BOX1, BOX2, ..., BOXn obtained from the learning model MD as detection area information in the storage unit 102. The image processing device 100 extracts a predetermined number (e.g., 20) of bounding boxes BOXn, BOXn-1, ..., BOXn-19, going back from the most recent bounding box BOXn. The center coordinates of each bounding box BOXn, BOXn-1, ..., BOXn-19 are defined as Pn, Pn-1, ..., Pn-19.

[0041] The image processing device 100 sets a search area RA for the latest bounding box BOXn. The image processing device 100 can set the search area RA based on the center coordinate Pn of the bounding box BOXn. As shown in Fig. 6, if Pn-1, ..., Pn-19 extend downward and to the left with respect to Pn, for example, a rectangular area cut out by boundaries 50 pixels to the right and upward, 350 pixels to the left, and 250 pixels downward from Pn is set as the search area RA.

[0042] The image processing device 100 selects, from among the predetermined number of extracted bounding boxes BOXn, BOXn-1, ..., BOXn-19, those whose center coordinates Pn, Pn-1, ..., Pn-19 are inside the search area RA as detection areas to be used for tracking the vehicle position. In the example of Fig. 6, center coordinates Pn to Pn-17 are inside the search area RA, and so bounding boxes BOXn to BOXn-17 are selected as detection areas to be used for tracking the vehicle position. On the other hand, center coordinates Pn-18 and Pn-19 are outside the search area RA, so bounding boxes BOXn-18 and BOXn-19 are excluded from the detection areas to be used for tracking the vehicle position.

[0043] 6, 20 bounding boxes are extracted going back from the most recent bounding box BOXn, but the number of bounding boxes to be extracted may be set appropriately depending on the angle of view of the image capture device 120, the size of the facility, etc. The size of the search area RA is not limited to the above, and may be set appropriately depending on the angle of view of the image capture device 120, the size of the facility, etc.

[0044] The image processing device 100 derives the vehicle movement trajectory based on the selected detection area. FIG. 7 is a schematic diagram showing an example of derivation of the vehicle movement trajectory. When the image processing device 100 selects bounding boxes BOXn to BOXn-17 as detection areas used to track the vehicle position, it extracts these center coordinates Pn to Pn-17 as the vehicle movement trajectory. In this embodiment, the detection areas are extracted by tracing back from the most recent captured image, so the center coordinate Pn is the start point of the search, and the center coordinate Pn-17 is the end point of the search. In this case, the trajectory connecting the center coordinates Pn-17, Pn-16, ..., Pn-1, Pn in order represents the vehicle movement trajectory.

[0045] In this embodiment, the center coordinates of the detection area are identified as the vehicle position, but the coordinates of a point offset from the center may also be identified as the vehicle position, and the center of gravity of the area detected as a vehicle by the learning model MD may also be identified as the vehicle position.

[0046] As described above, it is possible to derive the vehicle's movement trajectory by tracking the center coordinates of the selected bounding box. However, in this embodiment, since an image captured from diagonally above the vehicle is used, when two vehicles are lined up side by side, the vehicle at the back may be hidden by the shadow of the vehicle at the front. In this case, it becomes difficult to obtain the information necessary for type discrimination, and there is a possibility that the vehicle at the back may not be detected. For this reason, if a method is adopted in which entry and exit are determined based on whether or not a vehicle has crossed the boundary line between the parking area PA and the slow-down area S, a problem occurs in that it is not possible to count vehicles that have entered or exited if a vehicle is not detected immediately before or immediately after crossing the boundary line.

[0047] Therefore, in this embodiment, the movement trajectory of the vehicle is tracked, and entry and exit into the parking area PA is determined by determining whether the movement trajectory crosses the area between the two reference lines.

[0048] 8 is an explanatory diagram illustrating an example of setting reference lines. In this embodiment, two reference lines BL1 and BL2 are set at a distance from each other. In the following description, when the two reference lines are to be distinguished, the reference line BL1 closer to the parking area PA will be referred to as the first reference line BL1, and the reference line BL2 located at a distance from the first reference line BL1 on the opposite side of the parking area PA will be referred to as the second reference line. The reference line BL0 shown for reference is a reference line that is equidistant from both the first reference line BL1 and the second reference line BL2.

[0049] In order to determine whether parking has occurred, it is preferable that the reference lines BL1 and BL2 be set near the parking area PA. On the other hand, if the garbage collection vehicle GT2 at the back is hidden in the shadow of the garbage collection vehicle GT1 at the front and cannot be detected, it may not be possible to determine whether the movement trajectory has crossed the area between the reference lines BL1 and BL2, so it is preferable that the reference lines BL1 and BL2 be set at a position some distance away from the parking area PA (a position away from the slow-down space).

[0050] More specifically, it is preferable to set the reference lines BL1 and BL2 based on the following points: (1) With respect to the first reference line BL1, it is preferable that the centers of the detection boxes (center coordinates of the bounding boxes) of all vehicles parked in the parking spaces PS1 to PS3 are located on the parking area PA side of the first reference line BL1. (2) Even if another vehicle is parked in the lane in front, it is preferable that the center of the detection box can be confirmed on the parking area PA side of the first reference line BL1.

[0051] (3) Regarding the second reference line BL2, when a vehicle in parking spaces PS1 to PS3 moves into the slow driving area SA, it is preferable that the center of the detection box of the vehicle moving into the slow driving area SA can be confirmed on the slow driving area SA side of the second reference line BL2, even if another vehicle is parked in the lane in front of it.

[0052] (4) With regard to the overall position of the reference lines BL1 and BL2, it is preferable that both reference lines BL1 and BL2 be located as close as possible to the parking area PA so that the center of the detection box of a vehicle detected in the slow-down area SA does not straddle the reference lines BL1 and BL2.

[0053] (5) With regard to the width of the area between the reference lines BL1 and BL2, it is preferable that the area between the reference lines BL1 and BL2 includes at least one point that constitutes the vehicle's movement trajectory, based on the vehicle's movement speed and the frame rate of the imaging device 120.

[0054] In this embodiment, two reference lines BL1 and BL2 are set based on the above points (1) to (5).

[0055] 9 is an explanatory diagram illustrating a method for determining entry and exit into the parking area PA. The image processing device 100 according to this embodiment determines whether the derived movement trajectory of the vehicle crosses the area between the first reference line BL1 and the second reference line BL2, thereby determining whether the vehicle has entered or left the parking area PA.

[0056] More specifically, if the search start point (Pn in the example of FIG. 9) is further inside the first reference line BL1 and the search end point (Pn-17 in the example of FIG. 9) is further outside the second reference line BL2, it is determined that the vehicle has entered the parking area PA. Conversely, if the search start point is further outside the second reference line BL2 and the search end point is further inside the first reference line BL1, it is determined that the vehicle has exited the parking area PA.

[0057] If the image processing device 100 tracks the vehicle position and determines that the vehicle has crossed the area between the second reference line BL2 and the first reference line BL1 from the slow driving area SA side, it determines that the vehicle has parked and adds +1 to the number of parked vehicles; if the image processing device 100 determines that the vehicle has crossed the area between the first reference line BL1 and the second reference line BL2 from the parking area PA side, it determines that the vehicle has left the parking area PA and subtracts 1 from the number of parked vehicles, thereby counting the number of parked vehicles in the parking area PA.

[0058] Furthermore, since the image processing device 100 can identify the class of object (garbage collection vehicle / general vehicle) from the calculation results of the learning model MD, the number of parked vehicles may be counted for each garbage collection vehicle or for each general vehicle.

[0059] The operation of the image processing device 100 will now be described. 10 is a flowchart illustrating the procedure for the entry determination process executed by the image processing device 100. When the control unit 101 of the image processing device 100 acquires a captured image frame by frame through the input unit 104 (step S101), the control unit 101 inputs the acquired captured image into the learning model MD and executes calculations using the learning model MD (step S102). The captured image acquired in step S101 is set as the latest frame. The control unit 101 executes calculations using the learning model MD on the acquired latest frame to detect vehicles included in that frame and acquire information such as bounding boxes. The control unit 101 stores the bounding box information acquired in the latest frame in the storage unit 102 as information on the detection area (step S103).

[0060] The control unit 101 obtains the center coordinates of the detection area acquired from the latest frame and determines whether the search start point is located further inside the parking area PA than the first reference line BL1 (step S104). If the search start point is not located inside the first reference line BL1 (S104: NO), the control unit 101 returns the process to step S101.

[0061] If the search start point is inside the first reference line BL1 (S104: YES), the control unit 101 reads information on a predetermined number (e.g., 20) of detection areas from the storage unit 102 (step S105). The control unit 101 sets a search area based on the center coordinates of the detection area extracted from the latest frame, and selects detection areas included within the set search area (step S106).

[0062] The control unit 101 derives the vehicle movement trajectory by sequentially selecting the detection area with the smallest center-to-center distance from the selected detection areas (step S107). That is, the control unit 101 may sequentially select the detection area with the smallest center-to-center distance from the detection area extracted from the latest frame. Note that even if the center-to-center distance is the smallest, it may be excluded from selection if the distance is equal to or greater than a threshold value.

[0063] Next, the control unit 101 determines whether the search end point is outside the second reference line BL2 (step S108). If the search end point is not outside the second reference line BL2 (S108: NO), the control unit 101 returns the process to step S101.

[0064] If the search end point is outside the second reference line BL2 (S108: YES), the control unit 101 determines that the vehicle has entered the parking area PA (step S109) and adds +1 to the number of parked vehicles (step S110).

[0065] The control unit 101 outputs information on the counted number of parked vehicles from the output unit 105, and causes the display device 140 to display the information (step S111).

[0066] 11 is a flowchart illustrating the procedure for the exit determination process executed by the image processing device 100. The procedure for the exit determination is the same as the procedure for the entry determination process. In the same procedure as in the flowchart of FIG. 10, the control unit 101 determines whether the search start point is outside the second reference line BL2 in step S124, and determines whether the search end point is inside the first reference line in step S128. If it is determined that the search start point is outside the second reference line BL2 (S124: YES) and the search end point is inside the first reference line BL1 (S128: YES), the control unit 101 determines that the vehicle has exited the parking area (step S129) and decrements the number of parked vehicles by 1 (step S130).

[0067] The control unit 101 outputs information on the counted number of parked vehicles from the output unit 105, and causes the display device 140 to display the information (step S131).

[0068] Fig. 12 is a schematic diagram showing an example of displaying the number of parked vehicles. Fig. 12 shows an example of displaying the number of parked vehicles by vehicle type (refuse collection vehicle / general vehicle).

[0069] As described above, in this embodiment, the first reference line BL1 and the second reference line BL2 are set to be biased toward the parking area PA with respect to the boundary line BL0 that separates the parking area PA from the slow driving area SA, and entry and exit to the parking area PA is determined when it is determined that the vehicle's movement trajectory has crossed the area between the first reference line BL1 and the second reference line BL2. Therefore, compared to when a single boundary line is used for determination, erroneous detection due to fluctuations in the detection position can be avoided, and entry and exit can be determined with high accuracy.

[0070] (Embodiment 2) In the second embodiment, a configuration will be described in which, when the detected position of the vehicle vibrates in the direction in which the vehicle's movement trajectory extends, derivation of the movement trajectory is stopped.

[0071] Fig. 13 is a schematic diagram showing a detection example when the detected position of a vehicle vibrates. The example in Fig. 13 shows the result of searching past frames up to Pn-14, with the center coordinate of the bounding box BOX detected from the captured image (latest frame) at time t = tn as Pn, starting from Pn. Since the search start point (Pn) is inside the first reference line BL1 and the search end point (Pn-14) is outside the second reference line BL2, the example in Fig. 13 shows a case in which it is determined that the vehicle has entered the parking area PA in the first embodiment.

[0072] 13, the search starts from Pn, goes as far as Pn-4, turns back, and returns to the inside of the first reference line BL1 at Pn-7. In this case, parking should have been determined based on the movement trajectory from Pn-7 to Pn-14, so if parking is determined based on both the movement trajectory starting from Pn-7 and the movement trajectory starting from Pn, there is a risk that the number of parked vehicles will be overcounted.

[0073] Therefore, the image processing apparatus 100 according to the second embodiment performs processing to stop the search at that point when the captured image is searched in order from the search start point and the area returns to the same area as the search start point.

[0074] 14 is a flowchart illustrating the procedure for determining entry in accordance with the second embodiment. The control unit 101 of the image processing device 100 executes steps S201 to S207 in the same manner as in the first embodiment, and sequentially selects detection areas within the set search range that have the shortest center-to-center distance. As a result, the control unit 101 determines whether the center coordinates of the detection areas return to a position inside the first reference line BL1 during the search (step S208).

[0075] If it is determined that the object does not return to the inside of the first reference line BL1 during the search (S208: NO), the control unit 101 executes steps S209 to S212 in the same manner as in the first embodiment.

[0076] On the other hand, if it is determined that the object will return to the inside of the first reference line BL1 during the search (S208: YES), the control unit 101 stops the search at that point and ends the process of this flowchart. At this time, the information on the detection area read out in step S205 is erased.

[0077] 15 is a flowchart illustrating the procedure for determining whether or not an exit is made in accordance with the second embodiment. The control unit 101 of the image processing device 100 executes steps S221 to S227 in the same manner as in the first embodiment, and sequentially selects detection areas within the set search range that have the shortest center-to-center distance. As a result, the control unit 101 determines whether or not the center coordinates of the detection areas return to a position outside the second reference line BL2 during the search (step S228).

[0078] If it is determined that the object does not return to the outside of the second reference line BL2 during the search (S228: NO), the control unit 101 executes steps S229 to S232 in the same manner as in the first embodiment.

[0079] On the other hand, if it is determined that the object will return to the inside of the second reference line BL2 during the search (S228: YES), the control unit 101 stops the search at that point and ends the process of this flowchart. At this time, the information on the detection area read out in step S225 is erased.

[0080] As described above, in the second embodiment, if the vehicle returns to the search start point during the search, the search is stopped, so that it is possible to avoid double counting of the number of parked vehicles.

[0081] In this embodiment, a method for determining entry and exit to a parking area PA provided at a waste disposal facility has been described, but the determination method of this embodiment can be used to determine entry and exit to any area, including a flat parking lot and a multi-story parking lot, not limited to parking areas at waste disposal facilities. Furthermore, the method of this embodiment can be used to count not only garbage collection trucks and general vehicles, but also vehicles used for car sharing or autonomous driving.

[0082] Furthermore, in this embodiment, a configuration is adopted in which vehicles are detected using the learning model MD, but a configuration is also possible in which vehicles are detected using a combination of the learning model MD and template matching. For example, a template image may be generated by cutting out an image within the bounding box from a captured image with reference to information about the bounding box extracted by the learning model MD, and vehicle detection may be performed by supplementally using template matching using this template image. In this case, even if a detection miss occurs in a particular frame using the learning model MD, it is possible to avoid the detection miss by using the results of template matching.

[0083] The embodiments disclosed herein should be considered in all respects as illustrative and not restrictive. The scope of the present invention is defined by the claims, not by the above meaning, and is intended to include all modifications within the meaning and scope of the claims. [Explanation of symbols]

[0084] 1. Vehicle Tracking System 100 Image processing device 101 Control section 102 Storage section 103 Operation section 104 Input section 105 Output section 106 Communications Department 120 Imaging device 140 Display device PG Vehicle Tracking Program MD learning model

Claims

1. Acquire captured images from an imaging device that captures images of a vehicle tracking area in time series; Using a learning model for object detection, object detection is performed for each of the images captured by the imaging device at each time point, and a detection area including the vehicle to be detected is extracted from the image captured at each time point; Deriving a movement trajectory of the vehicle from the extracted plurality of detection areas; Entry and exit into the predetermined area are determined by determining whether the derived movement trajectory crosses an area between a first reference line set closer to the predetermined area and a second reference line set away from the first reference line on the opposite side of the predetermined area. A computer program that causes a computer to execute a process.

2. The movement trajectory is derived by sequentially searching a predetermined number of captured images, going back from the most recent captured image. The computer program product according to claim 1, for causing the computer to execute a process.

3. determining that the person has entered the predetermined area when the starting point of the movement trajectory is at a position deviated from the first reference line toward the inside of the predetermined area and the ending point of the movement trajectory is at a position deviated from the second reference line toward the outside of the predetermined area; If the start point of the movement trajectory is located at a position deviated from the second reference line toward the outside of the predetermined area and the end point of the movement trajectory is located at a position deviated from the first reference line toward the inside of the predetermined area, it is determined that the person has left the predetermined area.

3. A computer program product according to claim 2, for causing the computer to execute a process.

4. the learning model is configured to output, when a captured image is input, information about a bounding box surrounding a vehicle included in the captured image; Extracting a detection area including a vehicle to be detected based on information output from the learning model.

4. A computer program according to claim 1, for causing a computer to execute a process.

5. The trajectory of the attention point in each detection area is derived as the movement trajectory of the vehicle.

5. A computer program according to claim 1, for causing a computer to execute a process.

6. When the moving trajectory returns to the area where the starting point of the moving trajectory exists halfway along the moving trajectory, derivation of the moving trajectory is stopped.

6. A computer program according to claim 5, for causing the computer to execute a process.

7. an acquisition unit that acquires captured images from an imaging device that captures images of a vehicle tracking area in time series; an extraction unit that performs object detection for each of the images captured by the imaging device at each time point using a learning model for object detection, and extracts a detection area including a vehicle to be detected from the image captured at each time point; a derivation unit that derives a movement trajectory of the vehicle from the extracted plurality of detection areas; a determination unit that determines whether the derived movement trajectory crosses an area between a first reference line that is set closer to the predetermined area and a second reference line that is set away from the first reference line on the opposite side of the predetermined area, thereby determining whether the person has entered or left the predetermined area; An image processing device comprising:

8. an imaging device that captures images of a vehicle tracking area in time series; an acquisition unit that acquires a captured image from the imaging device; an extraction unit that performs object detection for each of the captured images acquired at each time point by the acquisition unit using a learning model for object detection, and extracts a detection area including a vehicle to be detected from the captured image at each time point; a derivation unit that derives a movement trajectory of the vehicle from the extracted plurality of detection areas; and a determination unit that determines whether the derived movement trajectory crosses an area between a first reference line that is set closer to the predetermined area and a second reference line that is set away from the first reference line on the opposite side of the predetermined area, thereby determining whether the person has entered or left the predetermined area. an image processing device comprising: An image processing system comprising:

9. Acquire captured images from an imaging device that captures images of a vehicle tracking area in time series; Using a learning model for object detection, object detection is performed for each of the images captured by the imaging device at each time point, and a detection area including the vehicle to be detected is extracted from the image captured at each time point; Deriving a movement trajectory of the vehicle from the extracted plurality of detection areas; Entry and exit into the predetermined area are determined by determining whether the derived movement trajectory crosses an area between a first reference line set closer to the predetermined area and a second reference line set away from the first reference line on the opposite side of the predetermined area. An image processing method in which processing is performed by a computer.

Images