Trailer tracking method, trailer tracking device, and trailer tracking system

Abstract

To track a trailer without interruption even when a plurality of trailers are crowded in a yard.SOLUTION: Detecting a plurality of objects from a captured image including one or more trailers captured by a camera provided to be able to capture an image of a yard into which a trailer including the plurality of objects enters, creating a traffic line of each of the plurality of objects by performing object tracking on each of the plurality of detected objects, and selecting a plurality of objects satisfying a first condition for selecting an object from among the plurality of objects; Among the plurality of selected objects, a plurality of objects satisfying a second condition for linking the objects are linked to each other, and based on the flow line of any one specific object among the plurality of objects linked to each other, the flow line of one or more objects linked to the specific object is updated.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present disclosure relates to a trailer tracking method, a trailer tracking device, and a trailer tracking system. [Background technology]

[0002] Patent Document 1 discloses a control device that extracts edges representing the outline of a vehicle from an image generated by capturing an image of a parking area, and determines the presence or absence of a vehicle in the parking area based on the extracted edges. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 11-201741 Summary of the Invention [Problem to be solved by the invention]

[0004] Conventionally, temporal changes in the position of trailers may be recorded within a large business premises (hereinafter sometimes referred to as a "yard") of a transportation company or the like where tractors (so-called transport trucks) towing trailers (an example of a towed object) loaded with many loads are accumulated. In this case, the transport truck (i.e., the tractor) enters the yard with a driver who drives the tractor and parks the trailer in a designated parking space within the yard. A yard worker (hereinafter sometimes referred to as a "yard jockey"), who is a different person from the tractor driver, then gets on the tractor and tows the designated trailer. However, when the driver and the yard jockey are different people, the parking space where the driver thinks the trailer is parked may not match the parking space the yard jockey is heading to. In other words, the yard jockey may get on the tractor and head to the designated parking space, but may not be able to find the desired trailer. Particularly when the yard has a large area, it is often difficult for the yard jockey to visually find the desired trailer. Therefore, there is a need for technology that can detect and track each trailer parked within a yard.

[0005] When multiple trailers are closely spaced in an image captured within a yard, the edges representing the trailer outlines may overlap, making detection difficult. For example, when detecting trailers within a yard using edges representing the trailer outlines, as in Patent Document 1, if multiple trailers are closely spaced in the captured image, part of the trailer to be detected may be hidden by other trailers, making it difficult to detect the trailer to be detected. Furthermore, if the trailer to be detected cannot be detected, tracking of that trailer may be interrupted.

[0006] The present disclosure has been devised in view of the above-described conventional situation, and aims to track trailers without interruption even when multiple trailers are crowded together in a yard. [Means for solving the problem]

[0007] The present disclosure provides a trailer tracking method that detects multiple objects from an image including one or more trailers captured by a camera that is installed to be able to capture images of a yard into which a trailer containing the multiple objects enters, performs object tracking on each of the detected multiple objects to create a movement path for each of the multiple objects, selects multiple objects from the multiple objects that satisfy a first condition for selecting the objects, links multiple objects from the selected multiple objects that satisfy a second condition for linking the objects to each other, and updates the movement path of one or more objects linked to any one specific object from the multiple objects that are linked to the specific object based on the movement path of the specific object.

[0008] The present disclosure also provides a trailer tracking device that includes a processor and a memory, and in which the processor and the memory work together to detect a plurality of objects from an image including one or more trailers captured by a camera that is configured to capture an image of a yard into which the trailer containing the plurality of objects enters, perform object tracking on each of the detected plurality of objects to create a movement line for each of the plurality of objects, select a plurality of objects from the plurality of objects that satisfy a first condition for selecting the objects, link a plurality of objects from the selected plurality of objects that satisfy a second condition for linking the objects to each other, and update the movement line of one or more objects linked to any one specific object from the plurality of objects that are linked to the specific object based on the movement line of the specific object.

[0009] The present disclosure also provides a trailer tracking system comprising a processor, a memory, and a camera capable of capturing images of a yard into which a trailer containing multiple objects enters, wherein the processor and the memory work together to detect the multiple objects from an image captured by the camera that includes one or more of the trailers, perform object tracking for each of the detected multiple objects, create a movement line for each of the multiple objects, select multiple objects from the multiple objects that satisfy a first condition for selecting the objects, link multiple objects from the selected multiple objects that satisfy a second condition for linking the objects to each other, and update the movement lines of one or more objects linked to any one specific object from the multiple objects that are linked to the specific object.

[0010] Any combination of the above components, and conversion of the expression of the present disclosure into a method, device, system, storage medium, computer program, etc., are also valid aspects of the present disclosure. [Effects of the Invention]

[0011] According to the present disclosure, even when multiple trailers are crowded together in a yard, the trailers can be tracked without interruption. [Brief explanation of the drawings]

[0012] [Figure 1] FIG. 1 is a block diagram showing a configuration example of a trailer tracking system according to a first embodiment. [Figure 2A] FIG. 1 is a schematic diagram illustrating an example of an object to be detected according to the first embodiment; [Figure 2B] FIG. 1 is a schematic diagram illustrating an example of an object to be detected according to the first embodiment; [Figure 2C] FIG. 1 is a schematic diagram illustrating an example of an object to be detected according to the first embodiment; [Figure 2D]FIG. 1 is a schematic diagram illustrating an example of an object to be detected according to the first embodiment; [Figure 3] FIG. 1 is a schematic diagram for explaining an outline of trailer tracking according to a first embodiment; [Figure 4] FIG. 1 is a schematic diagram for explaining an outline of trailer tracking according to a first embodiment; [Figure 5] FIG. 1 is a schematic diagram illustrating object detection according to a first embodiment. [Figure 6] FIG. 1 is a schematic diagram for explaining detection of circumscribing of a plurality of trailers according to the first embodiment; [Figure 7A] FIG. 1 is a table illustrating the IoU of the circumscribed area of ​​a trailer according to the first embodiment. [Figure 7B] FIG. 1 is a table illustrating the IoU of the circumscribed area of ​​a trailer according to the first embodiment. [Figure 8] FIG. 1 is a schematic diagram for explaining detection of the front of a plurality of trailers according to the first embodiment; [Figure 9A] FIG. 1 is a table illustrating IoU in front of a trailer according to the first embodiment. [Figure 9B] FIG. 1 is a table illustrating IoU in front of a trailer according to the first embodiment. [Figure 10] FIG. 1 is a schematic diagram for explaining a method for identifying the circumscribing direction and the front of a trailer according to the first embodiment; [Figure 11] FIG. 1 is a schematic diagram illustrating an area for selecting an identification target according to the first embodiment; [Figure 12] FIG. 1 is a schematic diagram illustrating a speed limit for selecting an identification target according to the first embodiment; [Figure 13] FIG. 1 is a schematic diagram for explaining restrictions on overlap rates for selecting identification targets according to the first embodiment; [Figure 14] FIG. 1 is a table illustrating an overlap rate for selecting an identification target according to the first embodiment. [Figure 15] FIG. 1 is a schematic diagram illustrating a distance restriction for selecting an identification target according to the first embodiment; [Figure 16] 1 is a flowchart of an object tracking process of a trailer tracking device according to a first embodiment. [Figure 17] FIG. 1 is a schematic diagram for explaining object tracking by a trailer tracking device according to a first embodiment; [Figure 18] Flowchart of overall processing of the trailer tracking device according to the first embodiment [Figure 19] Flowchart of identification target selection processing of the trailer tracking device according to the first embodiment DETAILED DESCRIPTION OF THE INVENTION

[0013] Hereinafter, with reference to the drawings as appropriate, detailed descriptions will be given of embodiments that specifically disclose a trailer tracking method, trailer tracking device, and trailer tracking system according to the present disclosure. However, unnecessary detailed descriptions may be omitted. For example, detailed descriptions of already well-known matters and redundant descriptions of substantially identical configurations may be omitted. This is to avoid unnecessary redundancy in the following description and to facilitate understanding by those skilled in the art. Note that the accompanying drawings and the following description are provided to enable those skilled in the art to fully understand the present disclosure, and are not intended to limit the subject matter recited in the claims.

[0014] <1. Definition of terms> First, terms used in the following embodiments will be defined. (1) Trailer: A container-like object with a rectangular parallelepiped casing carrying one or more cargo items, towed by a tractor. The trailer has identification information ("trailer ID (ID)") to identify the trailer. (2) Tractor: A vehicle part of a transport truck that can be docked (coupled) with a trailer or detached from the trailer. A tractor has identification information ("Tractor ID") to identify the tractor. (3) Yard: A large office or site of a transportation company or the like where tractors pulling trailers (so-called transport trucks) are gathered. (4) Yard jockey: A worker who performs tasks within the yard, such as driving a tractor within the yard to dock a desired trailer in a designated parking space with the tractor and move it to the dock. The yard jockey may also perform tasks such as taking out or carrying in cargo from a trailer that has been moved to the dock. In the following embodiments, the yard jockey and the driver of a transport truck entering the yard or the driver of a transport truck leaving the yard are treated as different persons.

[0015] <First Embodiment> [System Configuration] First, a configuration example of a trailer tracking system 1 according to the first embodiment will be described with reference to Fig. 1. Fig. 1 is a block diagram showing a configuration example of a trailer tracking system 1 according to the first embodiment.

[0016] The trailer tracking system 1 is configured to include at least a camera CM, a trailer tracking device 10, and a management system 20. The trailer tracking system 1 is a system that enables tracking of trailers within a yard.

[0017] The camera CM is installed so that it can capture images inside the yard. The trailer tracking system 1 may be configured to include not only one camera CM but also multiple cameras. The camera CM captures images inside the yard and generates captured images. The camera CM also transmits the generated captured images to the trailer tracking device 10. The camera CM installed inside the yard is a so-called network camera, and is connected to the trailer tracking device 10 via the network NW1 so that data communication is possible.

[0018] The camera CM is configured to generate a captured image by at least optical elements, including a lens (not shown) and an image sensor (not shown). The lens receives light reflected by an object within the field of view of the area captured by the camera CM and forms an optical image of the object on the light-receiving surface of the image sensor, i.e., the imaging surface. The image sensor is a solid-state imaging element, such as a Charged Coupled Device (hereinafter referred to as "CCD") or a Complementary Metal Oxide Semiconductor (hereinafter referred to as "CMOS"). The image sensor converts the optical image formed on the imaging surface via the lens into an electrical signal at predetermined intervals. For example, if the predetermined interval is 1 / 30 of a second, the frame rate of the camera CM is 30 fps. The camera CM may also generate captured images by performing predetermined signal processing on the electrical signal at the aforementioned predetermined intervals. The captured images generated by the camera CM include still images and videos.

[0019] The trailer tracking device 10 is configured using a general-purpose computer device such as a personal computer or a server computer. The trailer tracking device 10 includes a processor 11, a memory 12, a communication interface device 13, and a display device 14. The components of the trailer tracking device 10 are connected to each other via an internal bus (not shown) so that they can communicate with each other.

[0020] The processor 11 is configured using, for example, a central processing unit (hereinafter referred to as "CPU"), a graphics processing unit (hereinafter referred to as "GPU"), a micro processing unit (hereinafter referred to as "MPU"), a digital signal processor (hereinafter referred to as "DSP"), or a field programmable gate array (hereinafter referred to as "FPGA"), etc. The processor 11 realizes the functions of the trailer tracking device 10 by reading and executing various data and programs stored and held in the memory 12.

[0021] The memory 12 is a storage area for storing and holding various data, programs, etc. The memory 12 is composed of, for example, a read only memory (hereinafter referred to as "ROM"), which is a non-volatile storage area, a solid state drive (hereinafter referred to as "SSD"), a hard disk drive (hereinafter referred to as "HDD"), and a random access memory (hereinafter referred to as "RAM"), which is a volatile storage area. The RAM is, for example, a work memory used during operation of the trailer tracking device 10. The ROM stores and holds, for example, programs for controlling the trailer tracking device 10 in advance.

[0022] The communication interface device 13 communicates with the camera CM via the network NW1 and receives captured images acquired by the camera CM. The communication interface device 13 also transmits information generated by the trailer tracking process to the management system 20 via the network NW2. The communication interface device 13 may support either wired communication or wireless communication. The communication method used by the communication interface device 13 may be, for example, a Wide Area Network (hereinafter referred to as "WAN"), a Local Area Network (hereinafter referred to as "LAN"), Long Term Evolution (hereinafter referred to as "LTE"), mobile communication such as 5G, power line communication, short-range wireless communication such as Wi-Fi (registered trademark) and Bluetooth (registered trademark), or a combination thereof.

[0023] The display device 14 is a device that displays information, images, etc. Examples of the display device 14 include a liquid crystal display, an organic electroluminescence (hereinafter referred to as "EL") display, and the like.

[0024] The management system 20 is a system that supports yard management operations such as trailer entry and exit into the yard by workers such as yard jockeys or managers. The management system 20 is sometimes referred to as a yard management system. The hardware configuration of the management system 20 may be the same as the hardware configuration of the trailer tracking device 10. A user such as a worker or manager can use the management system 20 to check the movement line of a trailer being tracked within the yard.

[0025] The trailer tracking device 10 detects an object from the captured image received from the camera CM. The trailer tracking device 10 then performs object tracking processing, which will be described later, on the detected object to create a path of movement of the object. This allows tracking of the object. Next, examples of objects to be detected will be described with reference to FIGS. 2A, 2B, 2C, and 2D.

[0026] [Trailer Object] 2A, 2B, 2C, and 2D are schematic diagrams showing examples of objects to be detected according to the first embodiment.

[0027] First, it is assumed that the tracking target is a trailer, and that the object to be detected is included in the trailer. Figures 2A, 2B, 2C, and 2D show a trailer TR1. The trailer TR1 includes a circumscription of the trailer TR1 as an object to be detected. Circumscription of the trailer TR1, in other words, refers to the entire trailer TR1. When the trailer tracking device 10 detects the circumscription of the trailer TR1 from the captured image, it generates a circumscription detection frame O1, which is the smallest rectangle that encloses the circumscription area of ​​the trailer TR1 in the captured image (see Figure 2A). The smallest rectangle that encloses the area of ​​the object to be detected is sometimes called a bounding box. In addition, in the first embodiment, the bounding box may also be referred to as a detection frame. The generated rectangle, such as the circumscription detection frame O1, may be displayed on the display device 14, or may be displayed on a display device (not shown) provided in the management system 20.

[0028] Furthermore, the trailer TR1 includes the front of the trailer TR1 as an object to be detected. When the trailer tracking device 10 detects the front of the trailer TR1 from the captured image, it generates a front detection frame O2, which is the smallest rectangle that encloses the area of ​​the front of the trailer TR1 in the captured image (see FIG. 2B). Note that in the first embodiment, the front of the trailer TR1 and the rear of the trailer TR1 do not need to be distinguished.

[0029] The trailer TR1 also includes the top surface of the trailer TR1 as an object to be detected. In other words, the top surface of the trailer TR1 refers to the ceiling of the trailer TR1. When the trailer tracking device 10 detects the top surface of the trailer TR1 from the captured image, it generates a top surface detection frame O3, which is the smallest rectangle that encloses the area of ​​the top surface of the trailer TR1 in the captured image (see FIG. 2C).

[0030] Furthermore, the trailer TR1 includes a side surface of the trailer TR1 as an object to be detected. When the trailer tracking device 10 detects the side surface of the trailer TR1 from the captured image, the trailer tracking device 10 generates a side surface detection frame O4, which is the smallest rectangle that encloses the area of ​​the side surface of the trailer TR1 in the captured image (see FIG. 2D).

[0031] As described with reference to Figures 2A, 2B, 2C, and 2D, a trailer may include multiple objects. However, in the first embodiment, two types of objects are treated as objects to be detected: those circumscribing the trailer and those in front of the trailer.

[0032] [Example of trailer tracking] Next, an overview of trailer tracking will be described with reference to Fig. 3 and Fig. 4. Fig. 3 and Fig. 4 are schematic diagrams for explaining an overview of trailer tracking according to the first embodiment. Trailer tracking device 10 performs object tracking processing (described later) using the bounding box of the detected object to create a line of movement of the object. This allows tracking of the object, that is, the trailer including the object. This is because the line of movement of the object is, in other words, the line of movement of the trailer including the object.

[0033] A parking space PS for parking trailers is provided within the yard. The example of FIG. 3 shows the driver of trailer TR1 moving trailer TR1 to the parking space PS. Note that the example of FIG. 3 shows a bird's-eye view of the yard. Trailer TR1 moves, for example, to position P4, position P3, position P2, and then to position P1 within the parking space PS. The movement of trailer TR1 at this time is represented by a flow line FL1. The flow line FL1 may be created by trailer tracking device 10 and displayed, for example, on display device 14 or a display device (not shown) of management system 20.

[0034] FIG. 4 shows the state of the yard as captured by the camera CM, in other words, as displayed on the screen of the display device 14, when the camera CM is installed as shown in the example of FIG. 3. Note that the installation position of the camera CM is an example and is not limited to this. Furthermore, one or more cameras other than the camera CM may be installed to capture images outside the angle of view of the camera CM.

[0035] In the example of FIG. 4, the circumscribing of the trailer TR1 is detected regardless of the position of the trailer TR1 (for example, position P1, position P2, position P3, or position P4). At this time, for example, the circumscribing detection frame O1 is displayed continuously on the display device 14 of the trailer tracking device 10. On the other hand, the front of the trailer TR1 may not be detected depending on the angle or orientation of the trailer TR1. For example, when the trailer TR1 is at position P1, position P2, or position P4, the front of the trailer TR1 is detected. When the trailer TR1 is at position P4, the rear of the trailer TR1 is detected, but in the first embodiment, the front and rear of the trailer are not distinguished. However, when the trailer TR1 is at position P3, the front of the trailer TR1 is not detected. At this time, for example, the front detection frame O2 displayed on the display device 14 of the trailer tracking device 10 is interrupted while the trailer TR1 is moving near position P3. Therefore, it is preferable to use the trailer's circumscribing detection frame rather than the trailer's front detection frame when tracking a trailer. However, if the trailer's circumscribing detection frame is detected and the trailer is tracked using the circumscribing detection frame, tracking of the trailer may be interrupted in areas where multiple trailers are crowded together. For example, if multiple trailers are parked in a parking space, it may become difficult to determine where a particular trailer is parked in the parking space. Details will be described with reference to Figures 5, 6, 7A, and 8.

[0036] [Trailer circumferential detection] 5 is a schematic diagram for explaining object detection according to the first embodiment. The trailer tracking device 10 may use known techniques for object detection. In the first embodiment, it is assumed that the trailer tracking device 10 uses Non-Maximum Suppression (hereinafter referred to as "NMS") processing when detecting an object.

[0037] For example, when the trailer tracking device 10 detects the circumscribing of a trailer TR1, it obtains multiple detection results. That is, the trailer tracking device 10 generates multiple circumscribing detection frames, such as a circumscribing detection frame O1, a circumscribing detection frame O1a, and a circumscribing detection frame O1b. Here, it is assumed that each of the multiple circumscribing detection frames is assigned a degree of certainty that a trailer is circumscribing within that detection frame. The trailer tracking device 10 selects the circumscribing detection frame with the highest degree of certainty. Here, the description will be given assuming that the circumscribing detection frame O1 has the highest degree of certainty. The trailer tracking device 10 calculates the intersection over union (hereinafter referred to as "IoU") between the circumscribing detection frame O1 with the highest degree of certainty and the other circumscribing detection frames. The IoU is a value that represents the proportion of overlap between detection frames. The trailer tracking device 10 calculates the IoU between the circumscribed detection frame O1 and other circumscribed detection frames. If the IoU value is equal to or greater than a specified threshold, the other circumscribed detection frame is determined to overlap with the circumscribed detection frame O1 and is deleted. The trailer tracking device 10 performs this process on all circumscribed detection frames other than the circumscribed detection frame O1. As a result, circumscribed detection frames that overlap with the circumscribed detection frame O1 are deleted, leaving only the most reliable circumscribed detection frame O1. Note that the specified threshold for the IoU value may be set in advance by, for example, a user. Simply put, when multiple detection frames overlap, one or more detection frames may be deleted depending on the degree of overlap between the multiple detection frames. When a detection frame is deleted, the object corresponding to that detection frame goes undetected. Hereinafter, for simplicity's sake, it is assumed that there is only one detection frame for one object. For example, the detection frame for the circumscribed detection frame of the trailer TR1 is one of the circumscribed detection frames O1.

[0038] Fig. 6 is a schematic diagram for explaining the detection of the circumscribing of multiple trailers according to the first embodiment. The state of multiple trailers parked in the parking space PS shown in Fig. 6 is an image captured by the camera CM. Therefore, the state shown in Fig. 6 may be displayed on the screen of the display device 14 of the trailer tracking device 10.

[0039] Here, the following description focuses on trailer TRA, trailer TRB, trailer TRC, trailer TRD, trailer TRE, trailer TRF, trailer TRG, trailer TRH, and trailer TRI among the multiple trailers. The trailer tracking device 10 generates circumscribed detection frames for each of the trailers TRA, trailer TRB, trailer TRC, trailer TRD, trailer TRE, trailer TRF, trailer TRG, trailer TRH, and trailer TRI. Some of the circumscribed detection frames for each trailer overlap. For example, circumscribed detection frame O1H and circumscribed detection frame O1I overlap. Figure 7A shows the area of ​​the region of each circumscribed detection frame for each trailer.

[0040] 7A and 7B are table diagrams illustrating the IoU of the circumscribed detection frames of trailers according to the first embodiment. Table TBL1 shown in FIG. 7A shows the area of ​​the region of each circumscribed detection frame of each trailer. To avoid redundant explanation, the area of ​​the region of the trailer detection frame may be simply referred to as the area of ​​the trailer detection frame. From table TBL1, it can be seen that in the example of FIG. 6, the circumscribed detection frame of a trailer closer to the edge of the captured image has a larger area than the circumscribed detection frame of a trailer closer to the center of the captured image. Next, FIG. 7B shows the area of ​​the overlapping region between the circumscribed detection frames of trailers and the calculated IoU value. Table TBL2 shown in FIG. 7B shows the area of ​​the overlapping region between two adjacent circumscribed detection frames and the IoU value. For example, the IoU value of the circumscribed detection frame of each trailer TRH and trailer TRI can be calculated using the following formula (1).

[0041]

number

[0042] For example, when the threshold value of the IoU value is 0.45, either the circumscribed detection frame O1H of the trailer TRH or the circumscribed detection frame O1I of the trailer TRI is deleted. That is, in this case, either the trailer TRH or the trailer TRI goes undetected. In this way, in an area where multiple trailers are densely packed, a trailer may go undetected. For example, when a trailer TRH is not detected in the parking space PS, it may take a long time for the yard jockey to search for the trailer TRH in the yard.

[0043] Next, with reference to FIGS. 8, 9A and 9B, detection of a trailer by detecting the front of the trailer in an area where multiple trailers are densely packed will be described.

[0044] [Detection of the front of the trailer] FIG. 8 is a schematic diagram for explaining detection of the front faces of multiple trailers according to the first embodiment. The state of multiple trailers parked in the parking space PS shown in FIG. 8 is the same as that shown in FIG. 6, as seen in an image captured by the camera CM. The trailer tracking device 10 generates front face detection frames for each of the trailers TRA, TRB, TRC, TRD, TRE, TRF, TRG, TRH, and TRI. Unlike the circumscribing detection frames shown in FIG. 6, the front face detection frames do not overlap each other. FIG. 9A shows the area of ​​the front face detection frame region for each trailer.

[0045] 9A and 9B are table diagrams illustrating the IoU of the front of the trailer according to the first embodiment. Table TBL3 shown in FIG. 9A shows the area of ​​the region of the front detection frame of each trailer. From table TBL3, it can be seen that in the example of FIG. 8, the front detection frame of the trailer closer to the edge of the captured image has a larger area than the front detection frame of the trailer near the center of the captured image. However, compared to the case of the circumscribed detection frames, the difference in area between the front detection frames is smaller. Next, FIG. 9B shows the area of ​​the region where the front detection frames of the trailers overlap and the calculated IoU value. Table TBL2 shown in FIG. 9B shows the area of ​​the region where two adjacent circumscribed detection frames overlap and the IoU value. For example, the IoU value of the front detection frame of each trailer TRH and trailer TRI can be calculated using the following formula (2).

[0046]

number

[0047] For example, assume that the threshold value of the IoU value is 0.45. In this case, neither the front detection frame O2H of the trailer TRH nor the front detection frame O2I of the trailer TRI is deleted. In other words, in this case, both the trailer TRH and the trailer TRI are detected. Furthermore, trailers other than the trailer TRH and the trailer TRI are also detected. As such, in an area where multiple trailers are densely packed, detecting trailers using the front detection frame is more likely to accurately detect each trailer than detecting trailers using the circumscribing detection frame. This is because the trailer front detection frames are less likely to overlap than the circumscribing detection frames. Therefore, it is preferable to mainly use the trailer circumscribing detection frame for trailer detection and tracking, but it is preferable to mainly use the trailer front detection frame for trailer detection in an area where trailers are densely packed. However, the circumscribing and front of the same trailer must be linked. Hereinafter, linking the circumscribing and front of a trailer may be referred to as identifying the circumscribing and front of a trailer.

[0048] [Identification of trailer circumscription and front] FIG. 10 is a schematic diagram for explaining a method for identifying the circumscribing direction and the front of the trailer according to the first embodiment.

[0049] The following explanation will be given using three coordinate systems: a camera coordinate system, a screen coordinate system, and a world coordinate system. The camera coordinate system is a coordinate system based on the position and orientation of the camera CM. The camera coordinate system consists of three axes: X, Y, and Z, with the Z axis parallel to the optical axis of the camera. The X axis is horizontal and the Y axis is vertical. In the example of FIG. 10, the camera CM captures an image of a trailer TR1. The camera CM is preferably installed so that it can capture the front (rear) of the trailer parked in the parking space PS. More specifically, the camera CM is preferably installed so that it can generate captured images that capture the front (rear) of each of multiple trailers, as shown in FIG. 6 or FIG. 8.

[0050] The screen coordinate system is a coordinate system that represents the position on the image captured by the camera CM, and is defined by the u axis and The screen coordinate system consists of two axes, the a and v axes. The screen coordinate system is used, for example, to specify the position of an object on a captured image. The trailer tracking device 10 detects the circumscription of the trailer TR1 from the captured image of the trailer TR1 by the camera CM, and generates a circumscription detection frame O1. The trailer tracking device 10 also detects the front of the trailer TR1, and generates a front detection frame O2. The trailer tracking device 10 further calculates the center coordinate QS1 of the circumscription detection frame O1 and the center coordinate QS2 of the front detection frame O2.

[0051] The trailer tracking device 10 converts the screen coordinate system into a world coordinate system. The world coordinate system is a coordinate system that corresponds to a map of the yard and consists of two axes, the x-axis and the y-axis. The trailer tracking device 10 calculates the center coordinate QW1 by converting the center coordinate QS1 of the circumscribed detection frame O1 of the trailer TR1 into the world coordinate system. The trailer tracking device 10 also calculates the center coordinate QW2 by converting the center coordinate QS2 of the front detection frame O2 of the trailer TR1 into the world coordinate system.

[0052] Although only one trailer, trailer TR1, is shown in FIG. 10, there may be trailers on both sides of trailer TR1. Furthermore, the center coordinates of the circumscribing detection frame and the front detection frame may also be determined for trailers not shown. When identifying the circumscribing of the trailer and the front of the trailer, trailer tracking device 10 may determine a combination to be identified from among combinations of the circumscribing of the trailer and the front of the trailer. For example, if trailer tracking device 10 detects one circumscribing of the trailer and three fronts of the trailer, it may identify the circumscribing of the trailer and the front of the trailer that make up one of the three combinations.

[0053] Specifically, the trailer tracking device 10 may identify the circumscribed detection frame and the front detection frame whose center coordinates are closest to each other. In the example of FIG. 10, the trailer tracking device 10 determines the center coordinate of the front detection frame whose distance is closest to the center coordinate QW1 of the circumscribed detection frame O1, and identifies the front detection frame and the circumscribed detection frame O1. The trailer tracking device 10 calculates that the distance D1 between the center coordinate QW1 and the center coordinate QW2 is the smallest. Then, the trailer tracking device 10 identifies the circumscribed detection frame O1 and the front detection frame O2. In other words, the trailer tracking device 10 identifies the circumscribed detection frame and the front of the trailer TR1.

[0054] For simplicity, FIG. 10 shows only one trailer, trailer TR1. However, in an actual yard, multiple trailers may be densely packed, and the captured image may also contain numerous circumscribing trailers (in other words, circumscribing detection frames and center coordinates of the circumscribing detection frames) and fronts of trailers (in other words, front detection frames and center coordinates of the front detection frames). If the trailer tracking device 10 selects these numerous objects as objects to be identified and determines the combination of circumscribing and fronts to be identified, efficiency may be poor in terms of processing load, time, and the like. Therefore, the trailer tracking device 10 may select objects to be identified based on predetermined conditions. The predetermined conditions for selecting objects to be identified by the trailer tracking device 10 will be described with reference to FIGS. 11, 12, 13, 14, and 15. Hereinafter, the predetermined conditions for selecting objects may be referred to as first conditions. The condition for identifying the selected object, in other words, the condition for identifying the combination of the circumscribing of the trailer and the front face of the trailer, is that, among the combinations of the circumscribing of the trailer and the front face of the trailer, the combination of the circumscribing of the trailer and the front face of the trailer is the combination whose center coordinates in the world coordinate system are closest to each other, as explained with reference to Fig. 10. Hereinafter, the condition for identifying (in other words, linking) the object may be referred to as the second condition.

[0055] FIG. 11 is a schematic diagram illustrating an area for selecting an identification target according to the first embodiment. As shown in FIG. 11, an area R1 for selecting an identification target may be set within a yard. Trailer detection using the front of the trailer is particularly useful in locations within the yard where multiple trailers are crowded together, such as near parking space PS. Therefore, area R1 for selecting an identification target may be set near parking space PS as shown in FIG. 11.

[0056] If the center coordinates of the trailer's circumscribing detection frame in the world coordinate system are within area R1, the trailer tracking device 10 selects the circumscribing of the trailer as the identification target. Also, if the center coordinates of the trailer's front detection frame in the world coordinate system are within area R1, the trailer tracking device 10 selects the front of the trailer as the identification target. Using trailer TR1 as an example, if trailer TR1 is at position P5, the trailer tracking device 10 selects the circumscribing and front of trailer TR1 as the identification target. However, if trailer TR1 is at position P6, the trailer tracking device 10 does not select the circumscribing and front of trailer TR1 as the identification target.

[0057] In this way, the first condition for selecting an identification target may include that the object detected by the trailer tracking device 10 is present within a specified area R1 in the yard.

[0058] Furthermore, even if an object exists within area R1, if the speed of the object is equal to or greater than a specified speed limit, trailer tracking device 10 does not need to select the object as an identification target. A specific example will be described with reference to Fig. 12. Fig. 12 is a schematic diagram for explaining the speed limit for selecting an identification target according to the first embodiment.

[0059] In the example of Fig. 12, trailer TR1 is traveling within area R1 at a speed of 10 km / h. The trailer tracking device 10 may treat the speed of trailer TR1 and the speed of each object included in trailer TR1 as the same speed. In other words, the trailer tracking device 10 may treat the speed of an object of trailer TR1 (for example, an object circumscribing trailer TR1) as 10 km / h. The trailer tracking device 10 may acquire the speed of the trailer using a sensor (not shown) or may calculate it based on a captured image.

[0060] For example, if the trailer tracking device 10 sets the speed limit for selecting an identification target to 5 km, the trailer tracking device 10 will not select an object of the trailer TR1 (e.g., a circumscription of the trailer TR1) traveling at a speed of 10 km / h as an identification target. In this way, the first condition for selecting an identification target may include that the speed of the object detected by the trailer tracking device 10 is less than the specified speed limit.

[0061] Furthermore, a first condition for selecting an identification target may include that the overlap rate of the detection frames circumscribing the trailer and in front of the trailer is equal to or greater than a specified threshold. The overlap rate and specific examples will be described with reference to Fig. 13 and Fig. 14. Fig. 13 is a schematic diagram for explaining limitations on the overlap rate for selecting an identification target according to the first embodiment. Fig. 14 is a table for explaining the overlap rate for selecting an identification target according to the first embodiment.

[0062] The following description focuses on the trailer TRJ and trailer TRK. The trailer tracking device 10 detects the circumscribing and front of the trailer TRJ, and generates a circumscribing detection frame O1J and a front detection frame O2J. The trailer tracking device 10 also detects the circumscribing and front of the trailer TRK, and generates a circumscribing detection frame O1K and a front detection frame O2K.

[0063] Table TBL5 shown in Figure 14 shows the area of ​​the overlapping region between the circumscribed detection frame O1J and the front detection frame O2J, the area of ​​the overlapping region between the circumscribed detection frame O1J and the front detection frame O2K, the area of ​​the overlapping region between the circumscribed detection frame O1K and the front detection frame O2J, and the area of ​​the overlapping region between the circumscribed detection frame O1K and the front detection frame O2K.

[0064] Table TBL6 shown in FIG. 14 shows the areas of the front detection frame O2J, the front detection frame O2K, the circumscribed detection frame O1J, and the circumscribed detection frame O1K.

[0065] The overlap rate between the circumscribing detection frame and the front detection frame is calculated by the following formula (3).

[0066]

number

[0067] According to the above formula (3), for example, the overlap rate between the circumscribed detection frame O1J and the front detection frame O2J is calculated to be 1.0. The overlap rate between the circumscribed detection frame O1J and the front detection frame O2K is calculated to be 0.0. The overlap rate between the circumscribed detection frame O1K and the front detection frame O2J is calculated to be 0.26. The overlap rate between the circumscribed detection frame O1K and the front detection frame O2K is calculated to be 0.94. Here, a predetermined threshold value for the overlap rate may be set in advance. If the predetermined threshold value for the overlap rate is, for example, 0.7, the trailer tracking device 10 may select, as an identification target, a combination of a circumscribed detection frame and a front detection frame having an overlap rate of 0.7 or more. In other words, if the predetermined threshold value for the overlap rate is, for example, 0.7, the trailer tracking device 10 excludes, from identification targets, a combination of a circumscribed detection frame and a front detection frame having an overlap rate of less than 0.7. In the example of Figure 13, the trailer tracking device 10 excludes the combination of the circumscribing detection frame O1K and the front detection frame O2J from the identification target. Also, the trailer tracking device 10 excludes the combination of the circumscribing detection frame O1J and the front detection frame O2K from the identification target. On the other hand, the trailer tracking device 10 may select the combination of the circumscribing detection frame O1J and the front detection frame O2J, and the combination of the circumscribing detection frame O1K and the front detection frame O2K, as the identification target. As a result, the circumscribing and the front of the same trailer are selected as the identification target.

[0068] Furthermore, a first condition for selecting an identification target may include that the distance between the front of the trailer and an end point of the traffic line circumscribing the trailer is less than a specified limit distance. This will be described with reference to Fig. 15. Fig. 15 is a schematic diagram for explaining the distance limit for selecting an identification target according to the first embodiment.

[0069] The flow line FL2 shown in Fig. 15 is the flow line of the trailer TRN. Here, the tracking of the trailer TRN is performed by detecting and tracking the circumscribing of the trailer TRN. Therefore, in Fig. 15, the trailer tracking device 10 treats the circumscribing flow line of the trailer TRN as the flow line FL2 of the trailer TRN.

[0070] Assume that the circumscribing detection frames of multiple trailers (e.g., trailers TRL and TRM) overlap with the circumscribing detection frame of trailer TRN near parking space PS, resulting in trailer TRN not being detected. Furthermore, assume that the flow line FL2 ends near parking space PS. In this case, it is desirable to link the flow line FL2 with trailer TRN.

[0071] The trailer tracking device 10 may select the front of the trailer whose distance from the end point EP of the flow line FL2 is less than a specified limit distance as the target for identification as a circumscribed part of the trailer TRN.

[0072] Alternatively, the trailer tracking device 10 may select the trailer front face that is closest to the end point EP of the flow line FL2 among the front faces of the trailers that are less than the specified limit distance as the target for identification with the circumscribed area of ​​the trailer TRN. For example, the front faces of the trailers that are less than the specified limit distance from the end point EP of the flow line FL2 are the front faces of the trailer TRN, the trailer TRM, and the trailer TRL. The distance D2 between the end point EP of the flow line FL2 and the front face of the trailer TRN is 1.1. The distance D3 between the end point EP of the flow line FL2 and the front face of the trailer TRM is 1.2, and the distance D4 between the end point EP of the flow line FL2 and the front face of the trailer TRL is 1.5. In this case, the trailer tracking device 10 may select the front face of the trailer TRN as the target for identification with the circumscribed area of ​​the trailer TRN.

[0073] [Object Tracking Processing] Next, referring to FIGS. 16 and 17, the object tracking process by the trailer following device 10 will be described. FIG. 16 is a flowchart of the object tracking process of the trailer following device 10 according to Embodiment 1. Each process in the flowchart of FIG. 16 will be described in conjunction with FIG. 17. FIG. 17 is a schematic diagram for explaining the object tracking of the trailer following device 10 according to Embodiment 1. When the camera CM generates an imaging image, it is assumed that the camera CM transmits the imaging image to the trailer following device 10. That is, it is assumed that the trailer following device 10 receives and acquires the imaging image from the camera CM in real time. Also, in the example of FIG. 17, it is assumed that the trailer following device 10 is following the object 50. In FIG. 17, the reference numerals with parentheses indicate the components of the reference numerals at the specific time described in the parentheses. That is, in FIG. 17, the object 50 at time t1, the object 50 at time t2, and the object 60 at time t2 are shown. Note that the magnitude relationship between t1 and t2 is t1 < t2. That is, time t2 is a time after time t1. The detection frame 51 is the detection frame of the object 50 at time t1. The movement line FL3 shown in FIG. 17 is the movement line of the object 50.

[0074] The trailer following device 10 detects an object from the acquired captured image (step S100). In the example of FIG. 17, the trailer following device 10 receives and acquires the captured image at time t2, and detects an object from the captured image. The trailer following device 10 detects the object 50 and the object 60, and generates a detection frame 52 of the object 50 and a detection frame 61 of the object 60. At this point, the trailer following device 10 does not treat the object 50 at time t2 and the object 50 at time t1, that is, the object 50 being followed, as the same object.

[0075] The trailer tracking device 10 predicts the position of the object being tracked (step S101). The trailer tracking device 10 may predict the position of the object being tracked based on the speed, acceleration, past position or most recent position of the object being tracked, etc. In the example of FIG. 17, the trailer tracking device 10 predicts the position of the object 50 being tracked at time t2. The trailer tracking device 10 generates a detection window corresponding to the predicted position of the object 50 at time t2. Hereinafter, the detection window corresponding to the predicted position of the object may be referred to as a prediction window. In the example of FIG. 17, the trailer tracking device 10 generates a prediction window 51A.

[0076] The trailer tracking device 10 calculates the IoU between the detection frame of the object detected in the process of step S100 and the prediction frame of the object whose position was predicted in the process of step S101 (step S102). In the example of Fig. 17, the trailer tracking device 10 calculates the IoU between the detection frame 52 and the prediction frame 51A. The trailer tracking device 10 also calculates the IoU between the detection frame 61 and the prediction frame 51A.

[0077] Based on the IoU calculated in step S102, the trailer tracking device 10 performs a process of linking the object detected in step S100 with the object being tracked (step S103). The trailer tracking device 10 may perform a process of linking the object detected in the process of step S100, which has the largest IoU between the detection window and the prediction window, with the object being tracked. Note that an IoU condition may be set for the process of linking the detected object with the object being tracked. For example, if the IoU between the detection window and the prediction window is less than a specified threshold, the linking process may not be performed even if the IoU between the detection window and the prediction window is the largest among the calculated IoUs, in order to avoid erroneous linking. In the example of FIG. 17, the trailer tracking device 10 performs a process of linking the object 50 detected at time t2 in the process of step S100 with the object 50 being tracked. As a result, the flow line FL3 is linked to the object 50 detected in the processing of step S100. The processing of linking the detected object with the object being tracked will also be described later in the explanation of step S108.

[0078] The trailer tracking device 10 determines whether or not there is any object that has not been linked in the processing of step S103 among the objects detected in the processing of step S100 (step S104). In the example of Fig. 17, object 60 has not been linked in the processing of step S103. If the trailer tracking device 10 determines that there is no object that has not been linked in the processing of step S103 among the objects detected in the processing of step S100 (step S104: NO), the trailer tracking device 10 proceeds to the processing of step S106.

[0079] If the trailer tracking device 10 determines that there is an object that was not linked in the processing of step S103 among the objects detected in the processing of step S100 (step S104: YES), it creates a flow line of the object (step S105). In the example of Fig. 17, the trailer tracking device 10 creates a flow line FL4 of the object 60.

[0080] The trailer tracking device 10 determines whether there is a flow line that has not been linked for a certain period of time (step S106). This certain period of time may be set in advance by, for example, a user. With reference to FIG. 17, an example has been described in which the flow line FL3 is linked to the object 50 at time t2, but there is a possibility that the flow line FL4 will not be linked for a certain period of time in the future.

[0081] If the trailer tracking device 10 determines that there is no flow line that has not been linked for a certain period of time (step S106: NO), the process proceeds to step S108.

[0082] If the trailer tracking device 10 determines that there is a traffic line that has not been linked for a certain period of time (step S106: YES), it discards the traffic line (step S107). This is to prevent an erroneously detected object from remaining as a detection result. For example, if the traffic line FL4 has not been linked for a certain period of time, the trailer tracking device 10 discards the traffic line FL4.

[0083] The trailer tracking device 10 outputs the tracking information of the object (step S108). The output destination of the tracking information of the object may be, for example, the management system 20, the display device 14, or a user terminal (not shown). The tracking information of the object may include, for example, the following information (1) to (12). (1)tm: time (2) id: Traffic flow ID (3) u: The u coordinate of the center of the bounding box in the screen coordinate system (4) v: v coordinate of the center of the bounding box in the screen coordinate system (5) w: The width of the bounding box in screen coordinates. (6) h: Height of the bounding box in screen coordinates (7) x: x coordinate in the world coordinate system (8) y: y coordinate in the world coordinate system (9) cl: Class ID of the object (10) vx: velocity in the x-axis direction in the world coordinate system (11) vy: Velocity in the y-axis direction in the world coordinate system (12) bn: 1 (linked) / 0 (unlinked)

[0084] In the tracking information, "tm" indicates the time at which the tracking information is recorded.

[0085] In addition, "id" in the tracking information indicates identification information for identifying a flow line.

[0086] In addition, "u" and "v" in the tracking information together indicate the center coordinates of the bounding box of the object in the screen coordinate system.

[0087] In addition, "w" in the tracking information indicates the width of the bounding box of the object in the screen coordinate system. The width of the bounding box in the screen coordinate system may be, for example, the length of the bounding box in the u-axis direction in the screen coordinate system.

[0088] In addition, "h" in the tracking information indicates the height of the bounding box of the object in the screen coordinate system. The height of the bounding box in the screen coordinate system may be, for example, the length of the bounding box in the v-axis direction in the screen coordinate system.

[0089] In addition, "x" in the tracking information indicates the x coordinate of the object in the world coordinate system. Note that "x" may also indicate the x coordinate of the center of the bounding box of the object in the world coordinate system.

[0090] In addition, "y" in the tracking information indicates the y coordinate of the object in the world coordinate system. Note that "y" may also indicate the y coordinate of the center of the bounding box of the object in the world coordinate system.

[0091] In addition, "cl" in the tracking information, that is, the class ID of the object, indicates the type of object, such as the circumscription of the trailer or the front of the trailer.

[0092] In addition, "vx" in the tracking information indicates the velocity of the object in the x-axis direction in the world coordinate system.

[0093] In addition, "vy" in the tracking information indicates the velocity of the object in the y-axis direction in the world coordinate system.

[0094] Furthermore, "bn" in the tracking information indicates whether the linking process of step S103 has been performed for the flow line indicated by the flow line ID included in the tracking information. For example, in the tracking information of object 50 at time t2 shown in FIG. 17, bn is 1, but in the tracking information of object 60 at time t2, bn is 0.

[0095] After processing step S108, the trailer tracking device 10 ends this processing flow. Note that in the explanation with reference to Figures 16 and 17, it is assumed that the trailer tracking device 10 is tracking the object 50. If there is no object being tracked by the trailer tracking device 10, the trailer tracking device 10 may proceed to processing step S105 after processing step S100.

[0096] For example, the trailer tracking device 10 performs object tracking processing on the circumscribed and frontal surfaces of a trailer that has entered a yard. Assume that the trailer drives to a parking space within the yard and stops there. The trailer tracking device 10 tracks the trailer by performing object tracking processing on the circumscribed surface of the trailer and creates a circumscribed flow line of the trailer. The circumscribed flow line of the trailer may be treated as the flow line of the trailer. As described with reference to FIG. 4, even if object tracking processing is performed on the frontal surface of the trailer, the trailer tracking device 10 may not be able to detect the frontal surface of the trailer, and an accurate flow line of the trailer may not be created. Therefore, the trailer tracking device 10 tracks the trailer using the circumscribed surface of the trailer. However, as described with reference to FIG. 6, it may be difficult to detect a specific trailer in an area where multiple trailers are densely packed. To detect a specific trailer in an area where multiple trailers are densely packed, it is effective to use the frontal surface of the trailer, as described with reference to FIG. 7. The trailer tracking device 10 seamlessly tracks the trailer within the yard by identifying the circumscribing direction and the front of the trailer. A specific processing flow will be described with reference to Figs.

[0097] [Processing flow] Fig. 18 is a flowchart of the overall processing of the trailer tracking device 10 according to embodiment 1. In the explanation of Fig. 18, an example is assumed in which the trailer tracking device 10 tracks a trailer that enters a yard, drives to a parking space within the yard, and stops in the parking space.

[0098] The trailer tracking device 10 receives and acquires the captured image from the camera CM (step S200). For convenience of explanation, Fig. 18 shows that the trailer tracking device 10 acquires the captured image in step S200, but the timing at which the trailer tracking device 10 receives and acquires the captured image is not limited to the timing of the processing in step S200. For example, the trailer tracking device 10 may be able to constantly receive and acquire the captured image in real time while the camera CM is capturing images inside the yard.

[0099] The trailer tracking device 10 detects the circumscribing edge of the trailer and the front of the trailer from the captured image acquired in the processing of step S200 (step S201). Note that it is assumed that the captured image acquired by the trailer tracking device 10 in step S200 shows the circumscribing edge of the trailer and the front of the trailer.

[0100] The trailer tracking device 10 may detect the circumscribing edges of multiple trailers and the front of multiple trailers.

[0101] The trailer tracking device 10 tracks the circumscribing of the trailer detected in the processing of step S201 (step S202). The trailer tracking device 10 also tracks the front of the trailer detected in the processing of step S201 (step S203). The trailer tracking device 10 performs the object tracking processing described with reference to FIGS. 16 and 17 on each of the circumscribing of the trailer and the front of the trailer detected in the processing of step S201. In this way, the trailer tracking device 10 creates a circumscribing flow line of the trailer and a front-of-the-trailer flow line. For ease of explanation, the circumscribing flow line of the trailer shows the continuous movement of the trailer from when it enters the yard until it drives near the parking space. The trailer tracking device 10 may treat the circumscribing flow line of the trailer as the movement line of the trailer. It is also assumed that the front-of-the-trailer flow line shows the continuous movement of the trailer at least near the parking space and in the parking space.

[0102] The trailer tracking device 10 selects the circumscribing object and the front of the trailer to be identified from the detected circumscribing object and the front of the trailer (step S204). Note that the object detected by the trailer tracking device 10 refers to an object that appears in the captured image and for which a detection frame has been generated. Details of the processing of step S204 will be described with reference to FIG. 19.

[0103] 19 is a flowchart of the identification target selection process of the trailer tracking device 10 according to embodiment 1. The trailer tracking device 10 performs each process included in this processing flow for each of one or more circumscribing faces and front faces of trailers detected by the trailer tracking device 10. This allows the trailer tracking device 10 to narrow down the circumscribing faces and front faces of trailers to be identified, even when there are many circumscribing faces and front faces of trailers that have been detected.

[0104] The trailer tracking device 10 determines whether the center coordinates in the world coordinate system of the detection frame of the detected trailer object are within an area for selecting an object to be identified (step S300). For ease of explanation, the circumscribing part of the trailer and the front of the trailer are collectively referred to as the object. The area for selecting an object to be identified is, for example, area R1 described with reference to FIG. 11.

[0105] If the trailer tracking device 10 determines that the center coordinates in the world coordinate system of the detection frame of the trailer object are not within the area for selecting an identification target (step S300: NO), it does not select the object as an identification target and terminates this processing flow for the object.

[0106] If the trailer tracking device 10 determines that the center coordinates in the world coordinate system of the detection frame of the trailer object are within the area for selecting the identification target (step S300: YES), it proceeds to step S301 for processing the object.

[0107] The trailer tracking device 10 determines whether the speed of the trailer object is less than a specified speed limit (step S301).

[0108] If the trailer tracking device 10 determines that the speed of the trailer object is not less than the specified speed limit, i.e., is greater than or equal to the specified speed limit (step S301: NO), it does not select the object as an identification target and terminates this processing flow for the object.

[0109] When the trailer tracking device 10 determines that the speed of the trailer object is less than the specified speed limit (step S301: YES), the process for the object proceeds to step S302.

[0110] In step S302, the trailer tracking device 10 performs a determination process for each combination of the circumscribing of the trailer and the front of the trailer. For example, if there is one circumscribing of the trailer to be processed and five fronts of the trailer, the trailer tracking device 10 performs a determination process for each of the five combinations.

[0111] The trailer tracking device 10 determines whether or not the overlap rate of the detection frames of the circumscribing area of ​​the trailer and the front of the trailer is equal to or greater than a threshold value (step S302).

[0112] If the trailer tracking device 10 determines that the overlap rate of the detection frames of the circumscribing perimeter of the trailer and the front of the trailer is less than the threshold (step S302: NO), it does not select the combination of the circumscribing perimeter of the trailer and the front of the trailer as an identification target, and ends this processing flow for that combination. Note that even if the combination of the circumscribing perimeter of the trailer and the front of the trailer is not selected as an identification target, for example, another combination including the circumscribing perimeter of the trailer or the front of the trailer may be selected as an identification target.

[0113] If the trailer tracking device 10 determines that the overlap rate of the detection frames of the circumscribed area of ​​the trailer and the front of the trailer is greater than or equal to a threshold value (step S302: YES), it executes the processing of step S303 for the combination of the circumscribed area of ​​the trailer and the front of the trailer.

[0114] In step S303, the trailer tracking device 10 performs a determination process for each combination of the circumscribed area of ​​the trailer and the front of the trailer. For example, if the determination process is performed for five combinations in step S302 and the five combinations are narrowed down to three, the determination process in step S303 is performed for each of the three combinations.

[0115] The trailer tracking device 10 determines whether the distance between the front of the trailer and the end point of the trailer's flow line is less than a specified limit distance (step S303). For example, when the processing of step S303 is performed for a certain combination of the circumscribing edge of the trailer and the front of the trailer, the trailer tracking device 10 treats the circumscribing edge of the trailer as the flow line of the trailer. The trailer tracking device 10 then determines whether the distance between the end point of the trailer's flow line and the front of the trailer is less than a specified limit distance.

[0116] If the trailer tracking device 10 determines that the distance between the front of the trailer and the end point of the trailer's traffic line is not less than the specified limit distance, i.e., is greater than or equal to the specified limit distance (step S303: NO), it does not select the combination of the front of the trailer and the circumscribed part of the trailer linked to the traffic line as an identification target, and terminates this processing flow for that combination.

[0117] If the trailer tracking device 10 determines that the distance between the front of the trailer and the end point of the trailer's traffic line is less than the specified limit distance (step S303: YES), it selects the combination of the front of the trailer and the circumscribed trailer associated with the traffic line as an identification target (step S304).Then, the trailer tracking device 10 proceeds to step S205 of the flowchart shown in FIG.

[0118] The trailer tracking device 10 identifies the circumscribing center coordinate of the trailer and the front side of the trailer that constitutes the combination in which the distance between the circumscribing center coordinate of the trailer and the front side center coordinate of the trailer in the world coordinate system is the shortest among the combinations selected as the identification target in step S304 (step S205). The circumscribing center coordinate of the trailer in the world coordinate system is, more precisely, the center coordinate of the circumscribing center frame in the screen coordinate system that has been coordinate-transformed into the world coordinate system. The center coordinate of the front side of the trailer in the world coordinate system is, more precisely, the center coordinate of the front side detection frame in the screen coordinate system that has been coordinate-transformed into the world coordinate system.

[0119] The trailer tracking device 10 updates the identified circumscribing or front-of-trailer flow line ID (step S206). Specifically, the trailer tracking device 10 replaces the front-of-trailer flow line ID with the circumscribing flow line ID of the trailer. Alternatively, the trailer tracking device 10 replaces the circumscribing flow line ID of the trailer with the front-of-trailer flow line ID. This allows the trailer tracking device 10 to treat the circumscribing and front-of-trailer as the circumscribing and front-of-trailer, respectively, of a single trailer.

[0120] The trailer tracking device 10 outputs tracking information of the object (step S207). The output destination of the tracking information of the object may be, for example, the management system 20, the display device 14, or a user terminal (not shown), etc. The tracking information of the object may include tracking information of the circumscribed area of ​​the trailer and tracking information of the front of the trailer. The tracking information of the circumscribed area of ​​the trailer and the tracking information of the front of the trailer include the same flow line ID. The tracking information of the circumscribed area of ​​the trailer and the tracking information of the front of the trailer may be output together as tracking information of the trailer.

[0121] In this manner, the trailer tracking device 10 may receive and acquire captured images including one or more trailers captured by a camera CM installed to capture an image of a yard into which a trailer containing multiple objects enters. The trailer tracking device 10 may also detect multiple objects from the acquired captured images and perform object tracking processing on each of the detected multiple objects. The trailer tracking device 10 can create a movement line for each of the multiple objects through the object tracking processing. The trailer tracking device 10 may also select multiple objects from the multiple objects that satisfy a first condition for selecting the objects. As a result, for example, one or more circumscribing edges of the trailer and one or more front faces of the trailer are selected. In other words, one or more combinations of the circumscribing edges of the trailer and the front faces of the trailer are selected. The trailer tracking device 10 may then link the selected multiple objects, in other words, the multiple objects that satisfy the first condition, that satisfy a second condition for linking the objects, to each other. As a result, for example, the circumscribing edges of the trailer and the front faces of the trailer are linked. The trailer tracking device 10 may update the flow lines of one or more objects linked to a specific object based on the flow line of the specific object among multiple objects linked to each other. For example, the trailer tracking device 10 may use the flow line ID of the trailer's circumscribing to replace the flow line ID of the front of the trailer linked to the trailer's circumscribing. This allows the trailer tracking device 10 to track a trailer using the trailer's circumscribing, and even if the trailer goes undetected near a parking space, for example, it can link the front of the trailer detected near the parking space to the trailer's circumscribing. This allows the trailer to be tracked seamlessly within the yard.

[0122] (Summary of the first embodiment) The above description of the first embodiment discloses at least the following techniques. Note that the components corresponding to the first embodiment are shown in parentheses, but the present invention is not limited to these.

[0123] (Technology 1) The trailer tracking method detects multiple objects from an image including one or more trailers captured by a camera (e.g., camera CM) that is installed so as to be able to capture images of a yard into which a trailer containing multiple objects (e.g., trailer TR1) enters, performs object tracking on each of the detected multiple objects, creates a movement line (e.g., movement line FL1) for each of the multiple objects, selects multiple objects from the multiple objects that satisfy a first condition for selecting the objects, links multiple objects from the selected multiple objects that satisfy a second condition for linking the objects to each other, and updates the movement lines of one or more objects linked to any one specific object from the multiple objects that are linked to the specific object.

[0124] This allows the trailer tracking method to track trailers without interruption even when multiple trailers are crowded together in a yard.

[0125] (Technology 2) In the trailer tracking method described in Technology 1, if an object is not present within a specified area in a yard set for selection, the trailer tracking method may determine that the object does not satisfy the first condition and exclude the object from selection.

[0126] This allows the trailer tracking method to select objects that are present within a specified area.

[0127] (Technology 3) In the trailer tracking method according to Technique 1 or 2, when the speed of the object is equal to or greater than a specified speed limit, the trailer tracking method may determine that the object does not satisfy the first condition and exclude the object from selection.

[0128] This allows the trailer tracking method to select objects whose speed is below the specified speed limit.

[0129] (Technology 4) In the trailer tracking method described in any one of techniques 1 to 3, the multiple objects include the circumscribing of the trailer and the front of the trailer, and if the overlap rate of the bounding boxes of the circumscribing of the trailer and the front of the trailer is less than a specified threshold, the trailer tracking method may determine that the combination of the circumscribing of the trailer and the front of the trailer does not satisfy the first condition, and exclude the combination from selection.

[0130] This allows the trailer tracking method to track the trailer using both the circumscribed and frontal surfaces of the trailer. Also, the trailer tracking method can select, for example, a combination of the circumscribed and frontal surfaces of the trailer that overlap by a certain amount or more in the captured image.

[0131] (Technology 5) In the trailer tracking method described in Technology 4, if the distance between the front of the trailer and the end point of the trailer's circumscribed movement line is equal to or greater than a specified limit distance, the trailer tracking method may determine that the combination of the front of the trailer and the circumscribed movement line of the trailer does not satisfy the first condition, and may exclude the combination from selection.

[0132] This makes it possible for the trailer tracking method to select, for example, a combination of the front of a trailer detected at a position close to where the circumscribing traffic line of the trailer ends and the circumscribing line of the trailer.

[0133] (Technology 6) In the trailer tracking method described in Technology 5, the trailer tracking method may determine that, among the selected combinations of the circumscribing of the trailer and the front face of the trailer, the combination of the circumscribing of the trailer and the front face of the trailer that has the closest distance between the center coordinate of the circumscribing of the trailer and the center coordinate of the front face of the trailer in a coordinate system corresponding to the map within the yard satisfies the second condition.

[0134] This allows the trailer tracking method to link the circumscribed area of ​​the trailer with the closest center coordinates to the front of the trailer in a coordinate system corresponding to the map within the yard.

[0135] (Technology 7) In the trailer tracking methods described in Techniques 1 to 3, the multiple objects may include the circumscribed area of ​​the trailer, the front of the trailer, the top surface of the trailer, and the side surface of the trailer.

[0136] This allows the trailer tracking method to track the trailer using each of the circumscribing surface of the trailer, the front surface of the trailer, the top surface of the trailer, and the side surface of the trailer.

[0137] (Technology 8) A trailer tracking device (e.g., trailer tracking device 10) includes a processor (e.g., processor 11) and a memory (e.g., memory 12), and the processor and memory work together to detect multiple objects from an image including one or more trailers captured by a camera (e.g., camera CM) that is configured to capture images of a yard into which a trailer including multiple objects (e.g., trailer TR1) enters, and perform object tracking on each of the detected multiple objects to create a movement line (e.g., movement line FL1) for each of the multiple objects, select multiple objects from the multiple objects that satisfy a first condition for object selection, link multiple objects from the selected multiple objects that satisfy a second condition for linking the objects to each other, and update the movement lines of one or more objects linked to any one specific object from the multiple objects that are linked to that specific object based on the movement line of the specific object.

[0138] This allows the trailer tracking device to obtain the same effect as that of Technology 1.

[0139] (Technology 9) A trailer tracking system (e.g., trailer tracking system 1) includes a processor (e.g., processor 11), a memory (e.g., memory 12), and a camera (e.g., camera CM) that is capable of capturing images of a yard into which a trailer containing multiple objects (e.g., trailer TR1) enters. The processor and memory work together to detect multiple objects from an image captured by the camera that includes one or more trailers, and perform object tracking on each of the detected multiple objects to create a movement line (e.g., movement line FL1) for each of the multiple objects. Among the multiple objects, multiple objects that satisfy a first condition for object selection are selected, and among the selected multiple objects, multiple objects that satisfy a second condition for linking the objects are linked to each other. Based on the movement line of any one specific object among the multiple linked objects, the system updates the movement lines of one or more objects linked to the specific object.

[0140] This allows the trailer tracking system to achieve the same effect as Technology 1.

[0141] The functions of the above-described embodiments can also be realized by supplying programs and applications for realizing the functions of the above-described embodiments to a system or device using a network or storage medium, etc., and having one or more processors in the computer of the system or device read and execute the programs.

[0142] Furthermore, the functions of the above-described embodiments may be realized by a circuit that realizes one or more functions (for example, an Application Specific Integrated Circuit (hereinafter referred to as "ASIC") or an FPGA).

[0143] Although the embodiments of the present disclosure have been described above with reference to the drawings, it goes without saying that the present disclosure is not limited to such examples. It is clear that a person skilled in the art can conceive of various modifications, alterations, substitutions, additions, deletions, and equivalents within the scope of the claims, and it is understood that these also naturally fall within the technical scope of the present disclosure. Furthermore, the components of the above-described embodiments may be combined in any manner as long as they do not deviate from the spirit of the invention. [Industrial Applicability]

[0144] The present disclosure is useful as a trailer tracking method, a trailer tracking device, and a trailer tracking system. [Explanation of symbols]

[0145] 1 Trailer Tracking System 10 Trailer Tracking Device 11 processors 12 Memory 13 Communication interface device 14 Display device 20 Management System 50,60 objects 51, 52, 61 Detection frame 51A Prediction Frame TR1,TRA,TRB,TRC,TRD,TRE,TRF,TRG,TRH,TRI,TRJ,TRK,TRL,TRM,TRN Trailer O1, O1a, O1b, O1H, O1I, O1J, O1K Circumference detection frame O2, O2H, O2I, O2J, O2K Front detection frame O3 Upper detection frame O4 Side detection frame QS1,QS2,QW1,QW2 Center coordinates R1 Area Commercial camera FL1,FL2,FL3,FL4 Flow line EP endpoint PS Parking Space P1,P2,P3,P4,P5,P6 position NW1,NW2 network

Claims

1. Detecting the plurality of objects from a captured image including one or more trailers captured by a camera installed so as to be able to capture an image of a yard into which a trailer including the plurality of objects enters; performing object tracking for each of the detected objects to generate a movement line for each of the detected objects; selecting a plurality of objects that satisfy a first condition for selecting the objects from the plurality of objects; linking a plurality of objects that satisfy a second condition for linking the objects among the selected plurality of objects to each other; updating the flow lines of one or more objects linked to any one specific object among the plurality of objects linked to one another, based on the flow line of the specific object; Trailer tracking method.

2. if the object is not present within a specified area in the yard set for the selection, determining that the object does not satisfy the first condition, and excluding the object from the selection; 2. The trailer tracking method of claim 1.

3. If the speed of the object is equal to or greater than a specified speed limit, it is determined that the object does not satisfy the first condition, and the object is excluded from the selection.

3. The trailer tracking method according to claim 2.

4. the plurality of objects includes a circumscription of the trailer and a front of the trailer; If the overlap rate of the bounding boxes of the circumscribing box of the trailer and the front of the trailer is less than a specified threshold, it is determined that the combination of the circumscribing box of the trailer and the front of the trailer does not satisfy the first condition, and the combination is excluded from the selection.

4. The trailer tracking method according to claim 3.

5. If the distance between the front of the trailer and the end point of the circumscribing traffic line of the trailer is equal to or greater than a specified limit distance, it is determined that the combination of the front of the trailer and the circumscribing traffic line of the trailer does not satisfy the first condition, and the combination is excluded from the selection.

5. The trailer tracking method according to claim 4.

6. Among the selected combinations of the circumscribing of the trailer and the front face of the trailer, the combination of the circumscribing of the trailer and the front face of the trailer having the closest distance between the center coordinate of the circumscribing of the trailer and the center coordinate of the front face of the trailer in a coordinate system corresponding to the map within the yard is determined to satisfy the second condition.

6. The trailer tracking method according to claim 5.

7. the plurality of objects include a circumscribed area of ​​the trailer, a front surface of the trailer, a top surface of the trailer, and a side surface of the trailer; 2. The trailer tracking method of claim 1.

8. a processor and a memory, The processor and the memory cooperate to: Detecting the plurality of objects from a captured image including one or more trailers captured by a camera installed so as to be able to capture an image of a yard into which a trailer including the plurality of objects enters; performing object tracking for each of the detected objects to generate a movement line for each of the detected objects; selecting a plurality of objects that satisfy a first condition for selecting the objects from the plurality of objects; linking a plurality of objects that satisfy a second condition for linking the objects among the selected plurality of objects to each other; updating the flow lines of one or more objects linked to any one specific object among the plurality of objects linked to one another, based on the flow line of the specific object; Trailer tracking device.

9. a processor, a memory, and a camera that is provided to be able to capture an image of a yard into which a trailer including a plurality of objects enters; The processor and the memory cooperate to: Detecting the plurality of objects from a captured image including one or more of the trailers captured by the camera; performing object tracking for each of the detected objects to generate a movement line for each of the detected objects; selecting a plurality of objects that satisfy a first condition for selecting the objects from the plurality of objects; linking a plurality of objects that satisfy a second condition for linking the objects among the selected plurality of objects to each other; updating the flow lines of one or more objects linked to any one specific object among the plurality of objects linked to one another, based on the flow line of the specific object; Trailer tracking system.

Images