Object-tracking device, object-tracking system, and object-tracking method

Abstract

This object-tracking device acquires first detection information detected by a first detection device that detects an object in a first predetermined region and second detection information detected by a second detection device that detects an object in a second predetermined region; acquires a first feature amount based on the first detection information and a second feature amount based on the second detection information; and if it is determined, on the basis of the first feature amount and the second feature amount, that the object detected by the first detection device and the object detected by the second detection device are the same, estimates and outputs the position information of the object, which is based on the first detection information and the second detection information.

Description

Object tracking device, object tracking system, and object tracking method

[0001] The present disclosure relates to an object tracking device, an object tracking system, and an object tracking method.

[0002] Patent Literature 1 discloses a moving object tracking device including one or more target area cameras that capture images of a target area, a recognition unit that recognizes a unique identification code displayed on a moving object moving within the target area based on an image captured by the target area camera, an image position acquisition unit that acquires the position of the moving object within the target area based on the captured image, and a position processing unit that determines the identity of the moving object based on the recognized identification code and records changes in the position of the moving object over time in a recording unit. The target area cameras include, for example, an entry camera that captures images upon entry into the target area. The moving object tracking device tracks the behavior of a worker wearing a helmet with an identification code attached at the work site, using a work site as the target area and a worker as the moving object as an example.

[0003] Japanese Patent Application Publication No. 2020-98590

[0004] In Patent Document 1, it is assumed that the moving object whose change in position over time is to be recorded is a human (a worker moving within a work site). Therefore, it is easy to track the change in the worker's position over time. However, if the moving object whose change in position over time is to be recorded is something other than a human (for example, an object such as a load), tracking may be difficult.

[0005] For example, consider a case where the moving object whose position changes over time is a trailer within a large business premises (hereinafter sometimes referred to as a "yard") of a transportation company or the like, where tractors (so-called transport trucks) towing trailers (an example of a towed object) loaded with a large amount of cargo are gathered. In this case, the transport truck (i.e., the tractor) enters the yard (checks in) and parks the trailer in a designated parking space within the yard. Subsequently, a yard worker (hereinafter sometimes referred to as a "yard jockey"), who is different from the tractor driver, may get into the same tractor, tow another designated tractor, and then leave the yard (check out). When the driver and the yard jockey are different people, the parking space where the driver thinks the trailer is parked may not match the parking space the yard jockey is heading to. In other words, there is a problem in that the yard jockey may be unable to find the desired trailer even when getting into the tractor and heading to the designated parking space. Particularly when the yard has a large area, it is often difficult for the yard jockey to visually find the desired trailer. Therefore, there is a need for a technology that can estimate the parking position of each trailer parked in such a yard.

[0006] The present disclosure has been devised in view of the above-described conventional circumstances, and aims to provide an object tracking device, an object tracking system, and an object tracking method that more efficiently manage the positions of objects moving within a specified area.

[0007] The present disclosure provides an object tracking device including: a communication unit that acquires first detection information detected by at least one first detection device that detects an object in a first predetermined area, and second detection information detected by at least one second detection device that detects the object in a second predetermined area using a detection method different from that of the first detection device; a feature extraction unit that acquires first features of the object based on the first detection information, and acquires second features of the object based on the second detection information; a determination unit that determines whether the object detected by the first detection device and the second detection device are the same based on the first features and the second features; and an estimation unit that, when it is determined that the object detected by the first detection device and the second detection device are the same, estimates and outputs position information of the object based on the first detection information and the second detection information.

[0008] The present disclosure also provides an object tracking system including at least one first detection device that detects an object in a first predetermined area, at least one second detection device that detects the object in a second predetermined area using a detection method different from that of the first detection device, and at least one device communicably connected between the first detection device and the second detection device, wherein the device acquires first detection information detected by the first detection device and second detection information detected by the second detection device, acquires first feature amounts of the object based on the first detection information, acquires second feature amounts of the object based on the second detection information, determines whether the objects detected by the first detection device and the second detection device are the same based on the first feature amounts and the second feature amounts, and if it is determined that the objects detected by the first detection device and the second detection device are the same, estimates and outputs position information of the object based on the first detection information and the second detection information.

[0009] The present disclosure also provides an object tracking method performed by at least one processor, which acquires first detection information detected by at least one first detection device that detects an object in a first predetermined area, and second detection information detected by at least one second detection device that detects the object in a second predetermined area using a detection method different from that of the first detection device, acquires first feature amounts of the object based on the first detection information, acquires second feature amounts of the object based on the second detection information, determines whether the objects detected by the first detection device and the second detection device are the same based on the first feature amounts and the second feature amounts, and if it is determined that the objects detected by the first detection device and the second detection device are the same, estimates and outputs position information of the object based on the first detection information and the second detection information.

[0010] According to the present disclosure, the positions of moving objects within a given area can be managed more efficiently.

[0011] FIG. 1 is a top view showing an example of the arrangement of cameras and sensors according to the present embodiment; FIG. 2 is a diagram showing an example of a connection graph for cameras and sensors; FIG. 3 is a block diagram showing an example of a system configuration of an object tracking system according to an embodiment; FIG. 4 is a flowchart showing an example of the overall procedure of an object tracking system according to an embodiment; FIG. 5 is a flowchart showing an example of an object tracking procedure using a camera in an object tracking system according to an embodiment; FIG. 6 is a flowchart showing an example of a stopping position determination process using a camera in an object tracking system according to an embodiment; FIG. 7 is a diagram explaining an example of conversion for converting detection data from a camera into common data;

[0012] Hereinafter, with reference to the accompanying drawings as appropriate, detailed descriptions of embodiments specifically disclosing an object tracking system, an object tracking device, and an object tracking method according to the present disclosure will be provided. However, unnecessary detailed descriptions may be omitted. For example, detailed descriptions of well-known matters or redundant descriptions of substantially identical configurations may be omitted. This is to avoid unnecessary redundancy in the following description and to facilitate understanding by those skilled in the art. Note that the accompanying drawings and the following description are provided to enable those skilled in the art to fully understand the present disclosure, and are not intended to limit the subject matter recited in the claims.

[0013] First, the terms used in the following embodiments are defined. (1) Trailer: A container-like object having a rectangular parallelepiped casing loaded with one or more cargo items and towed by a tractor. (2) Tractor: A vehicle part of a transport truck that can be docked (coupled) to a trailer or detached from the trailer. (3) Yard: A large office or site of a transport company or the like where tractors (so-called transport trucks) towing trailers are accumulated. (4) Yard jockey: A worker who performs yard work such as driving a tractor within the yard to dock a desired trailer in a designated parking space to the tractor and move it to the dock. Note that the yard jockey may also perform work such as removing and carrying cargo from a trailer moved to the dock. In this embodiment, the yard jockey and the driver of the transport truck that checks in to the yard or the driver of the transport truck that checks out to the yard are treated as different people.

[0014] Possible targets to which the invention in the present disclosure is applicable include moving objects such as vehicles, people, and aircraft. However, in the present disclosure, examples of moving targets such as trailers, tractors, and a combination of a trailer and a tractor will be described as examples. Furthermore, the invention in the present disclosure can identify whether the type of target is a trailer, a tractor, or a combination of a trailer and a tractor. Therefore, to make the explanation of the invention easier to understand, the following explanation will focus on an example in which the target is a trailer, but this does not exclude tractors or a combination of a trailer and a tractor from the list of targets.

[0015] First, referring to Figure 1, an example of a yard YRD in which an object tracking system 100 (see Figure 2) according to an embodiment is installed, and cameras C1, C2, C3, C4, C5, C6, C7, and C8 and sensors S1, S2, S3, S4, S5, S6, S7, and S8 installed in the yard YRD will be described. Figure 1 is a top view showing an example of the arrangement of cameras C1 to C8 and sensors S1 to S8 in this embodiment. The yard YRD shown in Figure 1 is shown as viewed from above. Note that the configuration of the yard YRD and the number and arrangement of cameras shown in Figure 1 are merely examples and are not limited to these.

[0016] The yard YRD is a vast site owned by a business such as a transportation company. Specifically, the yard YRD includes within its site a gate GT for carrying out procedures for checking in (entering) and checking out (exiting) trailers TR into the yard YRD, at least one parking area with a plurality of parking spaces, and a warehouse WH where trailers TR are loaded or unloaded. It goes without saying that the yard YRD shown in FIG. 1 is merely an example and is not limited to this.

[0017] The gate GT is used by trailers TR entering the yard YRD and trailers TR leaving the yard YRD. At least one camera C1 is installed at the gate GT to capture images of trailers TR entering and leaving the yard YRD.

[0018] Each of the plurality of parking areas has parking spaces for parking one or more trailers TR.

[0019] The warehouse stores cargo unloaded from the trailer TR or cargo to be loaded onto the trailer TR. The warehouse includes at least one dock (not shown) for loading or unloading cargo to or from the trailer TR.

[0020] Each of the cameras C2 to C8 is installed within the yard YRD and captures images of angles of view ARC2, ARC3, ARC4, ARC5, ARC6, ARC7, and ARC8. Each of the cameras C2 to C8 captures, for example, an image of a road within the yard YRD, a road fork, or a part of a parking area.

[0021] Each of the sensors S1 to S8 has a detection area ARS1, ARS2, ARS3, ARS4, ARS5, ARS6, ARS7, and ARS8 that detects the passage of a trailer TR. Each of the sensors S1 to S8 is installed, for example, so that it can detect trailers TR outside the viewing angles ARC1 to ARC8 of the cameras C1 to C8, respectively. This allows the sensors S1 to S8 to detect trailers TR in areas that cannot be monitored by the cameras C1 to C8 alone, thereby collecting information for more accurately estimating the positions of multiple trailers TR moving within the vast yard YRD. The detection areas ARS1 to ARS8 of the sensors S1 to S8 may partially overlap with the viewing angles ARC1 to ARC8 of the cameras C1 to C8.

[0022] Next, the connection graph CNMP of the cameras C1 to C8 and the sensors S1 to S8 will be described with reference to Fig. 2. Fig. 2 is a diagram showing an example of the connection graph CNMP of the cameras C1 to C8 and the sensors S1 to S8.

[0023] The connection graph CNMP is data showing the relationship between the positions and distances of the cameras C1 to C8 and the sensors S1 to S8 installed in the yard YRD. The connection graph CNMP is used to track trailers TR detected by the cameras C1 to C8 or the sensors S1 to S8, and to estimate the parking areas of the trailers TR.

[0024] In the connection graph CNMP shown in Fig. 2, icons "C1" to "C8" represent cameras C1 to C8, and icons "S1" to "S8" represent sensors S1 to S8, as shown in Fig. 1. In the connection graph CNMP, the connections between icons indicate the positional relationship of the cameras C1 to C8 and sensors S1 to S8 installed on the roads within the yard YRD.

[0025] 1, for example, in the yard YRD, a camera C1 is installed at the gate GT, a camera C2 is installed at the branching point of the road where the trailer TR entering through the gate GT will move, and sensors S1, S3, and S5 are installed on each of the three roads beyond this branching point. The connection graph CNMP indicates which camera C1 and which sensor S1, S3, and S5 are installed adjacent to the camera C2 by connecting the camera C1 and the sensors S1, S3, and S5 to the camera C2. The connection graph CNMP also records numerical values ​​indicating the distances between the camera C2 and the camera C1 and the sensors S1, S3, and S5, corresponding to the connection lines connecting the camera C2 and the camera C1 and the sensors S1, S3, and S5.

[0026] 2 is the end of the monitoring network for the trailer TR, which is constructed by the cameras C1 to C8 and the sensors S1 to S8. The connection graph CNMP shows the connection destination of one end (end side) of the camera C8, which is the end of the monitoring network, with a loop line. The connection graph CNMP also records the round-trip distance between the camera C8 and the end of the road, in association with the loop line.

[0027] Next, an example of the system configuration of the object tracking system 100 will be described with reference to Fig. 3. Fig. 3 is a block diagram showing an example of the system configuration of the object tracking system 100 according to the embodiment.

[0028] The object tracking system 100 includes at least one camera C1 to C8, at least one sensor S1 to S8, and a server SB. The object tracking system 100 detects a trailer TR using the cameras C1 to C8 and the sensors S1 to S8. Note that, as an example, the object tracking system 100 in the present disclosure includes eight cameras C1 to C8 and eight sensors S1 to S8 as shown in FIGS. 1 and 2, but is not limited to this.

[0029] The server SB includes a communication unit 10, a processor 11, a memory 12, and a monitor 13. The server SB may be realized by, for example, a personal computer (hereinafter referred to as "PC"), a notebook PC, a tablet terminal, or the like.

[0030] The communication unit 10 receives various data or information transmitted from the cameras C1 to C8 and the sensors S1 to S8 via a network (not shown), and outputs the data or information to the processor 11.

[0031] The network NW here refers to a wired network, a wireless network, or a combination of a wired network and a wireless network. The wired network corresponds to at least one of a wired local area network (hereinafter referred to as "LAN"), a wired wide area network (hereinafter referred to as "WAN"), and a power line communication (PLC), and may also be other network configurations capable of wired communication. On the other hand, the wireless network corresponds to at least one of a wireless LAN such as Wi-Fi (registered trademark), a wireless WAN, and a mobile communication network such as 4G or 5G, and may also be other network configurations capable of wireless communication.

[0032] The processor 11 is configured using, for example, a central processing unit (hereinafter referred to as "CPU"), a digital signal processor (hereinafter referred to as "DSP"), or a field programmable gate array (hereinafter referred to as "FPGA"). The processor 11 performs various processes and controls in cooperation with the memory 12. Specifically, the processor 11 references programs and data stored in the memory 12 and executes the programs to realize various functions such as a feature extraction unit 111, an evaluation unit 112, an estimation unit 113, and an output unit 114.

[0033] The feature extraction unit 111 detects the detection target, a trailer TR, a tractor, or a tractor towing a trailer TR, from the captured images captured by the multiple cameras C1 to C8. For simplicity's sake, this disclosure will describe an example in which a trailer TR is detected as the detection target. The feature extraction unit 111 extracts and acquires feature values ​​that indicate the individuality of the detected trailer TR.

[0034] The evaluation unit 112 performs a matching process of the features acquired based on the images captured by different cameras, i.e., a trailer TR identification process, based on the features of the trailer TR extracted by the feature extraction unit 111. The evaluation unit 112 evaluates the similarity of the features of the trailers detected by the different cameras by matching the features. The evaluation unit 112 outputs the evaluation result to the estimation unit 113.

[0035] The estimation unit 113 estimates whether or not the trailer TR detected using one of the cameras or sensors is the same trailer as a trailer detected using another camera or another sensor, using the evaluation results based on the matching of the feature amounts output from the evaluation unit 112, or the detection information detected by the cameras C1 to C8 or the sensors S1 to S8, etc. If the estimation unit 113 estimates that the trailer TR detected using one of the cameras or sensors is the same trailer as a trailer detected using another camera or another sensor, the estimation unit 113 estimates position information such as the movement path, stopping position (stopping area), or parking space of the trailer TR based on the detection information of the trailer TR detected by the other camera or another sensor.

[0036] The output unit 114 outputs information about the parking area or parking space for the trailer TR estimated by the estimation unit 113 to the monitor 13. As shown in FIG. 1 , the output unit 114 may generate a screen in which each of the estimated movement routes TR11, TR12, TR13, TR14, and TR15 of the trailer TR is superimposed on a map of the yard YRD, and output the screen to the monitor 13. The movement route displayed on this screen may be any section of the movement route, such as the movement route from the gate GT to the stopping position, the movement route between the camera or sensor that last detected the trailer TR and the stopping position, or the movement route from the camera that last captured an image of the trailer TR to the stopping position.

[0037] The memory 12 includes a read-only memory (hereinafter referred to as "ROM") and a random access memory (hereinafter referred to as "RAM"). The ROM stores programs that define the processing (operations) of the processor 11, as well as data referenced when the programs are executed. The RAM is a work memory used when the processor 11 executes its processing (operations), and temporarily stores data or information generated or acquired during each process. The memory 12 also stores map data (see Figure 1) of the yard YRD.

[0038] The monitor 13 is configured using, for example, a Liquid Crystal Display (LCD) or an organic electroluminescence (EL). The monitor 13 outputs a screen including the position information of the trailers TR output from the output unit 114. The screen output here may be generated by superimposing the movement route of each trailer TR, the stopping area, parking area, or parking space on a map of the yard YRD.

[0039] Next, the overall procedure for tracking a trailer TR using multiple cameras C1 to C8 and multiple sensors S1 to S8 will be described with reference to Fig. 4. Fig. 4 is a flowchart showing an example of the overall procedure of the object tracking system 100 according to the embodiment.

[0040] In the following description, the camera identification information "cam_id" values ​​1 to 8 correspond to the cameras C1 to C8, respectively. Similarly, the sensor identification information "sens_id" values ​​1 to 8 correspond to the sensors S1 to S8, respectively.

[0041] The server SB acquires data of images captured by a camera C1, which is a reference camera provided at the gate GT and captures all trailers TR entering the yard YRD (St1).

[0042] The server SB executes a single camera feature acquisition process to acquire feature amounts of the trailer TR that has entered the yard YRD based on the data of the image captured by the camera C1 (reference camera) (St2). The single camera feature acquisition process in step St2 will be described with reference to Fig. 5. The server SB acquires a representative feature amount ret = (cam_id, local_id, Stop_flag, feature, track[ ]) of the trailer TR through the single camera feature acquisition process.

[0043] The feature amounts of the trailer TR acquired here include the identification information of the camera C1 "cam_id", the trailer identification information for identifying the trailer detected by the camera C1 "local_id", "Stop_flag" indicating whether the trailer TR is stopped, the feature amount of the trailer TR "feature", and the detection information of the trailer TR "track[ ]". Note that the information (data) included in the representative feature amount described above is an example and is not limited to this.

[0044] The server SB converts the trailer TR identification information "local_id" used to identify the trailer detected by camera C1 from the acquired representative feature data into trailer TR identification information "global_id" that is used in common by all cameras C1 to C8 and sensors S1 to S8 and is used to identify all trailers TR entering the yard YRD (St3).

[0045] Based on the connection graph CNMP of Figure 2, the server SB identifies adjacent cameras located adjacent to the camera that detected the trailer TR from which the representative feature was acquired, on the route on which the trailer TR can move (St4).

[0046] The server SB acquires distance information to the identified adjacent camera based on the connection graph CNMP, and calculates a search time, which is the time when the trailer TR whose representative feature was acquired by the identified adjacent camera is likely to have been captured, for searching whether the trailer TR has been detected by the adjacent camera. Based on the calculated search time, the server SB determines from which time (or time period) of the captured video image captured by the adjacent camera to search for the trailer TR, and acquires the captured video image from the determined time (time period) (St5).

[0047] For example, the server SB calculates a search time for the trailer TR to be imaged (detected) by the second camera based on the image capture time of the trailer TR captured by the first camera, the moving speed of the trailer TR based on the image capture times of the multiple captured images captured by the first camera and the position of the trailer TR, and the distance between the first camera and the second camera adjacent to the first camera. The server SB acquires captured video data from the second camera for a certain period of time centered on the calculated search time (for example, a time period of 3 minutes before and after the search time).

[0048] The server SB executes a single-camera feature acquisition process using each of the captured images included in the captured video acquired from the adjacent cameras (St6). The single-camera feature acquisition process in step St2 will be described with reference to FIG. 5.

[0049] The server SB determines whether or not the single camera feature acquisition process for this trailer TR has been completed for all the adjacent cameras (St7).

[0050] In step St7, if the server SB determines that the single camera feature acquisition process for this trailer TR has not been completed for all adjacent cameras (St7, NO), the server SB returns to step St5.

[0051] On the other hand, if the server SB determines in step St7 that the single camera feature acquisition process for this trailer TR has been completed in all adjacent cameras (St7, YES), it determines whether there is any detection information (data) that indicates that any of the trailers has been detected in all adjacent cameras (St8).

[0052] If the server SB determines in step St8 that there is no detection information (data) indicating that any trailer is detected by any of the adjacent cameras (St8, YES), the server SB executes a process for determining the stopping position of the trailer TR using the sensors S1 to S8 (St9). The process for determining the stopping position using the sensors S1 to S8 will be described later with reference to FIG. 8.

[0053] If the server SB determines in step St8 that at least one adjacent camera has detected a trailer (NO in St8), it performs feature matching between the detection information (data) including the representative feature of the trailer detected by the adjacent camera and the representative feature of the trailer TR currently being searched (St10). Through the feature matching process, the server SB generates a representative feature ret=(global_id, (cam_id, Stop_flag, feature, track[ ])[ ]) that is common to all cameras C1 to C8 and sensors S1 to S8 (St10).

[0054] In the feature matching process of step St10, the server SB estimates whether or not a stop flag "Stop_flag" indicating that no match has been found or that the trailer TR is stopped is set (Stop_flag=true) (St11).

[0055] If the server SB determines in step St11 that there is no match or that the stop flag is set (YES in St11), it executes a process for determining the stop position of the trailer TR using the cameras C1 to C8 (St12). The process for determining the stop position using the cameras C1 to C8 will be described later with reference to FIG. 6.

[0056] On the other hand, if the server SB determines in step St11 that a match has been found and the stop flag is not set (NO in St11), the process returns to step St4. In this case, the server SB identifies the "adjacent camera" of the camera that was previously the "adjacent camera" in step St4.

[0057] Next, the single camera feature acquisition process will be described with reference to Fig. 5. Fig. 5 is a flowchart showing an example of an object tracking procedure using cameras C1 to C8 in object tracking system 100 according to the embodiment.

[0058] The server SB acquires an image captured by a camera (St21). The server SB determines whether or not a trailer TR (target) is detected from the acquired image (St22).

[0059] If the server SB determines in step St22 that the trailer TR (target) has not been detected from the captured image (St22, NO), it determines whether there is a next captured image captured after the current captured image (St23).

[0060] If the server SB determines in step St23 that there is a next captured image (St23, NO), the process returns to step St21. On the other hand, if the server SB determines in step St23 that there is no next captured image (St23, YES), the server SB determines that the trailer TR (object) has not been detected by this camera (None) (St24).

[0061] On the other hand, if the server SB determines in step St22 that a trailer TR (object) has been detected (St22, YES), it acquires detection information "track (fr, xp, yp, wp, hp, cls)" including information such as the position coordinates, size, time information (frame number), and class information of a bounding box (detection frame) surrounding the trailer TR (object) in the captured image (St25). Note that the class information here indicates the type of the detected object. The class information indicates, for example, whether the detected object is a trailer TR alone, a tractor alone, or a trailer TR and a tractor connected together.

[0062] The server SB acquires the feature "feat" of the trailer TR (object) included in the detection frame (Step 26). The server SB associates the trailer TR's identification information "local_id" with the trailer TR's detection information "track (fr, xp, yp, wp, hp, cls)" (Step 27). Here, a single camera tracking technique (various filters) may be used.

[0063] The server SB determines whether or not there is a next captured image captured after the current captured image (St28).

[0064] If the server SB determines in step St28 that there is a next captured image (YES in St28), the server SB acquires the next captured image (St29). The server SB determines whether a trailer TR (object) is detected in the next captured image (St30).

[0065] If the server SB determines that the trailer TR (object) is detected in the next captured image (St30, YES), the server SB returns to step St25. On the other hand, if the server SB determines that the trailer TR (object) is not detected in the next captured image (St30, NO), the server SB determines that the trailer TR is detached from the tractor and is stopped alone, sets the stop flag indicating that the trailer TR (object) is stopped to Stop_flag = True (St31), and generates a representative feature of the trailer TR (St33).

[0066] On the other hand, if the server SB determines in step St28 that there is no next captured image (St30, NO), it determines whether or not a trailer TR (object) has been detected from the edge area of ​​the camera's field of view (St32).

[0067] If the server SB determines in step St32 that the trailer TR is not detected from the edge area (St32, NO), it determines that the trailer TR has stopped within the field of view of this camera, proceeds to step St31, and sets a stop flag.

[0068] On the other hand, if the server SB determines in step St32 that the trailer TR has been detected from the edge area (YES in St32), it determines that the trailer TR (object) has passed through the field of view (area) captured by the camera, and generates a representative feature of the trailer TR (St33). The generated representative feature is stored (memorized) in the memory 12.

[0069] Next, a stop position determination process using cameras C1 to C8 will be described with reference to Fig. 6. Fig. 6 is a flowchart showing an example of a stop position determination process procedure using cameras C1 to C8 in object tracking system 100 according to an embodiment. Fig. 7 is a diagram explaining an example of conversion of detection data from cameras C1 to C8 into common data.

[0070] The server SB acquires the latest detection data (representative feature amount) of the trailer TR generated in step St10 (St41). Based on the acquired latest detection data (representative feature amount), the server SB determines whether a stop flag is set for the trailer TR (St42).

[0071] When the server SB determines that the stop flag for the trailer TR is set (YES in St42), it determines the stop position (absolute position) of the trailer TR based on the position of the trailer TR captured in the image captured by the camera (St43). Note that the server SB uses conversion parameters for each camera to convert the camera-specific detection information (data) into detection information (shared data) common to all cameras C1 to C8 and sensors S1 to S8.

[0072] Specifically, the server SB converts the frame number indicating the image capture time into real-time information, and converts the camera-specific detection information (data) expressed in pixels into world coordinates set on the yard YRD. When converting the frame number into real-time information, the server SB calculates the real-time of each captured image (frame) based on the real-time of the reference captured image (frame). Furthermore, the server SB converts the camera-specific data expressed in pixels (i.e., coordinates on the angle of view of the camera alone) into world coordinates set on the yard YRD based on the position information of each camera.

[0073] On the other hand, when the server SB determines that the stop flag is not set for the trailer TR (St42, NO), it executes a stop position determination process using the sensors S1 to S8 (St44).

[0074] Next, a stop position determination process using sensors S1 to S8 will be described with reference to Fig. 8. Fig. 8 is a flowchart showing an example of an object tracking procedure using sensors S1 to S8 in object tracking system 100 according to an embodiment.

[0075] The server SB refers to the connection graph CNMP in FIG. 2 to search for an adjacent sensor of the camera that last captured (detected) the trailer TR (object) (St51), and determines whether or not there is an adjacent sensor (St52).

[0076] If the server SB determines in step St52 that there is no adjacent sensor (St52, NO), it determines that the trailer TR (object) is parked in the parking area between the camera or sensor that last detected the trailer TR and its adjacent sensor (St53).

[0077] If the server SB determines in step St52 that there is an adjacent sensor (St52, YES), the server SB acquires detection information (data) of the trailer TR (object) within a certain time period based on the detection information from the adjacent sensor (St54). Based on the acquired detection information (data), the server SB executes a detection determination as to whether the trailer TR (object) has been detected (St54).

[0078] The certain period of time referred to here can be calculated from the distance between the camera or sensor that last detected the trailer TR and an adjacent sensor, or from the estimated average speed of the trailer TR (object) based on previous detection information of the trailer TR (the position of the detected camera or sensor, or the detection time, etc.).

[0079] Here, when a trailer is detected by sensors S1 to S8, it is more difficult to identify an individual trailer TR (object) than when it is detected by cameras C1 to C8. Therefore, when there are detection results of one or more trailers (objects) detected by one or more adjacent sensors within a certain period of time, server SB executes a narrowing-down process to identify the detection information of the trailer that is most likely to be the trailer TR most recently detected by a camera or sensor among the detection results of these trailers (objects) (St55).

[0080] The server SB determines through a narrowing-down process whether there is detection information for a trailer identical to the last detected trailer TR, i.e., whether a trailer identical to the last detected trailer TR has been detected from an adjacent sensor (St56).

[0081] If the server SB determines in step St56 that the same trailer as the last detected trailer TR is not detected by an adjacent sensor (St56, NO), it determines that the trailer TR (object) is parked in the parking area between the camera or sensor that last detected the trailer TR (object) and the adjacent sensor identified in step St52 (St57).

[0082] On the other hand, if the server SB determines in step St56 that the same trailer TR as the last detected trailer TR has been detected by the adjacent sensor (St56, YES), it further determines whether the trailer TR has passed the adjacent sensor (St58).

[0083] If the server SB determines in step St58 that the trailer TR has not passed the adjacent sensor (St58, NO), it determines that the trailer TR (object) is stopped in a parking area within the detection area of ​​the adjacent sensor (St59).

[0084] On the other hand, if the server SB determines in step St58 that the trailer TR has passed the adjacent sensor (St58, YES), the process returns to step St51.

[0085] Next, feature matching using captured images captured by cameras C1 to C8 will be described with reference to FIG. 9. FIG. 9 is a diagram illustrating an example of feature matching processing based on captured images. Note that the matching table TB11 shown in FIG. 9 is provided to clearly explain the feature matching processing and is not an essential component. Server SB does not need to generate the matching table TB11 in the feature matching processing or execute the feature matching processing using the matching table TB11.

[0086] Matching table TB11 includes captured images of five trailers taken by a camera assigned identification information "cam_id=1" and captured images of five trailers taken by a camera assigned identification information "cam_id=2." Server SB matches representative feature amounts obtained from each of the trailer images taken by the camera assigned identification information "cam_id=1" with representative feature amounts obtained from each of the trailer images taken during a certain period of time (time period) by a camera assigned identification information "cam_id=2," and calculates the similarity between these trailers.

[0087] The matching table TB11 stores the similarity of the representative features of trailers captured by different cameras through feature matching, i.e., the likelihood that a trailer captured by a camera with the identification information "cam_id=2" is the same as a trailer captured by a camera with the identification information "cam_id=1." The server SB determines that trailers with the calculated similarity equal to or greater than a predetermined value (e.g., 0.9), or trailers with the highest similarity equal to or greater than the predetermined value, are the same trailer.

[0088] For example, in the example shown in Figure 9, the server SB determines that the trailer shown in the captured image with the identification information "cam_id = 1" and "local_id = 0" is the same as the trailer shown in the captured image with the identification information "cam_id = 2" and "local_id = 0" for which a similarity of "0.928" has been calculated.

[0089] Next, a tracking data structure of single camera feature amounts acquired by one camera will be described with reference to Fig. 10. Fig. 10 is a diagram showing an example of the tracking data structure.

[0090] The server SB generates detection information "track (frame, xp, yp, wp, hp, cls) [ ]" and feature "feature" of the trailer TR based on a plurality of captured images taken at different times by a single camera. The server SB performs an identification process of the trailer TR detected by the single camera and appearing in the plurality of captured images taken at different times based on the detection information and feature of the trailer TR. The server SB performs tracking of the trailer TR moving within the angle of view of the same camera based on the time-series identification process results.

[0091] The tracking data table TB12 shown in FIG. 10 stores various pieces of information (cam_id, local_id, Stop_flag, frame, xp, yp, wp, hp, cls, feature) that make up the representative feature ret=(cam_id, local_id, Stop_flag, feature, track[ ]) of the trailer TR, which allows the trailer TR to be identified and tracked within each single camera (single camera).

[0092] The server SB assigns tracking information indicating that the trailers are the same to tracking data that is determined to be the same trailer among the feature quantities of the trailer TR acquired by a single camera. For example, in the tracking data table TB12, if the server SB determines that the feature quantities "feat_c1l1f0" and "feat_c1l1f1" detected by a camera with camera identification information "cam_id=1" are the same trailer, the server SB assigns tracking information "track_c1l1". Also, if the server SB determines that the feature quantities "feat_c3l1f0" and "feat_c3l1f1" detected by a camera with camera identification information "cam_id=3" are the same trailer, the server SB assigns tracking information "track_c1l2". Here, the tracking information functions as tracking data for tracking trailers detected within a single camera.

[0093] The server SB calculates a representative feature of the trailer TR based on each of the multiple feature amounts to which the same tracking information is assigned. For example, the server SB generates a representative feature "feature_c1l1" based on the feature "feat_c1l1f0" and the feature "feat_c1l1f1" detected by a camera with camera identification information "cam_id=1." The server SB also generates a representative feature "feature_c3l1" based on the feature "feat_c3l1f0" and the feature "feat_c3l1f1" detected by a camera with camera identification information "cam_id=3."

[0094] Next, an example of estimating a stopping area for the trailer TR will be described with reference to Fig. 11. Fig. 11 is a diagram for explaining an example of estimating a stopping area.

[0095] The server SB estimates the stopping area of ​​the trailer TR entering the yard YRD based on the detection results of the trailer TR by each of the multiple cameras C2 to C8 or multiple sensors S1 to S8 installed within the yard YRD, and the positional relationship between the parking area within the yard YRD where the trailer TR can be parked and the field of view (position) of the multiple cameras C2 to C8 or the detection area (position) of the multiple sensors S1 to S8.

[0096] (i) In step St42, when the server SB determines that tracking of the trailer TR has been completed (i.e., that the trailer has stopped) within the angles of view ARC1 to ARC8 of any of the cameras C2 to C8, the server SB identifies a stopping area (parking space) corresponding to the position of the trailer TR detected within the angles of view of the cameras C2 to C8. Note that the absolute position of the stopped object in the stopping area (parking space) identified by the stopping area estimation process is acquired (step St43).

[0097] For example, as shown in Fig. 1, when the server SB determines that tracking of the trailer TR has been completed (i.e., that the trailer TR has stopped) within the angle of view ARC4 captured by the camera C4, the server SB determines that the trailer TR is stopped in the stopping area corresponding to the angle of view ARC4 of the camera C4. The process of determining the parking space will be described with reference to Fig. 18.

[0098] (ii) If the server SB determines in step St8 that the trailer TR, which is the detection target (tracking target), is not detected by the adjacent camera, and if the server SB determines in step St59 that a trailer estimated to be the same trailer as the trailer TR through the narrowing process has not passed through the detection area of ​​the adjacent sensor within a certain period of time (time period), the server SB estimates that the trailer TR is stopped within the detection area of ​​the adjacent sensor or in a stopping area adjacent to the detection area of ​​the adjacent sensor (step St59).

[0099] For example, as shown in Figure 1, if the server SB determines that the trailer TR is not detected by the adjacent camera (camera C3) after passing camera C2, and also determines that the trailer TR has not passed through the detection area of ​​the adjacent sensor (sensor S1) within a certain period of time (time period), it estimates that the trailer TR is stopped in the stopping area corresponding to the detection area ARS3 of this adjacent sensor.

[0100] (iii) If the server SB determines in step St8 that the trailer TR, which is the detection target (tracking target), is not detected by the adjacent camera, and determines in step St52 that there is no adjacent sensor adjacent to the camera that last detected the trailer TR, it estimates that the trailer TR is stopped in the stopping area between the camera that last detected the trailer TR and this adjacent camera (step St53).

[0101] For example, as shown in Figure 1, if the server SB determines that the trailer TR is not detected by an adjacent camera (camera C5) after passing camera C4 and also determines that there is no adjacent sensor adjacent to camera C5, it estimates that the trailer TR is stopped in the stopping area between camera C5 and this adjacent camera (camera C6).

[0102] (iv) If the server SB determines in step St8 that the trailer TR, which is the detection target (tracking target), is not detected by the adjacent camera, and if the server SB determines in step St56 that a trailer estimated to be the same trailer as the trailer TR through the narrowing process has not been detected in the detection area of ​​the adjacent sensor within a certain period of time (time period), it estimates that the trailer TR is stopped in the stopping area between the camera that last detected the trailer TR and this adjacent sensor (step St57).

[0103] For example, as shown in Figure 1, if the server SB determines that the trailer TR is not detected by the adjacent camera (camera C3) after passing camera C2, and also determines that the trailer TR is not detected in the detection area of ​​the adjacent sensors (sensors S3, S5) within a certain period of time (time period) based on the search time, it estimates that the trailer TR is stopped in the stopping area between camera C2 and sensors S3, S5.

[0104] (v) If the server SB determines in step St8 that the trailer TR, which is the detection target (tracking target), is not detected by the adjacent camera, and if the server SB determines in step St59 that a trailer estimated to be the same trailer as the trailer TR through the narrowing-down process has passed through the detection area of ​​the adjacent sensor within a certain period of time (time period), and then determines in step St52 that the trailer TR is not detected by an adjacent sensor adjacent to the sensor whose passage was detected, the server SB estimates that the trailer TR is stopped in a stopping area beyond the sensor whose last detection of the trailer TR was detected (step St53). Note that if the server SB determines in step St52 that the trailer TR is not detected by an adjacent sensor adjacent to the sensor whose passage was detected, the server SB may estimate the stopping area of ​​the trailer TR based on whether or not there is an adjacent camera adjacent to the sensor whose last detection of the trailer TR was detected and whether or not the adjacent camera detected (captured) the trailer TR.

[0105] For example, as shown in Figure 1, if the server SB determines that after passing camera C2, the trailer TR is not detected by the adjacent camera (camera C3), and after determining that the trailer TR has passed the detection area of ​​the adjacent sensor (sensor S5) within a certain time period (time period) based on the search time, the server SB estimates that the trailer TR is stopped in the stopping area beyond sensor S5.

[0106] Also, for example, as shown in Figure 1, if the server SB determines that the trailer TR is not detected by the adjacent camera (camera C8) after passing camera C7, and determines that the trailer TR has passed the detection area of ​​the adjacent sensor (sensor S8) within a certain time period (time period) based on the search time, and then determines that there is no adjacent sensor adjacent to sensor S5, the server SB may estimate the stopping area of ​​the trailer TR based on the adjacent camera (camera C8) adjacent to sensor S8 and whether or not the trailer TR is detected (imaged) by the adjacent camera (camera C8).

[0107] An example of estimating a stopping area of ​​the trailer TR outside the angles of view ARC1 to ARC8 of the cameras C1 to C8 will be described with reference to Fig. 12. Fig. 12 is a diagram illustrating an example of estimating a stopping area outside the angles of view ARC1 to ARC8 of the cameras C1 to C8.

[0108] 12 indicates the camera or sensor that last detected the trailer TR before stopping in the stopping area. Node (N-1) indicates the camera or sensor that is adjacent to the camera or sensor that last detected the trailer TR and that detected the trailer TR immediately before this camera or sensor. Also, in FIG. 12, "-" indicates that there is no camera or sensor that detected the trailer TR, or that this is the end of the road.

[0109] The server SB estimates the stopping area based on the camera or sensor corresponding to the node (N) and the camera or sensor corresponding to the node (N-1). Furthermore, if the passing direction (moving direction) of the trailer TR can be detected by the camera or sensor corresponding to the node (N), the server SB can estimate that the trailer TR is stopped in a stopping area that exists in the passing direction (moving direction) of the detected trailer TR.

[0110] For example, if node (N-1) does not exist and node (N) is camera (C), server SB estimates that the stopping area (thick line) in the passing direction of the trailer TR detected by the camera (C) corresponding to node (N) is the stopping area where the trailer TR is parked, based on the passing direction of the trailer TR.

[0111] Also, for example, if the server SB determines that node (N-1) and node (N) are cameras (C) and that the direction in which the trailer TR is passing cannot be detected, it estimates that the stopping area (thick line) that is outside the field of view of the camera (C) corresponding to node (N) and beyond the camera (C) corresponding to node (N) is the stopping area in which the trailer TR is stopped.

[0112] Also, for example, if node (N-1) does not exist and node (N) is sensor (S), server SB estimates that the stopping area (thick line) existing in the passing direction of the trailer TR detected by the sensor (S) corresponding to node (N) is the stopping area where the trailer TR is stopped, based on the passing direction of the trailer TR.

[0113] Also, for example, if the server SB determines that the node (N-1) is a camera (C) or a sensor (S), the node (N) is a sensor (S), and the direction in which the trailer TR is passing cannot be detected, it estimates that the stopping area (thick line) that exists beyond the sensor (S) is the stopping area in which the trailer TR is stopped.

[0114] In the above example, when the server SB can detect the passing direction of the trailer TR, it estimates the stopping position of the trailer TR using detection information from a camera or sensor located in the detected passing direction, but this is not limited to this. When the server SB cannot detect the passing direction of the trailer TR, it may estimate the movement path (passing direction) of the trailer TR based on the positional relationship of the camera or sensor that detected the trailer TR at each of node (N-1) and node (N), and estimate the stopping position of the trailer TR using detection information from a camera or sensor located in the estimated movement path.

[0115] Furthermore, if there are multiple estimated stopping areas, the server SB may output information on the multiple stopping areas. For example, if the node (N) is the sensor S6 and the node (N-1) is the camera C3, the server SB estimates and outputs the stopping areas of the trailer TR as the stopping areas between the sensors S4 and S6 and the stopping area between the sensors S5 and S6. In such a case, the server SB may also estimate the movement route of the trailer TR to each stopping area and output the multiple estimated movement routes in association with the multiple stopping areas.

[0116] Next, sensors in the present disclosure will be described with reference to Fig. 13 to Fig. 15. Fig. 13 is a diagram illustrating an example of the types and functions of sensors. Fig. 14 is a diagram illustrating an example of detection of a passing direction by a near-infrared sensor. Fig. 15 is a diagram illustrating an example of detection of an object by each of multiple sensors S1 to S3.

[0117] Although the sensor in the above description has been described as having a function of detecting whether an object has passed through the detection area of ​​the sensor, the present disclosure is not limited to this. Therefore, FIG. 13 describes examples of the types and functions of sensors of the present disclosure.

[0118] The sensor of the present disclosure may be realized by, for example, a reflective or blocking near-infrared sensor, a pyroelectric sensor, a vibration sensor, an ultrasonic sensor, a thermography sensor, a radar sensor, a Time of Flight (TOF) sensor, a Light Detection and Ranging (LiDAR) sensor, or a reflective or blocking paired near-infrared sensor.

[0119] The sensor of the present disclosure also has a passage detection function capable of detecting the passage of an object (target) within the detection area. The sensor is installed to be able to arbitrarily realize functions such as direction detection capable of detecting the passage direction of an object (target) passing through the detection area, retention detection capable of detecting an object (target) staying (stopping) within the detection area, or speed detection capable of detecting the moving speed of an object (target) passing through the detection area.

[0120] The combinations of sensor types and sensor functions shown in FIG. 13 are merely examples and are not limiting. Each type of sensor can be configured to realize or disable any desired detection function depending on the installation method or the combination of multiple sensors. Here, the congestion detection may detect "stay" when it is determined that a vehicle is not passing in the direction of passage travel. Furthermore, if the moving speed can be detected (calculated), the server SB may use the moving speed to calculate the search time for the same trailer TR (object) using another camera or another sensor.

[0121] For example, the sensor in the present disclosure may be configured to be able to detect passage, passage direction, and movement speed by using two near-infrared sensors in combination as shown in Fig. 14. Detection graphs Gp11 and Gp12 shown in Fig. 14 show the detection results of passage detection by "sensor 1" and "sensor 2" constituting the near-infrared sensor, respectively.

[0122] The pair of near-infrared sensors shown in Figure 14 are installed a predetermined distance apart, and calculate the average moving speed of the trailer TR (object) based on the time difference Δt1 between the passage detection timings detected by "sensor 1" and "sensor 2" and the distance between "sensor 1" and "sensor 2." Furthermore, the pair of near-infrared sensors determines whether the passage direction through the detection area is "moving from right to left" or "moving from left to right" based on the difference between the passage detection timings of "sensor 1" and "sensor 2."

[0123] Furthermore, for example, even if the sensors in the present disclosure only have a passage detection function, they can detect the presence of trailers TR detected by each sensor S1 to S3 or detect (calculate) their average moving speeds based on the detection results obtained at different times t.

[0124] The detection data table TB13 stores, for example, the detection results of sensors S1 to S3 that can detect passing direction, stay, and speed, as well as various information obtained from the detection results of sensors S1 to S3. The detection data table TB13 shown in Fig. 15 records the sensor ID (sens_id) assigned to each sensor, the passing detection result (det), the stay detection result (Stay), or the average moving speed result (vel) in sampling time units.

[0125] Server SB calculates the average moving speed "vel = 15" of trailers TR passing through the detection area of ​​sensor S3 based on the passage detection results of sensor S3 from time "00:01:36:200" to "00:01:36:800." Server SB also calculates the average moving speed "vel = 20" of trailers TR passing through the detection area of ​​sensor S2 based on the passage detection results of sensor S2 from time "00:01:36:400" to "00:01:36:800." Server SB also detects the retention of trailers TR based on the detection results of sensor S1 from time "00:01:37:000" to "00:01:37:200," and sets a retention flag.

[0126] Next, an example of detecting an object in an edge region of the angle of view will be described with reference to Fig. 16. Fig. 16 is a diagram showing an example of an edge region and an example of detecting the passing direction.

[0127] 16 is an image captured by camera C3 within the field of view ARC3. Server SB sets a mask area MSK within the field of view ARC3 of camera C3 that does not determine (detect) the direction in which the trailer TR passes, and edge areas AR11, AR12, and AR13 corresponding to directions in which the detection areas of other cameras or sensors are located. The mask area and edge areas are set in advance by a manager of the yard YRD, etc.

[0128] Server SB detects a trailer TR from image IMG1 captured by camera C3. If server SB determines that the center coordinate position of the detection frame FRM (bounding box) of the detected trailer TR is included in edge areas AR11 to AR13, server SB determines that the trailer TR has passed through the field of view ARC3 of camera C3. In the example shown in FIG. 16 , server SB determines that the center coordinate position of the detection frame FRM of the trailer TR is included in edge area AR11. In such a case, server SB determines that the trailer TR has passed in the direction in which sensor S2 is installed. This allows server SB to track the trailer TR based on the direction in which the trailer TR passed, even if the trailer TR has disappeared from the field of view ARC1 to ARC8 of cameras C1 to C8.

[0129] If the server SB determines that the position of the center coordinates of the detection frame FRM of the trailer TR is included in the edge area AR12, it determines that the trailer TR has passed in the direction where the sensor S4 or the sensor S6 is installed. If the server SB determines that the position of the center coordinates of the detection frame FRM of the trailer TR is included in the edge area AR13, it determines that the trailer TR has passed in the direction where the camera C4 is installed.

[0130] Next, an example of tracking a trailer TR (object) will be described with reference to Fig. 17. Fig. 17 is a diagram showing an example of tracking an object (trailer TR). Note that the detection data table TB14 shown in Fig. 17 is illustrated for explaining an example of tracking a trailer TR corresponding to a feature amount and estimating a stopping position of the trailer TR based on the feature amount commonized by assigning the trailer TR identification information "global_id", and is not essential to the object tracking system 100.

[0131] The server SB performs feature matching based on the feature amounts of trailers TR detected (imaged) by multiple different cameras. The server SB assigns identification information "global_id" to the feature amounts determined to be the same trailer TR, making it possible to identify each trailer TR entering the yard YRD. The server SB tracks trailers TR within the yard YRD by chronologically arranging detection information related to trailers TR having the same identification information "global_id."

[0132] For example, the server SB tracks the trailer TR based on the time-series tracking information "track_c1l1," "track_c2l1," and "track_c3l1." Based on the stop flag "Stop_flag=1" corresponding to the latest tracking information "track_c3l1," the server SB estimates that the trailer TR is stopped in the angle of view ARC3 of the camera C3.

[0133] Furthermore, for example, the server SB tracks the trailer TR based on the time-series tracking information “track_c112” and “track_c212.” Based on the latest tracking information “track_c212,” the server SB estimates that the trailer TR has passed the camera C2 but has not yet reached the camera C3.

[0134] Furthermore, for example, the server SB tracks the trailer TR based on the time-series tracking information “track_c1l3,” “track_c2l3,” “track_c3l3,” and “track_c4l3.” Based on the latest tracking information “track_c4l3,” the server SB estimates that the trailer TR has passed the camera C4 but has not yet reached the camera C5.

[0135] Next, an example of specifying a parking space or parking area for a trailer TR will be described with reference to Fig. 18 and Fig. 19. Fig. 18 is a diagram showing an example of defining a parking space in the stopping area determination process. Fig. 19 is a diagram showing an example of defining a parking area in the stopping area determination process. Note that the parking space table TB15 shown in Fig. 19 is illustrated to explain an example of specifying a parking space, and is not essential to the object tracking system 100.

[0136] Next, an example of specifying a parking space when a stopping area is determined will be described. In this disclosure, an example will be described in which identification information "lot_num" that can identify a parking space is assigned to each parking space in advance.

[0137] As shown in Figure 19, each of the multiple parking spaces is managed by associating the area information of the parking space "xc, yc, w, h, θ", the identification information of the parking area to which the parking space belongs "area_num", and the identification information assigned to the parking space "lot_num".

[0138] The area of ​​a parking space is defined as position information relative to the camera's angle of view. For example, parking space PAR305 with "lot_num" = 305 shown in Fig. 18 is defined as an area rotated by an angle θ around the xw axis (= 0 (zero)°) with the origin being the intersection of the xw axis and the yw axis, which are the coordinate axes of the world coordinate system set on the yard YRD, and with center coordinates (xc, yc) and a width x and height h.

[0139] The parking space table TB15 shown in Figure 19 stores the identification information of each parking space "lot_num", the area information of the parking space "xc, yc, w, h, θ", and the identification information of the parking area to which the parking space belongs "area_num", in association with each other.

[0140] If the stopping position of the trailer TR is within the camera's angle of view (detection pattern (i) in FIG. 11 ), the server SB determines whether the position information (xw, yw) included in the absolute position (World coordinates) of the trailer TR generated in step St43 is included in the area information "xc, yc, w, h, θ" of any parking space. If the server SB determines that the absolute position of the trailer TR is included in the area information "xc, yc, w, h, θ" of any parking space, it determines the stopping area of ​​the trailer TR based on the identification information "lot_num" of this parking space.

[0141] On the other hand, if the stopping position of the trailer TR is not within the camera's field of view (detection patterns (ii) to (v) in Figure 11), the server SB identifies one or more parking areas where the trailer TR is estimated to be stopping, or one or more parking spaces "lot_num" included in the parking area, based on the camera or sensor that last detected the trailer TR and the direction of movement of the trailer TR.

[0142] For example, in the example shown in Figure 19, if the trailer TR is not detected by sensor S7 after it was last detected (photographed) by camera C6, the server SB will assume that the trailer TR is parked in a parking space in parking area PAR4 other than the angle of view ARC6 of camera C6.

[0143] Furthermore, for example, if the vehicle is detected (photographed) by the camera C6 and then lastly detected by the sensor S7, the server SB will estimate that the vehicle is parked in one of the parking spaces included in the parking area PAR5.

[0144] (Additional Notes) The above description of each embodiment discloses the following techniques.

[0145] (Technology 1) A communication unit 10 that acquires first detection information detected by at least one first detection device (cameras C1 to C8) that detects an object (e.g., a trailer TR) in a first predetermined area (angles of view ARC1 to ARC8) and second detection information detected by at least one second detection device (sensors S1 to S8) that detects the object (trailer TR) in a second predetermined area (detection areas ARS1 to ARS8) using a detection method different from that used by the first detection device (cameras C1 to C8); a feature extraction unit 111 that acquires a first feature amount of the object (trailer TR) based on the first detection information and acquires a second feature amount of the object (trailer TR) based on the second detection information; an evaluation unit (112) that determines whether the object (trailer TR) detected by the first detection device (cameras C1 to C8) and the second detection device (sensors S1 to S8) is the same based on the first feature amount and the second feature amount; and an estimation unit (113, output unit 114) that estimates and outputs position information of the object (trailer TR) based on the first detection information and the second detection information when it is determined that the object (trailer TR) detected by the first detection device (cameras C1 to C8) and the second detection device (sensors S1 to S8) is the same. As a result, the server SB can perform an identification process between the trailer TR detected (imaged) by the cameras C1 to C8 and the trailer TR detected by the sensors S1 to S8 based on the detection information from the cameras C1 to C8, which can detect the trailer TR identifiably through image analysis, and the sensors S1 to S8, which only detect whether the trailer TR is present or not and have difficulty identifying the trailer TR itself. Therefore, the server SB can reduce the number of installed cameras and track the location information of the trailer TR. Furthermore, by tracking the location of the trailer TR, the server SB can assist in estimating and managing the stopping area where the trailer TR is stopped.

[0146] (Technology 2) The object tracking device (server SB) described in (Technology 1) is configured such that the first detection device is a camera C1 to C8, and the second detection device is a sensor S1 to S8 that detects the presence or absence of the target object (trailer TR). This allows the server SB to monitor and track trailers TR by combining cameras C1 to C8 with sensors S1 to S8 that are less expensive than the cameras C1 to C8. Therefore, the server SB can reduce the number of cameras installed and reduce the costs required for monitoring and tracking trailers TR.

[0147] (Technology 3) The object tracking device (server SB) according to (Technology 1), wherein the second detection device (sensors S1 to S8) is installed between the plurality of first detection devices (cameras C1 to C8). This allows the server SB to obtain approximate position information of the trailer TR between the plurality of cameras based on the presence or absence (detection result) of detection by the sensors S1 to S8.

[0148] (Technology 4) The object tracking device (server SB) according to (Technology 2) or (Technology 3), wherein the second detection device (sensors S1 to S8) is installed outside the first predetermined area (angles of view ARC1 to ARC8) captured by the first detection device (cameras C1 to C8). This allows the server SB to obtain approximate position information of the trailer TR outside the angles of view ARC1 to ARC8 of the cameras C1 to C8 based on whether or not the sensors S1 to S8 have detected it (detection results).

[0149] (Technology 5) The object tracking device (server SB) described in (Technology 1) further includes a storage unit (memory 12) that stores position information for the first detection device (cameras C1 to C8) and the second detection device (sensors S1 to S8) and multiple stopping areas where the object (trailer TR) can stop, wherein the determination unit (evaluation unit 112) estimates whether the object (trailer TR) is stopped based on the first detection information and the second detection information, and the estimation unit 113, if it is determined that the object (trailer TR) is stopped, estimates the stopping position of the object (trailer TR) based on the first detection information and the second detection information. This allows the server SB to estimate the stopping area (parking area) for the trailer TR and support management of the position information of trailer TR entering a yard YRD, etc.

[0150] (Technology 6) The object tracking device (server SB) according to (Technology 5), wherein each of the first detection information and the second detection information includes information on a detection time of the object (trailer TR), and the estimation unit 113 estimates a movement path of the object (trailer TR) between a plurality of the first detection devices (cameras C1 to C8) based on the second detection information. This allows the server SB to estimate the movement path of the trailer TR by tracking the position of the trailer TR along a time series of when the detection information detected by each of the cameras C1 to C8 or each of the sensors S1 to S8 was detected.

[0151] (Technology 7) The storage unit (memory 12) stores information about a route along which the target object (trailer TR) can move in association with position information of the first detection device (cameras C1 to C8) and the second detection device (sensors S1 to S8), and the determination unit (evaluation unit 112) determines an adjacent detection device among the first detection device (cameras C1 to C8) and the second detection device (sensors S1 to S8) that is adjacent to a detection device that detected the latest detection information of the target object (trailer TR), and estimates a movement route of the target object (trailer TR) when it is determined that the target object (trailer TR) detected by the detection device and the adjacent detection device are the same based on a feature amount of the target object (trailer TR) based on the latest detection information and a feature amount based on adjacent detection information detected by the adjacent detection device. This allows the server SB to track the trailer TR with higher accuracy by tracing the detection devices (each camera C1 to C8 or each sensor S1 to S8) that detected the trailer TR in chronological order, and to estimate the movement path of the trailer TR based on the position information of the detection device that detected the trailer TR.

[0152] (Technology 8) The determination unit (evaluation unit 112) determines a detection time period (i.e., a fixed time period with the search time as the center time) of the adjacent detection information used to determine whether the target (trailer TR) in the adjacent detection information is the same based on the distance between the detection device that detected the latest detection information and the adjacent detection device, and determines whether the target (trailer TR) is the same based on a feature amount of the target (trailer TR) based on the latest detection information and a feature amount based on the adjacent detection information detected during the detection time period. This enables the server SB to track the trailer TR with higher accuracy by chronologically tracing the detection devices (cameras C1 to C8 or sensors S1 to S8) that detected the trailer TR, and to estimate the movement path of the trailer TR based on the position information of the detection device that detected the trailer TR.

[0153] (Technology 9) The object tracking device (server SB) according to (Technology 7), wherein the determination unit (evaluation unit 112) determines that the object (trailer TR) is stopped between the detection device and the adjacent detection device when it determines that the object (trailer TR) has not been detected by any of the adjacent detection devices based on the feature amount of the object (trailer TR) based on the latest detection information and the feature amount based on the adjacent detection information detected by the adjacent detection device. This allows the server SB to estimate with high accuracy the stopping area of ​​the trailer TR by chronologically tracing the detection devices (each of the cameras C1 to C8 or each of the sensors S1 to S8) that detected the trailer TR, even if the trailer TR is no longer detected.

[0154] (Technology 10) The object tracking device (server SB) according to (Technology 5), wherein the storage unit (memory 12) stores an area map in which the first detection devices (cameras C1 to C8) and the second detection devices (sensors S1 to S8) are installed, and the estimation unit 113 generates and outputs an image showing a movement path of the target object (trailer TR) on the area map. This enables the server SB to visualize the movement path of the trailer TR to an administrator, thereby supporting the administrator's management of the trailer TR.

[0155] (Technology 11) An object tracking system 100 includes at least one first detection device (cameras C1 to C8) that detects an object (trailer TR) in a first predetermined area (angles of view ARC1 to ARC8), at least one second detection device (sensors S1 to S8) that detects the object (trailer TR) in a second predetermined area (detection areas ARS1 to ARS8) using a detection method different from that used by the first detection device (cameras C1 to C8), and at least one device (server SB) that is communicably connected between the first detection device (cameras C1 to C8) and the second detection device (sensors S1 to S8), wherein the device (server SB) acquires first detection information detected by the first detection device (cameras C1 to C8) and second detection information detected by the second detection device (sensors S1 to S8), The object tracking system 100 acquires a first feature amount of the object (trailer TR) based on the first detection information, acquires a second feature amount of the object (trailer TR) based on the second detection information, determines whether the object (trailer TR) detected by the first detection device (cameras C1 to C8) and the second detection device (sensors S1 to S8) are the same based on the first feature amount and the second feature amount, and if it is determined that the object (trailer TR) detected by the first detection device (cameras C1 to C8) and the second detection device (sensors S1 to S8) are the same, estimates and outputs position information of the object (trailer TR) based on the first detection information and the second detection information. As a result, the object tracking system 100 can perform an identification process between the trailer TR detected (imaged) by the cameras C1 to C8 and the trailer TR detected by the sensors S1 to S8 based on the detection information from the cameras C1 to C8, which can detect the trailer TR identifiably through image analysis, and the sensors S1 to S8, which only detect whether the trailer TR is present or not and have difficulty identifying the trailer TR itself. Therefore, the object tracking system 100 can reduce the number of installed cameras and track the position information of the trailer TR. Furthermore, by tracking the position of the trailer TR, the object tracking system 100 can assist in estimating and managing the stopping area where the trailer TR is stopped.

[0156] (Technology 12) An object tracking method performed by at least one processor 11, comprising: acquiring first detection information detected by at least one first detection device (cameras C1 to C8) that detects an object (trailer TR) in a first predetermined area (angles of view ARC1 to ARC8); and second detection information detected by at least one second detection device (sensors S1 to S8) that detects the object (trailer TR) in a second predetermined area (detection areas ARS1 to ARS8) using a detection method different from that used by the first detection device (cameras C1 to C8); acquiring a first feature amount of the object (trailer TR) based on the first detection information; acquiring a second feature amount of the object (trailer TR) based on the second detection information; an object tracking method for determining whether the object (trailer TR) detected by the first detection device (cameras C1 to C8) and the second detection device (sensors S1 to S8) is the same based on the first feature amount and the second feature amount, and if it is determined that the object (trailer TR) detected by the first detection device (cameras C1 to C8) and the second detection device (sensors S1 to S8) is the same, estimating and outputting position information of the object (trailer TR) based on the first detection information and the second detection information. This allows the server SB to perform an identification process between the trailer TR detected (imaged) by the cameras C1 to C8 and the trailer TR detected by the sensors S1 to S8 based on the detection information from the cameras C1 to C8 that can detect the trailer TR in an identifiable manner by image analysis and the sensors S1 to S8 that can only detect whether the trailer TR is present but have difficulty identifying the trailer TR itself. Therefore, the server SB can reduce the number of installed cameras and track the location information of the trailer TR. Furthermore, by tracking the location of the trailer TR, the server SB can support estimation and management of the stopping area where the trailer TR is stopped.

[0157] Although various embodiments have been described above with reference to the accompanying drawings, the present disclosure is not limited to such examples. It is clear that those skilled in the art can conceive of various modifications, alterations, substitutions, additions, deletions, and equivalents within the scope of the claims, and it is understood that these also fall within the technical scope of the present disclosure. Furthermore, the components of the various embodiments described above may be combined in any manner without departing from the spirit of the invention.

[0158] This application is based on a Japanese patent application (Patent Application No. 2024-110987) filed on July 10, 2024, the contents of which are incorporated herein by reference.

[0159] The present disclosure is useful for an object tracking system, an object tracking device, and an object tracking method that more efficiently manage the positions of objects moving within a predetermined area.

[0160] 100 Object tracking system 10 Communication unit 11 Processor 111 Feature extraction unit 112 Evaluation unit 113 Estimation unit 114 Output unit C1, C2, C3, C4, C5, C6, C7, C8 Camera GT Gate SB Server S1, S2, S3, S4, S5, S6, S7, S8 Sensor TR Trailer

Claims

1. An object tracking device comprising: a communication unit that acquires first detection information detected by at least one first detection device that detects an object in a first predetermined area, and second detection information detected by at least one second detection device that detects the object in a second predetermined area using a detection method different from that of the first detection device; a feature extraction unit that acquires a first feature of the object based on the first detection information, and acquires a second feature of the object based on the second detection information; a determination unit that determines whether the object detected by the first detection device and the second detection device are the same based on the first feature and the second feature; and an estimation unit that, when it is determined that the object detected by the first detection device and the second detection device are the same, estimates and outputs position information of the object based on the first detection information and the second detection information.

2. The object tracking device according to claim 1, wherein the first detection device is a camera, and the second detection device is a sensor that detects the presence or absence of the object.

3. The object tracking device according to claim 1, wherein the second detection device is installed between a plurality of the first detection devices.

4. The object tracking device according to claim 2 or 3, wherein the second detection device is installed outside the first predetermined area imaged by the first detection device.

5. An object tracking device as described in claim 1, further comprising a memory unit that stores position information of the first detection device and the second detection device, and a plurality of stopping areas in which the object can stop, wherein the determination unit estimates whether the object is stopped based on the first detection information and the second detection information, and when it is determined that the object is stopped, the estimation unit estimates the stopping position of the object based on the first detection information and the second detection information.

6. The object tracking device described in claim 5, wherein each of the first detection information and the second detection information includes information on the detection time of the object, and the estimation unit estimates the movement path of the object between multiple first detection devices based on the second detection information.

7. The object tracking device described in claim 5, wherein the memory unit stores information about a route along which the object can move in association with position information of the first detection device and the second detection device, and the determination unit determines which of the first and second detection devices is an adjacent detection device adjacent to the detection device that detected the latest detection information of the object, and estimates the movement route of the object if it determines that the object detected by the detection device and the adjacent detection device are the same based on features of the object based on the latest detection information and features based on adjacent detection information detected by the adjacent detection device.

8. The object tracking device described in claim 7, wherein the determination unit determines a detection time period of adjacent detection information to be used for determining whether the objects in the adjacent detection information are the same or not, based on the distance between the detection device that detected the latest detection information and the adjacent detection device, and determines whether the objects are the same or not based on the feature amount of the object based on the latest detection information and the feature amount based on the adjacent detection information detected during the detection time period.

9. The object tracking device described in claim 7, wherein the determination unit determines that the object is stopped between the detection device and the adjacent detection device when it determines that the object has not been detected by any of the adjacent detection devices based on the feature of the object based on the latest detection information and the feature based on the adjacent detection information detected by the adjacent detection device.

10. The object tracking device described in claim 5, wherein the memory unit stores an area map in which the first detection device and the second detection device are installed, and the estimation unit generates and outputs an image showing the movement path of the object on the area map.

11. An object tracking system comprising: at least one first detection device that detects an object in a first predetermined area; at least one second detection device that detects the object in a second predetermined area using a detection method different from that of the first detection device; and at least one device communicably connected between the first detection device and the second detection device, wherein the device acquires first detection information detected by the first detection device and second detection information detected by the second detection device; acquires a first feature amount of the object based on the first detection information, and acquires a second feature amount of the object based on the second detection information; determines whether the object detected by the first detection device and the second detection device are the same based on the first feature amount and the second feature amount; and if it is determined that the object detected by the first detection device and the second detection device are the same, estimates and outputs position information of the object based on the first detection information and the second detection information. Object tracking system.

12. An object tracking method performed by at least one processor, comprising: acquiring first detection information detected by at least one first detection device that detects an object in a first predetermined area, and second detection information detected by at least one second detection device that detects the object in a second predetermined area using a detection method different from that of the first detection device; acquiring a first feature amount of the object based on the first detection information; acquiring a second feature amount of the object based on the second detection information; determining whether the object detected by the first detection device and the second detection device are the same based on the first feature amount and the second feature amount; and if it is determined that the object detected by the first detection device and the second detection device are the same, estimating and outputting position information of the object based on the first detection information and the second detection information.

Images