Device, program, and method for tracking terminal based on communication compatibility of detected object

Abstract

To provide a technique for continuing tracking a tracked terminal even if occlusion occurs in the tracked terminal.SOLUTION: A terminal tracking device includes: a target detection information determination unit that detects a predetermined target and determines target position information; a terminal communication information determination unit that determines terminal communication information based on information from a communication interface; means that determines target communication information related to a communication state to be embodied if a target includes a terminal; and a tracking target determination and management unit that determines whether or not there is a correspondence relation between the terminal and the target based on the degree of accordance between the target position information and predicted target position information and / or the degree of correspondence between the terminal communication information and the target communication information. When occlusion is eliminated for at least the target, the tracking target determination and management unit makes a determination based on the degree of communication compatibility when the degree of communication compatibility is larger than a predetermined level or makes a determination by increasing a degree based on the degree of communication compatibility as the degree of communication compatibility is larger or when it is larger than the predetermined level.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a technology for acquiring information about a terminal as a communication target, and in particular to a technology for tracking the terminal. [Background technology]

[0002] The fifth-generation mobile communication system (5G), currently being introduced, utilizes millimeter-wave radio waves (mmWaves) to enable high-performance communications, including high speeds, large capacity, low latency, and multi-terminal connections. However, mmWaves have a high degree of directionality and are less susceptible to diffraction, making their propagation more likely to be blocked or attenuated by objects between the terminal and the base station. Therefore, a major problem with 5G is that the presence of such objects can cause a sudden drop in received power, potentially resulting in the loss of communication connections altogether.

[0003] Therefore, if it were possible to predict a decrease in received power due to such objects, it would be possible to maintain communication connections without interruption by controlling and securing the communication paths of the terminals, etc. As a technique for predicting received power, Non-Patent Document 1 discloses a technique for predicting received power in a communication device through machine learning, using time-series three-dimensional environmental information acquired by an RGB-D camera that can generate depth images in addition to RGB images.

[0004] Specifically, this technology constructs a machine learning model using training data that links time-series environmental information resulting from capturing an image of the communication environment with time-series information on received power measured (at the same time as capturing the image) at a communication terminal that receives radio waves from a transmitting station as correct answer data, and then uses this constructed machine learning model to determine a predicted value for the received power at the communication terminal from the time-series environmental information acquired by the RGB-D camera.

[0005] Here, in such millimeter-wave received power prediction, it is important to continuously grasp the position of the communication terminal. However, if the position is grasped using the positioning result by the sensor on the communication terminal side, a mechanism for notifying the positioning result from the communication terminal to the base station is required, and furthermore, a large power consumption and computational load on the communication terminal side become problems.

[0006] As a technique that does not depend on the positioning result on the communication terminal side, Patent Document 1 including the inventor of the present application discloses a technique for estimating the position of a communication terminal using time-series information of the received power of millimeter waves and time-series information of the shielding relationship between objects estimated from, for example, environmental image information. Furthermore, Non-Patent Document 2 discloses a technique for continuously capturing the position of an object in an image by utilizing the consistency between the detection position of the object acquired from the image and the appearance features.

Prior Art Documents

Patent Documents

[0007]

Patent Document 1

Non-Patent Documents

[0008]

Non-Patent Document 1

Non-Patent Document 2

Summary of the Invention

[0009] If such conventional technology can be used to continue tracking a communication terminal, it will be possible to continue the communication connection or temporarily suspend it and then immediately resume it, even if the reception power suddenly drops due to the presence of an object.

[0010] However, object tracking technology based on image information such as that disclosed in Non-Patent Document 2 may fail to track a target when a phenomenon known as occlusion occurs, in which an object in the foreground at least partially covers an object in the background. In other words, conventional object tracking technology based on image information has problems such as "lost" when occlusion occurs, in which the target is hidden by another object and cannot be detected, causing tracking to be interrupted, or an "ID switch" when the target intersects with another object, causing the other object to be mistaken for the target being tracked, resulting in a deviation from the intended tracking.

[0011] Therefore, even if the target tracking technology disclosed in Non-Patent Document 2 is applied to tracking of a communication terminal, if such an occlusion problem occurs, it will be impossible to accurately determine the position of the communication terminal. Furthermore, the communication terminal position estimation technology described in Patent Document 1 also assumes that the terminal has not been lost or an ID switch has not occurred, and does not propose any measures to be taken when such problems occur.

[0012] Therefore, an object of the present invention is to provide a terminal tracking device, program, and method that can continue to track a terminal more appropriately even when occlusion occurs for the terminal being tracked. [Means for solving the problem]

[0013] According to the present invention, an object detection information determination means for detecting a predetermined object that may include the terminal from environmental information related to an environment in which the terminal may be present, generated by the sensor, and determining object position information that is information related to the detected position of the detected object; a terminal communication information determination means for determining terminal communication information, which is information relating to a communication state with the terminal, based on information relating to communication acquired from a communication means that communicates with the terminal; a target communication information determination means for determining, from the environmental information, target communication information which is information relating to a communication state to be realized with the target if the detected target includes a terminal; a tracking target determination means for determining whether or not the tracking terminal and the detected target are in a correspondence relationship based on at least one of a position coincidence degree, which is a degree of coincidence between the target location information and predicted target location information predicted from the target location information of the target determined to be related to the tracking terminal, and a communication compatibility degree, which is a degree of correspondence between the terminal communication information of the tracking terminal and the target communication information, and determining the target determined to be in a correspondence relationship as the target related to the tracking terminal; and The tracking target determination means makes the determination based on the communication compatibility degree, not on the positional coincidence degree, when the communication compatibility degree is large enough to satisfy a predetermined condition, or makes the determination by increasing the degree based on the communication compatibility degree as the communication compatibility degree becomes larger or when the communication compatibility degree becomes large enough to satisfy a predetermined condition. A terminal tracking device is provided.

[0014] In one embodiment of the terminal tracking device according to the present invention, the tracking target determination means makes the judgment without basing the communication compatibility degree on the degree of communication compatibility when the communication compatibility degree is small enough to satisfy a predetermined condition and the position matching degree is small enough to satisfy the predetermined condition, or it is also preferable that the smaller the position matching degree is, the smaller the degree based on the communication compatibility degree is when making the judgment.

[0015] In another embodiment of the terminal tracking device according to the present invention, the object detection information determination means also determines object appearance information, which is information related to the detected appearance of the detected object, from the environmental information; When the tracking target determination means makes the determination based at least on the degree of positional coincidence, it is also preferable that the means also make the determination based on the degree of appearance coincidence, which is the degree of coincidence between the target appearance information and past target appearance information of the target determined to be related to the terminal being tracked.

[0016] Furthermore, in the embodiment relating to the above-mentioned target appearance information, the tracking target determination means When the degree of positional matching is smaller than a predetermined condition, the determination is made based on at least the degree of appearance matching and the degree of positional matching, and at that time, the degree based on the degree of appearance matching is made larger, or The determination is made based on at least the degree of appearance match and the degree of position match, and the smaller the degree of position match, the larger the degree based on the degree of appearance match. It is also preferable.

[0017] In addition, as one embodiment of the target communication information determination means, it is also preferable that the target communication information determination means determines the target communication information by assuming that the target is present at a position interpolated using detection positions determined before and after the occlusion during a time period in which an occlusion occurs and the detection position of the target cannot be determined.

[0018] Furthermore, as an embodiment related to the degree of positional agreement, the environment information is information related to an image of the environment, and the target position information includes a position and a range within the image of an image area related to the detected target, It is also preferable that the degree of positional agreement is the degree of overlap between the range of the detected target and the predicted range in the image of the target determined by the device being tracked, or the degree of proximity between the position of the detected target and the predicted position in the image of the target determined by the device being tracked.

[0019] In addition, as a preferred embodiment of the tracking target determination process according to the present invention, the tracking target determination means determining a weighting factor that is a monotonically increasing function of the communication capability level, the weighting factor being a zero value; a weighted sum of an environmental information cost including a location cost that is a monotonically decreasing function of the degree of location coincidence and a communication cost that is a monotonically decreasing function of the degree of communication compatibility, the communication cost being weighted by the weighting coefficient; information The tracking cost of the target is calculated as a weighted sum in which the cost is weighted by another weighting factor that can take a zero value, and the target for which the tracking cost is small enough to satisfy a predetermined condition is determined to be the target related to the terminal being tracked. It is also preferable.

[0020] Furthermore, in the embodiment for determining the weighting coefficients, it is also preferable that the tracking target determination means determines the weighting coefficients that are monotonically increasing functions for both the communication compatibility degree and the positional agreement degree.

[0021] According to the present invention, an object detection information determination means for detecting a predetermined object that may include the terminal from environmental information generated by the sensor and relating to an environment in which the terminal may be present, determining object position information that is information relating to the detected position of the detected object, and also determining object appearance information that is information relating to the detected appearance of the detected object from the environmental information; a terminal communication information determination means for determining terminal communication information, which is information relating to a communication state with the terminal, based on information relating to communication acquired from a communication means that communicates with the terminal; a target communication information determination means for determining, from the environmental information, target communication information which is information relating to a communication state to be realized with the target if the detected target includes a terminal; a tracking target determination means for determining whether or not the tracking terminal and the detected target are in a correspondence relationship based on at least one of: a position matching degree, which is a degree of matching between the target location information and predicted target location information predicted from target location information of the target determined to be related to the tracking terminal; an appearance matching degree, which is a degree of matching between the target appearance information and past target appearance information of the target determined to be related to the tracking terminal; and a communication compatibility degree, which is a degree of correspondence between the terminal communication information of the tracking terminal and the target communication information; and determining the target determined to be in a correspondence relationship as the target related to the tracking terminal; and The tracking target determination means, when the occlusion of at least the detected target is resolved, makes the determination based on the communication compatibility degree, not on the positional coincidence degree, if the communication compatibility degree is large enough to satisfy a predetermined condition, or makes the determination by increasing the degree based on the communication compatibility degree as the communication compatibility degree becomes larger or becomes large enough to satisfy a predetermined condition. A terminal tracking device is provided.

[0022] According to the present invention, further an object detection information determination means for detecting a predetermined object that may include the terminal from environmental information related to an environment in which the terminal may be present, generated by the sensor, and determining object position information that is information related to the detected position of the detected object; a terminal communication information determination means for determining terminal communication information, which is information relating to a communication state with the terminal, based on information relating to communication acquired from a communication means that communicates with the terminal; a target communication information determination means for determining, from the environmental information, target communication information which is information relating to a communication state to be realized with the target if the detected target includes a terminal; a tracking target determination means for determining whether or not the tracking terminal and the detected target are in a correspondence relationship based on at least one of a position coincidence degree, which is a degree of coincidence between the target location information and predicted target location information predicted from the target location information of the target determined to be related to the tracking terminal, and a communication compatibility degree, which is a degree of correspondence between the terminal communication information of the tracking terminal and the target communication information, and determining the target determined to be in a correspondence relationship as the target related to the tracking terminal; to make the computer function, The tracking target determination means, when the occlusion of at least the detected target is resolved, makes the determination based on the communication compatibility degree, not on the positional coincidence degree, if the communication compatibility degree is large enough to satisfy a predetermined condition, or makes the determination by increasing the degree based on the communication compatibility degree as the communication compatibility degree becomes larger or becomes large enough to satisfy a predetermined condition. A terminal tracking program is provided.

[0023] According to the present invention, furthermore, a first step of detecting a predetermined object that may include the terminal from environmental information generated by a sensor relating to an environment in which the terminal may be present, determining object location information which is information relating to the detected position of the detected object, determining terminal communication information which is information relating to the communication state with the terminal based on information relating to communication obtained from a communication means that communicates with the terminal, and further determining object communication information from the environmental information which is information relating to the communication state that will be realized with the object if the detected object includes the terminal; a second step of determining whether or not the tracked terminal and the detected target are in a correspondence relationship based on at least one of a position match degree, which is a degree of match between the target location information and predicted target location information predicted from the target location information of the target determined to be related to the tracked terminal, and a communication compatibility degree, which is a degree of correspondence between the terminal communication information of the tracked terminal and the target communication information, and determining the target determined to be in a correspondence relationship as the target related to the tracked terminal; and In the second step, when the occlusion of at least the detected object is resolved, if the communication compatibility degree is large enough to satisfy a predetermined condition, the determination is made based on the communication compatibility degree, not on the positional coincidence degree, or the larger the communication compatibility degree is, or if the communication compatibility degree is large enough to satisfy a predetermined condition, the determination is made by increasing the degree based on the communication compatibility degree. A computer-implemented terminal tracking method is provided, comprising: [Effects of the Invention]

[0024] According to the terminal tracking device, program, and method of the present invention, even if an occlusion occurs in a terminal being tracked, it is possible to continue tracking the terminal more appropriately. [Brief explanation of the drawings]

[0025] [Figure 1] 1 is a functional block diagram showing a functional configuration of an embodiment of a terminal tracking device according to the present invention; [Figure 2] 3A to 3C are schematic diagrams for explaining two modes of occlusion determination processing in the occlusion determination unit according to the present invention. [Figure 3] FIG. 10 is a schematic diagram for explaining a specific example of a calculation process of communication costs according to the present invention. [Figure 4] 10A and 10B are schematic diagrams for explaining a specific example of a calculation process of a position cost and an appearance cost according to the present invention; [Figure 5] 3A to 3C are schematic diagrams for explaining various embodiments of the tracking target determination process in the tracking target determination and management unit according to the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0026] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

[0027] [Device tracking device] FIG. 1 is a functional block diagram showing the functional configuration of an embodiment of a terminal tracking device according to the present invention.

[0028] The base station 1 shown in Figure 1, which is one embodiment of a terminal tracking device according to the present invention, is a communication relay device compatible with 5G (fifth generation mobile communication system) in this embodiment, and is a device capable of communicating with multiple terminals 2, which are also communication terminals compatible with 5G.

[0029] In addition, as one embodiment of a terminal tracking device, base station 1 is also a device that can detect specified objects, including terminal 2, from image data as ``environmental information'' obtained by camera 103, which acts as a sensor that senses the environment in which terminal 2 may be present, and which is information related to the environment and may include specified objects, and track terminal 2.

[0030] In this embodiment, the camera 103 is an RGB camera that can capture images of the communication area with a predetermined angle of view. The image data (video data) generated by the camera 103 includes the following: (a) A person who possesses or carries Terminal 2, (b) Mobile objects such as automobiles, motorcycles, railway vehicles, robots, drones, etc., which are equipped with, include, or have Terminal 2 on board, (c) Equipment, facilities, buildings and fixed objects installed on Terminal 2 There is a possibility that "objects" such as the above may be included (imaged). Furthermore, "objects" such as people, moving objects, equipment, facilities, buildings, and fixed objects (including plants such as trees) that are not related to terminal 2 may also be included.

[0031] Incidentally, while such an "object" may be something for which the correspondence with the terminal 2 being tracked needs to be determined (in other words, something for which it needs to be determined whether it is a tracking object that should be tracked), its presence between the terminal 2 and the base station 1 may also become an obstacle to communication radio waves.

[0032] In particular, 5G, which is the communication method of this embodiment, uses millimeter-wave radio waves (millimeter waves) for communication, but these millimeter waves have high directivity and are not easily diffracted, so there is a high possibility that their propagation will be blocked or attenuated by an "object" that exists between the terminal 2 and the base station 1. Therefore, with 5G, the presence of such an "object" can cause a sudden drop in the received signal power at the base station 1, which can lead to, for example, a situation in which the communication connection itself is interrupted, which is a serious problem.

[0033] On the other hand, even if "occlusion" occurs for the terminal 2 being tracked, if tracking of this terminal 2 can be continued appropriately, it will be possible to continue the communication connection with that terminal 2 or to temporarily suspend and then immediately resume the connection.

[0034] Incidentally, the above-mentioned "occlusion" will be hereinafter defined as a phenomenon in which an "object" in the foreground as viewed from the base station 1 obscures at least a portion of an "object" in the background. For example, a phenomenon in which multiple "objects" intersect or at least one "object" is blocked by at least one other "object" will also be treated as "occlusion" hereinafter. When such "occlusion" occurs, there is a high possibility that a specific object (e.g., a "person") cannot be detected from "environment information" (image data in this embodiment) or can only be detected inaccurately.

[0035] In order to solve the serious problem when occlusion occurs as described above, the base station 1 specifically: (A) an object detection information determination unit 113 that detects a predetermined object (e.g., a "person") that may include the terminal 2 from "environment information" (image data in this embodiment) generated by the sensor (the camera 103 in this embodiment) and determines "object position information" that is information related to the detection position of the detected object ("person"); (B) a terminal communication information determination unit 111 that determines "terminal communication information," which is information related to the communication status with the terminal 2, based on information related to communication (e.g., communication history information) acquired from a communication means for communicating with the terminal 2 (in this embodiment, the communication interface 101, communication control unit 116, and communication history information storage unit 102 provided in the terminal 2 itself); (C) a target communication information determination unit 112 that determines, from the "environment information," "target communication information" which is information related to the communication state that will be realized between the detected target (e.g., a "person") and the terminal 2 if the detected target (e.g., a "person") includes the terminal 2; and (D) A tracking target determination and management unit 115 determines whether or not the tracking target terminal 2 and the detected target ("person") correspond to each other based on at least one of (d1) a "position agreement degree" which is the degree of agreement between the "target location information" and predicted target location information predicted from the "target location information" of the target (e.g., a "person") determined to be related to the tracking target terminal 2, and (d2) a "communication compatibility degree" which is the degree of correspondence between the "terminal communication information" of the tracking target terminal 2 and the "target communication information", and determines the target ("person") determined to be related to the tracking target terminal 2 as the target related to the tracking target terminal 2. It has the following characteristics.

[0036] Furthermore, the tracking target determination and management unit 115 of (D) above, when an occlusion occurs and is resolved for at least a detected target (for example, a "person"), (D1) When the "communication compatibility level" is large enough to satisfy a predetermined condition (in this embodiment, when a representative value (e.g., a maximum value) of the communication compatibility level of each detected target exceeds a predetermined threshold), the above determination is made based on the "communication compatibility level" and not on the "position agreement level," or (D2) The larger the "communication readiness level" (in this embodiment, the larger the representative value (e.g., the maximum value) of the communication readiness level of each detected target) or when it is large enough to satisfy a predetermined condition (in this embodiment, when the representative value (e.g., the maximum value) exceeds a predetermined threshold value), the degree based on the "communication readiness level" is made larger and the above determination is made. This is a special processing.

[0037] Here, the "predetermined target" detected from the "environmental information" in (A) above can be a "person" or a "mobile object" that carries, is equipped with, or may contain the terminal 2, as will be explained in detail later. Furthermore, in some cases, the "predetermined target" may be an "equipment," "facility," "building," "fixed object," etc. that may contain the terminal 2. Furthermore, the "target position information" in (A) above is information related to the target detection position, and is position-related information that may include the position and speed of the detected target (e.g., a "person"), etc.

[0038] Furthermore, in a preferred embodiment, the "terminal communication information" in (B) above can be (α) information relating to the strength of radio waves received from terminal 2 (for example, received signal power, RSSI (Received Signal Strength Indicator)). Alternatively, it can be (β) information relating to the presence or absence of a communication connection with terminal 2. Furthermore, the "target communication information" in (C) above can be, in accordance with the "terminal communication information" above, (α') information relating to the strength of radio waves received from a predetermined target ("person") if the detected target (for example, "person") includes terminal 2 (for example, RSSI). Alternatively, it can be (β') information relating to the presence or absence of a communication connection with the detected target ("person"), which is determined using the detected position of the detected target (for example, "person").

[0039] Furthermore, the "degree of communication compatibility" in (D) above can be specifically, for example, the degree of agreement between time series data of quantities related to radio wave strength (e.g., RSSI values) as "terminal communication information" and time series data of quantities related to radio wave strength (e.g., RSSI values) as "target communication information." Alternatively, it can also be the degree of agreement between time series data related to the presence or absence of communication connection as "terminal communication information" and time series data related to the possibility of communication connection as "target communication information."

[0040] With such settings for "communication compatibility degree" and "position matching degree," the base station 1 as a terminal tracking device according to the present invention, at least when occlusion is resolved, makes the above judgment by increasing the degree based on the "communication compatibility degree" (after checking the magnitude of the communication compatibility degree of each detected target), as shown in (D1) and (D2) above, or uses only the "communication compatibility degree" as the judgment criterion for the above judgment.

[0041] Here, the "communication readiness level" can be determined from the communication status of terminal 2 and the estimated communication status of the target to be determined, which is obtained from "environmental information," without depending on whether the target to be determined (e.g., a "person") is detected during the process. In other words, the "communication readiness level" is likely to be used as a more reliable determination criterion even when occlusion occurs. Therefore, the terminal tracking device (base station 1) according to the present invention makes it possible to continue tracking the tracked terminal 2 more appropriately, even when occlusion occurs for the terminal 2 being tracked.

[0042] Furthermore, in this embodiment, even if a situation occurs in which the base station 1 has a person or other object between itself and the terminal 2 and the received power drops suddenly, the base station 1 can continue or immediately resume communication connection with the terminal 2.

[0043] In this embodiment, the base station 1 (terminal tracking device) performs at least the following during a period when no occlusion occurs, excluding a predetermined period immediately after the occlusion is resolved: (a) The above-mentioned "degree of positional agreement" and (b) "Appearance Matching Degree" determined from "Environmental Information" (e.g., image data), which will be explained in detail later; It is also preferable to determine whether or not there is a correspondence between the tracked terminal 2 and the detected object (for example, a "person") based on either or both of the above. (c) The "degree of communication compatibility" mentioned above It is also preferable to make the determination based on the following.

[0044] Although 5G is used as the communication method between base station 1 and terminal 2 in this embodiment, other communication methods, such as LTE or future mobile communication methods using shorter wavelength radio waves, may also be used. Furthermore, communication between the terminal tracking device of the present invention and the communication terminal may be based on various other wireless communication standards. For example, even in communication methods where the problem of obstruction by objects is not as significant as in 5G, there are many situations where it is necessary to accurately identify and track a subordinate communication terminal using objects detected from camera image data. Furthermore, there are needs for terminal identification and tracking in various fields, such as determining the relationship between the pages viewed by a user determined to be in correspondence with a certain communication terminal and the user's movement path, and using this information for marketing and management. In contrast, the terminal tracking device of the present invention makes it possible to perform such terminal identification and tracking processing with high accuracy.

[0045] Furthermore, the terminal tracking device according to the present invention is not limited to a communication relay device such as a base station. For example, it may be provided as a dedicated device for terminal tracking processing, connected to a communication relay device or facility such as a base station. Furthermore, the terminal tracking device according to the present invention may be a cloud server, a non-cloud server device, a personal computer (PC), a notebook or tablet computer, a smartphone, or the like, equipped with the terminal tracking program according to the present invention.

[0046] Furthermore, it is not impossible that at least one of the components (A) to (D) of the terminal tracking device (base station 1) according to the present invention is a separate device from the other components. For example, it is also possible for a plurality of servers to implement the functions of (A) to (D) above. Even in such a case, these components as a whole can be considered to be a terminal tracking device or system that implements the terminal tracking method according to the present invention and is operated by the terminal tracking program according to the present invention.

[0047] [Functional configuration of terminal tracking device, terminal tracking program] 1, base station 1 includes a communication interface 101, a communication history information storage unit 102, a camera 103, an environmental information storage unit 104, a tracking information storage unit 105, and a processor / memory (a processing system with a memory function) as an embodiment of a terminal tracking device and a communication relay device according to the present invention. Here, the processor / memory stores an embodiment of a communication relay program including a terminal tracking program according to the present invention, and has a computer function, and executes this communication relay program to perform terminal tracking processing and communication relay processing.

[0048] The processor memory also has, as functional components, a terminal communication information determination unit 111, a target communication information determination unit 112 including a position interpolation unit 112a, a target detection information determination unit 113, an occlusion determination unit 114, a tracking target determination and management unit 115 including a communication cost calculation unit 115a, a position cost calculation unit 115b, an appearance cost calculation unit 115c, and a weight determination unit 115d, and a communication control unit 116. These functional components can be considered to be functions of a terminal tracking program and a communication relay program stored in the processor memory, and the processing flow shown by connecting the functional components of the base station 1 in the functional block diagram of Figure 1 with arrows can also be understood as one embodiment of a terminal tracking method and a communication relay method according to the present invention.

[0049] Also in the functional block diagram of FIG. 1 , the communication control unit 116 functions as a base station by controlling wireless communication operations between the communication interface 101 and each terminal 2, and further acquires and records various information related to communications with each terminal 2, generates communication history information by organizing the information in chronological order, and causes the communication history information storage unit 102 to store and manage it.

[0050] Furthermore, the camera 103 may be a visible light, near-infrared, or infrared compatible imaging device equipped with a solid-state imaging element such as a CCD image sensor or a CMOS image sensor, and may also be an RGB camera, an RGB-D camera, a stereo camera, an omnidirectional camera, etc. Of course, instead of the camera 103, it is also possible to employ a sensor capable of sensing the environment in which the terminal 2 may be present and generating environmental information, such as a LiDAR, a laser / infrared positioning device, a TOF camera, or a thermography device.

[0051] In this embodiment, the camera 103 is an RGB camera capable of generating RGB image data (RGB video data) as environmental information, and the generated RGB image data is saved and managed in the environmental information accumulation unit 104. Note that, although the camera 103 is installed inside the base station 1 in this embodiment, it may also be a camera installed at a location remote from the base station 1, such as a surveillance camera in a city, which is connected to the base station 1 for communication.

[0052] Furthermore, a plurality of cameras 103 may be installed at different locations, either inside or outside the base station 1. For example, the base station 1 may receive and acquire "target location information" and "target communication information" generated by the cameras 103 installed at each of a plurality of base stations 1 present around the base station 1, and perform the tracking target determination process. In this case, the plurality of base stations 1 cooperate to more suitably perform the terminal tracking process according to the present invention.

[0053] <Target detection information determination process> Also in the functional block diagram of FIG. 1, the object detection information determination unit 113 determines, from image data (environmental information) that may include a “predetermined object” acquired from the environmental information accumulation unit 104, (A) Target location information, which is information related to the detected location of the "predetermined target"; and (a) Object appearance information, which is information related to the detected appearance of a "predetermined object" At least one, and preferably both, of

[0054] As mentioned above, the "predetermined subject" is, for example, (a) The person (user) who owns or carries the terminal 2, (b) A vehicle equipped with a terminal 2 having a drive recorder function, a CAN information transfer function, an interface function for automatic driving control by a server, etc., and further, (c) An autonomous mobile robot or an autonomous flying drone equipped with a terminal 2 having an interface function for autonomous movement control by a server. In some cases, a "predetermined target" may be a "person" or a "mobile object" that may include Terminal 2, which is the communication destination related to the communication history information. In addition, in some cases, a "facility," a "building," a "fixed object," etc. that may include Terminal 2 may also be a "predetermined target."

[0055] Naturally, the object detection information determination unit 113 of this embodiment can simultaneously detect multiple types of objects, including a "predetermined object." Here, when one "predetermined object" is detected as a tracking object from one piece of image data, other objects (including other "predetermined objects") detected from this image data are treated as objects that may cause occlusion of the detected "predetermined object," that is, as objects that may cause interference.

[0056] Here, the process of determining the target position information (a) and the target appearance information (b) will be explained. In this embodiment, the target detection information determination unit 113 sets a plurality of image regions in the acquired image data, calculates a score representing the likelihood of each image region being a predetermined target using a corresponding object detector, and determines an image region having a score high enough to satisfy a predetermined condition as the detected image region in which the predetermined target has been detected. Next, (a) The position (detection position) of the detected image area relating to the detected object, the range of the detected image area (for example, coordinate values ​​of the four corners), and further, the detection speed as the change in the detection position between successive image data are used as the object position information of (a) above, (b) The image feature quantity of the detected image area relating to the detected object, calculated using a known image feature extractor (e.g., a CNN (Convolutional Neural Networks) image feature extractor), is used as the object appearance information in (a) above.

[0057] The object detector may be, for example, the one described in Non-Patent Document: Alexey Bochkovskiy, Chien-Yao Wang, and Hong-Yuan Mark Liao, "YOLOv4: Optimal Speed ​​and Accuracy of Object Detection," arXiv:2004.10934v1, 2020. Furthermore, it is also possible to determine the target position information (A) and the target appearance information (B) using various other object detectors and image feature extractors known in the field of image recognition technology.

[0058] Furthermore, when a stereo camera is used as the camera 103, an object detector such as that described in the non-patent document Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott Reed, Cheng-Yang Fu, Alexander C. Berg, “SSD: single shot multibox detector”, European Conference on Computer Vision, Computer Vision-ECCV 2016, pp. 21-37, 2016 may be applied to the generated environment information, which is the stereo camera image data.

[0059] In yet another embodiment, when a LiDAR is used instead of the camera 103 and three-dimensional point cloud data is used as environmental information, an object detector for detecting a predetermined target can be, for example, the one disclosed in non-patent document: Charles R. Qi, Hao Su, Kaichun Mo, and Leonidas J. Guibas, “PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation,” 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), vol. 1, pp. 77-85, 2017.

[0060] Furthermore, in this embodiment in which image data is used as environmental information, instead of the above-mentioned image feature extractor, for example, a feature extractor described in non-patent document: Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun, “Deep Residual Learning for Image Recognition,” 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Vol. 1, pp. 770-778, 2016 may be used.

[0061] Incidentally, in this embodiment, the target detection information determination unit 113 stores the determined "target position information" and "target appearance information" for each detected predetermined target for a predetermined period of time in the past, and can provide them to the tracking target determination and management unit 115 as appropriate.

[0062] <Device communication information determination process> 1, the terminal communication information determination unit 111 acquires communication history information from the communication history information storage unit 102, and determines "terminal communication information" which is information related to communication with the terminal 2 based on this communication history information. Here, in this embodiment, this "terminal communication information" is (α) time-series data of received signal strength (RSSI) of the radio wave signal received from terminal 2, and (β) Time series data of information regarding the presence or absence of communication connection with terminal 2 It is either one or both of the above.

[0063] Of these, for the above (α), the terminal communication information determination unit 111 may generate time-series data of a received signal power variable E_t indicating the signal power value at time t of a radio wave signal received from terminal 2. Furthermore, for the above (β), the terminal communication information determination unit 111 may generate time-series information on whether or not a communication connection with terminal 2 is established at each time point, that is, time-series data of a binary communication connection presence / absence variable B_t(∈{-1, 1}) that takes the value 1 if a communication connection is established at time point t and the value -1 if a communication connection is not established. Here, taking into consideration the accuracy of the degree of match (degree of communication compatibility) with "target communication information" calculated later, it is preferable that the "terminal communication information" include at least the time-series data of the above (α) received signal power variable E_t.

[0064] Incidentally, in this embodiment, the terminal communication information determination unit 111 stores the "terminal communication information" determined for each terminal 2 being tracked for a predetermined period of time in the past, and can provide it as appropriate to the tracking target determination and management unit 115. Furthermore, it is also preferable that the terminal communication information determination unit 111 acquires, for example, an IMSI (International Mobile Subscriber Identity) from each terminal 2 present under the control of the base station 1, compiles the determined "terminal communication information" for each terminal 2 identified by the IMSI, and performs subsequent processing separately for each terminal 2.

[0065] <Target communication information determination process> Also in the functional block diagram of Figure 1, the target communication information determination unit 112 determines "target communication information," which is information related to the communication state that will be realized with a target if the detected target includes terminal 2, from image data (environmental information) that may include a specified target, obtained from the environmental information accumulation unit 104.

[0066] (When using received signal strength data as "target communication information") The "target communication information" determined by the target communication information determination unit 112 is (α') Information on the strength of radio waves received from the detected object if the detected object includes terminal 2 (received radio wave strength data) In this case, the target communication information determination unit 112 determines, based on the detected position of the target detected from the image data (by the target detection information determination unit 113) and the detected positions of "other targets" (which may cause interference), as shown on the right side of FIG. 3, which will be used for explanation later, (a) determining a line segment connecting the detected object and the base station 1 (antenna thereof); (b) calculating occlusion degree information relating to the degree to which the "other object" occludes a predetermined area including the line segment; (c) From the calculated degree of obstruction information, if the detected target includes terminal 2, received radio wave intensity data that will be received from this target is calculated as "target communication information."

[0067] Specifically, the above-mentioned (b) occlusion degree information is calculated as follows: First, as shown on the right side of Fig. 3 (which will be used later in the explanation), a predetermined size (a cross-sectional area at the detection position having a predetermined area) is set in advance for the "other object" that may cause an obstacle according to its type (class), and then a rotating body area of ​​a predetermined size including the above-mentioned straight line segment as a rotation axis is set, and then at a certain time t (= t0 - M, t0 - M + 1, , t0 - 1, t0), (b1) When any of the "other objects" has a cross-sectional area that is not partly included in the above-mentioned rotating body area at its detection position (the cross-sectional area of ​​the other object does not overlap with the rotating body area), it takes a value of 1. (b2) If a part of the cross-sectional area of ​​any of the "other objects" is included in the above-mentioned rotating body area at the detection position (the cross-sectional area of ​​the other object and the rotating body area have an overlapping portion), the following formula (1) R=(S a ∩S b- ) / S a The R value is calculated using The time series data of such a variable E'_t can be used as the occlusion degree information.

[0068] Here, in the above equation (1), S a is a set representing the cross-sectional area (perpendicular to the axis of rotation) at the detection position of the "other object" in the rotating body area (for example, a set of unit areas forming the area), and S b is a set representing the cross-sectional area at the detection position of the "other object" (for example, a set of unit areas forming the area). b - is S b The " / " is the operator indicating that the area value of the cross-sectional area (e.g., the number of unit areas) related to the numerator set is divided by the area value of the cross-sectional area (e.g., the number of unit areas) related to the denominator set.

[0069] Next, the target communication information determination unit 112 determines the time series data of this variable E'_t (or the variable E'_t multiplied by a predetermined constant) as "target communication information" as received radio wave intensity data. Hereinafter, the variable E'_t will be referred to as a received power equivalent variable to distinguish it from the above-mentioned received signal power variable E_t.

[0070] In addition, if there are two or more "other targets" that have the above R value, S b may be the area formed by all of the cross-sectional areas of these "other objects," i.e., the area expressed as the union of these cross-sectional areas. Furthermore, the above-mentioned body of revolution area may be a cylindrical area, or may be one that can be formed by rotating a predetermined curve around a rotation axis, such as an ellipsoid.

[0071] As a further modification, the target communication information determination unit 112 may use a trained "target radio wave information estimation model" that outputs information related to the strength of received radio waves in accordance with input image data to derive information related to the strength of radio waves that will be received from the detected target if the detected target includes terminal 2 from the acquired image data, and determine the information related to the strength of radio waves as the "target communication information."

[0072] Here, the above-mentioned "target radio wave information estimation model" can be a publicly known trained model such as that disclosed in Non-Patent Document 1. Specifically, image data at a certain time t (=t0-M, t0-M+1, . . . , t0-1, t0) is input to this trained model to output an estimated value of received signal power, and this estimated value is set as the value of the received power equivalent variable E'_t, and the time-series data of this received power equivalent variable E'_t is used as the "target communication information."

[0073] (When using communication connection availability data as "target communication information") The "target communication information" determined by the target communication information determination unit 112 is (β') Information regarding whether or not communication connection with the detected target is possible (communication connection possible data) In this case, the target communication information determination unit 112 determines, based on the detected position of the target detected from the acquired image data and the detected positions of the "other targets" that may cause a problem, (a) determining a line segment connecting the detected object and the base station 1 (antenna thereof); (b) calculating communication connection availability data indicating whether or not the "other object" is located on the line segment or within a predetermined area including the line segment, or whether or not the other object is blocking the line segment; (c) The calculated communication connection availability data is determined as the "target communication information." It is also preferable.

[0074] More specifically, as the communication connection availability data of (b) above, a binary communication connection availability variable is used, which takes the value of -1 if the detected position of any of the "other targets" is located on the above line segment or within a region of a rotating body of a predetermined size that includes this line segment as the rotation axis at a certain time t (==t0-M, t0-M+1, , t0-1, t0), and takes the value of 1 otherwise: (2) B'_t∈{-1, 1} The time series data may be generated and used as the "target communication information." Here, the above-mentioned rotating body region may be a cylindrical region, or may be one that can be formed by rotating a predetermined curve around a rotation axis, such as an ellipsoid.

[0075] As a modified embodiment, the target communication information determination unit 112 may pre-set a predetermined size (cross-sectional area at the detection position) for the "other target" according to its type (class), and define B'_t in (2) above as a communication connection feasibility variable that takes the value -1 if any of the "other targets" blocks the above line segment with that size (cross-sectional area) at the detection position (the above line segment penetrates the cross section of the "other target"), and 1 if not.

[0076] The above "target communication information" is (α') Time series data of the received power equivalent variable E'_t (received radio wave intensity data), or (β') Time series data of communication connection availability variable B'_t (communication connection availability data) Here, when the "terminal communication information" includes both of the above (α) and (β), it is also preferable that the "target communication information" includes both of the above (α') and (β'). Furthermore, in consideration of the accuracy of the degree of match with the "terminal communication information" (degree of communication compatibility) calculated later, it is preferable that the "target communication information" includes time-series data of the received power equivalent variable E'_t of the above (α').

[0077] Incidentally, in this embodiment, the target communication information determination unit 112 stores the determined "target communication information" for each detected predetermined target for a predetermined period of time in the past, and can provide it to the tracking target determination and management unit 115 as appropriate.

[0078] <Occlusion detection processing> Also in the functional block diagram of FIG. 1, the occlusion determination unit 114 determines whether or not an occlusion has occurred (or is predicted to occur) at each time point for an object (e.g., a "person") related to the terminal 2 being tracked, based on information from the object detection information determination unit 113, and outputs this determination result to the tracking object determination and management unit 115, which will be described in detail later.

[0079] FIG. 2 is a schematic diagram for explaining two modes of the occlusion determination process in the occlusion determination unit 114. In FIG.

[0080] First, according to the determination mode shown in FIG. 2(A), the occlusion determination unit 114 obtains from the target detection information determination unit 113 the detected position and detection speed (target position information) of the target (e.g., a "person") related to the tracked terminal 2 at a time point earlier than time t0 (e.g., time t0-1), and determines the predicted position of this target ("person") at time t0 from this information.

[0081] Next, if the occlusion determination unit 114 detects another specified object (e.g., a "person") at time t0 within a specified range including the determined predicted position (e.g., an image area corresponding to a real-space circular range of a specified size centered on the predicted position), it can determine that occlusion has occurred at this time t0.

[0082] Next, in the determination mode shown in FIG. 2(B), the occlusion determination unit 114 first determines the predicted position of the target (e.g., a "person") associated with the terminal 2 being tracked at time t0, in the same manner as in the determination mode shown in FIG. 2(A) described above.

[0083] Next, the occlusion determination unit 114 can determine that an occlusion has occurred at time t0 if no object, including this object, is detected at time t0 within a predetermined range including the determined predicted position (for example, an image area corresponding to a real-space circular range of a predetermined size centered on the predicted position). In this case, it is highly likely that an occlusion has occurred in which the tracked object is completely blocked.

[0084] As yet another mode of determining occlusion, when the target detection information determination unit 113 is unable to determine one or both of the target position information and target appearance information of a tracked target (e.g., a "person") from image data at a certain time t0, the occlusion determination unit 114 can receive a report to that effect and determine that occlusion has occurred at this time t0.

[0085] Furthermore, as yet another determination mode, the occlusion determination unit 114 can determine that occlusion has occurred at time t0 if the target detection information determination unit 113 detects a tracked target (e.g., a "person") and another predetermined target (e.g., a "person") from image data at time t0 and the distance between their detected positions (e.g., a distance in image space corresponding to a predetermined distance in real space) is less than a predetermined threshold. In this case, it is highly likely that partial occlusion has occurred to the extent that the detection position can be determined.

[0086] <Tracking target determination process / Device tracking process> Returning to the functional block diagram of FIG. 1, the tracking target determination and management unit 115 of this embodiment performs at least the following during a time period in which no occlusion occurs for a detected target (e.g., a person) related to tracking and target position information and target appearance information are determined: (A) "Position matching degree" which is the degree to which the determined target position information matches the predicted target position information predicted from the past target position information, and (a) "Appearance matching degree" which is the degree to which the determined target appearance information matches the past target appearance information Based on either one or both (both in this embodiment), it is determined whether or not there is a correspondence between the terminal 2 being tracked and the detected object ("person"), and the object ("person") that is determined to have a correspondence is determined to be the object related to the terminal 2 being tracked, and terminal tracking processing is performed.

[0087] In this embodiment, as will be explained later, the weight (λ) is adjusted, but (c) "Communication compatibility level" which is the degree to which the terminal communication information determined for the tracked terminal 2 corresponds to the target communication information determined for the detected target ("person"). The correspondence determination process may also be performed based on the above.

[0088] In addition, in this embodiment, when an occlusion occurs for a detected target related to tracking, in which the detection position cannot be determined, and then the occlusion is resolved, the tracking target determination and management unit 115 always uses the "communication compatibility degree" described above (c), and performs a correspondence relationship determination process (tracking target determination process) between the terminal 2 and the target ("person"), for example, based only on this "communication compatibility degree."

[0089] This is because, in order to calculate the "degree of positional agreement" in (A) above and the "degree of appearance agreement" in (B) above at the time when an occlusion whose detection position cannot be determined occurs and then resolves, "(past) target position information" and "(past) target appearance information" for the time period in which this occlusion occurred are required, but this information cannot be obtained precisely because such an occlusion occurred. In other words, in this case, neither the "degree of positional agreement" in (A) above nor the "degree of appearance agreement" in (B) above can be calculated in the first place.

[0090] A specific correspondence determination process (tracking target determination process) using the above-mentioned "degree of positional match," "degree of appearance match," and "degree of communication compatibility" will be described below.

[0091] Similarly, in the functional block diagram of FIG. 1, the communication cost calculation unit 115a, the location cost calculation unit 115b, and the appearance cost calculation unit 115c of the tracking target determination and management unit 115 calculate the above-mentioned position match degree, appearance match degree, and communication compatibility degree for the detected target, respectively, and then: "Location cost" Cgeo, which is a monotonically decreasing function of the degree of location agreement; "Appearance cost" Capp, which is a monotonically decreasing function of the degree of appearance agreement, and "Communication cost" Cradio, which is a monotonically decreasing function of the degree of communication compatibility Calculate.

[0092] The smaller these costs are, the higher the possibility that the terminal 2 and the detected object are in a correspondence relationship. Here, the monotonically decreasing function of each degree may be, for example, a function that reverses the sign of the degree (multiplying the degree by -1), or a function that takes the reciprocal of the degree. Note that hereinafter, the location cost Cgeo and the appearance cost Capp may also be collectively referred to as the "environment information cost."

[0093] Next, the tracking target determination and management unit 115 of this embodiment, which will be described in detail later, (a) For each of the detected targets (e.g., a person), a tracking cost Cost is calculated, which is a weighted sum of the calculated location cost Cgeo, appearance cost Capp, and communication cost Cradio, using a "weighting factor λ" that may be zero, where the "weighting factor λ" is applied to the communication cost Cradio; (b) The target (“person”) whose tracking cost is small enough to satisfy a predetermined condition, for example, whose tracking cost is the smallest, is determined to be the target related to the terminal 2 being tracked.

[0094] A specific example of calculation of the communication cost Cradio, the location cost Cgeo, and the appearance cost Capp required to calculate the tracking cost Cost will be described below.

[0095] (Communication cost calculation) FIG. 3 is a schematic diagram for explaining a specific example of the communication cost calculation process according to the present invention.

[0096] In the specific example of calculating the communication cost Cradio shown in FIG. 3, the communication cost calculation unit 115a first calculates: (a) Time series data of the received signal power variable E_t of the terminal 2 being tracked (as terminal communication information) determined by the terminal communication information determination unit 111; (b) Time series data of the received power equivalent variable E'_t relating to the detected target (as the target communication information) determined by the target communication information determination unit 112; The degree of correspondence Cr (degree of communication compatibility) between the two is calculated by the cross-correlation (Σ t E_t*E'_t).

[0097] Incidentally, when the time series data of the communication connection availability variable B_t and the communication connection availability variable B'_t are used as the terminal communication information and the target communication information, respectively, the degree of agreement Cr (degree of communication compatibility) is calculated by the cross-correlation (Σ t B_t*B'_t). Also, the cross-correlation (Σ t E_t*E'_t) and cross-correlation (Σ t The weighted sum of B_t*B'_t can also be used as the degree of match Cr.

[0098] In this specific example, the communication cost calculation unit 115a calculates the degree of match Cr at time t0 when it is determined that no occlusion has occurred, which prevents the detection position from being determined (when the detection position (object position information) of the object detected by the object detection information determination unit 113 is saved), as the cross-correlation between the received signal power variable E_t and the received power equivalent variable E'_t in a predetermined past time interval (the time interval t=t0-M, t0-M+1, . . . , t0-N), using the following equation: (3) Cr=C0·Σ t=t0-M t0-N E_t*E'_t Here, in the above formula (3), Σ t=t0-N t0-Nis an operator indicating summation for time t (= t0-M, t0-M+1, . . . , t0-N). Furthermore, C0 is a coefficient used to adjust the Cr value so that it becomes 1 in the event of a perfect match. Furthermore, M is an integer set so that time t0-M is a time in the past that is sufficiently far in the past to obtain cross-correlation (for example, 10 seconds ago).

[0099] Furthermore, N is an integer equal to or greater than 1 such that time t0-N is the latest (closest to time t0) time at which the detected position (target position information) of the target detected (in the target detection information determination unit 113) is stored, as viewed from time t0. Therefore, when N=1 and the detected position of the target detected at time t0-1, immediately before time t0, is stored in the target detection information determination unit 113, that is, when no occlusion with an undetermined detection position has occurred recently, the degree of coincidence Cr at time t0 is expressed by the following formula: (3') Cr=C0·Σ t=t0-M t0-1 E_t*E'_t It will be calculated using

[0100] On the other hand, when N>1, the detected position of the detected object in the time interval from time (t0-N+1) to time (t0-1) is not determined due to the occurrence of occlusion to the extent that detection processing is impossible, and as a result, the received power equivalent variable E'_t is not calculated in the time interval where such occlusion occurred. Therefore, in this case, the degree of match Cr expressed by the above formula (3) cannot be calculated as it is.

[0101] Therefore, in response to the fact that the detected positions in the time interval from time point (t0-N+1) to time point (t0-1) have not been determined and saved (the received power equivalent variable E'_t has not been calculated), the position interpolation unit 112a (FIG. 1) of the target communication information determination unit 112 determines the positions of the detected target at each time point (time point t0-N+1, time point t0-N+2, . . . , time point t0-1) in the time interval in which such occlusion occurred, by interpolation processing. Specifically, the position interpolation unit 112a determines the positions at time point t0-N+1, time point t0-N+2, . . . and time point t0-1 by interpolating the detected position at the acquired time point (t0-N) (or the detected position acquired before time point t0-N) and the detected position at the acquired time point t0 by a known interpolation method, for example, linear interpolation or spline interpolation.

[0102] Next, the target communication information determination unit 112 (FIG. 1) determines the variable E'_t (target communication information) equivalent to the received power at each time point determined as the interpolation position (time point t0-N+1, time point t0-N+2, . . . , time point t0-1), assuming that the detected target exists at that position. This enables the communication cost calculation unit 115a to calculate the degree of match Cr using the above formula (3) even at time point t0 when the occlusion that occurred is resolved.

[0103] Furthermore, in this specific example, the communication cost calculation unit 115a inverts the sign of the degree of match Cr calculated as described above (multiplies the degree of match Cr by -1), and determines this sign inversion result (-Cr) as the communication cost Cradio. Therefore, the larger this communication cost Cradio, the higher the possibility that the detected target does not correspond to the terminal 2 being tracked.

[0104] Incidentally, in Figure 3, the communication cost Cradio of target A is smaller than the communication cost Cradio of target B because the waveform of the received power equivalent variable E'_t is more similar to the waveform of the received signal power variable E_t. Therefore, looking at the communication cost Cradio alone, it can be said that target A is more likely to correspond to the terminal 2 being tracked.

[0105] (Location cost calculation, appearance cost calculation) FIG. 4 is a schematic diagram for explaining a specific example of the calculation process of the position cost and the appearance cost according to the present invention.

[0106] First, in the specific example of calculating the location cost Cgeo shown in FIG. 4(A), the location cost calculation unit 115b calculates the location cost Cgeo for the detected object as follows: (a) The determined range of the detected image area (as target position information) (for example, coordinate values ​​of the four corners), (b) the range of the detected image area (as predicted target position information) predicted from the past detected position (as target position information) The degree of match (degree of positional match) Cg is calculated.

[0107] More specifically, at time t0 when it is determined that no occlusion has occurred that prevents a detection position from being determined (when the detection position (target position information) of the target detected by the target detection information determination unit 113 is saved), the position cost calculation unit 115b calculates (a) The range S1 of the detected image area for the detected object at this time t0; (b) The range S2 of the predicted detected image area at time t0, which is predicted from the detection position at the latest (closest to time t0) time (t0-N) for which the detection position (object position information) of the object detected (in the object detection information determination unit 113) is stored, using the detection speed at the same time (t0-N). The area of ​​the overlapping portion is defined as S1∩S2, and the area occupied by the range S1 of the detected image area and the range S2 of the predicted detected image area is defined as S1∪S2. (4) Cg=(S1∩S2) / (S1∪S2) The degree of match (degree of positional match) Cg at time t0 is calculated by the above formula.

[0108] As a modified example, the position cost calculation unit 115b can also use the degree of proximity between the detected position of the detected object at time t0 and the predicted position predicted from the detected position at the latest time (t0-N) (using the detection speed at time t0-N) as the degree of agreement (degree of position agreement) Cg at time t0, for example, the inverse of the distance between the two positions.

[0109] In any case, according to the calculation process of the degree of coincidence Cg described above, even when there is a time point at which occlusion occurs and the detection position cannot be determined, i.e., when N>1, it is possible to calculate the degree of coincidence Cg at time t0.

[0110] In this embodiment, the location cost calculation unit 115b inverts the sign of the thus calculated degree of match Cg (multiplies the degree of match Cg by -1) and determines this sign-inverted result (-Cg) as the location cost Cgeo. Therefore, the larger this location cost Cgeo, the higher the possibility that the detected target does not correspond to the terminal 2 being tracked.

[0111] Next, in the specific example of calculating the appearance cost Capp shown in FIG. 4(B), the appearance cost calculation unit 115c calculates the appearance cost Capp of the detected object as follows: (a) Determined image features of the detected image area (as target appearance information); (b) Image features of the detected image area (as target appearance information) The degree of match (degree of appearance match) Ca is calculated.

[0112] More specifically, at time t0 when it is determined that no occlusion has occurred that would prevent the detection position from being determined (when the target appearance information of the detected target is stored in the target detection information determination unit 113), the appearance cost calculation unit 115c calculates (a) Image features derived using a known CNN image feature extractor for the detected image region of the detected object at this time point t0; (b) Image feature amounts derived using the same CNN image feature extractor as in (a) above for the detected image region of the detected object at the latest (closest to time t0) time t0-N for which object appearance information of the detected object (in the object detection information determination unit 113) is saved, as viewed from this time t0. The similarity between the feature quantities is calculated using a known method for calculating similarity between feature quantities, for example, the method described in Non-Patent Document 2, and the calculated similarity is set as the coincidence Ca at time t0.

[0113] According to this calculation process of the degree of coincidence Ca, even when there is a time point at which occlusion occurs and the detection position cannot be determined, i.e., when N>1, the degree of coincidence Ca at time t0 can be calculated.

[0114] In this embodiment, the appearance cost calculation unit 115c inverts the sign of the degree of match Ca calculated as described above (multiplies the degree of match Ca by -1), and determines this sign inversion result (-Ca) as the appearance cost Capp. Therefore, the larger the appearance cost Ca, the more likely it is that the detected target does not correspond to the device 2 being tracked.

[0115] The calculation process for the communication cost Cradio, the location cost Cgeo, and the appearance cost Capp has been explained above. Below, we will explain the calculation process for the tracking cost Cost using these costs, and the correspondence determination process (tracking target determination process) using the calculated tracking cost Cost.

[0116] (Tracking cost calculation, tracking target determination processing) FIG. 5 is a schematic diagram for explaining a specific example of the tracking cost calculation process and the tracking target determination process according to the present invention.

[0117] As shown in FIG. 5, in this embodiment, the tracking target determination and management unit 115: (a) For each of the detected multiple targets (targets A and B in FIG. 5), a tracking cost Cost is calculated, which is a weighted sum of the calculated location cost Cgeo, appearance cost Capp, and communication cost Cradio, using a "weighting factor λ" that may be zero, where the "weighting factor λ" is applied to the communication cost Cradio; (b) The target (“person”) whose tracking cost is small enough to satisfy a predetermined condition, for example, whose tracking cost is the smallest, is determined to be the target related to the terminal 2 being tracked.

[0118] In this embodiment, the tracking cost is expressed by the following formula: (5) Cost=(1-λ)·(α·Cgeo+β·Capp)+λ·Cradio In the above formula (5), the weighting coefficient λ is a value greater than or equal to 0 and less than or equal to 1. Furthermore, α(≧0) and β(≧0) satisfy α+β=1 and are weighting coefficients that determine the weighting of the position cost Cgeo and the appearance cost Capp, and may take, for example, a preset value.

[0119] As a modification, the weighting coefficient (1-λ) of the first term (environmental information cost) on the right side of the above equation (5) can be replaced with another weighting coefficient, for example, a constant. Furthermore, although this may result in accuracy problems, the above equation (5) can also be modified to use only either the position cost Cgeo or the appearance cost Capp. For example, it is possible to use only the position cost Cgeo as the environmental information cost. Incidentally, when the tracking cost Cost described above is greater than a predetermined threshold C_th, the correspondence determination process may be set not to be performed.

[0120] Here, we will explain the correspondence relationship determination process (tracking target determination process) using the tracking cost Cost of the above formula (5) when occlusion occurs for a detected target. When such occlusion is resolved, in this embodiment, the weight determination unit 115d of the tracking target determination and management unit 115 determines a weighting factor λ that is a monotonically increasing function of the maximum value (maxCr) of the degree of match Cr (communication compatibility degree) calculated for each detected target. That is, the value of the weighting factor λ that is preset as such a monotonically increasing function is determined by substituting the maximum value (maxCr) of the calculated degree of match Cr. For example, the following formula (6) When maxCr>K, λ=λ1 When maxCr≦K, λ=λ0(λ0<λ1) The weighting factor λ may be determined by:

[0121] Here, K is a predetermined threshold that defines the boundary for whether the communication cost Cradio should be reflected more significantly in the tracking cost Cost. For example, if λ1 = 1 and λ0 = 0 in the above formula (6), then depending on whether the maximum value (maxCr) of the matching score Cr (communication compatibility level) exceeds the K value, the tracking cost Cost will either be the communication cost Cradio itself or the environmental information cost (α·Cgeo + β·Capp) itself.

[0122] In another embodiment, the weight determining unit 115d uses the following equation using a sigmoid function: (7) λ=1 / (1+e -a(maxCr+k) ) Here, a is the gain of the sigmoid function. Furthermore, k is an offset value relative to the maximum value (maxCr) of the degree of coincidence Cr, and is set in advance. Incidentally, when using the above formula (7), the communication cost calculation unit 115a uses the cross-correlation (for example, Σ t It is preferable to use a value less than 0 in E_t*E'_t rounded up to 0.

[0123] Here, the degree of match Cr (communication readiness) can be determined from the communication state of terminal 2 and the estimated communication state of the target to be determined (target A or target B) obtained from environmental information (e.g., image data), regardless of whether the target to be determined (target A or target B in FIG. 5) is detected during the process. In other words, the degree of match Cr (communication readiness) is likely to be used as a more reliable determination criterion even when occlusion occurs. As a result, as described above, in this embodiment, by determining the weighting coefficient λ that is a monotonically increasing function of the maximum value (maxCr) of the degree of match Cr (communication readiness), it is possible to calculate and determine a more appropriate (actually appropriate) tracking cost (Cost) even when occlusion occurs. This also makes it possible to improve the accuracy of terminal tracking.

[0124] Of course, when the degree of match Cr (communication compatibility degree) is considerably low, if a tracking target determination process is performed that places emphasis on the communication cost Cradio or that takes only the communication cost Cradio into consideration, there is a concern that the accuracy of terminal tracking may decrease. In response to this, the tracking target determination and management unit 115 of this embodiment determines a smaller weighting factor λ in such a situation, thereby avoiding or minimizing the decrease in the accuracy of terminal tracking. On the other hand, when the degree of match Cr (communication compatibility degree) is higher, a larger weighting factor λ is determined and a tracking target determination process that places a greater emphasis on the communication cost Cradio is performed, thereby maintaining or improving the accuracy of terminal tracking even when occlusion occurs.

[0125] Here, the weighting factor λ, which is a monotonically increasing function of the maximum value (maxCr) of the degree of match Cr (communication readiness) for each detected object, as in Equation (6) or Equation (7), may be set to be used for a predetermined period of time immediately after occlusion is resolved. In this case, outside of this predetermined period, the weighting factor λ can be set to a predetermined value (e.g., a small value such as 0.1) or zero. For example, if the weighting factor λ is set to zero, the correspondence determination process (tracking target determination process) is performed based on the environmental information cost (Cgeo, Capp) that is determined to be highly reliable in a situation where no occlusion occurs. Note that the weighting factor λ, as in Equation (6) or Equation (7), can also be set at all times, not just when occlusion is resolved.

[0126] Next, an embodiment will be described in which the degree of match Cg (degree of positional match) is also taken into consideration when determining the weighting factor λ. In this case, when the occlusion is resolved, the weight determining unit 115d determines a weighting factor λ that is a monotonically increasing function of both the maximum value (maxCr) of the degree of match Cr (degree of communication compatibility) of each detected object and the maximum value (maxCg) of the degree of match Cg (degree of positional match) of each detected object (when one is a constant, the other is a monotonically increasing function). For example, (a) When the maximum value (maxCg) of the degree of match Cg (degree of positional match) of each detected object is greater than a predetermined threshold Kg (maxCg>Kg), the above formula (6) or (7) is followed, but (b) If the maximum value (maxCg) of the match degree Cg (degree of positional match) of each detected object is equal to or less than a predetermined threshold Kg (maxCg≦Kg) and the maximum value (maxCr) of the match degree Cr (degree of communication compatibility) of each detected object is also equal to or less than a predetermined threshold K' (maxCr≦K' (>K)), the value is set to zero or a small value less than a predetermined value (here, if maxCr>K' even if maxCg≦Kg, the above formula (6) or (7) is followed). In another embodiment, it is also possible to determine a weighting factor λ that is a sigmoid function for both the maximum value (maxCr) of the degree of coincidence Cr and the maximum value (maxCg) of the degree of coincidence Cg in accordance with the above formula (7) (when one is a constant, the other becomes a sigmoid function).

[0127] By determining such a weighting coefficient λ, the proportion of the communication cost Cradio in the tracking cost Cost can be set to zero or reduced in situations where the degree of match Cg (degree of positional match) is considerably low. In such situations, even if the degree of match Cr (degree of communication compatibility) is high, the reliability of the communication cost Cradio calculated from the degree of match Cr (degree of communication compatibility) is deemed to be reduced because the target detection position is used to calculate the degree of match Cr. Therefore, by determining the weighting coefficient λ as described above and setting to zero or reducing the proportion of the communication cost Cradio in the tracking cost Cost, it is possible to maintain or improve the accuracy of terminal tracking.

[0128] To summarize the correspondence determination process described above, the tracking target determination and management unit 115 performs the correspondence determination process without basing the degree of match Cr (communication compatibility degree) on the degree of match Cr (communication compatibility degree) when the degree of match Cg (positional match degree) is small enough to satisfy the predetermined condition (in this embodiment, when the maximum value maxCr of the degree of match Cr of each detected target is equal to or less than a predetermined threshold value), or performs the correspondence determination process by making the degree based on the degree of match Cr (communication compatibility degree) smaller the degree of match Cg (positional match degree) (in this embodiment, the smaller the maximum value maxCg of the degree of match Cg of each detected target is), the smaller the degree of match Cr (communication compatibility degree).

[0129] As a preferred modification, the tracking target determination and management unit 115 (a) When the degree of match Cg (degree of positional match) is smaller than a predetermined condition (for example, when the maximum value maxCg of the degree of match Cg of each detected object is equal to or smaller than a predetermined threshold), a correspondence determination process is performed based on at least the degree of match Cg (degree of positional match) and the degree of match Ca (degree of appearance match), and at that time, it is preferable to increase the degree based on the degree of match Ca (degree of appearance match), specifically, to increase the value of the weighting coefficient β in the above formula (5) (i.e., to decrease the value of the weighting coefficient α (= β-1)), or (b) It is also preferable to perform a correspondence determination process based on at least the degree of match Cg (degree of positional match) and the degree of match Ca (degree of appearance match), and the smaller the degree of match Cg (degree of positional match) (for example, the smaller the maximum value maxCg of the degree of match Cg of each detected object), the greater the degree based on the degree of match Ca (degree of appearance match), specifically the greater the value of the weighting coefficient β.

[0130] In addition, such adjustment of the weighting coefficient β based on the degree of coincidence Cg (degree of positional coincidence) can be performed when performing correspondence relationship determination processing based on at least the degree of coincidence Cg (degree of positional coincidence) and the degree of coincidence Ca (degree of appearance coincidence), including when it is also based on the degree of coincidence Cr (degree of communication compatibility).

[0131] In any case, when the degree of match Cg (degree of location match) is considerably low, it is determined that the reliability of not only the communication cost Cradio calculated from the degree of match Cr (degree of communication compatibility) but also the value of the location cost Cgeo calculated from this degree of match Cg itself is reduced. Therefore, as mentioned above, by increasing the value of the weighting coefficient β and lowering the weight of the location cost Cgeo (increasing the weight of the appearance cost Capp calculated from the degree of match Ca), it is possible to maintain and improve the accuracy of terminal tracking.

[0132] Also referring to FIG. 5, the tracking target determination and management unit 115 calculates the location costs Cgeo_A and Cgeo_B, appearance costs Capp_A and Capp_B, and communication costs Cradio_A and Cradio_B of the target A (“person”) and the target B (“person”) at time t0, and calculates the tracking costs Cost_A and Cost_B at time t0 using the following equations: (8) Cost_A=(1-λ)·(α·Cgeo_A+β·Capp_A)+λ·Cradio_A (9) Cost_B=(1-λ)·(α·Cgeo_B+β·Capp_B)+λ·Cradio_B Here, the weighting coefficient λ and the weighting coefficients α and β are determined by the determination process described above.

[0133] Next, in this embodiment, the tracking target determination and management unit 115 determines target A ("person") with the smallest tracking cost (the one with the smaller tracking cost in the case of FIG. 5) as the target related to the terminal 2 being tracked, and continues tracking that terminal 2. Incidentally, in FIG. 5, there is no difference between the two in appearance cost Capp, but both the location cost Cgeo and communication cost Cradio are clearly smaller for target A ("person").

[0134] As described above, the weighting coefficient λ and the weighting coefficients α and β are set to more suitable values ​​by taking into consideration the degree of coincidence Cr (degree of communication compatibility) in particular, even when occlusion occurs with respect to a detected object whose correspondence with terminal 2 is to be determined. As a result, even when occlusion occurs with respect to terminal 2 being tracked, it becomes possible to continue tracking terminal 2 more appropriately (in FIG. 5, between target A and target B, the one that is holding or carrying terminal 2 is determined to be the tracking target).

[0135] Returning to the functional block diagram of Fig. 1, the tracking target determination and management unit 115 stores and manages the results of the tracking target determination process (correspondence determination process) for each tracked terminal 2, which have been explained above with reference to various embodiments and specific examples, in the tracking information storage unit 105, and also outputs the results to the communication control unit 116 constantly, periodically, or as needed in response to a request. Here, the results of this tracking target determination process (correspondence determination process) can be time-series data in which the location and image features at each point in time are linked for each terminal ID of the terminal 2, for example, and are thus suitable in that the adverse effects of occlusion (for example, loss, ID switch, etc.) are avoided or reduced.

[0136] The communication control unit 116 can more accurately grasp the situation including the current location and communication path of the connected terminal 2 by using the result of the tracking target determination process (correspondence determination process) of such a suitable terminal 2. This also makes it possible to continue the communication connection with this terminal 2 or immediately resume it, even if a situation occurs in which the received power from the connected terminal 2 drops suddenly.

[0137] As described above in detail, according to the present invention, when an occlusion occurs, a process of determining whether a tracked terminal and a detected target are in a correspondence relationship can be performed based on the degree of communication correspondence between the terminal and the detected target. As a result, even when an occlusion occurs for the tracked terminal, it is possible to continue tracking the terminal more appropriately.

[0138] Furthermore, the terminal tracking process according to the present invention can be applied to various fields and situations related to the currently being introduced 5G and future mobile communication systems that use shorter wavelength radio waves. For example, it can solve the problem of communication failure due to obstructions in the communication path, notify or warn automobiles equipped with terminals of approaching people or other vehicles, and even determine the relationship between the pages viewed by a user determined to be in correspondence with a certain terminal and the user's movement path, which can be used for marketing and management.

[0139] Furthermore, by popularizing information and communication systems including the device tracking device of the present invention, which can stably realize a huge number of high-capacity communication connections, it will be possible to widely deploy mobile and cellular communication services that strengthen people's connections and empower communities. Furthermore, information and communication systems including the device tracking device of the present invention are expected to be particularly effective in urban areas where the global population is growing. In other words, the present invention can contribute to achieving Goal 9 of the United Nations' Sustainable Development Goals (SDGs), "Build resilient infrastructure, promote sustainable industrialization and foster innovation," and Goal 11, "Make cities inclusive, safe, resilient and sustainable."

[0140] In the various embodiments of the present invention described above, various changes, modifications, and omissions within the scope of the technical idea and perspective of the present invention can be easily made by those skilled in the art. The above description is merely an example and is not intended to limit the present invention in any way. The present invention is limited only by the scope of the claims and their equivalents. [Explanation of symbols]

[0141] 1 Base station (terminal tracking device, communication relay device) 101 Communication Interface 102 Communication history information storage unit 103 Camera (sensor) 104 Environmental Information Storage Department 105 Tracking Information Storage Unit 111 Terminal communication information determination unit 112 Target communication information determination unit 112a Position interpolation unit 113 Target detection information determination unit 114 Occlusion determination unit 115 Tracking Target Determination and Management Department 115a Communication cost calculation unit 115b Location cost calculation unit 115c Appearance cost calculation section 115d Weight determination unit 116 Communication control section 2. Terminal

Claims

1. an object detection information determination means for detecting a predetermined object which may include the terminal from environmental information related to an environment in which the terminal may be present, the environmental information being generated by the sensor, and determining object position information which is information related to a detection position of the detected object; a terminal communication information determination means for determining terminal communication information, which is information relating to a communication state between the terminal and the communication information determination means, based on information relating to communication acquired from a communication means for communicating with the terminal; a target communication information determination means for determining target communication information, which is information related to a communication state to be realized between the detected target and the target if the detected target includes a terminal, from the environmental information; a tracking target determination means for determining whether or not the tracking terminal and the detected target are in a corresponding relationship based on at least one of a position coincidence degree, which is a degree of coincidence between the target location information and predicted target location information predicted from the target location information of the target determined to be related to the tracking terminal, and a communication compatibility degree, which is a degree of correspondence between the terminal communication information of the tracking terminal and the target communication information, and determining the target determined to be in a corresponding relationship as the target related to the tracking terminal; having The tracking target determination means performs the determination based on the degree of communication compatibility, not on the positional agreement degree, when the degree of communication compatibility is large enough to satisfy a predetermined condition, at least when the occlusion of the detected target is resolved, or performs the determination by increasing the degree based on the degree of communication compatibility as the degree of communication compatibility becomes larger or when the degree of communication compatibility is large enough to satisfy a predetermined condition. A terminal tracking device comprising:

2. The tracking target determination means, when the degree of communication compatibility is not large enough to satisfy a specified condition, makes the determination without basing the degree of communication compatibility on the degree of positional matching being not large enough to satisfy a specified condition, or, the smaller the degree of positional matching is, the smaller the degree of communication compatibility is before making the determination.The terminal tracking device of claim 1,

3. The object detection information determination means also determines object appearance information, which is information related to a detected appearance of the detected object, from the environmental information; When the tracking target determination means performs the determination based on at least the position matching degree, the tracking target determination means also performs the determination based on an appearance matching degree, which is a degree of matching between the target appearance information and past target appearance information of the target determined to be related to the terminal being tracked.

3. The terminal tracking device according to claim 1 or 2.

4. The tracking target determination means When the degree of positional agreement is smaller than a predetermined condition, the determination is made based on at least the degree of appearance agreement and the degree of positional agreement, and at that time, the degree based on the degree of appearance agreement is made larger, or The determination is made based on at least the degree of appearance match and the degree of position match, and the smaller the degree of position match, the larger the degree based on the degree of appearance match.

4. The terminal tracking device according to claim 3.

5. The terminal tracking device according to any one of claims 1 to 4, characterized in that the target communication information determination means determines the target communication information by assuming that the target is present at a position interpolated using detection positions determined before and after the occlusion during a time period in which an occlusion occurs and the detection position of the target is not determined.

6. the environmental information is information relating to an image of the environment, and the target position information includes a position and a range within the image of an image area relating to the detected target; The degree of positional agreement is the degree of overlap between the range of the detected target and the predicted range in the image of the target determined by the device being tracked, or the degree of closeness between the position of the detected target and the predicted position in the image of the target determined by the device being tracked.

6. The terminal tracking device according to claim 1,

7. The tracking target determination means determining a weighting factor that is a monotonically increasing function of the degree of communication capability, the weighting factor including a zero value; A tracking cost of the target is calculated, which is a weighted sum of an environmental information cost including a location cost which is a monotonically decreasing function of the degree of location agreement, and a communication cost which is a monotonically decreasing function of the degree of communication compatibility, the communication cost being weighted by the weighting coefficient, and the environmental information cost being weighted by another weighting coefficient which may take a zero value, and a target for which the tracking cost is small enough to satisfy a predetermined condition is determined to be a target related to the terminal being tracked.

7. The terminal tracking device according to claim 1,

8. 8. The terminal tracking device according to claim 7, wherein the tracking target determination means determines the weighting coefficients which are monotonically increasing functions for both the communication compatibility degree and the position agreement degree.

9. an object detection information determination means for detecting a predetermined object which may include the terminal from environmental information related to an environment in which the terminal may be present, generated by a sensor, determining object position information which is information related to a detected position of the detected object, and also determining object appearance information which is information related to a detected appearance of the detected object from the environmental information; a terminal communication information determination means for determining terminal communication information, which is information relating to a communication state between the terminal and the communication information determination means, based on information relating to communication acquired from a communication means for communicating with the terminal; a target communication information determination means for determining target communication information, which is information related to a communication state to be realized between the detected target and the target if the detected target includes a terminal, from the environmental information; a tracking target determination means for determining whether or not the tracking terminal and the detected target are in a corresponding relationship based on at least one of a position matching degree, which is a degree of matching between the target location information and predicted target location information predicted from the target location information of the target determined to be related to the tracking terminal, an appearance matching degree, which is a degree of matching between the target appearance information and past target appearance information of the target determined to be related to the tracking terminal, and a communication compatibility degree, which is a degree of compatibility between the terminal communication information of the tracking terminal and the target communication information, and determining the target determined to be in a corresponding relationship as the target related to the tracking terminal; having The tracking target determination means performs the determination based on the degree of communication compatibility, not on the positional agreement degree, when the degree of communication compatibility is large enough to satisfy a predetermined condition, at least when the occlusion of the detected target is resolved, or performs the determination by increasing the degree based on the degree of communication compatibility as the degree of communication compatibility becomes larger or when the degree of communication compatibility is large enough to satisfy a predetermined condition. A terminal tracking device comprising:

10. an object detection information determination means for detecting a predetermined object which may include the terminal from environmental information related to an environment in which the terminal may be present, the environmental information being generated by the sensor, and determining object position information which is information related to a detection position of the detected object; a terminal communication information determination means for determining terminal communication information, which is information relating to a communication state between the terminal and the communication information determination means, based on information relating to communication acquired from a communication means for communicating with the terminal; a target communication information determination means for determining target communication information, which is information related to a communication state to be realized between the detected target and the target if the detected target includes a terminal, from the environmental information; a tracking target determination means for determining whether or not the tracking terminal and the detected target are in a corresponding relationship based on at least one of a position coincidence degree, which is a degree of coincidence between the target location information and predicted target location information predicted from the target location information of the target determined to be related to the tracking terminal, and a communication compatibility degree, which is a degree of correspondence between the terminal communication information of the tracking terminal and the target communication information, and determining the target determined to be in a corresponding relationship as the target related to the tracking terminal; to make the computer function, The tracking target determination means performs the determination based on the degree of communication compatibility, not on the positional agreement degree, when the degree of communication compatibility is large enough to satisfy a predetermined condition, at least when the occlusion of the detected target is resolved, or performs the determination by increasing the degree based on the degree of communication compatibility as the degree of communication compatibility becomes larger or when the degree of communication compatibility is large enough to satisfy a predetermined condition. A terminal tracking program comprising:

11. a first step of detecting a predetermined target which may include the terminal from environmental information generated by a sensor and related to an environment in which the terminal may be present, determining target position information which is information related to a detection position of the detected target, and also determining terminal communication information which is information related to a communication state between the terminal and the detected target based on information related to communication acquired from a communication means for communicating with the terminal, and further determining target communication information from the environmental information which is information related to a communication state which will be realized between the detected target and the target if the detected target includes the terminal; a second step of determining whether or not the tracked terminal and the detected target are in a corresponding relationship based on at least one of a position match degree, which is a degree of match between the target location information and predicted target location information predicted from the target location information of the target determined to be related to the tracked terminal, and a communication compatibility degree, which is a degree of correspondence between the terminal communication information of the tracked terminal and the target communication information, and determining the target determined to be in a corresponding relationship as the target related to the tracked terminal; having In a second step, when the occlusion is resolved for at least the detected target, if the degree of communication compatibility is large enough to satisfy a predetermined condition, the determination is made based on the degree of communication compatibility, not on the degree of positional agreement, or, if the degree of communication compatibility is large enough to satisfy a predetermined condition, the degree based on the degree of communication compatibility is made larger.

13. A computer-implemented method for tracking a terminal, comprising:

Images