Information processing device, information processing method, program

Abstract

A decrease in the accuracy of tracking moving objects from images. [Solution] The information processing device of the present disclosure includes: a detection unit that detects objects from images over time and calculates the reliability of the detected objects; a linking unit that sets linking information that links the objects detected over time; a calculation unit that calculates the value of the linking information based on the reliability of the objects linked by the linking information; and a setting unit that decides whether or not to adopt the linking information that links the objects based on the value of the linking information, and sets the linking of the objects by the adopted linking information as the trajectory of the objects.

Description

[Technical Field]

[0001] This disclosure relates to an information processing device, an information processing method, and a program. [Background technology]

[0002] Tracking moving objects such as people using captured images is being practiced. For example, Patent Document 1 describes tracking people and mobile machines from images captured by cameras in workplaces and facilities to understand their actions. [Prior art documents] [Patent Documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2020-098590 [Overview of the project] [Problems that the invention aims to solve]

[0004] However, when tracking a moving object from an image, a problem arises: the tracking accuracy decreases depending on the image's capture environment. For example, if the image is distorted or the target moving object is hidden by other objects, the tracking accuracy of the object will decrease.

[0005] Therefore, one of the purposes of this disclosure is to solve the aforementioned problem of reduced accuracy in tracking moving objects from images. [Means for solving the problem]

[0006] An information processing device, which is one form of this disclosure, A detection unit that detects objects from images taken over time and calculates the reliability of the detected objects, A coupling unit that sets coupling information to link objects detected at time intervals, A calculation unit that calculates the value of the linked information based on the reliability of the objects linked by the linked information, Based on the value of the connection information, determine whether to adopt the connection information for connecting between objects, and set the connection between objects by the adopted connection information as the trajectory of the object, a setting unit; Comprising adopts the following configuration. Also, an information processing method according to an aspect of the present disclosure detects an object from an image for each time, and calculates the reliability of the detected object, sets connection information for connecting between objects detected for each time, calculates the value of the connection information based on the reliability of the objects connected by the connection information, based on the value of the connection information, determine whether to adopt the connection information for connecting between objects, and set the connection between objects by the adopted connection information as the trajectory of the object, adopts the following configuration. Also, a program according to an aspect of the present disclosure to an information processing apparatus detect an object from an image for each time, and calculate the reliability of the detected object, set connection information for connecting between objects detected for each time, calculate the value of the connection information based on the reliability of the objects connected by the connection information, based on the value of the connection information, determine whether to adopt the connection information for connecting between objects, and set the connection between objects by the adopted connection information as the trajectory of the object, cause to execute processing adopts the following configuration.

Advantages of the Invention

[0007] The present disclosure is configured as described above, and thus can improve the tracking accuracy of a moving object from an image.

Brief Description of the Drawings

[0008] [Figure 1] It is a block diagram showing an example of the configuration of an information processing apparatus according to the present disclosure. [Figure 2]This is a diagram showing an example of the processing state of the information processing apparatus according to the present disclosure. [Figure 3] This is a diagram showing an example of the processing state of the information processing apparatus according to the present disclosure. [Figure 4] This is a flowchart showing an example of the processing operation of the information processing apparatus according to the present disclosure. [Figure 5] This is a block diagram showing an example of the hardware configuration of the information processing apparatus according to the present disclosure. [Figure 6] This is a block diagram showing an example of the configuration of the information processing apparatus according to the present disclosure.

Embodiments for Carrying Out the Invention

[0009] <First Embodiment> The first embodiment of the present disclosure will be described with reference to the drawings. Note that the drawings may be relevant to any of the embodiments.

[0010] The information processing apparatus of the present disclosure is used to track a moving object such as a person reflected in an image using the image. As an example, the present disclosure is used to track a worker at a work site such as a factory and recognize or record the work content. However, the moving object to be tracked in the present disclosure is not limited to a person, and may be any object such as a work robot or an animal.

[0011] The information processing system consists of one or more information processing units equipped with an arithmetic unit and a memory device. As shown in Figure 1, the information processing system includes a video supply unit 1, an object detection unit 2, a graph construction unit 3, a value calculation unit 4, a confidence addition calculation unit 5, a constraint enumeration unit 6, an optimization unit 7, a linked part enumeration unit 8, a trajectory selection unit 9, a trajectory joining unit 10, and a duplicate trajectory integration unit 11. The video supply unit 1 is composed of a memory device. Furthermore, the functions of the object detection unit 2, graph construction unit 3, value calculation unit 4, confidence addition calculation unit 5, constraint enumeration unit 6, optimization unit 7, linked part enumeration unit 8, trajectory selection unit 9, trajectory joining unit 10, and duplicate trajectory integration unit 11 can be realized by the arithmetic unit executing a program for realizing each function stored in the memory device. The configuration and operation of each unit will be described in detail below.

[0012] The video supply unit 1 supplies image data captured by the camera at the shooting location, such as a work site, to the object detection unit 2 (step S1 in Figure 4). The image data consists of a time series of frame images taken at different points in time.

[0013] The object detection unit 2 (detection unit) detects objects in each frame image at each time interval and outputs a detection result (coordinates, object class, confidence level, feature quantity) for each object detected from each frame image, consisting of coordinates, object class, detection confidence level, and feature quantity (step S2 in Figure 4). At this time, the coordinates are real values ​​representing the position and size of the bounding box (e.g., the coordinates of the center pixel, width, and height (e.g., the coordinates of the top-left and bottom-right vertices of the bounding box)). The object class is the type of object, such as a person or a mobile machine. The detection confidence level is the degree of confidence in the detection result, for example, the degree of confidence that the detected object is an object or the degree of confidence that it belongs to the detected object class. At this time, the confidence level is a real number from 0 to 1, and the closer it is to 1, the higher the reliability of the detection result. The feature quantity is, for example, the coordinates, object class, and the pixels of the frame image region indicated by the coordinates. It is a fixed-dimensional vector calculated from the values. For example, the feature vector may be the result of processing a patch extracted from the object rectangular region of the frame image using a convolutional neural network. Alternatively, the feature vector may be the result of processing a vector of 0s and 1s representing coordinates or object classes using a multilayer perceptron. Alternatively, the feature vector may be the result of merging the results of the two examples above by concatenating or summing them. Note that the above examples of detection results (coordinates, object class, confidence level, feature vector) are just examples, and the information about the object detected from the frame image can be any kind of information.

[0014] The graph construction unit 3 (connection unit) constructs a graph by connecting objects detected from each frame image using connection information, as described above (step S3 in Figure 4). Specifically, the graph construction unit 3 constructs a graph where each object, which is a detection result included in a predetermined time window, is a vertex, and edges, which are connection information, are placed between pairs of vertices that may be the same object and may be temporally adjacent on the object's trajectory. For example, as shown in Figure 2(2-1), a graph is set up where vertices corresponding to objects (detection results) detected from the frame image at each time are nodes N, and edges E are edges that connect temporally adjacent objects.

[0015] In the process of setting edges E by the graph construction unit 3, the above-mentioned "temporally adjacent on the trajectory" also considers the detection of objects immediately before and after a gap in a trajectory with gaps (such as when an object is hidden behind something) as "adjacent." For example, pairs of objects that are less than or equal to a certain temporal interval may be considered adjacent, pairs of objects with a similarity of object features above a certain level may be considered adjacent, and pairs of objects with a coordinate difference of less than or equal to a certain level may be considered adjacent. Alternatively, instead of performing thresholding on the above three criteria or a combination thereof, the k nearest neighbors measured using the above criteria may be considered adjacent.

[0016] The graph construction method described above by the graph construction unit 3, that is, the method of constructing a graph by connecting nodes N corresponding to objects detected from each frame image with edges E, is not limited to the method described above, and the graph may be constructed by other methods.

[0017] The value calculation unit 4 (calculation unit) calculates the value of each edge E from the graph constructed as described above (step S4 in Figure 4). At this time, the value calculation unit 4 calculates the value of edge E based on the detection results of each object corresponding to each node N connected to both ends of edge E. For example, the value calculation unit 4 calculates the value of edge E based on the similarity of the feature quantities of the objects corresponding to each node N connected to both ends of edge E. As an example, the value calculation unit 4 calculates a higher value the higher the similarity of the feature quantities of the objects. Note that the value calculation unit 4 may calculate the value of edge E using any method or information. For example, it may input the detection results, such as the feature quantities of the objects corresponding to the nodes N connected to edge E, into a pre-trained neural network and use its output as the value.

[0018] The confidence addition calculation unit 5 (calculation unit) further calculates the value to be added to the value of each edge E from the graph constructed as described above (step S4 in Figure 4). In particular, the confidence addition calculation unit 5 calculates the value to be added to edge E based on the confidence of each object corresponding to each node N connected to both ends of edge E. For example, the confidence addition calculation unit 5 first checks whether the confidence of the two nodes N, i.e., objects connected to both ends of edge E, is above a predetermined threshold. Then, the value calculation unit 4 checks which of the three stages the confidence of the two objects corresponds to: both are above the threshold, only one is above the threshold, or both are below the threshold, and calculates the value according to each stage. In this case, as an example, the value is calculated to increase in the order of both confidence levels being above the threshold, only one being above the threshold, and both being below the threshold, and the higher the confidence of the connected objects, the higher the value is calculated. The value to be added is set to a large value such as 1000 or 100000 when the confidence levels of both objects are above the threshold or when only one of them is above the threshold, and is set to a value greater than the value of edge E calculated by the value calculation unit 4 as described above. The method for calculating the value to be added to edge E described above is just one example, and it may be calculated by other methods. For example, the value to be added may be calculated as a constant multiple (e.g., 1000) of the result of applying a temperature-controlled sigmoid function to the sum of the confidence levels of the two objects connected to edge E minus twice the threshold.

[0019] As described above, each edge E of the graph is associated with the value calculated by the value calculation unit 4 plus a value based on the reliability calculated by the reliability addition calculation unit 5. In this case, the value based on reliability is calculated with a greater weight, so edges E that connect nodes N of objects with high reliability will be associated with higher values.

[0020] The constraint enumeration section 6 enumerates the constraints that the graph representing the set of trajectories must satisfy. Examples of constraints include: "Each node N corresponding to an object representing a detection can be connected to at least one detection that occurs at a time earlier than that detection," and "Each node N corresponding to an object representing a detection can be connected to at least one detection that occurs at a time later than that detection." However, other conditions may also be enumerated as constraints.

[0021] The optimization unit 7 (setting unit) decides whether to adopt or reject edge E based on the value of edge E in the graph (step S5 in Figure 4). Specifically, the optimization unit 7 prioritizes adopting graphs that satisfy the above-mentioned constraints and have a high value for edge E. For example, the optimization unit 7 may adopt edge E whose value is above a predetermined threshold or above a predetermined rank from the top. As a result, edge E whose value is calculated to be high due to the high reliability of the objects connected to edge E will be prioritized for adoption. In this case, for example, edge E may also be adopted if its value is calculated to be high because the reliability of only one of the objects connected to edge E is above the threshold. In other words, edge E may be adopted even if the reliability of one of the objects connected to edge E is above the threshold and the reliability of the other object is below the threshold.

[0022] The process by which the optimization unit 7 accepts or rejects edge E can also be expressed as follows: The optimization unit 7 calculates the graph that satisfies all the constraints and has the maximum sum of values ​​for edge E. i For this, the 0 / 1 variable x determines whether or not to adopt that edge. i Assign a value to that edge and the value of that edge is v i Therefore, under the constraints, Σ i v i x i This becomes an integer linear programming problem that maximizes . Since the constraints are unimodular, it is guaranteed that it can be solved as a (continuous) linear programming problem, and therefore a linear programming solver can actually be used.

[0023] As described above, the optimization unit 7 decides whether to accept or reject edge E and outputs the graph containing the accepted edge E to the connected part enumeration unit 8. As an example, edge E as shown by the solid line in Figure 2(2-2) is accepted and a graph T consisting of the black circle node N and the solid line edge E is output.

[0024] The linked part enumeration unit 9 (setting unit) outputs a set of graphs that have been enumerated by the optimization unit 8. At this time, the graph output by the optimization unit 8, in which edge E is selected, is considered to be part of the trajectory of the object corresponding to node N.

[0025] The trajectory selection unit 10 (setting unit) takes the trajectory graph output by the connected part enumeration unit 9 as input and selects graphs to be used as trajectories by determining which graphs to discard from the trajectory based on the confidence level of the objects corresponding to the nodes N that make up the graph (step S6 in Figure 4). For example, for each graph input as a trajectory, the trajectory selection unit 10 calculates the trajectory confidence level from the confidence levels of the objects corresponding to the connected nodes N, and discards graphs whose trajectory confidence level is below a threshold. In this case, the trajectory confidence level is the average or maximum confidence level of the two objects corresponding to the nodes N connected to the graph. As a result, after adopting the edge E as described above to make the graph a trajectory, graphs with low confidence levels of the objects corresponding to the nodes N connected to the edge E are discarded from the trajectory. The trajectory selection unit 10 may also determine which graphs to discard by other methods based on the confidence levels of the objects corresponding to the nodes N that make up the graph. Then, as described above, the trajectory selection unit 10 discards the graphs from the trajectory and outputs the remaining graphs as trajectories.

[0026] The trajectory joining unit 10 (joining unit) joins multiple graphs that have been designated as trajectories to generate and output an object trajectory (joined trajectory) (step S7 in Figure 4). Specifically, the trajectory joining unit 10 joins each graph listed as a trajectory for different time windows and outputs a long-term object trajectory. At this time, the trajectory joining unit 10 considers the result of processing two overlapping time windows using the sliding window method, and if the overlap of trajectories included in the two time windows is large, it treats them as a single trajectory and joins them. Since each time window contains multiple trajectories, the joining is performed using Hungarian matching to maximize the sum of the overlaps of trajectory pairs.

[0027] The overlapping trajectory integration unit 11 (joining unit) takes the set of object trajectories output by the trajectory joining unit 10 as input and integrates and outputs the object trajectories with the greatest overlap among the input object trajectories. Specifically, the overlapping trajectory integration unit 11 selects the portion of one object trajectory and discards the portion of the other object trajectory from among the multiple object trajectories where the object trajectories overlap at the same time, that is, the portion where the trajectories coexist at the same time, and further connects and integrates the multiple object trajectories. At this time, the overlapping trajectory integration unit 11 selects the portion of the object trajectory with high object reliability and discards the portion of the object trajectory with low object reliability, and further connects and integrates the multiple object trajectories.

[0028] As an example, in the example shown in Figure 3, there are two object trajectories T1 and T2, shown by a dotted line and a dashed line respectively, and these trajectories coexist at the same time R. In this case, the reliability of the objects in each object trajectory T1 and T2 at the same time R is examined, and a portion of the object trajectory with the higher reliability is adopted. As a result, in the example shown in Figure 3, at the same time R, the portion of object trajectory T1 is adopted, the portion of object trajectory T2 (dashed line) is discarded, and the two portions are further connected and integrated to form object trajectory T10, as shown by the solid line.

[0029] Here, we will explain a specific example of the processing performed by the duplicate trajectory integration unit 11. The duplicate trajectory integration unit 11 repeats the following steps (1) to (3).

[0030] (1) The duplicate trajectory integration unit 11 extracts, from the set τ of object trajectories output by the trajectory junction unit 10 shown in Equation (1), a pair of trajectories (Traj j , Traj k ) that satisfies the duplication condition and has the largest duplication degree among those that satisfy the duplication condition. If there is no pair that satisfies the duplication condition, the iteration ends.

Equation

[0031] As an example of the duplication condition, as shown in Equation (2), there is at least one frame in which Traj j and Traj k overlap, and for all frames in which Traj j and Traj k overlap, the overlap (measured by the Intersection over Union (IoU) of the bounding boxes) between the object in frame t included in Traj j and the object in frame t included in Traj k is greater than the threshold θ l , and the ratio of the frames in which the above IoU is greater than or equal to the threshold θ h to the frames in which Traj j and Traj k overlap is greater than or equal to θ r .

Equation

Equation

[0032] Also, as an example of the duplication degree, it is the average value of the above IoU over all frames in which Traj j and Traj k overlap.

[0033] (2) For the pair Traj j , Traj​​​​k Integrating them into one trajectory Traj l Create an object in a certain frame t. j and Traj k If both are present, the one with higher confidence is adopted. For frames without detection overlap, the original trajectory (Traj) is used. j or Traj k We will use the objects included in it. (3) Let's use equation 4.

number

[0034] [Differentiation] Next, a modified version of the processing by the information processing device described above will be explained. In the above, node N of the graph was considered as an optimization problem on the graph where the detected object is located, but the object to be detected corresponding to node N may be a trajectory fragment of an object. Here, a trajectory fragment is defined as a short fragment of a trajectory consisting of the same object. In other words, in the processing by the information processing device described above, the object detected from the frame image may be a short fragment of the object's trajectory. Note that trajectory fragments can be generated using other trackers or the techniques described so far. As a result, the detection result of the trajectory fragment will have a time width in addition to coordinates and time, and motion features such as velocity can be calculated. Accordingly, the graph construction unit 3 and the value calculation unit 4 can also use criteria based on motion features. Note that the confidence level of trajectory fragment detection can be defined as the confidence level that the detection result is a trajectory fragment, and the maximum or average value of the confidence level of the detection result can be used.

[0035] Furthermore, the reliability addition calculation unit 5 and the trajectory selection unit 9 perform the same operation as above, except that the node N considered has changed from an object to a trajectory fragment, and the value of edge E etc. is calculated from the reliability of both nodes N. Also, the trajectory joining unit 10 and the overlapping trajectory integration unit 11 can join trajectory fragments to create an object trajectory, as described above.

[0036] As described above, the information processing device of this disclosure concatenates objects detected from each frame, calculates the value of the degree of connection based on the confidence level between objects, and generates the object's trajectory based on this connection value. Therefore, even with fluctuations in the shooting environment, such as when the image is distorted or an object is hidden by another object, it is possible to track objects with greater accuracy and improve tracking accuracy. Furthermore, since a decision to discard an object is made based on the confidence level of the object detection after setting up the graph representing the object's tracking, it is possible to suppress lowering the confidence threshold that serves as the basis for constructing the graph or selecting objects as trajectories, thereby suppressing excessive detection and tracking.

[0037] <Second Embodiment> Next, a second embodiment of the present disclosure will be described with reference to the drawings. This embodiment shows an outline of the information processing device, etc., described in the above-described embodiment. Note that the drawings may be relevant to any embodiment.

[0038] First, the hardware configuration of the information processing device 100 in this disclosure will be described. The information processing device 100 is composed of a general information processing device, and as an example, it is equipped with the following hardware configuration as shown in Figure 5. ·CPU(Central Processing Unit)101(Arithmetic unit) ROM (Read Only Memory) 102 (Storage Device) • RAM (Random Access Memory) 103 (Storage Device) • Program group 104 loaded into RAM 103 • Storage device 105 for storing the program group 104 • Drive device 106 for reading and writing to external storage medium 110 of the information processing device. • Communication interface 107 connecting to a communication network 111 outside the information processing device. • Input / output interface 108 for data input and output. • Bus 109 connecting each component

[0039] Figure 5 shows an example of the hardware configuration of the information processing device 100, and the hardware configuration of the information processing device is not limited to the case described above. For example, the information processing device may consist of only a part of the configuration described above, such as not having a drive device 106. In addition, the information processing device may use a GPU (Graphic Processing Unit), DSP (Digital Signal Processor), MPU (Micro Processing Unit), FPU (Floating point number Processing Unit), PPU (Physics Processing Unit), TPU (Tensor Processing Unit), quantum processor, microcontroller, or a combination thereof instead of the CPU described above.

[0040] The information processing device 100 can be equipped with the detection unit 121, connection unit 122, calculation unit 123, and setting unit 124 shown in Figure 6 by having the CPU 101 acquire the program group 104 and execute it. The program group 104 is, for example, stored in advance in a storage device 105 or ROM 102, and the CPU 101 loads it into RAM 103 and executes it as needed. The program group 104 may also be supplied to the CPU 101 via a communication network 111, or it may be stored in advance in a storage medium 110, and the drive device 106 reads the program and supplies it to the CPU 101. However, the detection unit 121, connection unit 122, calculation unit 123, and setting unit 124 described above may be constructed with dedicated electronic circuits to realize such means.

[0041] The detection unit 121 detects objects from time-based images and calculates the reliability of the detected objects. The linking unit 122 sets linking information to connect the objects detected at each time interval. The calculation unit 123 calculates the value of the linking information based on the reliability of the objects linked by the linking information. The setting unit 124 decides whether to adopt the linking information to connect the objects based on the value of the linking information, and uses the adopted linking information to connect the objects as the object trajectory.

[0042] This disclosure, configured as described above, connects objects detected from an image, calculates the value of the connection based on the confidence level between the objects, and generates the object's trajectory based on this connection value. Therefore, even with fluctuations in the shooting environment, such as when the image is distorted or an object is hidden by another object, it is possible to track objects with greater accuracy and improve tracking accuracy.

[0043] Furthermore, at least one of the functions of the detection unit 121, connection unit 122, calculation unit 123, and setting unit 124 described above may be performed on an information processing device installed and connected at any location on the network, that is, it may be performed using so-called cloud computing.

[0044] Furthermore, the aforementioned programs can be stored and supplied to a computer using various types of non-transitory computer-readable media. Non-transitory computer-readable media include various types of tangible storage media. Examples of non-transitory computer-readable media include magnetic recording media (e.g., flexible disks, magnetic tapes, hard disk drives), magneto-optical recording media (e.g., magneto-optical disks), CD-ROMs (Read Only Memory), CD-Rs, CD-R / Ws, and semiconductor memory (e.g., mask ROMs, PROMs (Programmable ROMs), EPROMs (Erasable PROMs), flash ROMs, and RAMs (Random Access Memory)). Programs may also be supplied to a computer using various types of transient computer-readable media. Examples of transient computer-readable media include electrical signals, optical signals, and electromagnetic waves. Transitory computer-readable media can be supplied to a computer via wired communication channels such as electric wires and optical fibers, or via wireless communication channels.

[0045] Although the present disclosure has been described above with reference to embodiments, the present disclosure is not limited to the embodiments described above. Various modifications to the structure and details of the present disclosure are possible, as can be understood by those skilled in the art within the scope of the present disclosure. Furthermore, each of the embodiments described above can be combined with other embodiments as appropriate.

[0046] <Note> Some or all of the above embodiments may also be described as follows. The general configuration of the information processing apparatus, information processing method, and program in this disclosure is described below. However, this disclosure is not limited to the configuration described in the following notes. Furthermore, some or all of the configurations and functions described in Appendices 2 to 8, which are dependent on Appendice 1 below, may also be dependent on Appendices 9 and 10 in the same way as Appendices 2 to 8. Moreover, not limited to Appendices 1, 9, and 10, some or all of the configurations and functions described as appendices may also be dependent on similar hardware, software, various recording means for recording software, or systems, without departing from the embodiments described above. (Note 1) A detection unit that detects objects from images taken over time and calculates the reliability of the detected objects, A coupling unit that sets coupling information to link objects detected at time intervals, A calculation unit that calculates the value of the linked information based on the reliability of the objects linked by the linked information, Based on the value of the aforementioned linking information, a setting unit determines whether to adopt the linking information that connects objects, and sets the connection between objects based on the adopted linking information as the trajectory of the objects. Equipped with an information processing device. (Note 2) The information processing device described in Appendix 1, The calculation unit calculates a higher value for the linked information the higher the reliability of the object linked to the linked information. The setting unit determines that the higher the value of the linked information, the more likely it is to prioritize the adoption of that linked information. Information processing device. (Note 3) The information processing device described in Appendix 1, The calculation unit calculates the value of the linked information based on the confidence level of each of the two objects linked to the linked information. Information processing device. (Note 4) The information processing device described in Appendix 1, The calculation unit calculates the value of the linked information based on the reliability of one of the two objects linked to the linked information. Information processing device. (Note 5) The information processing device described in Appendix 1, The setting unit discards the connection between the two objects linked to the linking information, based on the confidence level of each of the two objects linked to the linking information, without using it as the object's trajectory. Information processing device. (Note 6) The information processing device described in Appendix 5, The setting unit discards the connection between the two objects linked to the linking information without using it as the object's trajectory if the value based on the reliability of each of the two objects linked to the linking information is lower than a preset reference value. Information processing device. (Note 7) The information processing device described in Appendix 1, The system includes a joining section that generates a joined trajectory by joining multiple aforementioned trajectories, The joint further connects and joins the multiple joint trajectories by selecting a portion of one of the joint trajectories and discarding the other portion of the joint trajectory in the portion where the trajectories of objects at the same time coexist in the multiple joint trajectories. Information processing device. (Note 8) The information processing device described in Appendix 7, The joint portion selects the portion of the joint trajectory with high reliability of the object from among the portions of the multiple joint trajectories that coexist at the same time, discards the portion of the joint trajectory with low reliability of the object, and further connects and joins the multiple joint trajectories. Information processing device. (Note 9) Objects are detected from images taken over time, and the confidence level of the detected objects is calculated. Set connection information to link the objects detected at each time interval. Based on the reliability of the objects linked by the aforementioned linking information, the value of the linking information is calculated. Based on the value of the aforementioned linking information, a decision is made as to whether or not to adopt the linking information that connects the objects, and the connection between the objects based on the adopted linking information is defined as the trajectory of the objects. Information processing methods. (Note 9.1) The information processing method described in Appendix 9, Furthermore, a joined trajectory is generated by joining multiple trajectories, and in the portions of the joined trajectories where the trajectories of objects at the same time coexist, the portion of one of the joined trajectories is adopted and the portion of the other joined trajectory is discarded, thereby further joining the multiple joined trajectories. Information processing methods. (Note 10) In an information processing device, Objects are detected from images taken over time, and the confidence level of the detected objects is calculated. Set connection information to link the objects detected at each time interval. Based on the reliability of the objects linked by the aforementioned linking information, the value of the linking information is calculated. Based on the value of the aforementioned linking information, a decision is made as to whether or not to adopt the linking information that connects the objects, and the connection between the objects based on the adopted linking information is defined as the trajectory of the objects. A program that executes a process. [Explanation of Symbols]

[0047] 1. Video Supply Department 2. Object detection unit 3. Graph Construction Section 4. Value Calculation Unit 5. Confidence Addition Calculation Unit 6. Constraint Enumeration Section 7. Optimization Unit 8. Enumerated section of connecting parts 9. Track sorting section 10 Locus junction 11. Duplicate Trajectory Integration Unit 100 Information Processing Devices 101 CPU 102 ROM 103 RAM 104 Program Groups 105 Storage device 106 Drive unit 107 Communication Interface 108 Input / Output Interfaces 109 Bus 110 Storage medium 111 Communication Network 121 Detection unit 122 Connecting part 123 Calculation Section 124 Setting section

Claims

1. A detection unit that detects objects from images taken over time and calculates the reliability of the detected objects, A coupling unit that sets coupling information to link objects detected at time intervals, A calculation unit that calculates the value of the linked information based on the reliability of the objects linked by the linked information, Based on the value of the aforementioned linking information, a setting unit determines whether to adopt the linking information that connects objects, and sets the connection between objects based on the adopted linking information as the trajectory of the objects. Equipped with an information processing device.

2. An information processing apparatus according to claim 1, The calculation unit calculates a higher value for the linked information the higher the reliability of the object linked to the linked information. The setting unit determines that the higher the value of the linked information, the more likely it is to prioritize the adoption of that linked information. Information processing device.

3. An information processing apparatus according to claim 1, The calculation unit calculates the value of the linked information based on the confidence level of each of the two objects linked to the linked information. Information processing device.

4. An information processing apparatus according to claim 1, The calculation unit calculates the value of the linked information based on the reliability of one of the two objects linked to the linked information. Information processing device.

5. An information processing apparatus according to claim 1, The setting unit discards the connection between the two objects linked to the linking information, based on the confidence level of each of the two objects linked to the linking information, without using it as the object's trajectory. Information processing device.

6. An information processing device according to claim 5, The setting unit discards the connection between the two objects linked to the linking information without using it as the object's trajectory if the value based on the reliability of each of the two objects linked to the linking information is lower than a preset reference value. Information processing device.

7. An information processing apparatus according to claim 1, The system includes a joining section that generates a joined trajectory by joining multiple aforementioned trajectories, The joint further connects and joins the multiple joint trajectories by selecting a portion of one of the joint trajectories and discarding the other portion of the joint trajectory in the portion where the trajectories of objects at the same time coexist in the multiple joint trajectories. Information processing device.

8. An information processing apparatus according to claim 7, The joint portion selects the portion of the joint trajectory with high reliability of the object from among the portions of the multiple joint trajectories that coexist at the same time, discards the portion of the joint trajectory with low reliability of the object, and further connects and joins the multiple joint trajectories. Information processing device.

9. Objects are detected from images taken over time, and the confidence level of the detected objects is calculated. Set connection information to link the objects detected at each time interval. Based on the reliability of the objects linked by the aforementioned linking information, the value of the linking information is calculated. Based on the value of the aforementioned linking information, a decision is made as to whether or not to adopt the linking information that connects the objects, and the connection between the objects based on the adopted linking information is defined as the trajectory of the objects. Information processing methods.

10. In an information processing device, Objects are detected from images taken over time, and the confidence level of the detected objects is calculated. Set connection information to link the objects detected at each time interval. Based on the reliability of the objects linked by the aforementioned linking information, the value of the linking information is calculated. Based on the value of the aforementioned linking information, a decision is made as to whether or not to adopt the linking information that connects the objects, and the connection between the objects based on the adopted linking information is defined as the trajectory of the objects. A program that executes a process.

Images