Information processing device, information processing method, program, and circuit information

Abstract

To improve performance or efficiency of an information processing function by using a constraint violation solution obtained by solving a combinatorial optimization problem.SOLUTION: An information processing system according to an embodiment performs processing on data. An information processing apparatus generates a combinatorial optimization problem based on the data under a predetermined constraint condition and solves the generated combinatorial optimization problem while allowing the constraint condition to be unsatisfied and calculates a solution of the combinatorial optimization problem which is obtained by solving. If the solution does not satisfy the constraint condition, the information processing apparatus generates a constraint violation control signal based on a constraint violation part and performs predetermined first processing on the data on the basis of the constraint violation control signal and outputs data subjected to the first processing.SELECTED DRAWING: Figure 8

Description

[Technical Field]

[0001] The embodiments of the present invention relate to an information processing device and an information processing method. 、 program and circuit information Regarding. [Background technology]

[0002] Optimization of systems in various application fields such as control, finance, communications, logistics, and chemistry is often mathematically reduced to combinatorial optimization problems. Information processing systems have been proposed that solve combinatorial optimization problems to realize various information processing functions such as recognition, judgment, and planning, and to improve the efficiency of such functions.

[0003] A combinatorial optimization problem is a problem in which a cost function is defined whose arguments are multiple decision variables (e.g., discrete variables) that represent the state of the system to be optimized, and the combination of values ​​of the multiple decision variables that minimizes the defined cost function is solved. The state of the system represented by multiple decision variables is called a solution. In a combinatorial optimization problem, the number of states that can be taken as a solution increases exponentially as the number of decision variables increases. This increase in the number of states that can be taken as a solution is called combinatorial explosion. Combinatorial optimization, which involves selecting the best solution from all candidate solutions, is known to be a computationally difficult problem. Performing large-scale combinatorial optimization in a short period of time remains a challenging task.

[0004] A constrained combinatorial optimization problem is a problem in which certain constraints are imposed on the solution, and a solution that minimizes a cost function under the constraints is sought. A solution that satisfies the constraints is called a constraint-satisfying solution. A solution that satisfies the constraints is sometimes called a feasible solution. A solution that does not satisfy the constraints is called a constraint-violating solution. A solution that does not satisfy the constraints is sometimes called an infeasible solution.

[0005] Conventional information processing systems for solving constrained combinatorial optimization problems do not output solutions that violate constraints as solutions, or discard solutions that violate constraints even if they are found. However, even solutions that violate constraints may contain solutions that are effective for improving the performance or efficiency of information processing functions.

[0006] Conventionally, no technology has been known that efficiently outputs solutions that are effective for improving the performance or efficiency of an information processing function from among constraint-violating solutions, or that improves the performance or efficiency of an information processing function using constraint-violating solutions. Therefore, with conventional technology, even if the constraint-violating solutions include solutions that are effective for improving the performance or efficiency of an information processing function, it has not been possible to improve the performance or efficiency of the information processing function. [Prior art documents] [Patent documents]

[0007] [Patent Document 1] Japanese Patent Application Publication No. 2019-145010 [Patent Document 2] Japanese Patent Application Publication No. 2019-159566 [Patent Document 3] Japanese Patent Publication No. 2021-043667 [Patent Document 4] Japanese Patent Publication No. 2021-043589 [Patent Document 5] Patent Publication No. 2021-060864 [Non-patent literature]

[0008] [Non-Patent Document 1] A Bewley, et at., “Simple online and realtime tracking”, IEEE ICIP(2016), pp. 3464-3468, 2016 [Non-patent document 2] Hayato Goto, Kosuke Tatsumura and Alexander R. Dixon, “Combinatorial optimization by simulating adiabatic bifurcations in nonlinear Hamiltonian systems,” Science Advances 5, eaav2372, 2019 [Non-patent document 3] Hayato Goto, Kotaro Endo, Masaru Suzuki, Yoshisato Sakai, Taro Kanao, Yohei Hamakawa, Ryo Hidaka, Masaya Yamasaki and Kosuke Tatsumura, “High-performance combinatorial optimization based on classical mechanics”, Science Advances 7, eabe7953, 2021 [Non-patent document 4] Joseph Redmon, Santosh Divvala, Ross Girshick and Ali Farhadi, “You Only Look Once: Unified, Real-Time Object Detection”, Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016, pp. 779-788 [Non-patent document 5] Wei Liu, et al., “SSD: Single Shot MultiBox Detector”, Springer International Publishing AG 2016, ECCV 2016,Part I, LNCS vol 9905, pp. 21-37, 2016 Summary of the Invention [Problem to be solved by the invention]

[0009] The problem to be solved by the present invention is to improve the performance or efficiency of information processing functions by using constraint violation solutions obtained from combinatorial optimization problems. [Means for solving the problem]

[0010] The information processing system according to the embodiment executes processing on data. The information processing device stores one or more pieces of first information at a first time, and acquires one or more pieces of second information at the first time. The information processing device performs the following operations under predetermined constraints: and associating any one of the one or more pieces of second information at the first time with each of the one or more pieces of first information at the first time that are stored. Generate combinatorial optimization problems The information processing device The generated combinatorial optimization problem is solved while allowing the constraints not to be satisfied, and a solution to the combinatorial optimization problem obtained by solving the combinatorial optimization problem is calculated. The information processing device A constraint violation control signal based on a constraint-violating portion of the solution that does not satisfy the constraint conditions. and a constraint satisfaction control signal based on a constraint satisfaction portion of the solution that satisfies the constraint. Generate The information processing device selects, from the one or more pieces of first information stored at the first time, based on the constraint violation control signal Identifying first information Execute the first predetermined process on The information processing device executes a predetermined second process different from the first process on first information specified based on the constraint satisfaction control signal among the one or more pieces of first information stored at the first time. Output. [Brief explanation of the drawings]

[0011] [Figure 1] A diagram showing a model of the Ising problem. [Figure 2] FIG. 2 illustrates internal variables used by the simulated branching algorithm. [Figure 3] 10 is a flowchart showing the flow of processing of a simulated branching machine. [Figure 4] FIG. 1 is a diagram showing the configuration of an information processing system including an Ising machine. [Figure 5] FIG. 1 is a diagram showing the configuration of an information processing system equipped with a simulated branching machine. [Figure 6] FIG. 1 is a diagram showing the configuration of an object tracking system. [Figure 7] FIG. 1 is a diagram showing the configuration of an object tracking device. [Figure 8] FIG. 2 is a diagram showing the configuration of an object tracking unit together with a solver unit and a storage unit. [Figure 9] FIG. 1 is a diagram showing an example of multiple decision variables included in a combinatorial optimization problem. [Figure 10] FIG. 1 is a diagram showing multiple coefficients included in a combinatorial optimization problem. [Figure 11] FIG. 1 shows a set of solutions for a cost function and a set of solutions that satisfy constraints. [Figure 12] 10 is a flowchart showing the flow of processing by the object tracking device. [Figure 13] FIG. 1 shows a data flow graph of an object tracking device. [Figure 14] 10 is a flowchart showing the flow of processing by an arbitration unit. [Figure 15] A diagram showing the bounding box and solution for the (k-1)th frame. [Figure 16] A diagram showing the bounding box and solution for the kth frame. [Figure 17] A diagram showing the bounding box and solution for the (k+1)th frame. [Figure 18] FIG. 10 shows constraint-satisfying portions identified in the k-th frame. [Figure 19] FIG. 10 is a diagram showing the processing of a formulation unit according to a first modified example. [Figure 20] FIG. 10 is a diagram showing the processing of an arbitration unit according to a second modified example. [Figure 21] A diagram showing the solution to the first combinatorial optimization problem. [Figure 22] A diagram showing the solution to the second combinatorial optimization problem. [Figure 23] FIG. 23 shows the boundary box when the solutions of FIGS. 21 and 22 are obtained. [Figure 24] FIG. 2 is a diagram showing the hardware configuration of a host unit. DETAILED DESCRIPTION OF THE INVENTION

[0012] (Premise) First, terms and techniques that are the basis for the description of the embodiments will be explained.

[0013] A combinatorial optimization problem is a problem in which a cost function is defined with arguments being multiple decision variables (e.g., discrete variables) that represent the state of a system to be optimized, and a combination of values ​​of the multiple decision variables that minimizes the defined cost function is sought. The cost function includes multiple decision variables that represent the state of the system as arguments, and is a linear or higher-order function of the multiple decision variables. For example, the cost function is expressed by a linear or quadratic function of the multiple decision variables. The cost function may also be a cubic or higher-order function of the multiple decision variables. In other words, the cost function is a function that sums up multiple terms. Each of the multiple terms is a function that multiplies one or more, but not more than a predetermined number of, the multiple decision variables by a coefficient.

[0014] The state of the system, represented by multiple decision variables, is called a solution. The entire set of possible solutions for the state of the system is called the solution space. The solution that gives the minimum value of the cost function is called an exact solution. The solution that gives a value close to the minimum value of the cost function is called a good solution.

[0015] An exact solution method is a method for finding an exact solution that gives the minimum value of a cost function for a combinatorial optimization problem, and is a method that guarantees that the solution is an exact solution.

[0016] A heuristic solution is a solution method for combinatorial optimization problems that finds an exact solution that minimizes the cost function or a good solution that approaches the minimum value of the cost function. Heuristic solution methods are also called discovery solutions. Heuristic solution methods do not guarantee the accuracy of the solution, that is, an index that indicates how close the solution is to the minimum value of the cost function. Heuristic solution methods can output solutions with practically significant accuracy in a shorter solution time than exact solution methods.

[0017] The computational complexity of a solution to a combinatorial optimization problem is the number of operations required to obtain a solution, such as the number of multiply-and-accumulate operations. The solution time required to obtain a solution to a combinatorial optimization problem depends not only on the computational complexity but also on the configuration of the computer that processes the operations. For example, the solution time becomes shorter as the computer's degree of parallelism and operating frequency increase.

[0018] A QUBO (Quadratic Unconstrained Binary Optimization) problem is an unconstrained quadratic optimization problem in which the decision variables are binary. In a QUBO problem, each of multiple terms included in a cost function is expressed by a quadratic or linear expression of the decision variables. A narrowly defined QUBO problem has binary decision variables of 0 or 1. In this embodiment, the QUBO problem represents a narrowly defined problem in which the decision variables are 0 or 1. In a narrowly defined QUBO problem, the decision variables are sometimes called bit variables.

[0019] The cost function for the QUBO problem is H in Eq. (1). total_QUBO It is expressed by:

number

[0020] Figure 1 shows a model of the Ising problem. The Ising problem is a problem of searching for the ground state of the Ising model. The Ising problem is one of the QUBO problems. The decision variables of the Ising problem represent discrete variables of -1 or +1.

[0021] The cost function for the Ising problem is H in Eq. (2). total_Ising It is expressed by:

number

[0022] The decision variable is s i The QUBO problem corresponds to the problem of searching for the ground state of the Ising model, which is one of the magnetic models in statistical mechanics. i is sometimes called the spin variable. Also, the number of decision variables, N, is sometimes called the spin number. Also, H total_Ising is sometimes called the Zing energy. total_Ising N s for which is the smallest value i The vector represented by is sometimes called the ground state (ground spin configuration).

[0023] The cost function of the QUBO problem and the cost function of the Ising problem differ only in the value of the constant. Therefore, the QUBO problem and the Ising problem are the same as combinatorial optimization problems. In other words, the QUBO problem and the Ising problem can be converted into each other. For example, the Ising problem and the QUBO problem can be converted into each other using formulas (3-1), (3-2), (3-3), and (3-4).

number

[0024] QUBO and Ising problems are known to be NP-complete. That is, many NP-hard problems can be transformed into QUBO or Ising problems in polynomial time. Therefore, many practical combinatorial optimization problems can be transformed into QUBO or Ising problems.

[0025] An Ising machine is a device that solves Ising problems. Most Ising machines solve Ising problems using heuristic solutions. Ising machines based on various principles, such as electronics, optics, quantum mechanics, and statistical mechanics, have been proposed. Many Ising machines can output exact or good solutions in a short time.

[0026] The simulated branching algorithm is an algorithm for solving combinatorial optimization problems. The simulated branching algorithm is a heuristic solution algorithm.

[0027] Simulated bifurcation algorithms are disclosed, for example, in Non-Patent Document 2, Non-Patent Document 3, Patent Document 1, Patent Document 2, Patent Document 3, Patent Document 4, and Patent Document 5. The simulated bifurcation algorithm is also called a quantum-inspired algorithm because it was discovered inspired by quantum mechanical optimization techniques based on the quantum adiabatic theorem. The simulated bifurcation algorithm can solve combinatorial optimization problems in which the cost function is a quadratic function of multiple decision variables. The simulated bifurcation algorithm can also solve combinatorial optimization problems in which the cost function is a cubic or higher function of multiple decision variables, i.e., HUBO (Higher Order Binary Optimization) problems. For example, a simulated bifurcation algorithm for solving HUBO problems is disclosed in Patent Document 3. The simulated bifurcation algorithm can also solve combinatorial optimization problems in which some or all of the multiple decision variables are continuous-valued variables. A simulated bifurcation algorithm for solving combinatorial optimization problems in which some or all of the multiple decision variables are continuous-valued variables is disclosed in Patent Document 4.

[0028] A simulated bifurcation machine is a computing device that executes processing according to a simulated bifurcation algorithm. A simulated bifurcation machine that solves a QUBO problem or an Ising problem is an example of an Ising machine. In this embodiment, the simulated bifurcation machine solves an Ising problem.

[0029] FIG. 2 is a diagram illustrating the internal variables used by the simulated branching algorithm.

[0030] The simulated bifurcation algorithm uses N decision variables (s1 to s N When solving a combinatorial optimization problem in which the cost function is expressed using N ) and N momentum variables (y1~y N ) is used. That is, the simulated bifurcation algorithm uses 2×N internal variables.

[0031] N location variables (x i ) is the number of N decision variables (s i ) in a one-to-one correspondence. That is, the i-th position variable (x i ) is the i-th decision variable (s i ) and N momentum variables (y i ) is the number of N decision variables (s i ) corresponds to the i-th momentum variable (y i ) is the i-th decision variable (s i ) corresponds to

[0032] 3 is a flowchart showing the flow of processing by the simulated branching machine. The simulated branching machine executes processing according to the simulated branching algorithm, as shown in FIG.

[0033] First, in S11, the simulated bifurcation machine obtains an Ising problem. Specifically, the simulated bifurcation machine obtains J, which is a matrix including N×N coefficients, and h, which includes N bias coefficients.

[0034] Next, in S12, the simulated branching machine calculates 2×N internal variables, i.e., N position variables (x1 to x N ) and N momentum variables (y1~y N ) is initialized. The simulated branching machine has N position variables (x1 to x N ) and the initial values ​​of N momentum variables (y1~y N ) may be obtained from the outside. N ) and the initial values ​​of N momentum variables (y1~y N The initial value of x1 to x2 may be generated by a random number generator or may be set to a predetermined value. Since the simulated branching machine is a heuristic, even for the same problem, the initial value of x1 to x2 may be set to a predetermined value. N ) and the initial values ​​of N momentum variables (y1~y N ) is different, a different good solution may be output.

[0035] Next, the simulated branching machine repeats the processing from S14 to S16 a preset number of times (loop processing between S13 and S17). The processing from S14 to S16 is performed by using N position variables (x1 to x N ) and N momentum variables (y1~y N ) is a time evolution process.

[0036] In S14, the simulated bifurcation machine calculates N momentum variables (y1 to y N ) in the y update process. The simulated bifurcation machine updates the i-th momentum variable (y i In the update process of N position variables (x1 to x N) and the i-th position variable (x i ) and the other (N-1) location variables (x 1~i-1, x i+1~N ) and N coefficients (J i,j ) and the i-th bias coefficient (h i ) and the i-th momentum variable (y i ) to update the

[0037] Next, in S15, the simulated branching machine calculates the N position variables (x1 to x N ) in the x update process. The simulated branch machine updates the i-th position variable (x i In the update process of the i-th momentum variable (y i ) to determine the i-th position variable (x i ) to update the

[0038] The simulated branching machine may execute the process of S14 and the process of S15 in reverse order.

[0039] Next, in S16, the simulated branching machine calculates the N position variables (x1 to x N ), the simulated branching machine performs wall processing on position variables whose absolute values ​​exceed 1. Furthermore, the simulated branching machine also performs wall processing on momentum variables corresponding to position variables whose absolute values ​​exceed 1. For example, in wall processing, the simulated branching machine changes the absolute value of a position variable whose absolute value exceeds 1 to 1 or a value smaller than 1 while keeping the sign the same. Also, for example, in wall processing, the simulated branching machine changes the value of a momentum variable corresponding to a position variable whose absolute value exceeds 1 to 0.

[0040] The simulated bifurcation algorithm has variations in the calculations of x-update processing, y-update processing, and wall processing. For example, the simulated bifurcation algorithm has variations such as the adiabatic simulated bifurcation (aSB) algorithm, the ballistic simulated bifurcation (bSB) algorithm, and the discrete simulated bifurcation (dSB) algorithm.

[0041] When executing processing according to the adiabatic simulated branching algorithm, the simulated branching machine executes the calculation shown in equation (4-1) in the y update processing (S14), and executes the calculation shown in equation (4-2) in the x update processing (S15). Note that when executing processing according to the adiabatic simulated branching algorithm, the simulated branching machine does not execute the wall processing (S15).

number

[0042] When processing according to the ballistic simulated branching algorithm, the simulated branching machine performs the calculation shown in equation (5-1) in the y update processing (S14), the calculation shown in equation (5-2) in the x update processing (S15), and the calculation shown in equation (5-3) in the wall processing (S16).

number

[0043] When executing processing according to the discrete simulated branching algorithm, the simulated branching machine performs the calculation shown in equation (6-1) in the y update processing (S14), the calculation shown in equation (6-2) in the x update processing (S15), and the calculation shown in equation (6-3) in the wall processing (S16).

number

[0044] In addition, in formulas (4-1), (4-2), (5-1), (5-2), (5-3), (6-1), (6-2), and (6-3), t k and t k+1 represents the time. k+1 is t k This is the time obtained by adding a unit time (Δt) to the above.

[0045] x i (t k ) is the time (t k ) the i-th position variable (x i ) value. i (t k+1 ) is the time (t k+1 ) the i-th position variable (x i ) value. i (t k ) is the time (t k ) the i-th momentum variable (y i ) value. i (t k+1 ) is the time (t k+1 ) the i-th momentum variable (y i ) value.

[0046] K, a0, η, and c0 are predetermined constants. k ) is a function that changes with time. k ) is a positive real number that increases as time increases, for example, a(t1)=0, and is a function that becomes a0 at the end time (T) (a(T)=a0). Also, sgn(x i (t k )) is the time (t k ) the i-th position variable (x i ) is a function that outputs the sign of x i (t k ) is 0 or greater, then +1, x i (t k ) is less than 0, it is set to -1.

[0047] When the simulated branching machine has executed the processes of S14 to S16 a predetermined number of times, i.e., when it has executed the calculations until time t reaches the final time T, it exits the loop processing between S13 and S17 and proceeds to S18.

[0048] In S18, the simulated branching machine calculates the N position variables (x1 to x N ), or N position variables (x1~x N ) are calculated based on N decision variables (s1 to s N ) The simulated branching machine outputs N decision variables (s1 to s N ) the i-th decision variable (s i ) and sgn(x i ) is calculated based on

[0049] When the simulated branching machine completes the process of S18, it ends the process according to the simulated branching algorithm.

[0050] The number of iterations of the time-evolution processing (the loop processing between S13 and S17) is determined in advance depending on the application. The amount of calculation required for one processing in the time-evolution processing (one processing of S14 to S16) does not vary. For this reason, the simulated branching machine can reduce fluctuations in the solution-finding time. Therefore, even when applied to a real-time system with time constraints that require processing to be completed by a certain time, the simulated branching machine can reliably output a solution by the specified time.

[0051] Furthermore, as shown in Patent Document 2, for example, a simulated branching machine can be configured using a dedicated parallel processing circuit including a large number of arithmetic units. This allows the simulated branching machine to significantly shorten the calculation time for a single process in time-evolving processing. Furthermore, unlike software processing, a simulated branching machine implemented in a dedicated hardware circuit does not generate any interrupt processes, so the solution time is strictly fixed. For example, a simulated branching machine implemented in a dedicated hardware circuit can fix the time until a solution is obtained in clock cycles. Therefore, when applied to a real-time system, a simulated branching machine implemented in a dedicated hardware circuit can output a solution while more reliably observing time constraints.

[0052] FIG. 4 is a diagram showing the configuration of an information processing system including an Ising machine and a host device. An information processing system having a function for solving a combinatorial optimization problem can be configured, for example, by an Ising machine and a host device. The host device executes processing other than that executed by the Ising machine. In this case, the Ising machine is introduced for the purpose of shortening the time required to solve the combinatorial optimization problem, and is considered to be an accelerator or offloader. The host device includes a general-purpose processor, an offloader other than the Ising machine, memory, storage, sensors, actuators, a communication interface, etc.

[0053] The host device provides at least information specifying the QUBO problem (matrix (Q)) or information for specifying the Ising problem (matrix (J) and bias coefficient (h)) to the Ising machine. The Ising machine may acquire information specifying the QUBO problem (matrix (Q)) and convert it into information for specifying the Ising problem (matrix (J) and bias coefficient (h)). After the optimization process, the Ising machine then outputs N decision variables (s1 to s2) as a solution. N ) opt ) is returned to the host device.

[0054] FIG. 5 is a diagram showing the configuration of an information processing system including a simulated branching machine and a host device.

[0055] An information processing system having a function of solving a combinatorial optimization problem may include a simulated bifurcation machine as an Ising machine. In this case, the host device provides information (matrix (J) and bias coefficient (h)) for identifying the Ising problem to the simulated bifurcation machine. Furthermore, the host device provides N position variables (x1 to x N ) and the initial values ​​of N momentum variables (y1~y N The host device may also provide initial values ​​for various constants and functions (e.g., K, a0, c) used in the simulated branching algorithm. 0、 a(t) and Δt), etc. may be given to the simulated branching machine. After the optimization process, the simulated branching machine outputs N decision variables (s1 to s N ) opt ) instead of N position variables (x1 to x N ) opt ) may be sent back to the host device.

[0056] (Embodiment) Next, an object tracking system 10 according to an embodiment will be described (FIG. 6).

[0057] The object tracking system 10 is an information processing system having a function of solving a constrained combinatorial optimization problem, and is an example of a real-time system. In particular, the object tracking system 10 is an example of an information processing system that detects valid constraint-violating solutions from among solutions to a constrained combinatorial optimization problem, and executes processing based on the valid constraint-violating solutions.

[0058] A real-time system is a system that is subject to time constraints, in that it must complete processing by a set time. Most real-time systems do not process only accumulated information from the past, but rather recognize the current, ever-changing situation, determine a response action based on the recognized situation, and immediately execute the response action.

[0059] The problem of determining the optimal response action for a recognized situation can sometimes be formulated as a combinatorial optimization problem. In a real-time system, a response action determined based on an exact or good solution to a combinatorial optimization problem can be considered a rational action. However, it is not easy to solve a combinatorial optimization problem in a short time. For this reason, conventional real-time systems often determine a response action based on simple conditional judgments.

[0060] FIG. 6 is a diagram showing the configuration of an object tracking system 10 according to the embodiment.

[0061] The object tracking system 10 includes a camera 12, an object tracker 14, a planner 16, and a controller 18.

[0062] The camera 12 captures images of the surrounding scenery and generates image data that may include one or more captured objects. The camera 12 generates the image data at regular intervals (for example, at each frame, which is the shooting interval). The camera 12 is, for example, an optical imaging camera. The camera 12 may be, for example, a camera system including a monocular camera (mono camera) and a stereo camera used in an ADAS (Advanced Driver Assistance System) or the like, or another sensor. The camera 12 supplies the image data generated at regular intervals to the object tracking device 14.

[0063] The object tracking device 14 acquires image data generated by the camera 12 at regular intervals. The object tracking device 14 then tracks the objects included in the image data. Specifically, the object tracking device 14 stores one or more pieces of tracking object information representing one or more tracking objects to be tracked, and periodically updates the one or more pieces of tracking object information representing the one or more tracking objects based on the acquired image data. Note that in the initial state or when no objects exist in the surrounding area, the object tracking device 14 does not need to store any tracking object information because no tracking objects exist. Each piece of tracking object information represents the position, size, etc. of the tracking object. Each piece of tracking object information may include information representing the type of tracking object, information representing the amount of change in position and size over time, and information representing the age, etc., representing the time elapsed since the object was added or modified. The object tracking device 14 periodically supplies the one or more pieces of tracking object information to the planning device 16.

[0064] The planning device 16 acquires one or more pieces of tracking object information from the object tracking device 14 at regular intervals. Furthermore, the planning device 16 may also acquire sensor information from other sensors besides the camera 12, such as LiDAR (Light detection and ranging). The planning device 16 generates behavior plan information representing a response behavior of the controlled object based on the acquired one or more pieces of tracking object information and sensor information. The planning device 16 supplies the generated behavior plan information to the control device 18.

[0065] The control device 18 acquires the behavior plan information from the planning device 16. The control device 18 controls the controlled object in accordance with the received behavior plan information so that the controlled object operates in accordance with the response behavior information.

[0066] Such an object tracking system 10 is applied, for example, to an automatic control mechanism that controls a moving object. An automatic control mechanism that controls a moving object generally periodically repeats a series of operations: sensing, situation recognition, action planning, and control. This series of operations is typically executed 10 to several tens of times per second. That is, this series of operations typically has a period of approximately 100 milliseconds or less. The throughput of the object tracking system 10 is limited by the slowest module among the multiple modules that make up the system. Therefore, when applied to an automatic control mechanism that controls a moving object, each of the multiple modules that make up the object tracking system 10 must achieve the required throughput while smoothly processing data (e.g., stream data) that is sequentially input and output. The object tracking system 10 is not limited to automatic control mechanisms that control moving objects, and may also be used in mechanisms that control other control objects.

[0067] Here, the object tracking device 14 (FIG. 7) performs a matching process, which is one process for solving a combinatorial optimization problem under constraint conditions. The object tracking device 14 reduces the matching process to an Ising problem and solves it using a combinatorial optimization solver. The combinatorial optimization solver used by the object tracking device 14 can output a constraint satisfying solution and a constraint violating solution. Furthermore, the object tracking device 14 performs an arbitration process to evaluate and determine the solutions output by the combinatorial optimization solver. In the arbitration process, if the object tracking device 14 obtains a valid control violation solution that is relatively close to the control satisfying solution among the constraint violating solutions, it generates a constraint violation control signal based on the valid control violation solution. Then, in response to the constraint violation control signal, the object tracking device 14 performs a predetermined constraint violation process on the tracked object information corresponding to the constraint violating portion of one or more pieces of tracked object information.

[0068] The phenomenon or situation in which a tracked object is temporarily hidden from the camera 12 by another tracked object or an obstructing object other than the tracked object is called occlusion. The ability to continue tracking of a tracked object without interruption even when occlusion occurs is called occlusion response capability. The object tracking device 14 can improve its occlusion response capability by performing constraint violation processing using a valid constraint violation solution.

[0069] FIG. 7 is a diagram showing the configuration of the object tracking device 14. As shown in FIG.

[0070] The object tracker 14 includes a solver portion 22 and a host portion 24 .

[0071] The solver unit 22 is a combinatorial optimization solver that acquires a combinatorial optimization problem and outputs a solution to the acquired combinatorial optimization problem. The solver unit 22 calculates a solution to the acquired combinatorial optimization problem using a heuristic solution method. The solver unit 22 also outputs a solution within a certain time after acquiring the combinatorial optimization problem. In this embodiment, the solver unit 22 is realized using a simulated bifurcation machine, which is a hardware computing device that uses a simulated bifurcation algorithm.

[0072] The host unit 24 is realized by an information processing device including an arithmetic processing unit (general-purpose processor), a RAM (Random Access Memory), a ROM (Read Only Memory), a storage device, a communication interface, etc., executing a program. The host unit 24 may further include an accelerator for object detection. The host unit 24 executes processes other than the process of solving the combinatorial optimization problem among the processes executed by the object tracking device 14. The host unit 24 may also be realized by an information processing device common to the planning device 16 and the control device 18.

[0073] The host unit 24 includes an object detection unit 26 , a storage unit 28 , and an object tracking unit 30 .

[0074] The object detection unit 26 acquires image data from the camera 12 at regular intervals, for example, for each frame. The object detection unit 26 generates one or more pieces of detected object information representing one or more objects included in the image data at regular intervals. During the processing at regular intervals, the object detection unit 26 does not output detected object information at times when the image data does not contain detected objects. Each of the one or more pieces of detected object information includes the position and size of a single object to be detected that is included in the image data. In this embodiment, each of the one or more pieces of detected object information is boundary box information. The boundary box information includes the horizontal and vertical pixel positions of the center of the box within the field of view of the image data, the area, and the aspect ratio. Alternatively, the boundary box information may include the horizontal and vertical pixel positions of the top left and bottom right of the box within the field of view of the image data instead of the center of the box. Each of the one or more pieces of detected object information includes identification information for distinguishing between multiple objects included in a single frame. Furthermore, each of the one or more pieces of detected object information may also include type information of the detected object (for example, distinction between a vehicle and a person, or distinction between colors, etc.) and distance information (distance from the camera 12 to the detected object).

[0075] For example, the object detection unit 26 generates one or more pieces of detected object information from image data using an existing method. In this embodiment, the object detection unit 26 generates one or more pieces of detected object information according to the YOLO technique described in Non-Patent Document 4. The object detection unit 26 may also generate one or more pieces of detected object information according to the SSD technique described in Non-Patent Document 5. The object to be detected may be a vehicle, a person, an aircraft, or the like.

[0076] The object detection unit 26 periodically supplies one or more pieces of generated detected object information to the object tracking unit 30. For example, the object detection unit 26 may d pieces(N d When an object (where n is an integer equal to or greater than 1) is detected, information (detections) shown in equation (7) is generated.

number

[0077] d i represents the i-th detected object information.

[0078] For example, the i-th detected object information (d i ,detection) is expressed as in equation (8).

number

[0079] u i represents the horizontal pixel position of the center of the bounding box within the field of view. i represents the vertical pixel position of the center of the bounding box in the field of view. i represents the area of ​​the bounding box within the field of view. i represents the aspect ratio of the bounding box.

[0080] The storage unit 28 stores internal state information. When there are one or more tracking objects to be tracked, the internal state information includes one or more tracking object information. For example, when the internal state information is N t pieces(N t When the tracking object information includes the internal state information (internal_state) of the tracking object, the internal state information (internal_state) is expressed as in Equation (9).

number

[0081] t i represents the i-th tracked object information, where T represents the transpose of a matrix.

[0082] The i-th tracking information (t i , tracker) is expressed as in equation (10).

number

[0083] u i, v i , s i and r i The symbol with a dot on it is u i , v i , s i and r i The change in each of these values ​​per unit time is shown. i represents the age. The age is incremented when there is no correspondence with the currently detected object, and represents the time elapsed since the tracking object information was added or modified. If the tracking object information's age exceeds a preset threshold (max_age), it is deleted from the internal state information.

[0084] The storage unit 28 continues to store one or more pieces of tracking object information included in the internal state information for a certain period of time, for example, a frame interval, unless the tracking object information is deleted from the internal state information because the age exceeds a threshold. Such a function in the storage unit 28 is called a state memory.

[0085] The object tracking unit 30 periodically acquires one or more pieces of detected object information generated by the object detection unit 26. The object tracking unit 30 periodically generates a combinatorial optimization problem that associates, with each of the one or more pieces of tracked object information, any of the one or more pieces of detected object information that is determined to represent the same object. In other words, the object tracking unit 30 periodically generates a combinatorial optimization problem that associates, with each of the one or more pieces of tracked object information, any of the one or more pieces of detected object information that is plausible as information representing the same object. Such a combinatorial optimization problem is realized as one type of combinatorial optimization problem that optimizes multiple pieces of data under predetermined constraints.

[0086] Next, the object tracking unit 30 provides the generated combinatorial optimization problem to the solver unit 22 and obtains a solution to the combinatorial optimization problem from the solver unit 22. Note that, at times when the image data does not contain a detected object and no detected object information is acquired, the object tracking unit 30 does not generate a combinatorial optimization problem or obtain a solution, and instead generates information indicating that the detected object information has not been associated with one or more pieces of tracked object information. Also, at times when a tracked object to be tracked does not exist and no tracked object information is stored, the object tracking unit 30 does not generate a combinatorial optimization problem or obtain a solution, and instead generates information indicating that the tracked object information has not been associated with one or more pieces of detected object information. Next, the object tracking unit 30 periodically updates one or more pieces of tracked object information included in the internal state information stored in the memory unit 28 based on the obtained solution and the generated information, in accordance with the Kalman filter framework. The object tracking unit 30 then supplies the updated one or more pieces of tracked object information to the downstream planning device 16.

[0087] FIG. 8 is a diagram showing the configuration of the object tracking unit 30 together with the solver unit 22 and the storage unit .

[0088] The object tracking unit 30 includes an acquisition unit 32, a prediction unit 34, a formulation unit 36, an arbitration unit 38, a correction unit 40, an addition unit 42, a life extension unit 44, a deletion unit 46, an output unit 48, and an exception handling unit 50.

[0089] The acquisition unit 32 acquires, at regular time intervals, one or more pieces of detection information (detections) representing one or more detected objects included in the image data from the object detection unit 26. For example, in processing image data at an arbitrary first time, the acquisition unit 32 acquires one or more pieces of detection information at the first time representing one or more detected objects included in the image data at the first time. Note that the acquisition unit 32 does not acquire detection object information at times when no detected objects are included in the image data.

[0090] The prediction unit 34 predicts, at regular time intervals, one or more pieces of tracking object information representing one or more current tracking objects based on one or more pieces of tracking object information (trackers) representing one or more past tracking objects included in the internal state information of the storage unit 28. For example, the prediction unit 34 predicts one or more pieces of tracking object information at the first time based on one or more pieces of tracking object information representing one or more tracking objects prior to the first time stored in the storage unit 28.

[0091] For example, the prediction unit 34 predicts the horizontal pixel position, vertical pixel position, and area included in the tracking object information, as well as the amount of change therein per unit time, using a constant velocity model, which is a predefined time evolution model. In this case, the prediction unit 34 may predict that the aspect ratio included in each of the one or more pieces of tracking object information will not change. Furthermore, the prediction unit 34 increases the age included in each of the one or more pieces of tracking object information by a predetermined value (for example, 1).

[0092] Then, the prediction unit 34 rewrites one or more pieces of past tracking object information included in the internal state information of the storage unit 28 with one or more pieces of current tracking object information. For example, the prediction unit 34 rewrites one or more pieces of tracking object information before the first time stored in the storage unit 28 with one or more pieces of tracking object information at the first time generated by prediction. Note that the prediction unit 34 does not make a prediction at a time when the internal state information does not include tracking object information.

[0093] The formulation unit 36 ​​acquires one or more pieces of detected object information (detections) and one or more pieces of tracked object information (trackers) at regular time intervals. For example, the formulation unit 36 ​​generates a combinatorial optimization problem that associates, with each of the one or more pieces of tracked object information, any of the one or more pieces of detected object information that are determined to represent the same object. For example, the formulation unit 36 ​​generates a combinatorial optimization problem that associates, with each of the one or more pieces of tracked object information at a first time, any of the one or more pieces of detected object information at the first time that are determined to represent the same object.

[0094] More specifically, the formulation unit 36 ​​generates a bipartite graph maximum matching problem that associates one or more pieces of detected object information with one or more pieces of tracked object information in a one-to-one correspondence. The bipartite graph maximum matching problem is a constrained combinatorial optimization problem. The bipartite graph maximum matching problem can be expressed as a combinatorial optimization problem that minimizes a total cost function that adds a cost function and a penalty function. The penalty function includes at least a portion of multiple decision variables and is a function that becomes a minimum value when the solution satisfies the constraints. The total cost function of the bipartite graph maximum matching problem is expressed by a quadratic function. Note that more specific descriptions of the cost function, penalty function, constraints, and total cost function will be described in detail below.

[0095] The formulation unit 36 ​​then supplies the generated combinatorial optimization problem to the solver unit 22 at regular time intervals. At times when the image data does not contain any detected object and no detected object information has been acquired, the formulation unit 36 ​​does not generate a combinatorial optimization problem, but instead generates information indicating that the detected object information has not been associated with any one or more pieces of tracked object information, and provides this information to the arbitration unit 38. At times when the tracked object to be tracked does not exist and no tracked object information is included in the internal state information, the formulation unit 36 ​​does not generate a combinatorial optimization problem, but instead generates information indicating that the tracked object information has not been associated with any one or more pieces of detected object information, and provides this information to the arbitration unit 38.

[0096] The solver unit 22 acquires a combinatorial optimization problem from the formulation unit 36 ​​at regular time intervals. The solver unit 22 solves the acquired combinatorial optimization problem while allowing for constraint conditions not to be satisfied. For example, the solver unit 22 acquires a combinatorial optimization problem at a first time point and solves the combinatorial optimization problem at the first time point while allowing for constraint conditions not to be satisfied. Since the solver unit 22 solves the combinatorial optimization problem while allowing for constraint conditions not to be satisfied, the solver unit 22 may output a constraint-satisfying solution that satisfies the constraint conditions, or may output a constraint-violating solution that does not satisfy the constraint conditions.

[0097] Then, the solver unit 22 outputs the solution of the combinatorial optimization problem to the arbitration unit 38 within a predetermined time limit. For example, the solver unit 22 outputs the solution to the arbitration unit 38 within a predetermined time that is shorter than a certain time (for example, the time between two consecutive frames) after acquiring the combinatorial optimization problem at the first time point.

[0098] In this embodiment, the solver unit 22 includes an Ising machine 52, a pre-processing unit 54, and a post-processing unit 56.

[0099] The Ising machine 52 acquires an Ising problem and outputs a solution to the Ising problem. In this embodiment, the Ising machine 52 is a simulated branching machine implemented in a hardware circuit. The simulated branching machine solves the Ising problem using a simulated branching algorithm, which is a heuristic solution. Therefore, the Ising machine 52 may output not only an exact solution but also a good solution, and may also output a solution that does not satisfy the constraints.

[0100] Furthermore, the simulated bifurcation machine implemented in a hardware circuit can fix the time from when an Ising problem is acquired until a solution is obtained in units of clock cycles. Therefore, the solver unit 22 including such an Ising machine 52 can output a solution to a combinatorial optimization problem at regular intervals within a predetermined time limit.

[0101] The preprocessing unit 54 acquires the combinatorial optimization problem from the formulation unit 36. The preprocessing unit 54 converts the acquired combinatorial optimization problem into an Ising problem and supplies the converted problem to the Ising machine 52. This enables the solver unit 22 to reduce the combinatorial optimization problem generated by the formulation unit 36 ​​to an Ising problem and solve it. Note that the preprocessing unit 54 may be included in the formulation unit 36 ​​instead of being included in the solver unit 22.

[0102] The post-processing unit 56 obtains a solution to the Ising problem from the Ising machine 52. The post-processing unit 56 converts the solution to the Ising problem into a solution to the combinatorial optimization problem generated by the formulation unit 36, and supplies the solution to the arbitration unit 38. Note that the post-processing unit 56 may be included in the arbitration unit 38 instead of being included in the solver unit 22.

[0103] The arbitration unit 38 periodically acquires a solution from the solver unit 22. The arbitration unit 38 determines whether the acquired solution satisfies the constraint conditions in the combinatorial optimization problem generated by the formulation unit 36. If the acquired solution does not satisfy the constraint conditions, the arbitration unit 38 further determines, based on predetermined evaluation criteria, whether the acquired solution is a valid constraint-violating solution that is close to a constraint-satisfying solution, or whether the acquired solution is far from a constraint-satisfying solution and is not a valid constraint-violating solution. An example of the evaluation criteria will be described in detail later.

[0104] In addition, if the acquired solution does not satisfy the constraint conditions and is a valid constraint-violating solution, the arbitration unit 38 identifies the constraint-satisfying part of the acquired solution that satisfies the constraint conditions and the constraint-violating part of the solution that does not satisfy the constraint conditions.

[0105] For example, the arbitration unit 38 identifies a constraint-satisfying portion by changing the values ​​of some decision variables that are the cause of a valid constraint-violating solution not satisfying the constraint conditions. Alternatively, for example, the arbitration unit 38 may identify a constraint-satisfying portion by excluding the values ​​of some decision variables that are the cause of a valid constraint-violating solution not satisfying the constraint conditions. The constraint-satisfying portion includes multiple decision variables whose values ​​satisfy the constraint conditions. Alternatively, for example, the arbitration unit 38 identifies a constraint-violating portion by extracting the values ​​of some decision variables that are the cause of a valid constraint-violating solution not satisfying the constraint conditions.

[0106] When a solution satisfies the constraint conditions, the arbitration unit 38 generates a constraint satisfaction control signal based on the solution that satisfies the constraint conditions. The constraint satisfaction control signal includes a match signal (matched), an unmatched detection signal (unmatched_detections), and an unmatched tracker signal (unmatched_trackers). The match signal (matched) is a signal indicating, for each piece of tracker information with which the detection information is associated, among one or more pieces of tracker information. The unmatched detection signal (unmatched_detections) is a signal indicating, among one or more pieces of tracker information, detection information with which the tracking object information is not associated. The unmatched tracker signal (unmatched_trackers) is a signal indicating, among one or more pieces of tracker information, detection information with which the tracking object information is not associated.

[0107] If the obtained solution does not satisfy the constraint conditions and is a valid constraint-violating solution, the arbitration unit 38 generates a constraint satisfaction control signal based on the identified constraint satisfaction portion. The constraint satisfaction control signal based on the constraint satisfaction portion includes a match signal based on the constraint satisfaction portion, an unmatched object signal based on the constraint satisfaction portion, and an unmatched tracked object signal based on the constraint satisfaction portion, as well as a constraint satisfaction control signal indicating a solution that satisfies the constraint conditions.

[0108] Furthermore, if the acquired solution does not satisfy the constraint conditions but is a valid constraint-violating solution, the arbitration unit 38 generates a constraint violation control signal based on the constraint-violating portion. The constraint violation control signal includes a potential match signal (potentially_matched). The potential match signal is a signal indicating tracking object information that corresponds to the constraint-violating portion of one or more pieces of tracking object information.

[0109] If the solution satisfies the constraints, the arbitration unit 38 provides a constraint satisfaction control signal to the modification unit 40, the addition unit 42, and the deletion unit 46. More specifically, the arbitration unit 38 provides a match signal to the modification unit 40, a mismatch detection signal to the addition unit 42, and a mismatch tracking signal to the deletion unit 46.

[0110] If the solution does not satisfy the constraint conditions and the obtained solution is a valid constraint-violating solution, the arbitration unit 38 supplies a constraint satisfaction control signal to the modification unit 40, the addition unit 42, and the deletion unit 46, and supplies a constraint violation control signal to the life extension unit 44. More specifically, the arbitration unit 38 supplies a match signal to the modification unit 40, a mismatch detection signal to the addition unit 42, a mismatch tracking signal to the deletion unit 46, and a potential match signal to the life extension unit 44.

[0111] If the solution does not satisfy the constraint conditions and the acquired solution is not a valid constraint-violating solution, the arbitration unit 38 generates an exception signal that causes exception processing to be executed, and supplies the signal to the exception processing unit 50 .

[0112] The arbitration unit 38 does not acquire a combinatorial optimization problem at a time when the image data does not contain a detected object and no detected object information has been acquired, or at a time when the tracked object does not exist and no tracked object information is included in the internal state information. At a time when the image data does not contain a detected object and no detected object information has been acquired, the arbitration unit 38 generates unmatched detected object information for all tracked object information included in the internal state information and supplies this to the deletion unit 46. At a time when the tracked object does not exist and no tracked object information is included in the internal state information, the arbitration unit 38 generates unmatched detected object information for all acquired detected object information and supplies this to the addition unit 42.

[0113] The modifying unit 40 periodically acquires a match signal included in the constraint satisfaction control signal. Based on the match signal, the modifying unit 40 periodically modifies, based on the match signal, tracking object information associated with one or more pieces of detected object information among one or more pieces of tracking object information included in the internal state information stored in the memory unit 28. For example, based on the match signal, the modifying unit 40 modifies tracking object information associated with one or more pieces of detected object information among one or more pieces of tracking object information at a first time point, using the associated detected object information. For example, the modifying unit 40 modifies the tracking object information of the target according to the Kalman filter theory, based on information included in the associated detected object information. This allows the modifying unit 40 to reflect the state of the detected object included in the image data in one or more pieces of tracking object information included in the internal state information. Furthermore, the modifying unit 40 changes the age included in the tracking object information of the target to an initial value (e.g., 0).

[0114] The adding unit 42 periodically acquires an unmatched detected object signal included in the constraint satisfaction control signal. Based on the unmatched detected object signal, the adding unit 42 periodically adds, as new tracking object information, detected object information that does not correspond to any tracking object information among the one or more pieces of detected object information to one or more pieces of tracking object information included in the internal state information stored in the storage unit 28. For example, based on the unmatched detected object signal, the adding unit 42 adds, as new tracking object information, detected object information that does not correspond to any tracking object information among the one or more pieces of detected object information at a first time, to the one or more pieces of tracking object information at the first time. This allows the adding unit 42 to reflect a detected object that has newly framed into the image data in one or more pieces of tracking object information included in the internal state information. Furthermore, the modifying unit 40 sets the age included in the newly added tracking object information to an initial value (e.g., 0).

[0115] The life extension unit 44 periodically acquires a potential match signal included in the constraint violation control signal. Based on the potential match signal, the life extension unit 44 periodically decreases, by a predetermined value, the age of one or more pieces of tracking object information included in the internal state information stored in the memory unit 28, the tracking object information corresponding to the constraint violation portion, which indicates the elapsed time since the tracking object information was added or modified. For example, based on the potential match signal, the life extension unit 44 decreases, by a predetermined value, the age of one or more pieces of tracking object information at a first time, the tracking object information corresponding to the constraint violation portion. This allows the life extension unit 44 to prevent the tracking object information from being deleted even if the target tracking object is temporarily not included in the image data due to occlusion. Therefore, the life extension unit 44 can continue tracking a tracking object that reappears in the image data after the occlusion is resolved.

[0116] The deletion unit 46 periodically acquires an unmatched tracking object signal included in the constraint satisfaction control information. Based on the unmatched tracking object signal, the deletion unit 46 periodically deletes tracking object information whose age exceeds a predetermined threshold from among one or more pieces of tracking object information included in the internal state information stored in the storage unit 28. For example, based on the unmatched tracking object signal, the deletion unit 46 deletes tracking object information whose age exceeds a predetermined threshold from among one or more pieces of tracking object information at a first time. This allows the deletion unit 46 to delete a tracking object that has framed out of the image data from the internal state information and update the internal state information to the latest state. Furthermore, the deletion unit 46 deletes information about objects that do not need to be tracked from the internal state information, thereby preventing one or more pieces of tracking object information included in the internal state information from continuously increasing.

[0117] The threshold value is set to a value that prevents the tracking object information, the age of which has been reduced by a predetermined value by the lifespan extension unit 44 at the first time, from being deleted at least at the first time. That is, the deletion unit 46 does not delete, at the first time, the tracking object information, the age of which has been reduced by a predetermined value by the lifespan extension unit 44 at the first time. This allows the lifespan extension unit 44 to prevent the tracking object information from being deleted when the target tracking object is temporarily not included in the image data due to, for example, occlusion.

[0118] The output unit 48 outputs one or more pieces of tracking object information included in the internal state information stored in the memory unit 28 to the planning device 16. For example, the output unit 48 outputs one or more pieces of tracking object information at a first time to the planning device 16. Note that the output unit 48 may output one or more pieces of tracking object information in response to a request from the planning device 16, rather than outputting one or more pieces of tracking object information at regular time intervals, or may output one or more pieces of tracking object information asynchronously with the time interval of the frames.

[0119] The output unit 48 may also output a portion of the information included in each of the one or more pieces of tracking object information. For example, the output unit 48 may output the horizontal pixel position, vertical pixel position, area, and identification information for each of the one or more pieces of tracking object information. The output unit 48 may also output type information of the tracking object.

[0120] The exception processing unit 50 receives an exception signal from the arbitration unit 38. When receiving the exception signal, the exception processing unit 50 instructs, for example, another device, to execute exception processing that is executed when, for example, the object tracking unit 30 is unable to properly track an object. For example, the exception processing unit 50 outputs warning information, outputs a signal indicating that tracking accuracy has deteriorated to the planning device 16 or the like, or initializes or resets the processing in the object tracking unit 30.

[0121] Fig. 9 is a diagram showing an example of multiple decision variables included in the combinatorial optimization problem generated by the formulation unit 36. Fig. 10 is a diagram showing an example of multiple coefficients included in the combinatorial optimization problem generated by the formulation unit 36.

[0122] The formulation unit 36 ​​generates a combinatorial optimization problem under predetermined constraints that associates, for each of one or more pieces of tracking object information at an arbitrary first time, any of the one or more pieces of detected object information at the first time that is determined to represent the same object. More specifically, the formulation unit 36 ​​generates a bipartite graph maximum matching problem as a combinatorial optimization problem that minimizes a cost function under predetermined constraints.

[0123] The bipartite graph maximum matching problem generated by the formulation unit 36 ​​will now be further described.

[0124] Each of the one or more pieces of detection unit information and each of the one or more pieces of tracking object information is assigned a unique identification information. Detection object information newly added to the one or more pieces of detection object information is assigned an identification information that does not overlap with identification information previously assigned. Tracking object information newly added to the one or more pieces of tracking object information is also assigned an identification information that does not overlap with identification information previously assigned.

[0125] 9 and 10, identification information d1, d2, d3, d4, and d5 are assigned to five pieces of detected object information. Also, in the example of Fig. 9 and 10, identification information t1, t2, t3, t4, and t5 are assigned to five pieces of tracked object information.

[0126] For example, if the number of detected object information pieces is N, d For example, if the number of pieces of tracking information is N t In this case, the formulation unit 36 ​​determines N t ×N d Define decision variables.

[0127] The decision variable is b t , d It is expressed as: b t , d is a bit variable, representing 0 or 1.

[0128] For more details, see b t , d is the t-th item (t is 1 or more, N t For the tracked object information, the dth (d is an integer greater than or equal to 1) of one or more detected object information d If the detected object information (an integer below) is determined to represent the same object and is associated, it represents 1, and if it is not associated, it represents 0.

[0129] Constraint-free cost function (H) in the bipartite graph maximum matching problem cost ) is expressed by equation (11).

number

[0130] IOU(t,d) is a real number equal to or greater than 0 and represents a coefficient. IOU(t,d) is a coefficient that represents a value that is larger the more likely it is that the t-th tracked object information and the d-th detected object information represent the same object. For example, IOU(t,d) becomes a larger value the greater the degree of match between the position and size of the tracked object represented by the t-th tracked object information and the position and size of the detected object represented by the d-th detected object information.

[0131] In this embodiment, IOU(t, d) is the value obtained by dividing the area of ​​the overlapping portion between the boundary box represented by the t-th tracked object information and the boundary box represented by the d-th detected object information by the area of ​​the rectangle containing the boundary box represented by the t-th tracked object information and the boundary box represented by the d-th detected object information.

[0132] The formulation unit 36 ​​generates IOU(t, d) for each pair of one or more pieces of detected object information and one or more pieces of tracked object information. d and the number of tracking information items is N t If so, the formulation unit 36 ​​determines N t ×N dCalculate the IOU(t,d).

[0133] The cost function described above is minimized when the sum of IOU(t, d) for pairs of tracking object information and detected object information determined to be associated by the decision variable, i.e., pairs of tracking object information and detected object information for which the decision variable is 1, is maximized.

[0134] FIG. 11 is a diagram showing the relationship between the range of a set of solutions that a decision variable can take and the range of a set of solutions that satisfy constraint conditions.

[0135] The possible solutions for the decision variables are 2(N t ×N d ) solutions exist. In contrast, the solution that satisfies the constraints in the bipartite graph maximum matching problem is the solution that satisfies the constraints in the 2(N t ×N d In this embodiment, among the set of solutions that the decision variables can take, a solution that satisfies the constraint conditions is called a constraint-satisfying solution, and a solution that does not satisfy the constraint conditions is called a constraint-violating solution.

[0136] The constraints in the bipartite graph maximum matching problem are expressed by the equality constraints shown in equations (12-1), (12-2), (13-1) and (13-2).

number

number

[0137] Equation (12-1) is N d ≧N t In the case of N t For each of the tracking information, the corresponding N d This represents the constraint that there is only one decision variable among the N decision variables whose value is 1. In other words, equation (12-1) d ≧N t In the case of N t For each of the tracking information, N dIt is to be noted that equation (12-1) indicates that any one of the pieces of detected object information is associated as information representing the same object. t It contains equality constraints.

[0138] Equation (12-2) is N d <N t In the case of N t For each of the tracking information, the corresponding N d Any two of the decision variables (b t , d ) and (b t , d´ ) is 0. In other words, equation (12-2) expresses the constraint that N d <N t In the case of N t For each of the tracking information, N d It means that two or more pieces of detected object information cannot be associated as information representing the same object. t It contains equality constraints.

[0139] Equation (13-1) is N d ≦N t In the case of N d For each of the detected object information, the corresponding N t This represents the constraint that there is only one decision variable among the N decision variables whose value is 1. In other words, equation (13-1) d ≦N t In the case of N d For each of the tracking information, N t It is to be noted that equation (13-1) indicates that any one of the tracking object information is associated as information representing the same object. d It contains equality constraints.

[0140] Equation (13-2) is N d >N t In the case of N d For each of the detected object information, the corresponding N t Any two of the decision variables (b t ,d ) and (b t´ , d ) is 0. In other words, equation (13-2) expresses the constraint that N d >N t In the case of N d For each of the detected object information, N t This means that two or more pieces of tracking object information cannot be associated as information representing the same object. d It contains equality constraints.

[0141] Equations (12-1), (12-2), (13-1) and (13-2) are t ×N d The decision variables are N t Row N d When expressed as a column matrix, this means the following:

[0142] N d =N t In the case of N, the sum of the decision variable values ​​for each row is 1. d =N t If , the sum of the decision variables for each column is 1.

[0143] N t <N d In the case of N, the sum of the decision variable values ​​for each row is 1. t <N d If , the sum of the decision variables for each column is either 0 or 1. t <N d , the number of columns that sum to 0 is (N d -N t ) pieces.

[0144] N t >N d In the case of N, the sum of the decision variable values ​​for each column is 1. t >N d If , the sum of the decision variables for each row is either 0 or 1. t >N d , the number of rows that sum to 0 is (N t -N d ) pieces.

[0145] In this way, the constraint conditions are expressed by one or more constraint equations. Each of the one or more constraint equations is expressed by an equality or inequality including one or more decision variables among the multiple decision variables. In this embodiment, each of the one or more constraint equations is expressed by an equality including a linear or quadratic term of the decision variable.

[0146] From the above, the multiple constraint equations expressed by equations (12-1), (12-2), (13-1) and (13-2) imply the following (first condition), (second condition) and (third condition) constraint conditions.

[0147] (First condition) N d =N t In the case of N d Detected object information and N t In other words, the N tracking object information currently being tracked by the object tracking unit 30 is t The number of tracking objects is N contained in the image data. d Each detected object is associated with a corresponding one.

[0148] (Second condition) N t <N d In the case of N d Of the detected object information (N d -N t ) pieces of detected object information are detected object information representing new detected objects. d N out of the detected object information t The information on detected objects is N t In other words, the N d (N d -N t ) detected objects are objects that have newly entered the image data and are currently being tracked. t On the other hand, the N d N out of detected objects t The detected objects are currently being tracked by N tIt is associated one-to-one with each tracked object.

[0149] (Third condition) N t >N d In the case of N t Of the tracking information (N t -N d ) tracked object information does not have corresponding detected object information. t N of the tracking information d The tracking information of N d In other words, N t (N of the traces t -N d ) tracked objects are objects that have gone out of the frame and are currently included in the image data. d On the other hand, N t N out of detected objects d The number of tracked objects is N, which is currently included in the image data. d Each detected object is associated with a corresponding one.

[0150] Therefore, the object tracking unit 30 can associate one or more pieces of tracking object information with any of the one or more pieces of detected object information that is determined to represent the same object, by searching for a combination of values ​​of multiple decision variables that minimizes the cost function under the constraints described above.

[0151] In this embodiment, the formulation unit 36 ​​incorporates the above-mentioned constraint conditions into the cost function to generate a combinatorial optimization problem in a form that can be solved by a solver that solves combinatorial optimization problems without constraints. This allows the solver unit 22 to output a solution to the combinatorial optimization problem supplied from the formulation unit 36, even if the solver unit 22 is a solver that solves combinatorial optimization problems without constraints.

[0152] In this embodiment, the formulation unit 36 ​​generates a combinatorial optimization problem that minimizes a total cost function that is the sum of a cost function and a penalty function, in a format that can be solved by a solver that solves unconstrained combinatorial optimization problems.

[0153] A penalty function is a function that includes at least some of the decision variables and that takes a minimum value when the solution satisfies the constraints. Specifically, the penalty function takes a minimum value (e.g., 0) when the solution satisfies one or more constraint equations that represent the constraints, and takes a value greater than the minimum value when the values ​​of the decision variables do not satisfy one or more constraint equations.

[0154] In this embodiment, the formulation unit 36 ​​generates the total cost function shown in equation (14).

number

[0155] (CH in formula (14) penalty1 +C2H penalty2 ) represents the penalty function. H penalty1 represents the first penalty function. H penalty2 represents the second penalty function.

[0156] The first penalty function (H penalty1 ) is expressed by equations (15-1) and (15-2).

number

[0157] The first penalty function (H penalty1 ) is expressed by a quadratic function of the decision variables. t If all of the constraints of the equality equations are satisfied, the minimum value (0 in this example) is obtained. t The larger the violation number, which is the number of constraints that are satisfied among the constraints of the equations, the larger the value.

[0158] The first penalty function (H penalty1) is expressed by a quadratic function of the decision variables. t If all the constraints of the equality equations are satisfied, the minimum value (0 in this example) is obtained. t The larger the violation number, which is the number of constraints that are satisfied among the constraints of the equations, the larger the value.

[0159] The second penalty function (H penalty2 ) is expressed by equations (16-1) and (16-2).

number

[0160] The second penalty function (H penalty2 ) is expressed by a quadratic function of the decision variables. d If all the constraints of the equality equations are satisfied, the minimum value (0 in this example) is obtained. d The larger the violation number, which is the number of constraints that are satisfied among the constraints of the equations, the larger the value.

[0161] The second penalty function (H penalty2 ) is expressed by a quadratic function of the decision variables. d If all the constraints of the equality equations are satisfied, the minimum value (0 in this example) is obtained. d The larger the number of violations, which is the number of constraint equations that are not satisfied among the equations, the larger the value.

[0162] In equation (14), C1 is a first coupling coefficient by which the first penalty function is multiplied, and is a positive real number greater than 0. C2 is a second coupling coefficient by which the second penalty function is multiplied, and is a positive real number greater than 0.

[0163] C1 and C2 are coefficients that determine the weight of the penalty function relative to the cost function. When C1 and C2 are sufficiently large, the values ​​of the multiple decision variables that minimize the total cost function are the values ​​of the multiple decision variables that minimize the cost function among the values ​​of the multiple decision variables that satisfy the constraints (constraint-satisfying solutions). When C1 and C2 are relatively small, the values ​​of the multiple decision variables that minimize the total cost function may be the values ​​of the multiple decision variables that do not satisfy the constraints (constraint-violating solutions). Therefore, C1 and C2 are appropriately adjusted based on both the perspectives of improving the accuracy of object tracking under normal conditions and improving occlusion response capabilities.

[0164] The total cost function shown in equation (14) is expressed by a quadratic function of multiple decision variables. Therefore, if the solver unit 22 is a solver that can solve quadratic unconstrained binary optimization problems, it can output a solution to the problem that minimizes the total cost function shown in equation (14).

[0165] The formulation unit 36 ​​may generate a combinatorial optimization problem that minimizes a total cost function different from the total cost function shown in Equation 14. For example, the formulation unit 36 ​​may generate a combinatorial optimization problem that minimizes a total cost function obtained by adding a penalty function to a cost function expressed as a quadratic function of multiple decision variables.

[0166] The formulation unit 36 ​​may also generate a higher-order unconstrained binary optimization (HUBO) problem that minimizes a total cost function obtained by adding a cost function, which is a cubic or higher function of multiple decision variables, and a penalty function. However, when the formulation unit 36 ​​generates a HUBO problem, the solver unit 22 must be a solver that can output a solution to the HUBO problem. A solver that can output a solution to the HUBO problem can be realized by executing a simulated bifurcation algorithm, as shown in Patent Document 3, for example.

[0167] In this embodiment, the constraints are expressed as equalities. However, the constraints may be inequalities. When the constraint conditions are expressed as multiple constraints, some of the multiple constraints may be equalities and the other may be inequalities.

[0168] Fig. 12 is a flowchart showing the processing flow of object tracking device 14. Fig. 13 is a diagram showing a data flow graph of object tracking device 14. The processing and data flow of object tracking device 14 will be explained below with reference to the flowchart of Fig. 12 and the data flow graph of Fig. 13. Note that in the explanations of Figs. 12 and 13, it is assumed that one or more pieces of detection unit information are detected in all frames, and that one or more pieces of tracked object information are included in the internal state information in all frames.

[0169] The object tracking device 14 repeatedly executes the processes of S32 to S41 for each frame (repeated processes between S31 and S42).

[0170] In S32, the object detection unit 26 of the object tracking device 14 performs an object detection process on the image data of the kth frame (k is an integer) acquired from the camera 12, and generates one or more pieces of detection information (detections) representing one or more detected objects included in the kth image data. Then, the acquisition unit 32 of the object tracking device 14 acquires the one or more pieces of detection information (detections) of the generated kth frame.

[0171] In parallel with S32, in S33, the prediction unit 34 of the object tracking device 14 predicts one or more pieces of tracking object information representing one or more tracking objects in the k-th frame based on one or more pieces of tracking object information (trackers) representing one or more tracking objects in the (k-1)-th frame, which are included in the internal state information of the storage unit 28 (state memory).The prediction unit 34 then stores the predicted one or more pieces of tracking object information for the k-th frame in the storage unit 28.In this case, the prediction unit 34 increases the age included in each of the one or more pieces of tracking object information for the k-th frame by a predetermined value (for example, 1).

[0172] Next, in S34, the formulation unit 36 ​​of the object tracking device 14 generates a combinatorial optimization problem for the k-th frame based on one or more pieces of tracked object information in the k-th frame and one or more pieces of detected object information in the k-th frame.

[0173] Next, in S35, the solver unit 22 of the object tracking device 14 acquires a combinatorial optimization problem for the k-th frame. Then, the solver unit 22 solves the acquired combinatorial optimization problem while allowing constraints not to be satisfied.

[0174] Next, in S36, the arbitration unit 38 of the object tracking device 14 executes arbitration processing. Specifically, the arbitration unit 38 acquires a solution to the combinatorial optimization problem for the k-th frame from the solver unit 22. Next, the arbitration unit 38 evaluates the acquired solution. If the acquired solution satisfies the constraint conditions, the arbitration unit 38 outputs a constraint satisfaction control signal based on the solution that satisfies the constraint conditions. The constraint satisfaction control signal includes a match signal (matched), an unmatched detection signal (unmatched_detections), and an unmatched trackers signal (unmatched_trackers).

[0175] Furthermore, if the acquired solution does not satisfy the constraint conditions but is a valid constraint-violating solution close to a constraint-satisfying solution, the arbitration unit 38 identifies a constraint-satisfying portion of the acquired solution that satisfies the constraint conditions and a constraint-violating portion of the solution that does not satisfy the constraint conditions. Then, if the acquired solution does not satisfy the constraint conditions but is a valid constraint-violating solution, the arbitration unit 38 generates a constraint satisfaction control signal based on the constraint satisfaction portion and a constraint violation control signal based on the constraint violation portion. The constraint violation control signal includes a potentially matched signal (potentially_matched).

[0176] Furthermore, the arbitration unit 38 outputs an exception signal when the acquired solution does not satisfy the constraint conditions and is not a valid solution violating the constraints.

[0177] Next, in S37, the modifying unit 40 of the object tracking device 14 acquires a match signal included in the constraint satisfaction control signal. Based on the match signal, the modifying unit 40 modifies the tracking object information associated with one or more pieces of detected object information among the one or more pieces of tracking object information of the k-th frame included in the internal state information stored in the memory unit 28, based on the associated detected object information. Furthermore, the modifying unit 40 modifies the age included in the associated tracking object information to an initial value (e.g., 0).

[0178] Next, in S38, the adding unit 42 of the object tracking device 14 acquires an unmatched detected object signal included in the constraint satisfaction control signal. Based on the unmatched detected object signal, the adding unit 42 adds the detected object information that does not correspond to any of the tracking object information among the one or more pieces of detected object information as new tracking object information to the one or more pieces of tracking object information of the k-th frame included in the internal state information stored in the storage unit 28. Furthermore, the modifying unit 40 sets the age included in the newly added tracking object information to an initial value (e.g., 0).

[0179] Next, in S39, the life extension unit 44 of the object tracking device 14 acquires a potential match signal included in the constraint violation control signal. Based on the potential match signal, the life extension unit 44 decreases by a predetermined value (for example, 1) the age in the tracking object information corresponding to the constraint violation portion among one or more pieces of tracking object information in the k-th frame included in the internal state information stored in the memory unit 28.

[0180] Next, in S40, the deletion unit 46 of the object tracking device 14 acquires an unmatched tracking object signal included in the constraint satisfaction control signal. Based on the unmatched tracking object signal, the deletion unit 46 deletes tracking object information whose age exceeds a predetermined threshold (e.g., 1) from among one or more pieces of tracking object information of the k-th frame included in the internal state information stored in the storage unit 28. Note that the threshold is set to a value such that tracking object information whose age has been reduced by a predetermined value in the process of S39 of the k-th frame is not deleted in S40 of at least the k-th frame.

[0181] Next, in S41, the output unit 48 of the object tracking device 14 outputs one or more pieces of tracking object information of the k-th frame included in the internal state information stored in the memory unit 28 to the planning device 16. Note that the output unit 48 may execute the output process for all frames, or may execute the output process for every predetermined number of frames or every time there is a request from the planning device 16.

[0182] The object tracking device 14 executes the above process for each frame (repeated process between S31 and S42).

[0183] 14 is a flowchart showing the flow of processing by the arbitration unit 38. When the arbitration unit 38 acquires a solution from the solver unit 22, the arbitration unit 38 executes arbitration processing, for example, according to the flow shown in the flowchart of FIG.

[0184] First, in S51, the arbitration unit 38 acquires the solution output from the solver unit 22.

[0185] Next, in S52, the arbitration unit 38 evaluates the acquired solution. Specifically, first, the arbitration unit 38 evaluates whether the solution satisfies the constraint conditions. If the solution does not satisfy the constraint conditions, the arbitration unit 38 further evaluates whether the solution is a valid constraint-violating solution.

[0186] For example, the constraints are expressed by one or more constraint equations, each of which is an equality or inequality including multiple decision variables as arguments. The arbitration unit 38 evaluates an acquired solution as satisfying the constraints if the acquired solution satisfies all of the one or more constraint equations. In other words, the arbitration unit 38 evaluates an acquired solution as a constraint-satisfying solution if the acquired solution satisfies all of the one or more constraint equations.

[0187] Furthermore, if the acquired solution does not satisfy at least one of one or more constraint equations, the arbitration unit 38 evaluates the acquired solution as a constraint-violating solution that does not satisfy the constraint conditions.

[0188] If the obtained solution is a constraint-violating solution, the arbitration unit 38 further evaluates whether the obtained solution is a valid constraint-violating solution that is close to a constraint-satisfying solution. The arbitration unit 38 evaluates whether the obtained solution is a valid constraint-violating solution based on predetermined evaluation criteria.

[0189] For example, the arbitration unit 38 assigns the obtained solution to each of one or more constraint equations, and calculates the number of violations, which is the number of constraint equations that do not satisfy the equation when the obtained solution is assigned to the one or more constraint equations. If the number of violations is equal to or less than a preset value, the arbitration unit 38 evaluates the solution as a valid constraint-violating solution.

[0190] Furthermore, for example, the arbitration unit 38 substitutes the obtained solution into a penalty function and calculates a penalty value, which is the value of the penalty function when the obtained solution is substituted. The arbitration unit 38 may evaluate the solution as a valid constraint-violating solution if the penalty value is equal to or less than a preset value.

[0191] In S53, the arbitration unit 38 determines whether the acquired solution satisfies the constraint conditions based on the evaluation result. If the acquired solution satisfies the constraint conditions (Yes in S53), the arbitration unit 38 proceeds to S54. Then, in S54, the arbitration unit 38 generates and outputs a constraint satisfaction control signal based on the acquired solution. After completing the process of S54, the arbitration unit 38 ends this flow.

[0192] If the acquired solution does not satisfy the constraint conditions (No in S53), the arbitration unit 38 proceeds to S55. In S55, the arbitration unit 38 determines whether the acquired solution is a valid solution that violates the constraints based on the evaluation result. If the acquired solution is not a valid solution that violates the constraints (No in S55), the arbitration unit 38 outputs an exception signal that causes exception processing to be executed, and ends this flow.

[0193] If the acquired solution is a valid constraint-violating solution (Yes in S55), the arbitration unit 38 advances the process to S56.

[0194] In S56, the arbitration unit 38 identifies constraint-satisfying parts and constraint-violating parts in the valid constraint-violating solution.

[0195] For example, based on the obtained solution, the arbitration unit 38 identifies a decision variable among the multiple decision variables that causes the constraint condition not to be satisfied. Then, the arbitration unit 38 identifies a constraint-satisfying portion by changing the value of the decision variable that causes the constraint condition not to be satisfied. Furthermore, the arbitration unit 38 extracts the value of the decision variable that causes the constraint condition not to be satisfied and identifies it as a constraint-violating portion.

[0196] For example, if the value of a decision variable that causes the constraint condition to not be satisfied is 0, the arbitration unit 38 changes it to 1, and if the value is 1, the arbitration unit 38 changes it to 0. Then, for example, if the solution satisfies the constraint condition after changing the value of the decision variable that causes the constraint condition to not be satisfied, the arbitration unit 38 identifies the solution after changing the value of the decision variable that causes the constraint condition to not be satisfied as a constraint-satisfying part. Also, for example, if the solution satisfies the constraint condition after changing the value of the decision variable that causes the constraint condition to not be satisfied, the arbitration unit 38 identifies the decision variable whose value has been changed as the decision variable that causes the constraint condition to not be satisfied.

[0197] For example, the arbitration unit 38 determines whether the values ​​of multiple decision variables arranged in a matrix as shown in FIG. 9 satisfy the constraints for each column (or row). The arbitration unit 38 then identifies a column (or row) that does not satisfy the constraints. The arbitration unit 38 then inverts the value of one of the multiple decision variables included in the column (or row) determined not to satisfy the constraints so that the value satisfies the constraints. If the inversion satisfies the constraints, the arbitration unit 38 identifies the decision variable whose value has been inverted as the cause of the non-satisfaction of the constraints. If the column (or row) determined not to satisfy the constraints contains two or more decision variables that change to satisfy the constraints by inverting the value, the arbitration unit 38 may determine the decision variable that is the cause of the non-satisfaction of the constraints based on the corresponding coefficients. For example, the arbitration unit 38 may select the decision variable that is the cause of the non-satisfaction of the constraints, giving priority to the decision variable with the smallest corresponding IOU.

[0198] Next, in S57, the arbitration unit 38 generates and outputs a constraint satisfaction control signal based on the constraint satisfaction portion and a constraint violation control signal based on the constraint violation portion. After completing the process of S57, the arbitration unit 38 ends this flow.

[0199] Next, the arbitration process, correction process, addition process, life extension process, deletion process, and occlusion countermeasure function will be described with reference to specific examples shown in FIGS.

[0200] Fig. 15 is a diagram showing an example of a boundary box indicated by each of the detected object information and the tracked object information, and a solution, in the (k-1)th frame. Fig. 16 is a diagram showing an example of a boundary box indicated by each of the detected object information and the tracked object information, and a solution, in the kth frame. Fig. 17 is a diagram showing an example of a boundary box indicated by each of the detected object information and the tracked object information, and a solution, in the (k+1)th frame. Fig. 18 is a diagram showing an example of a constraint-satisfying portion identified in the kth frame.

[0201] In addition, in Figs. 15 to 18, S C represents the sum of the values ​​of the decision variables in the column direction. R represents the sum of the values ​​of the decision variables in the row direction.

[0202] First, the arbitration process will be described.

[0203] In the k-th frame, as shown in Figure 16, d <N t N d <N t If the solution is a valid constraint-violating solution, the arbitration unit 38 can identify the constraint-violating portion of the solution.

[0204] First, in the k-th frame, N d <N t Therefore, the arbitration unit 38 determines whether the solution for the k-th frame satisfies the constraints based on the equations (12-2) and (13-1). The solution for the k-th frame is, as shown in FIG. 16, the column-wise sum (S C ) is 2, which does not satisfy equation (13-1).

[0205] Next, the arbitration unit 38 determines whether the solution for the kth frame is a valid constraint-violating solution. For example, if the number of violations in the solution for the kth frame is equal to or less than a preset value (max_violation), the arbitration unit 38 determines that the solution is a valid constraint-violating solution. In this example, the preset value (max_violation) is 1. Note that the preset value may be equal to or greater than 1. The number of violations in the solution for the kth frame is 1. Therefore, the arbitration unit 38 determines that the solution for the kth frame is a valid constraint-violating solution.

[0206] Next, the arbitration unit 38 identifies constraint-violating portions and constraint-satisfying portions from the solution of the k-th frame. The solution of the k-th frame is determined by the column-wise sum (S C) violates the constraint. In the fourth column (d4), both the decision variable for (t2, d4) and the decision variable for (t5, d4) are 1, making it impossible to determine whether the detected object information indicated in d4 corresponds to the tracked object information for t2 or t5. Therefore, the solution for the kth frame satisfies the constraint by inverting the value of either the decision variable for (t2, d4) or the decision variable for (t5, d4). For example, the arbitration unit 38 compares IOU(t2, d4) corresponding to the decision variable for (t2, d4) with IOU(t5, d4) corresponding to the decision variable for (t5, d4), and identifies the smaller decision variable as the constraint violating portion. In this example, it is assumed that IOU(t2, d4) > IOU(t5, d4). Therefore, in this example, the arbitration unit 38 identifies the decision variable for (t5, d4) as the constraint violating portion. Furthermore, the arbitration unit 38 identifies the solution after the value of the decision variable of (t5, d4) is inverted from 1 to 0 as shown in FIG. 18 as the constraint satisfying portion.

[0207] The arbitration unit 38 then generates and outputs a potential match signal (potentially_matched) included in the constraint violation control signal based on the tracking object information indicated in the constraint violation portion. The arbitration unit 38 also generates and outputs a match signal (matched) included in the constraint satisfaction control signal based on the detected object information and tracking object information associated one-to-one in the constraint satisfaction portion. The arbitration unit 38 also generates and outputs an unmatched tracking object signal (unmatched_trackers) included in the constraint satisfaction control signal based on the tracking object information with which the detected object information in the constraint satisfaction portion is not associated. The arbitration unit 38 also generates and outputs an unmatched detection object signal (unmatched_detections) included in the constraint satisfaction control signal based on the detected object information with which the tracking object information in the constraint satisfaction portion is not associated.

[0208] In the k-th frame shown in Fig. 18, the arbitration unit 38 generates a match signal indicating detected object information and tracked object information corresponding to the decision variables indicated by (t1, d3), (t2, d4), (t3, d1), and (t4, d2), respectively. Also, in the k-th frame shown in Fig. 18, the arbitration unit 38 generates an unmatched tracked object signal indicating tracked object information (t5) corresponding to the decision variable (t5, d4). Note that, in the k-th frame shown in Fig. 18, there is no detected object that has newly framed in the image data, and therefore the arbitration unit 38 does not output an unmatched detected object signal.

[0209] 18, the arbitration unit 38 generates a potential match signal indicating the tracking object information corresponding to the decision variables (t5, d4). That is, the tracking object information at t5 is identified as being in a state where there is no corresponding detected object information and at the same time, there is no detected object information that is temporarily associated with the tracking object information due to occlusion.

[0210] Next, the modification process, addition process, life extension process, and deletion process will be described.

[0211] At the start of the k-th frame, the internal state information includes five pieces of tracking object information (t1 to t5) as shown in FIG. 16. Each of the five pieces of tracking object information (t1 to t5) includes an age (age) immediately before the start of the k-th frame. i ) is 0. Then, for each of the five pieces of tracking object information (t1 to t5), the age (age i ) is incremented by 1.

[0212] The modifying unit 40 modifies each of the four pieces of tracked object information (t1, t2, t3, t4) indicated in the match signal, out of the five pieces of tracked object information (t1 to t5) included in the internal state information, using the associated detected object information (d1, d2, d3, d4) based on the Kalman filter theory. More specifically, the modifying unit 40 modifies the tracked object information indicated in t1 using the associated detected object information indicated in d3. The modifying unit 40 also modifies the tracked object information indicated in t2 using the associated detected object information indicated in d4. The modifying unit 40 also modifies the tracked object information indicated in t3 using the associated detected object information indicated in d1. The modifying unit 40 also modifies the tracked object information indicated in t4 using the associated detected object information indicated in d2.

[0213] Furthermore, the modifying unit 40 modifies the ages (ages) of the four pieces of tracking object information (t1, t2, t3, t4) indicated in the match signal. i ) to its initial value of 0.

[0214] The adding unit 42 is N d >N t In this case, the adding unit 42 acquires an unmatched detected object signal indicating tracking object information indicating a detected object that has newly framed into the image data. Then, the adding unit 42 adds the tracking object information indicated in the unmatched detected object signal to the internal state information as new tracking object information. The adding unit 42 also adds the age (age) of the added tracking object information to the internal state information. i ) is set to an initial value of 0. However, in the k-th frame shown in FIG. 16, since there is no detected object that has newly framed in the image data, the adding unit 42 does not receive the unmatched detected object signal and does not add new tracking object information to the internal state information.

[0215] The life extension unit 44 reduces the age of the tracking object information (t5) indicated in the potential match signal among the five tracking object information (t1 to t5) included in the internal state information by a predetermined value (anti-aging). In this example, the predetermined value (anti-aging) is 1. In the k-th frame shown in FIG. 16, the life extension unit 44 reduces the age (age) of the tracking object information indicated in t5. i) is subtracted by 1 to make it 0.

[0216] The deletion unit 46 deletes the tracking object information (t5) indicated in the unmatched tracking object signal, of the five pieces of tracking object information (t1 to t5) included in the internal state information, whose age is greater than a preset threshold (max_age). In this example, the threshold is assumed to be 1. In the k-th frame shown in FIG. 16, the unmatched tracking object signal indicates the tracking object information of t5. However, the age of the tracking object information indicated at t5 has been set to 0 by the life extension unit 44, which is smaller than the threshold. Therefore, the deletion unit 46 does not delete the tracking object information indicated at t5.

[0217] Note that, for example, unlike the example shown in FIG. 16, suppose that in the k-th frame, the arbitration unit 38 obtains the solution shown in FIG. 18 from the solver unit 22. In this case, the arbitration unit 38 does not output a potential match signal in the k-th frame. Therefore, the age of the tracking object information shown at t5 is not reduced by the life extension unit 44. Therefore, in this case, the deletion unit 46 deletes the tracking object information shown at t5.

[0218] Next, the occlusion countermeasure capability of the object tracking unit 30 will be described.

[0219] As shown in the boundary boxes of Figures 15, 16, and 17, the track object corresponding to the track object information at t2 is moving from left to right, and the track object corresponding to the track object information at t5 is moving from the bottom right to the top left.

[0220] As shown in FIG. 16, in the kth frame, the tracked object t2 and the tracked object t5 intersect. As a result, the tracked object t5 is hidden behind the tracked object t2, causing occlusion. Therefore, in the kth frame, the detected object d4 corresponding to the tracked object t2 is included in the image data. However, in the kth frame, the detected object corresponding to the tracked object t5 is not included in the image data. However, by detecting a valid constraint violation solution, the object tracking unit 30 does not delete the tracked object information indicated by t5 in the kth frame by decreasing the age of the tracked object information indicated by t5 by a predetermined value.

[0221] Next, as shown in FIG. 17 , in the k+1th frame, the occlusion of the tracked object at t5 is resolved. As a result, five detected objects are included in the image data in the k+1th frame. Therefore, a one-to-one correspondence can be established between the five pieces of tracked object information included in the internal state information and the five pieces of detected object information detected in the kth frame. The tracked object information at t5 in the k-1th frame and the tracked object information at t5 in the k+1th frame share the same identification information. Therefore, the object tracking unit 30 can identify the tracked object information at t5 in the k-1th frame of FIG. 15 and the tracked object information at t5 in the k+1th frame of FIG. 16 as information indicating the same object, regardless of the occlusion that occurred in the kth frame. Therefore, the object tracking unit 30 can continue to track the tracked object indicated in the tracked object information at t5 in the k+1th frame, continuing from the k-1th frame, regardless of the occlusion that occurred in the kth frame. In this way, the object tracking unit 30 can have an occlusion countermeasure capability.

[0222] On the other hand, unlike the above example, suppose that in the kth frame, the solver unit 22 outputs a solution such as that shown in FIG. 18 and the object tracking unit 30 does not detect a valid constraint-violating solution. In this case, the object tracking unit 30 deletes the tracked object information indicated at t5 in the kth frame. As a result, at the start of the k+1th frame, the internal state information includes four pieces of tracked object information. Therefore, in the k+1th frame, the object tracking unit 30 adds new tracked object information corresponding to the detected object information indicated at d5 to the internal state information with new identification information (e.g., t6). Therefore, the object tracking unit 30 cannot identify the tracked object information indicated at t5 in the k-1th frame and the tracked object information indicated at t6 in the k+1th frame as information indicating the same object due to the influence of occlusion that occurred in the kth frame. Therefore, if a valid constraint-violating solution is not detected, the object tracking unit 30 will discontinue tracking of the tracked object indicated in the tracked object information at t5 due to the influence of occlusion that occurred in the kth frame. In this way, if a valid constraint violating solution is not detected, the object tracking unit 30 will not have the capability to deal with occlusion.

[0223] As described above, the object tracking unit 30 according to this embodiment does not extend the lifespan of all tracking object information included in the internal state information, but rather extends the lifespan of tracking object information corresponding to a tracking object for which an event requiring occlusion countermeasures has occurred. This allows the object tracking unit 30 according to this embodiment to have occlusion countermeasure capabilities while suppressing an unnecessary increase in the number of tracking object information. Therefore, the object tracking unit 30 according to this embodiment suppresses the expansion of prediction and matching processes, and does not lead to a deterioration in object tracking processing performance.

[0224] Specifically, the object tracking unit 30 calculates the total cost function (H total) is solved using a heuristic solution, the tracking object information for which occlusion measures should be taken is identified. For example, in the constraint violation solution shown in Figure 16, the decision variable of (t5, d4) is 1 because IOU(t5, d4) is smaller than the second penalty function (H penalty2 ) (the product of Equation (16-2) and the coupling coefficient (C2)) or the second penalty function (H penalty2 ) is greater than the value of the total cost function (14). As shown in FIG. 16, a constraint violation solution in which the decision variables for (t2, d4) and (t5, d4) are both 1 corresponds to a judgment that it is reasonable to interpret that both the tracked object corresponding to the tracked object information at t2 and the tracked object corresponding to the tracked object information at t4 match the detected object corresponding to the tracked object information at d4 from the perspective of the total cost function. Such a constraint violation solution corresponds to a situation in which tracked objects for which occlusion countermeasures should be implemented intersect with each other. Therefore, the object tracking unit 30 uses the total cost function (H total By solving the unconstrained combinatorial optimization problem of minimizing ∑i=1 / i ...

[0225] As described above, the object tracking device 14 according to this embodiment can improve the performance or efficiency of information processing functions by using a constraint violation solution obtained from a combinatorial optimization problem.

[0226] In the above example, the arbitration unit 38 determines that a solution is a valid constraint-violating solution when the number of violations is 1 or less. Alternatively, the arbitration unit 38 may determine that a solution is a valid constraint-violating solution when the number of violations is equal to or less than a predetermined value greater than 2. This allows the object tracking unit 30 to have an occlusion prevention capability for multiple tracking objects.

[0227] In the above example, the life extension unit 44 reduces the age of the tracking object information indicated in the potential match signal by 1. Alternatively, the life extension unit 44 may reduce the age of the tracking object information indicated in the potential match signal by 2 or more. This allows the object tracking unit 30 to have long-term occlusion countermeasure capabilities.

[0228] In the above example, the deletion unit 46 sets the threshold (max_age) to 1. Alternatively, the deletion unit 46 may set the threshold (max_age) to a value of 2 or greater. In this case, the object tracking unit 30 can have a certain level of occlusion countermeasure capability for all tracking object information. Furthermore, the object tracking unit 30 has a longer-term occlusion countermeasure capability for tracking object information that requires occlusion countermeasures and is identified by a constraint violation solution. However, if the threshold (max_age) is set too large, the object tracking unit 30 will retain tracking object information that should essentially be deleted in the internal state information. Therefore, if the threshold (max_age) is set too large, the object tracking unit 30 will have an increased number of tracking object information, which will lead to bloated prediction and matching processes, and as a result, there is a possibility that this will lead to a deterioration in the processing performance of object tracking.

[0229] In addition, each of one or more pieces of tracking information has an individual threshold (max_age) for deleting that piece of tracking information. i ) may be included. When one or more pieces of tracking object information are added or modified, the individual threshold is set to an initial value (for example, 0). In this case, the deletion unit 46 compares the age included in the tracking object information indicated in the unmatched tracking object signal with the individual threshold, and deletes the tracking object information if the age exceeds the threshold. Also, in this case, the life extension unit 44 increases the threshold included in the tracking object information indicated in the potential match signal by a predetermined value, instead of decreasing the age included in the tracking object information indicated in the potential match signal by a predetermined value.

[0230] (First Modification) FIG. 19 is a diagram showing the processing of the formulation unit 36 ​​according to the first modified example.

[0231] The formulation unit 36 ​​calculates the total cost function (H total ) to the solver unit 22, the process shown in FIG. 19 may be executed.

[0232] First, in S61, the formulation unit 36 ​​calculates N d is the number of one or more pieces of tracking information included in the internal state information, N t Determine whether it is greater than N d <N t If so (Yes in S61), the formulation unit 36 ​​advances the process to S62.

[0233] In S62, the formulation unit 36 ​​calculates the total cost function (H total ) contains the second penalty function (H penalty2 After completing the process of S62, the formulation unit 36 ​​reduces the coupling coefficient (C2) of the total cost function (H total ) to the solver unit 22.

[0234] Also, N d <N t If not (No in S61), the formulation unit 36 ​​does not change the coupling coefficient (C2) and instead uses the total cost function (H total ) to the solver unit 22.

[0235] In this way, the formulation unit 36 ​​calculates N t >N d If N t ≦N d In this way, the formulation unit 36 ​​can formulate the situation where no occlusion occurs, i.e., N t ≦N d In the case where there is a possibility that occlusion has occurred and the lifetime of the tracked object information is extended using a valid constraint violation solution, that is, when N t <N dIf so, the influence of the second penalty function can be reduced to make it easier to generate a solution that violates the constraints.

[0236] The formulation unit 36 ​​uses N d <N t If not (No in S61), increase the coupling coefficient (C2) and d <N t 19 in place of the formulation unit 36. In this case, when converting the combinatorial optimization problem received from the formulation unit 36 ​​into an Ising problem, the formulation unit 36 ​​may convert N d <N t Determine whether or not N d <N t If so, decrease C2.

[0237] (Second Modification) FIG. 20 is a diagram showing the processing of the arbitration unit 38 according to the second modification.

[0238] The formulation unit 36 ​​calculates the total cost function (H total ) the second penalty function (H penalty2 ) with the standard value of the coupling coefficient (C2) and the total cost function (H total ) the second penalty function (H penalty2 ) may be relaxed. The relaxed value is smaller than the standard value. In other words, the ratio of the penalty function to the cost function in the first combinatorial optimization problem is larger than that in the second combinatorial optimization problem.

[0239] Then, the formulation unit 36 ​​supplies the first combinatorial optimization problem and the second combinatorial optimization problem to the solver unit 22.

[0240] The solver unit 22 solves each of the first and second combinatorial optimization problems, and then supplies the solutions of the first and second combinatorial optimization problems to the arbitration unit 38.

[0241] In the second modified example, the arbitration unit 38 executes the process according to the flow shown in FIG.

[0242] First, in S71, the arbitration unit 38 obtains the solution to the first combinatorial optimization problem output from the solver unit 22. Next, in S72, the arbitration unit 38 evaluates the obtained solution to the first combinatorial optimization problem. Specifically, the arbitration unit 38 evaluates whether or not the obtained solution to the first combinatorial optimization problem satisfies the constraint conditions.

[0243] Next, in S73, the arbitration unit 38 determines whether the solution to the first combinatorial optimization problem satisfies the constraints based on the evaluation results. If the solution to the first combinatorial optimization problem does not satisfy the constraints (No in S73), the arbitration unit 38 outputs an exception signal and ends this flow.

[0244] If the solution to the first combinatorial optimization problem satisfies the constraints (Yes in S73), the arbitration unit 38 proceeds to S74.

[0245] In S74, the arbitration unit 38 obtains the solution to the second combinatorial optimization problem output from the solver unit 22. Subsequently, in S75, the arbitration unit 38 evaluates the solution to the second combinatorial optimization problem. The arbitration unit 38 performs, for example, an evaluation process similar to S52 in Fig. 14. Note that the arbitration unit 38 may perform the processes of S74 and S75 in parallel with the processes of S71 and S72.

[0246] Next, in S76, the arbitration unit 38 determines whether the solution to the second combinatorial optimization problem satisfies the constraints based on the evaluation results. If the solution to the second combinatorial optimization problem satisfies the constraints (Yes in S76), an exception signal is output and this flow ends.

[0247] If the solution to the second combinatorial optimization problem does not satisfy the constraints (No in S76), the arbitration unit 38 proceeds to S78. In S78, the arbitration unit 38 determines whether the solution to the second combinatorial optimization problem is a valid solution that violates the constraints, based on the evaluation results. If the solution to the second combinatorial optimization problem is not a valid solution that violates the constraints (No in S78), the arbitration unit 38 outputs an exception signal and ends this flow.

[0248] If the solution to the second combinatorial optimization problem is a valid constraint violating solution (Yes in S78), the arbitration unit 38 proceeds to S79.

[0249] In S79, the arbitration unit 38 identifies the constraint satisfying portion and the constraint violating portion in the valid constraint violating solution. Specifically, the arbitration unit 38 identifies the constraint violating portion based on the difference between the solution to the first combinatorial optimization problem and the solution to the second combinatorial optimization problem. Furthermore, the arbitration unit 38 identifies the solution to the first combinatorial optimization problem as the constraint satisfying portion.

[0250] Next, in S80, the arbitration unit 38 generates and outputs a constraint satisfaction control signal based on the constraint satisfaction portion and a constraint violation control signal based on the constraint violation portion. After completing the process of S80, the arbitration unit 38 ends this flow.

[0251] Fig. 21 is a diagram showing an example of a solution to the first combinatorial optimization problem. Fig. 22 is a diagram showing an example of a solution to the second combinatorial optimization problem. Fig. 23 is a diagram showing an example of a boundary box when the solutions of Figs. 21 and 22 are obtained.

[0252] The object tracking unit 30 according to the second modification obtains solutions to two combinatorial optimization problems with different coupling coefficients (C2) of the penalty functions. A combinatorial optimization problem with a large coupling coefficient (C2) is less likely to produce a constraint-violating solution than a combinatorial optimization problem with a small coupling coefficient (C2). The object tracking unit 30 identifies the constraint-satisfying portion using the solution to the first combinatorial optimization problem with a large coupling coefficient (C2). The object tracking unit 30 also identifies the constraint-violating portion using the difference between the solution to the first combinatorial optimization problem with a large coupling coefficient (C2) and the solution to the second combinatorial optimization problem with a small coupling coefficient (C2). The object tracking unit 30 according to the second modification can easily identify the constraint-violating portion and the constraint-satisfying solution even when a valid constraint-violating solution contains decision variables that cause two or more constraint violations.

[0253] For example, as shown in Fig. 21 and Fig. 22, assume that the internal state information includes five pieces of tracked object information, and three pieces of detected object information are detected based on the image data. In this case, as shown in Fig. 23, the tracked objects at t2 and t5 intersect, and the tracked objects at t1 and t3 intersect. The solution to the second combinatorial optimization problem shown in Fig. 22 is the column-wise sum (S C ) and the column-wise sum in the third column (d3) (S C ) are 2, which does not satisfy equation (13-1). Even in such a case, the arbitration unit 38 can determine that the decision variables of (t3, d2) and (t5, d3) are the cause of the constraint violation by detecting the difference between the solution to the first combinatorial optimization problem shown in FIG. 21 and the solution to the second combinatorial optimization problem shown in FIG. 22. Therefore, the arbitration unit 38 can output a constraint violation control signal that includes a potential match signal (potentially_matched) that indicates the tracked object information of t3 and the tracked object information of t5.

[0254] In the second modification, the solver unit 22 may include, for example, two Ising machines 52, and may solve the first combinatorial optimization problem and the second combinatorial optimization problem simultaneously in parallel. This allows the solver unit 22 to execute two solution-finding processes in parallel and output a solution in a short time. Alternatively, the solver unit 22 may execute two solution-finding processes sequentially using one Ising machine 52. Such a solver unit 22 can reduce the circuit configuration and reduce manufacturing costs, etc.

[0255] (Application to information processing systems) As described above, the object tracking device 14 according to this embodiment executes the following series of processes.

[0256] First, the formulation unit 36 ​​of the object tracking device 14 according to this embodiment generates a combinatorial optimization problem based on data under predetermined constraints. For example, the object tracking device 14 generates a combinatorial optimization problem that minimizes a total cost function that adds a cost function and a penalty function. Then, the solver unit 22 of the object tracking device 14 according to this embodiment acquires the generated combinatorial optimization problem, solves the acquired combinatorial optimization problem while allowing the constraints not to be satisfied, and calculates a solution to the combinatorial optimization problem obtained by solving it.

[0257] Furthermore, the object tracking device 14 according to this embodiment generates a constraint violation control signal if the solution does not satisfy the constraint conditions. Then, the object tracking device 14 according to this embodiment executes a predetermined first process on the data based on the constraint violation control signal, and outputs the data after the first process. Furthermore, if the solution satisfies the constraint conditions, the object tracking device 14 according to this embodiment executes a predetermined second process on the data based on the solution, and outputs the data after the second process.

[0258] Furthermore, when a solution does not satisfy the constraint conditions, arbitration unit 38 of object tracking device 14 according to this embodiment identifies constraint-violating portions of the solution that do not satisfy the constraint conditions and constraint-satisfying portions of the solution that do satisfy the constraint conditions, and outputs a constraint violation control signal based on the constraint violation portions and a constraint satisfaction control signal based on the constraint satisfaction portions. When a solution does not satisfy the constraint conditions, object tracking device 14 according to this embodiment performs a first process on the data based on the constraint violation control signal, outputs the data on which the first process has been performed, and performs a second process on the data based on the constraint satisfaction control signal, and outputs the data on which the second process has been performed.

[0259] Furthermore, if the solution satisfies the constraint conditions, the object tracking device 14 according to this embodiment further executes a predetermined second process on the data based on the solution, and outputs the data on which the second process has been executed.

[0260] An information processing system that includes a process for solving a combinatorial optimization problem based on data under predetermined constraints while processing data can execute the above series of processes.

[0261] For example, a control device that analyzes large-scale point cloud data output from a LiDAR by solving a constrained combinatorial optimization problem and controls a controlled object based on the analysis results can execute the above series of processes.

[0262] Furthermore, for example, a wireless base station that solves a constrained combinatorial optimization problem for allocating wireless communication resources such as frequency channels and time slots to a large number of terminal devices and allocates wireless resources to the large number of terminal devices based on the solution of the constrained combinatorial optimization problem can execute the above series of processes.

[0263] Furthermore, for example, a control device in a wireless communication system that has multiple transmitting antennas and multiple receiving antennas and can use multiple frequency channels as wireless communication paths, and that selects the optimal wireless communication path by solving a constrained combinatorial optimization problem, can execute the above-described series of processes.

[0264] Furthermore, for example, a financial market analysis device that extracts the largest independent set of one or more stocks that are weakly correlated with each other based on a graph consisting of nodes corresponding to multiple stocks and edges corresponding to the correlation between two stocks can perform the above series of processes.

[0265] Furthermore, for example, a control device for a large-scale computer system having a large number of computing nodes that processes a large number of jobs simultaneously, and that dynamically allocates computing resources to jobs by solving a constrained combinatorial optimization problem, can perform the above series of processes.

[0266] (Hardware configuration) Fig. 24 is a diagram showing an example of the hardware configuration of the host unit 24. The host unit 24 is realized by, for example, a computer having the hardware configuration shown in Fig. 24. The host unit 24 includes a CPU (Central Processing Unit) 301, a RAM (Random Access Memory) 302, a ROM (Read Only Memory) 303, a storage device 304, and a communication interface device 305. These units are connected by a bus.

[0267] The CPU 301 is a processor that executes arithmetic processing, control processing, etc. in accordance with a program. The CPU 301 uses a predetermined area of ​​the RAM 302 as a work area and executes various processes in cooperation with programs stored in the ROM 303, the storage device 304, etc.

[0268] The RAM 302 is a memory such as an SDRAM (Synchronous Dynamic Random Access Memory), and functions as a work area for the CPU 301. The ROM 303 is a memory that stores programs and various types of information in a non-rewritable manner.

[0269] The storage device 304 is a device that writes and reads data to a semiconductor storage medium such as a flash memory, or a magnetically or optically recordable storage medium, etc. The storage device 304 writes and reads data to the storage medium in response to control from the CPU 301. The communication interface device 305 communicates with external devices via a network in response to control from the CPU 301.

[0270] A program executed by the computer causes the computer to function as the host unit 24. This program is loaded onto the RAM 302 by the CPU 301 (processor) and executed.

[0271] In addition, the program to be executed by a computer is provided as a file in a format that can be installed on a computer or in a format that can be executed by a computer, and is recorded on a computer-readable recording medium such as a CD-ROM, a flexible disk, a CD-R, or a DVD (Digital Versatile Disk).

[0272] This program may also be configured to be stored on a computer connected to a network such as the Internet and provided by being downloaded via the network. This program may also be configured to be provided or distributed via a network such as the Internet. The program executed by the host unit 24 may also be configured to be provided by being pre-installed in the ROM 303 or the like.

[0273] The solver unit 22 is realized by a reconfigurable semiconductor device such as a field-programmable gate array (FPGA). The solver unit 22 may be realized by a CPU, a microprocessor, a graphics processing unit (GPU), an application specific integrated circuit (ASIC), or an electronic circuit including these circuits. The solver unit 22 may be realized by an information processing device such as a computer, a computer system configured by multiple computers or servers communicating with each other via a network, or a PC cluster in which multiple computers cooperate to perform information processing.

[0274] Furthermore, when the solver unit 22 is realized by a reconfigurable semiconductor device such as an FPGA, the circuit information (configuration data) to be written into the reconfigurable semiconductor device to operate the reconfigurable semiconductor device as the solver unit 22 may be stored on a computer connected to a network such as the Internet and provided by being downloaded via the network. Furthermore, the circuit information (configuration data) to be written into the reconfigurable semiconductor device to operate the reconfigurable semiconductor device as the solver unit 22 may be provided by being recorded on a computer-readable recording medium.

[0275] Furthermore, when the solver unit 22 is realized by a semiconductor device such as an ASIC, the circuit information representing the configuration of the circuit described in a hardware description language may be stored on a computer connected to a network such as the Internet and provided by being downloaded via the network in order to operate the semiconductor device such as an ASIC as the solver unit 22. Furthermore, in order to operate the semiconductor device such as an ASIC as the solver unit 22, the circuit information representing the configuration of the circuit described in a hardware description language may be provided by being recorded on a computer-readable recording medium.

[0276] The solver unit 22 may be incorporated into the host unit 24. For example, the solver unit 22 may be incorporated as an accelerator into part of the host unit 24. The solver unit 22 may also include some of the functions of the host unit 24, such as the formulation unit 36 ​​or the arbitration unit 38.

[0277] Although several embodiments of the present invention have been described, these embodiments are presented as examples and are not intended to limit the scope of the invention. These novel embodiments can be embodied in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their modifications are included within the scope and spirit of the invention, and are also included in the scope of the invention and its equivalents as defined in the claims.

[0278] (Addendum) The above-described embodiments can be summarized as the following technical proposals.

[0279] [Technical proposal 1] An information processing device that executes processing on data, generating a combinatorial optimization problem based on the data under predetermined constraints; solving the generated combinatorial optimization problem while allowing the constraints not to be satisfied, and calculating a solution to the combinatorial optimization problem obtained by solving it; generating a constraint violation control signal if the solution does not satisfy the constraint; performing a predetermined first process on the data based on the constraint violation control signal; outputting the data on which the first process has been performed; Information processing device.

[0280] [Technical proposal 2] In generating the combinatorial optimization problem, the combinatorial optimization problem is generated to minimize a total cost function that is the sum of a cost function and a penalty function; the cost function includes, as arguments, a plurality of decision variables corresponding to the data, and is a first-order or higher-order function of the plurality of decision variables; The penalty function is a function that includes at least some of the plurality of decision variables and that takes a minimum value when the solution satisfies the constraints. An information processing device according to Technical Proposal 1.

[0281] [Technical proposal 3] In generating the constraint violation control signal, determining whether the solution to the combinatorial optimization problem satisfies the constraints; If the solution does not satisfy the constraint, output the constraint violation control signal. An information processing device according to Technical Proposal 2.

[0282] [Technical proposal 4] the constraints are expressed by one or more constraint equations, each of which is an equality or an inequality; In generating the constraint violation control signal, determining whether the solution satisfies the one or more constraints; If the solution does not satisfy the one or more constraints, output the constraint violation control signal. An information processing device according to Technical Proposal 2.

[0283] [Technical proposal 5] the constraints are expressed by one or more constraint equations, each of which is an equality or an inequality; In generating the constraint violation control signal, if the solution does not satisfy the one or more constraint equations and the number of unsatisfied constraint equations among the one or more constraint equations is equal to or less than a preset value, the constraint violation control signal is output. An information processing device according to Technical Proposal 2.

[0284] [Technical proposal 6] If the value obtained by substituting the solution into the penalty function is greater than the minimum value and equal to or less than a preset value, the constraint violation control signal is output. An information processing device according to Technical Proposal 2.

[0285] [Technical proposal 7] moreover, If the solution satisfies the constraints, performing a second predetermined process on the data based on the solution; outputting the data on which the second process has been performed; An information processing device according to any one of technical proposals 2 to 6.

[0286] [Technical proposal 8] If the solution does not satisfy the constraints, Identifying a constraint-violating portion of the solution that does not satisfy the constraint conditions and a constraint-satisfying portion of the solution that satisfies the constraint conditions; outputting the constraint violation control signal based on the constraint violation portion and the constraint satisfaction control signal based on the constraint satisfaction portion; performing the second operation on the data based on the constraint satisfaction control signal; executes the first processing on data corresponding to the constraint violation portion in the data based on the constraint violation control signal; An information processing device according to Technical Proposal 7.

[0287] [Technical proposal 9] In identifying the constraint violating portion and the constraint satisfying portion, identifying a decision variable among the plurality of decision variables that causes the constraint condition not to be satisfied based on the obtained solution, and changing the value of the decision variable that causes the constraint condition not to be satisfied to identify the constraint satisfying portion; Extracting the values ​​of the decision variables that cause the constraint conditions not to be satisfied and identifying the constraint violations An information processing device according to Technical Proposal 8.

[0288] [Technical proposal 10] generating a first combinatorial optimization problem and a second combinatorial optimization problem; The first combinatorial optimization problem has a larger ratio of the penalty function to the cost function than the second combinatorial optimization problem. In the solving of the solution, the solution is solved for each of the first combinatorial optimization problem and the second combinatorial optimization problem; In identifying the constraint violating portion and the constraint satisfying portion, identifying the solution of the first combinatorial optimization problem as the constraint satisfying portion; Identifying the constraint violation portion based on a difference between the solution of the first combinatorial optimization problem and the solution of the second combinatorial optimization problem. An information processing device according to Technical Proposal 8 or 9.

[0289] [Technical proposal 11] In generating the combinatorial optimization problem, a ratio of the penalty function to the cost function is changed according to the data on which the combinatorial optimization problem is based. An information processing device according to any one of technical proposals 2 to 10.

[0290] [Technical proposal 12] The information processing device is a device that tracks an object included in image data based on the image data, further storing one or more pieces of tracking object information representing one or more tracking objects being tracked prior to the first time; further acquiring one or more pieces of detected object information at the first time, the detected object information representing one or more detected objects included in the image data at the first time; generating the combinatorial optimization problem, under the constraint condition, to associate each of the one or more pieces of tracking object information at a first time with any of the one or more pieces of detected object information at the first time that is determined to represent the same object; In the execution of the first process, the elapsed time since the addition or modification of the tracking object information specified based on the constraint violation control signal among the one or more pieces of tracking object information at the first time is decreased by a predetermined value, or a threshold value for deletion when the elapsed time of the tracking object information specified based on the constraint violation control signal is exceeded is increased by a predetermined value; In the output of the data, the one or more pieces of tracking object information at the first time are output. An information processing device according to any one of technical proposals 1 to 11.

[0291] [Technical proposal 13] In generating the constraint violation control signal, a constraint satisfaction portion of the solution that satisfies the constraint conditions and a constraint violation portion of the solution that does not satisfy the constraint conditions are identified, and a constraint satisfaction control signal indicating the constraint satisfaction portion and a constraint violation control signal indicating the constraint violation portion are generated; Furthermore, based on the constraint satisfaction control signal, the tracking object information associated with any of the one or more pieces of detected object information among the one or more pieces of tracking object information at the first time is corrected using the associated detected object information; Furthermore, based on the constraint satisfaction control signal, add detected object information that is not associated with any of the one or more tracking object information among the one or more detected object information at the first time as new tracking object information to the one or more tracking object information at the first time; Furthermore, based on the constraint satisfaction control signal, delete tracking object information of which the elapsed time exceeds the threshold from among the one or more tracking object information at the first time. An information processing device according to Technical Proposal 12.

[0292] [Technical proposal 14] An information processing method for executing processing on data by an information processing device, comprising: The information processing device, generating a combinatorial optimization problem based on the data under predetermined constraints; solving the generated combinatorial optimization problem while allowing the constraints not to be satisfied, and calculating a solution to the combinatorial optimization problem obtained by solving it; generating a constraint violation control signal if the solution does not satisfy the constraint; performing a predetermined first process on the data based on the constraint violation control signal; outputting the data on which the first process has been performed; Information processing methods.

[0293] [Technical proposal 15] A program for causing an information processing device to function as an information processing device that executes processing on data, The information processing device generating a combinatorial optimization problem that optimizes the data under predetermined constraints; solving the generated combinatorial optimization problem while allowing the constraints not to be satisfied, and calculating a solution to the combinatorial optimization problem obtained by solving it; generating a constraint violation control signal if the solution does not satisfy the constraint; performing a predetermined first process on the data based on the constraint violation control signal; outputting the data on which the first process has been performed; A program to make it work like this. [Explanation of symbols]

[0294] 10. Object Tracking System 12 Camera 14 Object Tracking Device 16 Planning Device 18 Control Device 22 Solver section 24 Host Club 26 Object detection unit 28 Memory section 30 Object Tracking Unit 32 Acquisition Department 34 Prediction Department 36 Formulation part 38 Mediation Department 40 Correction section 42 Additional Section 44 Life extension section 46 Deleted section 48 Output section 50 Exception handling section 52 Ising machine 54 Pretreatment section 56 Post-processing section

Claims

1. An information processing device that executes processing on data, storing one or more pieces of first information at a first time; acquiring one or more pieces of second information at the first time; generating a combinatorial optimization problem that associates, under a predetermined constraint condition, each of the one or more pieces of first information at the first time with any one of the one or more pieces of second information at the first time; solving the generated combinatorial optimization problem while allowing the constraints not to be satisfied, and calculating a solution to the combinatorial optimization problem obtained by solving it; generating a constraint violation control signal based on a constraint violating portion of the solution that does not satisfy the constraint condition, and a constraint satisfaction control signal based on a constraint satisfying portion of the solution that satisfies the constraint condition; executing a predetermined first process on first information identified based on the constraint violation control signal from among the one or more pieces of first information stored at the first time; executing a predetermined second process different from the first process on first information identified based on the constraint satisfaction control signal among the one or more pieces of first information stored at the first time; outputting the one or more pieces of first information stored at the first time; Information processing device.

2. Storing the one or more pieces of first information before the first time; predicting the one or more pieces of first information at the first time based on the one or more pieces of first information at a time before the first time; The one or more pieces of first information stored before the first time are rewritten with the one or more pieces of first information at the predicted first time, thereby storing the one or more pieces of first information at the first time. The information processing device according to claim 1 .

3. In generating the combinatorial optimization problem, the combinatorial optimization problem is generated to minimize a total cost function that is the sum of a cost function and a penalty function; the cost function includes a plurality of decision variables corresponding to the data as arguments, and is a first or higher order function of the plurality of decision variables; The penalty function is a function that includes at least some of the plurality of decision variables and that takes a minimum value when the solution satisfies the constraints. The information processing device according to claim 1 .

4. In generating the constraint violation control signal, determining whether the solution to the combinatorial optimization problem satisfies the constraints; If the solution does not satisfy the constraint, output the constraint violation control signal. The information processing device according to claim 1 .

5. the constraints are expressed by one or more constraint equations, each of which is an equality or an inequality; In generating the constraint violation control signal, determining whether the solution satisfies the one or more constraints; If the solution does not satisfy the one or more constraints, output the constraint violation control signal. The information processing device according to claim 1 .

6. the constraints are expressed by one or more constraint equations, each of which is an equality or an inequality; In generating the constraint violation control signal, if the solution does not satisfy the one or more constraint equations and the number of unsatisfied constraint equations among the one or more constraint equations is equal to or less than a preset value, the constraint violation control signal is output. The information processing device according to claim 1 .

7. If the value obtained by substituting the solution into the penalty function is greater than the minimum value and equal to or less than a preset value, the constraint violation control signal is output. The information processing device according to claim 3 .

8. In identifying the constraint violating portion and the constraint satisfying portion, identifying a decision variable among the plurality of decision variables that causes the constraint condition not to be satisfied based on the obtained solution, and changing the value of the decision variable that causes the constraint condition not to be satisfied to identify the constraint satisfying portion; Extracting the values ​​of the decision variables that cause the constraint conditions not to be satisfied and identifying the constraint violations The information processing device according to claim 3 .

9. generating a first combinatorial optimization problem and a second combinatorial optimization problem; The first combinatorial optimization problem has a larger ratio of the penalty function to the cost function than the second combinatorial optimization problem. In the solving of the solution, the solution is solved for each of the first combinatorial optimization problem and the second combinatorial optimization problem; In identifying the constraint violating portion and the constraint satisfying portion, identifying the solution to the first combinatorial optimization problem as the constraint satisfying portion; Identifying the constraint violation portion based on a difference between the solution of the first combinatorial optimization problem and the solution of the second combinatorial optimization problem. The information processing device according to claim 3 .

10. In generating the combinatorial optimization problem, a ratio of the penalty function to the cost function is changed according to the data on which the combinatorial optimization problem is based. The information processing device according to claim 3 .

11. The information processing device is a device that tracks an object included in image data based on the image data, storing one or more pieces of tracking object information representing one or more tracking objects being tracked before the first time as the one or more pieces of first information before the first time; moreover, predicting the one or more pieces of tracking object information at the first time based on the one or more pieces of tracking object information prior to the first time; rewriting the one or more pieces of tracking object information stored before the first time with the one or more pieces of tracking object information predicted at the first time; acquiring, as the one or more pieces of second information at the first time, one or more pieces of detected object information at the first time that represent one or more detected objects included in the image data at the first time; generating the combinatorial optimization problem, under the constraint condition, to associate each of the one or more pieces of tracked object information at the first time with any detected object information determined to represent the same object among the one or more pieces of detected object information at the first time; In the execution of the first process, among the one or more pieces of tracking object information stored at the first time, the elapsed time since the addition or modification of the tracking object information specified based on the constraint violation control signal is decreased by a predetermined value, or a threshold value for deletion when the elapsed time of the tracking object information specified based on the constraint violation control signal is exceeded is increased by a predetermined value; In the output of the data, the one or more pieces of tracking object information stored at the first time are output. The information processing device according to claim 2 .

12. Based on the constraint satisfaction control signal, among the one or more pieces of tracked object information stored at the first time, tracked object information associated with any of the one or more pieces of detected object information is modified using the associated detected object information; adding, based on the constraint satisfaction control signal, detected object information that is not associated with any of the one or more tracking object information among the one or more detected object information at the first time as new tracking object information to the one or more stored tracking object information at the first time; Based on the constraint satisfaction control signal, delete tracking object information whose elapsed time exceeds the threshold from among the one or more pieces of tracking object information stored at the first time. The information processing device according to claim 11.

13. An information processing method for executing processing on data by an information processing device, comprising: The information processing device, storing one or more pieces of first information at a first time; acquiring one or more pieces of second information at the first time; generating a combinatorial optimization problem that associates, under a predetermined constraint condition, each of the one or more pieces of first information at the first time with any one of the one or more pieces of second information at the first time; solving the generated combinatorial optimization problem while allowing the constraints not to be satisfied, and calculating a solution to the combinatorial optimization problem obtained by solving it; generating a constraint violation control signal based on a constraint violating portion of the solution that does not satisfy the constraint condition, and a constraint satisfaction control signal based on a constraint satisfying portion of the solution that satisfies the constraint condition; executing a predetermined first process on first information identified based on the constraint violation control signal from among the one or more pieces of first information stored at the first time; outputting the one or more pieces of first information stored at the first time; Information processing methods.

14. A program for causing an information processing device to function as an information processing device that executes processing on data, The information processing device storing one or more pieces of first information at a first time; acquiring one or more pieces of second information at the first time; generating a combinatorial optimization problem that associates, under a predetermined constraint condition, each of the one or more pieces of first information at the first time with any one of the one or more pieces of second information at the first time; solving the generated combinatorial optimization problem while allowing the constraints not to be satisfied, and calculating a solution to the combinatorial optimization problem obtained by solving it; generating a constraint violation control signal based on a constraint violating portion of the solution that does not satisfy the constraint condition, and a constraint satisfaction control signal based on a constraint satisfying portion of the solution that satisfies the constraint condition; executing a predetermined first process on first information identified based on the constraint violation control signal from among the one or more pieces of first information stored at the first time; executing a predetermined second process different from the first process on first information identified based on the constraint satisfaction control signal among the one or more pieces of first information stored at the first time; outputting the one or more pieces of first information stored at the first time; A program to make it work like this.

15. Circuit information describing a circuit configuration written in a hardware description language, causing the circuit to function as an information processing device that performs processing on data; The information processing device includes: storing one or more pieces of first information at a first time; acquiring one or more pieces of second information at the first time; generating a combinatorial optimization problem that associates, under a predetermined constraint condition, each of the one or more pieces of first information at the first time with any one of the one or more pieces of second information at the first time; solving the generated combinatorial optimization problem while allowing the constraints not to be satisfied, and calculating a solution to the combinatorial optimization problem obtained by solving it; generating a constraint violation control signal based on a constraint violating portion of the solution that does not satisfy the constraint condition, and a constraint satisfaction control signal based on a constraint satisfying portion of the solution that satisfies the constraint condition; executing a predetermined first process on first information identified based on the constraint violation control signal from among the one or more pieces of first information stored at the first time; executing a predetermined second process different from the first process on first information identified based on the constraint satisfaction control signal among the one or more pieces of first information stored at the first time; outputting the one or more pieces of first information stored at the first time; Circuit information.

16. Circuit information written into a reconfigurable semiconductor device in order to operate the reconfigurable semiconductor device, causing the reconfigurable semiconductor device to function as an information processing device that performs processing on data; The information processing device includes: storing one or more pieces of first information at a first time; acquiring one or more pieces of second information at the first time; generating a combinatorial optimization problem that associates, under a predetermined constraint condition, each of the one or more pieces of first information at the first time with any one of the one or more pieces of second information at the first time; solving the generated combinatorial optimization problem while allowing the constraints not to be satisfied, and calculating a solution to the combinatorial optimization problem obtained by solving it; generating a constraint violation control signal based on a constraint violating portion of the solution that does not satisfy the constraint condition, and a constraint satisfaction control signal based on a constraint satisfying portion of the solution that satisfies the constraint condition; executing a predetermined first process on first information identified based on the constraint violation control signal from among the one or more pieces of first information stored at the first time; executing a predetermined second process different from the first process on first information identified based on the constraint satisfaction control signal among the one or more pieces of first information stored at the first time; outputting the one or more pieces of first information stored at the first time; Circuit information.

Images