Information processing device, information processing method, program, and circuit information

Abstract

To improve performance or efficiency of an information processing function by using a constraint violation solution obtained by solving a combinatorial optimization problem.SOLUTION: An information processing system according to an embodiment performs processing on data. An information processing apparatus generates a combinatorial optimization problem based on the data under a predetermined constraint condition and solves the generated combinatorial optimization problem while allowing the constraint condition to be unsatisfied and calculates a solution of the combinatorial optimization problem which is obtained by solving. If the solution does not satisfy the constraint condition, the information processing apparatus generates a constraint violation control signal based on a constraint violation part and performs predetermined first processing on the data on the basis of the constraint violation control signal and outputs data subjected to the first processing.SELECTED DRAWING: Figure 8

Description

[Technical field]

[0001] The present invention relates to an information processing device, an information processing method, and a program. [Background technology]

[0002] In many application fields, such as control, finance, communication, logistics, and chemistry, system optimization is often mathematically reduced to a combinatorial optimization problem. Information processing systems have been proposed that realize various information processing functions, such as recognition, judgment, and planning, and improve the efficiency of such functions by solving combinatorial optimization problems.

[0003] A combinatorial optimization problem is a problem in which a cost function is defined with arguments being multiple decision variables (e.g., discrete variables) that represent the state of the system to be optimized, and the combination of values ​​of multiple decision variables that minimizes the defined cost function is solved. The state of the system represented by multiple decision variables is called a solution. In a combinatorial optimization problem, the number of states that can be taken as a solution increases exponentially as the number of decision variables increases. The increase in the number of states that can be taken as a solution is called combinatorial explosion. Combinatorial optimization, which involves selecting an optimal solution from all candidate solutions, is known to be a computationally difficult problem. Performing large-scale combinatorial optimization in a short period of time remains a challenging task.

[0004] A constrained combinatorial optimization problem is a problem in which certain constraints are imposed on the solution, and a solution that minimizes a cost function under the constraints is sought. A solution that satisfies the constraints is called a constraint satisfying solution. A solution that satisfies the constraints is sometimes called a feasible solution. A solution that does not satisfy the constraints is called a constraint violating solution. A solution that does not satisfy the constraints is sometimes called an infeasible solution.

[0005] Conventional information processing systems for solving constrained combinatorial optimization problems do not output solutions that violate the constraints as solutions, or discard solutions that violate the constraints even if they are found. However, even solutions that violate the constraints may contain solutions that are effective for improving the performance or efficiency of information processing functions.

[0006] Conventionally, there has been no known technology for efficiently outputting an effective solution for improving the performance or efficiency of an information processing function among constraint violation solutions, or for improving the performance or efficiency of an information processing function using a constraint violation solution. Therefore, in conventional technologies, even if a solution effective for improving the performance or efficiency of an information processing function is included in the constraint violation solutions, it has not been possible to improve the performance or efficiency of the information processing function. [Prior art documents] [Patent documents]

[0007] [Patent Document 1] JP 2019-145010 A [Patent Document 2] JP 2019-159566 A [Patent Document 3] Patent Publication No. 2021-043667 [Patent Document 4] JP 2021-043589 A [Patent Document 5] JP 2021-060864 A [Non-patent literature]

[0008] [Non-Patent Document 1] A Bewley, et at., “Simple online and realtime tracking”, IEEE ICIP(2016), pp. 3464-3468, 2016 [Non-Patent Document 2] Hayato Goto, Kosuke Tatsumura and Alexander R. Dixon, “Combinatorial optimization by simulating adiabatic bifurcations in nonlinear Hamiltonian systems,” Science Advances 5, eaav2372, 2019 [Non-Patent Document 3] Hayato Goto, Kotaro Endo, Masaru Suzuki, Yoshisato Sakai, Taro Kanao, Yohei Hamakawa, Ryo Hidaka, Masaya Yamasaki and Kosuke Tatsumura, “High-performance combinatorial optimization based on classical mechanics”, Science Advances 7, eabe7953, 2021 [Non-Patent Document 4] Joseph Redmon, Santosh Divvala, Ross Girshick and Ali Farhadi, “You Only Look Once: Unified, Real-Time Object Detection”, Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016, pp. 779-788 [Non-Patent Document 5] Wei Liu, et al., “SSD: Single Shot MultiBox Detector”, Springer International Publishing AG 2016, ECCV 2016,Part I, LNCS vol 9905, pp. 21-37, 2016 Summary of the Invention [Problem to be solved by the invention]

[0009] The problem to be solved by the present invention is to improve the performance or efficiency of information processing functions by using a constraint violation solution obtained from a combinatorial optimization problem. [Means for solving the problem]

[0010] An information processing system according to an embodiment executes processing on data. The information processing device generates a combinatorial optimization problem based on the data under predetermined constraint conditions, solves the generated combinatorial optimization problem while allowing the constraint conditions not to be satisfied, calculates a solution to the combinatorial optimization problem obtained by solving it, generates a constraint violation control signal based on a constraint violating portion if the solution does not satisfy the constraint conditions, executes a predetermined first processing on the data based on the constraint violation control signal, and outputs the data on which the first processing has been executed. [Brief description of the drawings]

[0011] [Figure 1] A diagram showing a model of the Ising problem. [Diagram 2] FIG. 2 illustrates internal variables used by the simulated branching algorithm. [Diagram 3] 4 is a flowchart showing a process flow of a simulated branching machine. [Figure 4] FIG. 1 is a diagram showing a configuration of an information processing system including an Ising machine. [Diagram 5] FIG. 1 is a diagram showing a configuration of an information processing system including a simulated branching machine. [Figure 6] FIG. 1 is a diagram showing a configuration of an object tracking system. [Figure 7] FIG. 1 is a diagram showing a configuration of an object tracking device. [Figure 8] FIG. 2 is a diagram showing the configuration of an object tracking unit together with a solver unit and a storage unit. [Figure 9] FIG. 1 is a diagram showing an example of a plurality of decision variables included in a combinatorial optimization problem. [Figure 10] FIG. 13 is a diagram showing multiple coefficients included in a combinatorial optimization problem. [Figure 11] FIG. 1 shows a set of solutions for a cost function and a set of solutions that satisfy the constraints. [Figure 12] 4 is a flowchart showing a processing flow of the object tracking device. [Figure 13] FIG. 2 shows a data flow graph of the object tracking device. [Figure 14]13 is a flowchart showing a process flow of an arbitration unit. [Figure 15] A diagram showing the bounding box and solution for the (k-1)th frame. [Figure 16] A diagram showing the bounding box and solution for the kth frame. [Figure 17] A diagram showing the bounding box and solution for the (k+1)th frame. [Figure 18] FIG. 13 shows constraint-satisfying portions identified in the k-th frame. [Figure 19] FIG. 11 is a diagram showing a process of a formulation unit according to a first modified example. [Figure 20] FIG. 11 is a diagram showing a process of an arbitration unit according to a second modified example. [Figure 21] A diagram showing the solution to the first combinatorial optimization problem. [Figure 22] A diagram showing the solution to the second combinatorial optimization problem. [Figure 23] FIG. 23 shows the boundary box for which the solutions of FIGS. 21 and 22 are obtained. [Figure 24] FIG. 2 is a diagram showing the hardware configuration of a host unit. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] (premise) First, terms and techniques that are the basis for the description of the embodiments will be described.

[0013] A combinatorial optimization problem is a problem in which a cost function is defined with multiple decision variables (e.g., discrete variables) representing the state of a system to be optimized as arguments, and a combination of the values ​​of the multiple decision variables that minimizes the defined cost function is sought. The cost function includes multiple decision variables representing the state of the system as arguments, and is a linear or higher function of the multiple decision variables. For example, the cost function is expressed by a linear or quadratic function of the multiple decision variables. The cost function may be a cubic or higher function of the multiple decision variables. In other words, the cost function is a function that sums up multiple terms. Each of the multiple terms is a function that multiplies one or more, but not more than a predetermined number of, the multiple decision variables by a coefficient.

[0014] The state of the system, represented by the decision variables, is called a solution. The universe of possible solutions for the state of the system is called the solution space. The solution that gives the minimum of the cost function is called an exact solution. The solution that gives a value close to the minimum of the cost function is called a good solution.

[0015] An exact solution method is a method for finding an exact solution that gives a minimum value of a cost function for a combinatorial optimization problem, and is a method that guarantees that the solution is an exact solution.

[0016] A heuristic solution is a solution method for combinatorial optimization problems that finds an exact solution that minimizes the cost function or a good solution that is close to the minimum of the cost function. Heuristic solutions are also called discovery solutions. Heuristic solutions do not guarantee the accuracy of the solution, that is, an index that indicates how close the solution is to the minimum of the cost function. Heuristic solutions can output a solution with practically significant accuracy in a shorter solution time than exact solutions.

[0017] The computational complexity of a solution to a combinatorial optimization problem is the amount of calculations required to obtain a solution, such as the number of multiply-and-accumulate operations. The solution time required to obtain a solution to a combinatorial optimization problem depends not only on the computational complexity but also on the configuration of the computer that processes the calculations. For example, the solution time becomes shorter as the computer's computational parallelism and operating frequency increase.

[0018] A QUBO (Quadratic Unconstrained Binary Optimization) problem is an unconstrained quadratic optimization problem in which the decision variables are binary. In a QUBO problem, each of multiple terms included in a cost function is expressed by a quadratic or linear expression of the decision variables. In a narrowly defined QUBO problem, the decision variables are binary variables of 0 or 1. In this embodiment, the QUBO problem represents a narrowly defined problem in which the decision variables are 0 or 1. In a narrowly defined QUBO problem, the decision variables are sometimes called bit variables.

[0019] The cost function for the QUBO problem is H in Eq. (1). total_QUBO It is expressed by:

number

[0020] FIG. 1 is a diagram showing a model of the Ising problem. The Ising problem is a problem of searching for the ground state of the Ising model. The Ising problem is one of the QUBO problems. The decision variables of the Ising problem represent discrete variables of -1 or +1.

[0021] The cost function for the Ising problem is H in Eq. (2). total_Ising It is expressed by:

number

[0022] If the decision variable is s i The QUBO problem corresponds to the problem of searching for the ground state of the Ising model, which is one of the magnetic models in statistical mechanics. i is sometimes called the spin variable. The number of decision variables, N, is sometimes called the spin number. H total_Ising is sometimes called the Zing energy. total_Ising N s for which i The vector represented by is sometimes called the ground state (ground spin configuration).

[0023] The cost function of the QUBO problem and the cost function of the Ising problem differ only in the value of the constant. Therefore, the QUBO problem and the Ising problem are the same as combinatorial optimization problems. In other words, the QUBO problem and the Ising problem can be converted into each other. For example, the Ising problem and the QUBO problem are converted into each other by formula (3-1), formula (3-2), formula (3-3), and formula (3-4).

number

[0024] QUBO and Ising problems are known to be NP-complete. That is, many NP-hard problems can be transformed into QUBO or Ising problems in polynomial time. Therefore, many practical combinatorial optimization problems can be transformed into QUBO or Ising problems.

[0025] An Ising machine is a device that solves Ising problems. Most Ising machines solve Ising problems using heuristic solutions. Ising machines based on various principles, such as electronic engineering, optics, quantum mechanics, and statistical mechanics, have been proposed. Many Ising machines can output exact or good solutions in a short time.

[0026] The simulated branching algorithm is an algorithm for solving combinatorial optimization problems. The simulated branching algorithm is a heuristic solution algorithm.

[0027] The simulated bifurcation algorithm is shown in, for example, Non-Patent Document 2, Non-Patent Document 3, Patent Document 1, Patent Document 2, Patent Document 3, Patent Document 4, and Patent Document 5. The simulated bifurcation algorithm is also called a quantum-inspired algorithm because it was discovered inspired by a quantum mechanical optimization method based on the quantum adiabatic theorem. The simulated bifurcation algorithm can solve a combinatorial optimization problem in which the cost function is a quadratic function of multiple decision variables. The simulated bifurcation algorithm can also solve a combinatorial optimization problem in which the cost function is a cubic or higher function of multiple decision variables, that is, a HUBO (Higher Order Binary Optimization) problem. For example, a simulated bifurcation algorithm that solves a HUBO problem is shown in Patent Document 3. The simulated bifurcation algorithm can also solve a combinatorial optimization problem in which some or all of the multiple decision variables include variables with continuous values. A simulated bifurcation algorithm that solves a combinatorial optimization problem in which some or all of the multiple decision variables include variables with continuous values ​​is shown in Patent Document 4.

[0028] The simulated bifurcation machine is a computing device that executes processing according to a simulated bifurcation algorithm. The simulated bifurcation machine that solves the QUBO problem or the Ising problem is an example of an Ising machine. In this embodiment, the simulated bifurcation machine solves the Ising problem.

[0029] FIG. 2 is a diagram showing the internal variables used by the simulated branching algorithm.

[0030] The simulated branching algorithm is a method to determine the number of decision variables (s1 to s N When solving a combinatorial optimization problem in which the cost function is expressed usingN ) and N momentum variables (y1~y N ) is used. That is, the simulated bifurcation algorithm uses 2×N internal variables.

[0031] N location variables (x i ) is the number of N decision variables (s i ) in one-to-one correspondence. That is, the i-th position variable (x i ) is the i-th decision variable (s i ) corresponding to the N momentum variables (y i ) is the number of N decision variables (s i ) corresponds to the i-th momentum variable (y i ) is the i-th decision variable (s i )

[0032] 3 is a flow chart showing the flow of processing of the simulated branching machine. The simulated branching machine executes processing according to the simulated branching algorithm in the flow shown in FIG.

[0033] First, in S11, the simulated bifurcation machine obtains an Ising problem. Specifically, the simulated bifurcation machine obtains J, which is a matrix including N×N coefficients, and h, which includes N bias coefficients.

[0034] Next, in S12, the simulated branching machine calculates 2×N internal variables, i.e., N position variables (x1 to x N ) and N momentum variables (y1~y N ) is initialized. The simulated branching machine initializes N position variables (x1 to x N ) and the initial values ​​of N momentum variables (y1~y N The initial values ​​of N position variables (x1 to x N ) and the initial values ​​of N momentum variables (y1~y NThe initial value of x1 to x2 may be generated by a random number generator or may be set to a predetermined value. Since the simulated branching machine is a heuristic, the initial value of x1 to x2 may be set to a predetermined value even if the same problem is being solved. N ) and the initial values ​​of N momentum variables (y1~y N ) is different, a different good solution may be output.

[0035] Next, the simulated branching machine repeats the process from S14 to S16 a preset number of times (loop process between S13 and S17). The process from S14 to S16 is performed by iterating N position variables (x1 to x N ) and N momentum variables (y1~y N ) is a time evolution process.

[0036] In S14, the simulated bifurcation machine calculates N momentum variables (y1 to y N ) in the y update process. The simulated bifurcation machine updates the i-th momentum variable (y i In the update process of N position variables (x1~x N ) and the i-th position variable (x i ) and the other (N-1) location variables (x 1~i-1, x i+1~N ) and N coefficients (J i,j ) and the i-th bias coefficient (h i ) and the i-th momentum variable (y i ) to update the

[0037] Next, in S15, the simulated branching machine calculates N position variables (x1 to x N ) in the x update process. The simulated branch machine updates the i-th position variable (x i In the update process of the i-th momentum variable (y i ) to determine the i-th position variable (x i ) to update the

[0038] The simulated branching machine may execute the process of S14 and the process of S15 in an interchangeable order.

[0039] Next, in S16, the simulated branching machine calculates N position variables (x1 to x N ), the simulated branching machine performs wall processing on the position variables whose absolute values ​​exceed 1. Furthermore, the simulated branching machine also performs wall processing on the momentum variables corresponding to the position variables whose absolute values ​​exceed 1. For example, in wall processing, the simulated branching machine changes the absolute value of the position variable whose absolute value exceeds 1 to 1 or a value smaller than 1 while keeping the sign the same. Also, for example, in wall processing, the simulated branching machine changes the value of the momentum variable corresponding to the position variable whose absolute value exceeds 1 to 0.

[0040] The simulated bifurcation algorithm has variations in the calculation of x-update processing, y-update processing, and wall processing. For example, the simulated bifurcation algorithm has variations such as the adiabatic simulated bifurcation (aSB) algorithm, the ballistic simulated bifurcation (bSB) algorithm, and the discrete simulated bifurcation (dSB) algorithm.

[0041] When executing processing according to the adiabatic simulated branching algorithm, the simulated branching machine executes the calculation shown in formula (4-1) in the y update processing (S14), and executes the calculation shown in formula (4-2) in the x update processing (S15). Note that when executing processing according to the adiabatic simulated branching algorithm, the simulated branching machine does not execute the wall processing (S15).

number

[0042] When executing processing according to the ballistic simulated branching algorithm, the simulated branching machine performs the calculation shown in equation (5-1) in the y update processing (S14), the calculation shown in equation (5-2) in the x update processing (S15), and the calculation shown in equation (5-3) in the wall processing (S16).

number

[0043] When executing processing according to the discrete simulated branching algorithm, the simulated branching machine performs the calculation shown in equation (6-1) in the y update processing (S14), the calculation shown in equation (6-2) in the x update processing (S15), and the calculation shown in equation (6-3) in the wall processing (S16).

number

[0044] In addition, in formulas (4-1), (4-2), (5-1), (5-2), (5-3), (6-1), (6-2) and (6-3), t k and t k+1 represents the time. k+1 t k This is the time obtained by adding a unit time (Δt) to the current time.

[0045] x i (t k ) is the time (t k ) the i-th position variable (x i ) value. i (t k+1 ) is the time (t k+1 ) the i-th position variable (x i ) value. i (t k ) is the time (t k ) the i-th momentum variable (y i ) value. i (t k+1 ) is the time (t k+1) the i-th momentum variable (y i ) value.

[0046] K, a0, η, and c0 are predetermined constants. k ) is a function that changes with time. k ) is a positive real number that increases as time increases, for example, a(t1)=0, and is a function that becomes a0 at the end time (T) (a(T)=a0). Also, sgn(x i (t k )) is the time (t k ) the i-th position variable (x i ), which outputs the sign of x i (t k ) is 0 or more, then +1, x i (t k ) is less than 0, it becomes -1.

[0047] When the simulated branching machine has executed the processes of S14 to S16 a predetermined number of times, that is, when it has executed the operations until time t reaches the final time T, it exits the loop process between S13 and S17 and proceeds to S18.

[0048] In S18, the simulated branching machine calculates the N position variables (x1 to x N ), or N position variables (x1~x N ) are calculated based on N decision variables (s1 to s N ) The simulated branching machine outputs N decision variables (s1 to s N ) the i-th decision variable (s i ) to sgn(x i ) is used for the calculation.

[0049] When the simulated branching machine finishes the process of S18, it ends the process according to the simulated branching algorithm.

[0050] The number of times the time-evolution process (loop process between S13 and S17) is repeated is determined in advance depending on the application. The amount of calculation required for one process in the time-evolution process (one process from S14 to S16) does not vary. For this reason, the simulated branching machine can reduce fluctuations in the solution-finding time. Therefore, even if the simulated branching machine is applied to a real-time system with time constraints that require processing to be completed by a certain time, it can reliably output a solution by the certain time.

[0051] Also, as shown in Patent Document 2, for example, the simulated branching machine can be configured using a dedicated parallel processing circuit including a large number of arithmetic units. This allows the simulated branching machine to extremely shorten the calculation time for one processing in time-evolution processing. Also, unlike software processing, the simulated branching machine implemented in a dedicated hardware circuit does not generate any interrupt processing, so the solution-finding time is strictly fixed. For example, the simulated branching machine implemented in a dedicated hardware circuit can fix the time until a solution is obtained in clock cycle units. Therefore, when applied to a real-time system, the simulated branching machine implemented in a dedicated hardware circuit can output a solution while more reliably observing time constraints.

[0052] FIG. 4 is a diagram showing a configuration of an information processing system including an Ising machine and a host device. An information processing system having a function of solving a combinatorial optimization problem can be configured, for example, by an Ising machine and a host device. The host device executes processes other than the processes executed by the Ising machine. In this case, the Ising machine is introduced for the purpose of shortening the time required to solve the combinatorial optimization problem, and is considered to be an accelerator or offloader. The host device includes a general-purpose processor, an offloader other than the Ising machine, a memory, a storage, a sensor, an actuator, a communication interface, and the like.

[0053] The host device provides at least information for identifying the QUBO problem (matrix (Q)) or information for identifying an Ising problem (matrix (J) and bias coefficient (h)) to the Ising machine. The Ising machine may obtain the information for identifying the QUBO problem (matrix (Q)) and convert it into information for identifying an Ising problem (matrix (J) and bias coefficient (h)). Then, after the optimization process, the Ising machine outputs N decision variables (s1 to s N ) opt ) is returned to the host device.

[0054] FIG. 5 is a diagram showing a configuration of an information processing system including a simulated branching machine and a host device.

[0055] The information processing system having a function of solving a combinatorial optimization problem may include a simulated bifurcation machine as an Ising machine. In this case, the host device provides information (matrix (J) and bias coefficient (h)) for identifying an Ising problem to the simulated bifurcation machine. Furthermore, the host device provides N position variables (x1 to x N ) and the initial values ​​of N momentum variables (y1~y N The host device may also provide initial values ​​for various constants and functions (e.g., K, a0, c 0、 a(t) and Δt) may be given to the simulated branching machine. After the optimization process, the simulated branching machine outputs N decision variables (s1 to s N ) opt ), we use N position variables (x1~x N ) opt ) may be sent back to the host device.

[0056] (Embodiment) Next, the object tracking system 10 according to the embodiment will be described (FIG. 6).

[0057] The object tracking system 10 is an example of a real-time system, which is an information processing system having a function of solving a constrained combinatorial optimization problem. In particular, the object tracking system 10 is an example of an information processing system that detects valid constraint violation solutions among solutions of a constrained combinatorial optimization problem and executes processing based on the valid constraint violation solutions.

[0058] A real-time system is a system that has time constraints, in that it must complete processing by a certain time. Most real-time systems do not only process accumulated information from the past, but recognize the current, ever-changing situation, determine a response action according to the recognized situation, and immediately execute the determined response action.

[0059] The problem of determining the optimal response action according to a recognized situation can sometimes be formulated as a combinatorial optimization problem. In a real-time system, a response action determined based on an exact or good solution to a combinatorial optimization problem can be considered a rational action. However, it is not easy to solve a combinatorial optimization problem in a short time. For this reason, conventional real-time systems often determine a response action based on a simple condition judgment.

[0060] FIG. 6 is a diagram showing a configuration of an object tracking system 10 according to an embodiment.

[0061] The object tracking system 10 includes a camera 12, an object tracker 14, a planning device 16, and a controller 18.

[0062] The camera 12 captures an image of a surrounding scene and generates image data that may include one or more captured objects. The camera 12 generates image data at regular intervals (for example, for each frame that is an image capture interval). The camera 12 is, for example, an optical imaging camera. The camera 12 may be, for example, a camera system including a monocular camera (mono camera) and a stereo camera used in an ADAS (advanced driver assistance system) or the like, or another sensor. The camera 12 supplies the image data generated at regular intervals to the object tracking device 14.

[0063] The object tracking device 14 acquires image data generated from the camera 12 at regular intervals. Then, the object tracking device 14 tracks an object included in the image data. Specifically, the object tracking device 14 stores one or more pieces of tracking object information representing one or more tracking objects to be tracked, and updates the one or more pieces of tracking object information representing one or more tracking objects to be tracked based on the acquired image data at regular intervals. Note that in the initial state or in a state where no object exists in the surroundings, the object tracking device 14 does not need to store tracking object information since there is no tracking object to be tracked. Each of the one or more pieces of tracking object information represents the position and size of the tracking object to be tracked. Each of the one or more pieces of tracking object information may include information representing the type of the tracking object, information representing the amount of change in the position and size over time, and age representing the elapsed time since the object was added or modified. The object tracking device 14 supplies one or more pieces of tracking object information to the planning device 16 at regular intervals.

[0064] The planning device 16 acquires one or more pieces of tracking object information from the object tracking device 14 at regular time intervals. Furthermore, the planning device 16 may further acquire sensor information from other sensors, such as LiDAR (Light detection and ranging), other than the camera 12. The planning device 16 generates behavior plan information representing a response behavior of the controlled object based on the acquired one or more pieces of tracking object information and sensor information. The planning device 16 supplies the generated behavior plan information to the control device 18.

[0065] The control device 18 acquires the action plan information from the planning device 16. The control device 18 controls the controlled object in accordance with the received action plan information so as to make the controlled object behave in accordance with the response action information.

[0066] Such an object tracking system 10 is applied, for example, to an automatic control mechanism that controls a moving object. An automatic control mechanism that controls a moving object generally periodically repeats a series of operations including sensing, situation recognition, action planning, and control. These series of operations are typically executed 10 to several tens of times per second. That is, these series of operations typically have a period of about 100 milliseconds or less than 100 milliseconds. The throughput of the object tracking system 10 is limited by the slowest module among the multiple modules that configure the system. Therefore, when applied to an automatic control mechanism that controls a moving object, each of the multiple modules that configure the object tracking system 10 needs to achieve a required throughput while processing data (e.g., stream data) that is sequentially input and output without stagnation. Note that the object tracking system 10 is not limited to an automatic control mechanism that controls a moving object, and may be used in a mechanism that controls other control objects.

[0067] Here, the object tracking device 14 (FIG. 7) executes a matching process, which is one of processes for solving a combinatorial optimization problem under constraint conditions. The object tracking device 14 reduces the matching process to an Ising problem and solves it using a combinatorial optimization solver. The combinatorial optimization solver used by the object tracking device 14 can output a constraint satisfaction solution and a constraint violation solution. Furthermore, the object tracking device 14 executes an arbitration process for evaluating and determining the solution output by the combinatorial optimization solver. In the arbitration process, when a valid control violation solution that is relatively close to the control satisfaction solution among the constraint violation solutions is obtained, the object tracking device 14 generates a constraint violation control signal based on the valid control violation solution. Then, the object tracking device 14 executes a predetermined constraint violation process for the tracked object information corresponding to the constraint violation part among one or more pieces of tracked object information in response to the constraint violation control signal.

[0068] The phenomenon or situation in which a tracked object is temporarily hidden from the camera 12 by another tracked object or an obstructing object other than the tracked object is called occlusion. The ability to continue tracking of a tracked object without interruption even when occlusion occurs is called occlusion response capability. The object tracking device 14 can improve its occlusion response capability by performing constraint violation processing using a valid constraint violation solution.

[0069] FIG. 7 is a diagram showing the configuration of the object tracking device 14. As shown in FIG.

[0070] The object tracking device 14 includes a solver portion 22 and a host portion 24 .

[0071] The solver unit 22 is a combinatorial optimization solver that acquires a combinatorial optimization problem and outputs a solution to the acquired combinatorial optimization problem. The solver unit 22 calculates a solution to the acquired combinatorial optimization problem by a heuristic solving method. Furthermore, the solver unit 22 outputs the solution within a certain time after acquiring the combinatorial optimization problem. In this embodiment, the solver unit 22 is realized using a simulated bifurcation machine, which is a hardware computing device that utilizes a simulated bifurcation algorithm.

[0072] The host unit 24 is realized by an information processing device including an arithmetic processing unit (general-purpose processor), a RAM (Random Access Memory), a ROM (Read Only Memory), a storage device, a communication interface, and the like, executing a program. Furthermore, the host unit 24 may include an accelerator for object detection. The host unit 24 executes processes other than the process of solving the combinatorial optimization problem among the processes executed in the object tracking device 14. Furthermore, the host unit 24 may be realized by an information processing device common to the planning device 16 and the control device 18.

[0073] The host unit 24 includes an object detection unit 26 , a storage unit 28 , and an object tracking unit 30 .

[0074] The object detection unit 26 acquires image data from the camera 12 at regular time intervals, for example, for each frame. The object detection unit 26 generates one or more pieces of detection object information representing one or more objects included in the image data at regular time intervals. Note that, in the processing at regular time intervals, the object detection unit 26 does not output detection object information at a time when the image data does not include a detection object. Each of the one or more pieces of detection object information includes the position and size of one detection target object included in the image data. In this embodiment, each of the one or more pieces of detection object information is boundary box information. The boundary box information includes the horizontal and vertical pixel positions of the center of the box in the field angle of the image data, the area, and the aspect ratio. Also, the boundary box information may include the horizontal and vertical pixel positions of the upper left and lower right of the box in the field angle of the image data instead of the center of the box. Note that each of the one or more pieces of detection object information includes identification information for distinguishing between multiple objects included in one frame. Furthermore, each of the one or more pieces of detected object information may also include type information of the detected object (for example, distinction between a vehicle and a person, or distinction between colors, etc.) and distance information (the distance from the camera 12 to the detected object).

[0075] For example, the object detection unit 26 generates one or more pieces of detection object information from image data using an existing method. In this embodiment, the object detection unit 26 generates one or more pieces of detection object information according to the YOLO technique described in Non-Patent Document 4. The object detection unit 26 may also generate one or more pieces of detection object information according to the SSD technique described in Non-Patent Document 5. A vehicle, a person, an aircraft, etc. may be set as an object to be detected.

[0076] The object detection unit 26 supplies one or more pieces of generated detection object information to the object tracking unit 30 at regular time intervals. For example, the object detection unit 26 d pieces(N d When an object (where n is an integer equal to or greater than 1) is detected, information (detections) shown in equation (7) is generated.

number

[0077] d i represents the i-th detected object information.

[0078] For example, the i-th detected object information (d i ,detection) is expressed as in equation (8).

number

[0079] u i v represents the horizontal pixel position of the center of the bounding box in the field of view. i represents the vertical pixel position of the center of the bounding box in the field of view. i represents the area of ​​the bounding box in the field of view. i represents the aspect ratio of the bounding box.

[0080] The storage unit 28 stores the internal state information. When there are one or more tracking objects to be tracked, the internal state information includes one or more tracking object information. For example, when the internal state information is N t pieces(N t When the tracking object information includes the internal state information (internal_state) of the target object, the internal state information (internal_state) is expressed as shown in Equation (9).

number

[0081] t i represents the i-th tracked object information, where T represents the transpose of a matrix.

[0082] The i-th tracking information (t i , tracker) is expressed as in equation (10).

number

[0083] u i, v i , s i and r i The symbol with a dot on top is u i , v i , s i and r i It represents the amount of change per unit time. i represents the age. The age is incremented when there is no correspondence with the current detected object, and represents the time elapsed since the tracking object information was added or modified. If the tracking object information's age exceeds a preset threshold (max_age), it is deleted from the internal state information.

[0084] The storage unit 28 continues to store one or more pieces of tracking object information included in the internal state information for a certain period of time, for example, a frame interval, unless the tracking object information is deleted from the internal state information because the age exceeds a threshold. Such a function in the storage unit 28 is called a state memory.

[0085] The object tracking unit 30 periodically acquires one or more pieces of detected object information generated by the object detection unit 26. The object tracking unit 30 periodically generates a combinatorial optimization problem for associating one or more pieces of tracked object information with any one of the pieces of detected object information that is determined to represent the same object. That is, the object tracking unit 30 periodically generates a combinatorial optimization problem for associating one or more pieces of tracked object information with any one of the pieces of detected object information that is likely to represent the same object. Such a combinatorial optimization problem is realized as one of the combinatorial optimization problems for optimizing a plurality of data under a predetermined constraint condition.

[0086] Next, the object tracking unit 30 provides the generated combinatorial optimization problem to the solver unit 22 and obtains a solution to the combinatorial optimization problem from the solver unit 22. Note that, at a time when the image data does not include a detected object and the detected object information is not obtained, the object tracking unit 30 does not generate a combinatorial optimization problem or obtain a solution, and generates information indicating that the detected object information has not been associated with any of one or more pieces of tracked object information. Also, at a time when the tracked object to be tracked does not exist and the tracked object information is not stored, the object tracking unit 30 does not generate a combinatorial optimization problem or obtain a solution, and generates information indicating that the tracked object information has not been associated with any of one or more pieces of detected object information. Next, the object tracking unit 30 updates one or more pieces of tracked object information included in the internal state information stored in the storage unit 28 based on the obtained solution and the generated information at regular intervals, according to the framework of the Kalman filter. Then, the object tracking unit 30 supplies the updated one or more pieces of tracked object information to the subsequent planning device 16.

[0087] FIG. 8 is a diagram showing the configuration of the object tracking unit 30 together with the solver unit 22 and the storage unit .

[0088] The object tracking unit 30 includes an acquisition unit 32, a prediction unit 34, a formulation unit 36, an arbitration unit 38, a correction unit 40, an addition unit 42, a life extension unit 44, a deletion unit 46, an output unit 48, and an exception handling unit 50.

[0089] The acquisition unit 32 acquires one or more pieces of detection information (detections) representing one or more detected objects included in the image data at regular time intervals from the object detection unit 26. For example, in processing image data at an arbitrary first time, the acquisition unit 32 acquires one or more pieces of detection information at a first time representing one or more detected objects included in the image data at the first time. Note that the acquisition unit 32 does not acquire detection information at a time when no detected object is included in the image data.

[0090] The prediction unit 34 predicts, at regular time intervals, one or more pieces of tracking object information representing one or more tracking objects at the current time, based on one or more pieces of tracking object information (trackers) representing one or more past tracking objects, which are included in the internal state information of the storage unit 28. For example, the prediction unit 34 predicts one or more pieces of tracking object information at the first time, based on one or more pieces of tracking object information representing one or more tracking objects prior to the first time stored in the storage unit 28.

[0091] For example, the prediction unit 34 predicts the horizontal pixel position, vertical pixel position and area included in the tracking object information, as well as the amount of change therein per unit time, using a constant velocity model, which is a predefined time development model. In this case, the prediction unit 34 may predict that the aspect ratio included in each of the one or more pieces of tracking object information will not change. Furthermore, the prediction unit 34 increases the age included in each of the one or more pieces of tracking object information by a predetermined value (for example, 1).

[0092] Then, the prediction unit 34 rewrites one or more pieces of past tracking object information included in the internal state information of the storage unit 28 with one or more pieces of current tracking object information. For example, the prediction unit 34 rewrites one or more pieces of tracking object information before the first time stored in the storage unit 28 with one or more pieces of tracking object information at the first time generated by prediction. Note that the prediction unit 34 does not make a prediction at a time when the internal state information does not include tracking object information.

[0093] The formulation unit 36 ​​acquires one or more pieces of detection information (detections) and one or more pieces of tracked object information (trackers) at regular time intervals. For example, the formulation unit 36 ​​generates a combinatorial optimization problem in which one or more pieces of tracked object information are associated with any of the one or more pieces of detection object information that are determined to represent the same object. For example, the formulation unit 36 ​​generates a combinatorial optimization problem in which one or more pieces of tracked object information at a first time are associated with any of the one or more pieces of detection object information at a first time that are determined to represent the same object.

[0094] More specifically, the formulation unit 36 ​​generates a bipartite graph maximum matching problem that associates one or more pieces of detected object information with one or more pieces of tracked object information in a one-to-one relationship. The bipartite graph maximum matching problem is a constrained combinatorial optimization problem. The bipartite graph maximum matching problem can be expressed as a combinatorial optimization problem that minimizes a total cost function that adds a cost function and a penalty function. The penalty function is a function that includes at least a portion of multiple decision variables and becomes a minimum value when the solution satisfies the constraint conditions. The total cost function of the bipartite graph maximum matching problem is expressed by a quadratic function. Note that more specific descriptions of the cost function, penalty function, constraint conditions, and total cost function will be described in detail later.

[0095] Then, the formulation unit 36 ​​supplies the generated combinatorial optimization problem to the solver unit 22 at regular time intervals. Note that, at a time when the image data does not contain a detected object and the detected object information has not been acquired, the formulation unit 36 ​​does not generate a combinatorial optimization problem, but generates information indicating that the detected object information has not been associated with any of one or a plurality of pieces of tracked object information, and provides this information to the arbitration unit 38. Also, at a time when the tracked object to be tracked does not exist and the tracked object information is not included in the internal state information, the formulation unit 36 ​​does not generate a combinatorial optimization problem, but generates information indicating that the tracked object information has not been associated with any of one or a plurality of pieces of detected object information, and provides this information to the arbitration unit 38.

[0096] The solver unit 22 acquires a combinatorial optimization problem from the formulation unit 36 ​​at regular time intervals. The solver unit 22 solves the acquired combinatorial optimization problem while allowing for constraint conditions not to be satisfied. For example, the solver unit 22 acquires a combinatorial optimization problem at a first time and solves the combinatorial optimization problem at the first time while allowing for constraint conditions not to be satisfied. Since the solver unit 22 solves the combinatorial optimization problem while allowing for constraint conditions not to be satisfied, the solver unit 22 may output a constraint satisfying solution that satisfies the constraint conditions, or may output a constraint violating solution that does not satisfy the constraint conditions.

[0097] Then, the solver unit 22 outputs the solution of the combinatorial optimization problem to the arbitration unit 38 within a predetermined time limit. For example, the solver unit 22 outputs the solution to the arbitration unit 38 within a preset time that is shorter than a certain time (for example, the time between two consecutive frames) after acquiring the combinatorial optimization problem at the first time.

[0098] In this embodiment, the solver unit 22 includes an Ising machine 52, a pre-processing unit 54, and a post-processing unit 56.

[0099] The Ising machine 52 acquires an Ising problem and outputs a solution to the Ising problem. In this embodiment, the Ising machine 52 is a simulated branching machine implemented in a hardware circuit. The simulated branching machine solves the Ising problem by a simulated branching algorithm, which is a heuristic solution. Therefore, the Ising machine 52 may output not only an exact solution but also a good solution, and may also output a solution that does not satisfy the constraint conditions.

[0100] Furthermore, the simulated bifurcation machine implemented in the hardware circuit can fix the time from when an Ising problem is obtained until a solution is obtained in units of clock cycles. Therefore, the solver unit 22 including such an Ising machine 52 can output a solution to a combinatorial optimization problem at regular intervals within a predetermined time limit.

[0101] The pre-processing unit 54 acquires the combinatorial optimization problem from the formulation unit 36. The pre-processing unit 54 converts the acquired combinatorial optimization problem into an Ising problem and supplies it to the Ising machine 52. This enables the solver unit 22 to reduce the combinatorial optimization problem generated by the formulation unit 36 ​​to an Ising problem and solve it. Note that the pre-processing unit 54 may be included in the formulation unit 36 ​​instead of being included in the solver unit 22.

[0102] The post-processing unit 56 obtains a solution to the Ising problem from the Ising machine 52. The post-processing unit 56 converts the solution to the Ising problem into a solution to the combinatorial optimization problem generated by the formulation unit 36, and supplies the solution to the arbitration unit 38. Note that the post-processing unit 56 may be included in the arbitration unit 38 instead of being included in the solver unit 22.

[0103] The arbitration unit 38 acquires a solution from the solver unit 22 at regular time intervals. The arbitration unit 38 judges whether the acquired solution satisfies the constraint conditions in the combinatorial optimization problem generated by the formulation unit 36. If the acquired solution does not satisfy the constraint conditions, the arbitration unit 38 further judges whether the acquired solution is a valid constraint violating solution that is close to a constraint satisfying solution, or whether the acquired solution is far from the constraint satisfying solution and is not a valid constraint violating solution, based on a predetermined evaluation criterion. An example of the evaluation criterion will be described in detail later.

[0104] In addition, when the acquired solution does not satisfy the constraint conditions and is a valid constraint violating solution, the arbitration unit 38 identifies a constraint satisfying portion of the acquired solution that satisfies the constraint conditions and a constraint violating portion of the solution that does not satisfy the constraint conditions.

[0105] For example, the arbitration unit 38 identifies the constraint satisfying portion by changing the values ​​of some decision variables that are the cause of the valid constraint violation solution not satisfying the constraint conditions. Also, for example, the arbitration unit 38 may identify the constraint satisfying portion by excluding the values ​​of some decision variables that are the cause of the valid constraint violation solution not satisfying the constraint conditions. The constraint satisfying portion includes multiple decision variables whose values ​​satisfy the constraint conditions. Also, for example, the arbitration unit 38 identifies the constraint violating portion by extracting the values ​​of some decision variables that are the cause of the valid constraint violation solution not satisfying the constraint conditions.

[0106] When the solution satisfies the constraint condition, the arbitration unit 38 generates a constraint satisfaction control signal based on the solution that satisfies the constraint condition. The constraint satisfaction control signal includes a match signal (matched), an unmatched detection signal (unmatched_detections), and an unmatched trackers signal (unmatched_trackers). The match signal (matched) is a signal indicating the associated detection information for each piece of tracker information to which the detection information of one or more pieces of tracker information is associated. The unmatched detection signal (unmatched_detections) is a signal indicating the detection information to which the tracking object information of one or more pieces of tracker information is not associated. The unmatched trackers signal (unmatched_trackers) is a signal indicating the tracking object information to which the detection information of one or more pieces of tracker information is not associated.

[0107] When the obtained solution does not satisfy the constraint conditions and is a valid constraint violating solution, the arbitration unit 38 generates a constraint satisfaction control signal based on the identified constraint satisfaction portion. The constraint satisfaction control signal based on the constraint satisfaction portion includes a match signal based on the constraint satisfaction portion, an unmatch detection object signal based on the constraint satisfaction portion, and an unmatch tracker signal based on the constraint satisfaction portion, similar to the constraint satisfaction control signal indicating a solution that satisfies the constraint conditions.

[0108] Furthermore, when the acquired solution does not satisfy the constraint condition and is a valid constraint-violating solution, the arbitration unit 38 generates a constraint violation control signal based on the constraint violation part. The constraint violation control signal includes a potential match signal (potentially_matched). The potential match signal is a signal indicating tracking object information corresponding to the constraint violation part of one or more tracking object information.

[0109] If the solution satisfies the constraints, the arbitration unit 38 provides a constraint satisfaction control signal to the modification unit 40, the addition unit 42, and the deletion unit 46. More specifically, the arbitration unit 38 provides a match signal to the modification unit 40, an unmatched object signal to the addition unit 42, and an unmatched tracker signal to the deletion unit 46.

[0110] If the solution does not satisfy the constraint conditions and the obtained solution is a valid constraint-violating solution, the arbitration unit 38 provides a constraint satisfaction control signal to the modification unit 40, the addition unit 42 and the deletion unit 46, and provides a constraint violation control signal to the life extension unit 44. More specifically, the arbitration unit 38 provides a match signal to the modification unit 40, a mismatch detection signal to the addition unit 42, a mismatch tracking signal to the deletion unit 46, and a potential match signal to the life extension unit 44.

[0111] If the solution does not satisfy the constraint conditions and the acquired solution is not a valid constraint-violating solution, the arbitration unit 38 generates an exception signal for causing an exception process to be executed, and supplies the exception signal to the exception processing unit 50 .

[0112] The arbitration unit 38 does not acquire a combinatorial optimization problem at a time when the image data does not include a detected object and the detected object information has not been acquired, or at a time when the tracked object to be tracked does not exist and the tracked object information is not included in the internal state information. At a time when the image data does not include a detected object and the detected object information has not been acquired, the arbitration unit 38 generates unmatched detected object information for all tracked object information included in the internal state information and supplies it to the deletion unit 46. Also, at a time when the tracked object to be tracked does not exist and the tracked object information is not included in the internal state information, the arbitration unit 38 generates unmatched detected object information for all acquired detected object information and supplies it to the addition unit 42.

[0113] The modifying unit 40 acquires a match signal included in the constraint satisfaction control signal at regular intervals. The modifying unit 40 modifies, based on the match signal at regular intervals, the tracking object information associated with one or more pieces of detected object information among one or more pieces of tracking object information included in the internal state information stored in the storage unit 28, based on the associated detected object information. For example, the modifying unit 40 modifies, based on the match signal, the tracking object information associated with one or more pieces of detected object information among one or more pieces of tracking object information at the first time, by the associated detected object information. For example, the modifying unit 40 modifies the tracking object information of the target according to the Kalman filter theory, based on information included in the associated detected object information. This allows the modifying unit 40 to reflect the state of the detected object included in the image data in one or more pieces of tracking object information included in the internal state information. Furthermore, the modifying unit 40 changes the age included in the tracking object information of the target to an initial value (for example, 0).

[0114] The adding unit 42 acquires an unmatched detected object signal included in the constraint satisfaction control signal at regular intervals. The adding unit 42 adds, based on the unmatched detected object signal, detected object information that is not associated with any of the tracking object information among the one or more pieces of detected object information, to one or more pieces of tracking object information included in the internal state information stored in the storage unit 28 as new tracking object information at regular intervals. For example, the adding unit 42 adds, based on the unmatched detected object signal, detected object information that is not associated with any of the tracking object information among the one or more pieces of detected object information at the first time, to one or more pieces of tracking object information at the first time as new tracking object information. This allows the adding unit 42 to reflect a detected object that has newly framed in the image data in one or more pieces of tracking object information included in the internal state information. Furthermore, the modifying unit 40 sets the age included in the newly added tracking object information to an initial value (for example, 0).

[0115] The life extension unit 44 acquires a potential match signal included in the constraint violation control signal at regular intervals. The life extension unit 44 reduces, based on the potential match signal at regular intervals, the age of the tracking object information corresponding to the constraint violation portion among one or more pieces of tracking object information included in the internal state information stored in the storage unit 28, which indicates the elapsed time since the addition or modification, by a predetermined value. For example, the life extension unit 44 reduces, based on the potential match signal, the age of the tracking object information corresponding to the constraint violation portion among one or more pieces of tracking object information at the first time, by a predetermined value. This allows the life extension unit 44 to prevent the tracking object information from being deleted even if the target tracking object is temporarily not included in the image data due to occlusion. Therefore, the life extension unit 44 can continue to track a tracking object that has reappeared in the image data after the occlusion has been resolved.

[0116] The deletion unit 46 acquires an unmatched tracking object signal included in the constraint satisfaction control information at regular intervals. The deletion unit 46 deletes tracking object information whose age exceeds a predetermined threshold from among one or more pieces of tracking object information included in the internal state information stored in the storage unit 28, based on the unmatched tracking object signal at regular intervals. For example, the deletion unit 46 deletes tracking object information whose age exceeds a predetermined threshold from among one or more pieces of tracking object information at a first time, based on the unmatched tracking object signal. This allows the deletion unit 46 to delete a tracking object that has framed out of the image data from the internal state information and update the internal state information to the latest state. In addition, the deletion unit 46 can delete information about an object that does not need to be tracked from the internal state information, and suppress a continuous increase in one or more pieces of tracking object information included in the internal state information.

[0117] The threshold value is set to a value that prevents at least the tracking object information whose age has been reduced by a predetermined value by the life extension unit 44 at the first time from being deleted at the first time. That is, the deletion unit 46 does not delete, at the first time, the tracking object information whose age has been reduced by a predetermined value by the life extension unit 44 at the first time. This allows the life extension unit 44 to prevent the tracking object information from being deleted when the target tracking object is temporarily not included in the image data due to, for example, occlusion.

[0118] The output unit 48 outputs one or more pieces of tracking object information included in the internal state information stored in the storage unit 28 to the planning device 16. For example, the output unit 48 outputs one or more pieces of tracking object information at a first time to the planning device 16. Note that the output unit 48 may output one or more pieces of tracking object information in response to a request from the planning device 16, rather than outputting one or more pieces of tracking object information at regular time intervals, or may output one or more pieces of tracking object information asynchronously with the time interval of the frames.

[0119] The output unit 48 may also output a portion of information included in each of the one or more pieces of tracking object information. For example, the output unit 48 may output the horizontal pixel position, vertical pixel position, area, and identification information for each of the one or more pieces of tracking object information. The output unit 48 may also output type information of the tracking object.

[0120] The exception processing unit 50 receives an exception signal from the arbitration unit 38. When receiving the exception signal, the exception processing unit 50 instructs, for example, another device to execute exception processing that is executed when, for example, the object tracking unit 30 cannot properly track an object. For example, the exception processing unit 50 outputs warning information, outputs a signal indicating that tracking accuracy has deteriorated to the planning device 16 or the like, or initializes or resets the processing in the object tracking unit 30.

[0121] Fig. 9 is a diagram showing an example of a plurality of decision variables included in the combinatorial optimization problem generated by the formulation unit 36. Fig. 10 is a diagram showing an example of a plurality of coefficients included in the combinatorial optimization problem generated by the formulation unit 36.

[0122] The formulation unit 36 ​​generates a combinatorial optimization problem for associating, under a predetermined constraint condition, one or more pieces of tracking object information at an arbitrary first time with any detected object information determined to represent the same object among one or more pieces of detected object information at the first time. More specifically, the formulation unit 36 ​​generates a bipartite graph maximum matching problem as a combinatorial optimization problem for minimizing a cost function under a predetermined constraint condition.

[0123] The bipartite graph maximum matching problem generated by the formulation unit 36 ​​will now be described in further detail.

[0124] Each of the one or more detection unit information and each of the one or more tracking object information are assigned a unique identification information. Detection object information newly added to the one or more detection object information are assigned an identification information that does not overlap with any previously assigned identification information. Tracking object information newly added to the one or more tracking object information are also assigned an identification information that does not overlap with any previously assigned identification information.

[0125] In the example of Figures 9 and 10, identification information d1, d2, d3, d4, and d5 are assigned to five pieces of detected object information. Also, in the example of Figures 9 and 10, identification information t1, t2, t3, t4, and t5 are assigned to five pieces of tracked object information.

[0126] For example, if the number of pieces of detected object information is N d For example, if the number of pieces of tracking information is N t In this case, the formulation unit 36 ​​determines N t ×N d Define decision variables.

[0127] The decision variable is b t , d It is expressed as follows. t , d is a bit variable, representing 0 or 1.

[0128] For more information, see b t , d is the t-th item (t is 1 or more, N t For the tracked object information, the dth (d is an integer greater than or equal to 1) of one or more detected object information d It indicates 1 when the detected object information (an integer below) is determined to represent the same object and can be matched, and indicates 0 when they cannot be matched.

[0129] A constraint-free cost function (H cost ) is expressed by equation (11).

number

[0130] IOU(t,d) is a real number equal to or greater than 0, and represents a coefficient. IOU(t,d) is a coefficient that represents a larger value so that the t-th tracked object information and the d-th detected object information are determined to represent the same object. For example, IOU(t,d) becomes a larger value as the degree of agreement between the position and size of the tracked object represented by the t-th tracked object information and the position and size of the detected object represented by the d-th detected object information increases.

[0131] In this embodiment, IOU(t, d) is the area of ​​the overlapping portion between the boundary box represented by the tth tracking object information and the boundary box represented by the dth detected object information divided by the area of ​​a rectangle containing the boundary box represented by the tth tracking object information and the boundary box represented by the dth detected object information.

[0132] The formulation unit 36 ​​generates IOU(t,d) for each pair of one or more pieces of detected object information and one or more pieces of tracked object information. d and the number of tracking information items is N t If so, the formulation unit 36 ​​determines that N t ×N dCalculate IOU(t,d).

[0133] The above cost function is minimized when the sum of IOU(t,d) for pairs of tracking object information and detected object information that are determined to be associated by the decision variable, i.e., pairs of tracking object information and detected object information for which the decision variable is 1, is maximized.

[0134] FIG. 11 is a diagram showing the relationship between the range of a set of solutions that a decision variable can take and the range of a set of solutions that satisfy constraint conditions.

[0135] The possible solutions for the decision variables are 2(N t ×N d ) solutions exist. In contrast, the solution that satisfies the constraints in the bipartite graph maximum matching problem is the solution that satisfies the 2(N t ×N d In this embodiment, among the set of solutions that the decision variables can take, a solution that satisfies the constraint conditions is called a constraint satisfying solution, and a solution that does not satisfy the constraint conditions is called a constraint violating solution.

[0136] The constraints in the bipartite graph maximum matching problem are expressed by the equality constraints shown in equations (12-1), (12-2), (13-1) and (13-2).

number

number

[0137] Equation (12-1) is N d ≧N t If N t For each of the tracking information, the corresponding N d This represents the constraint that there is only one decision variable among the N decision variables whose value is 1. In other words, equation (12-1) d ≧N t If N t For each of the tracking information, N dIt is to be noted that any one of the pieces of detected object information is associated with the same object. t It contains equality constraints.

[0138] Equation (12-2) is N d <N t If N t For each of the tracking information, the corresponding N d Any two of the decision variables (b t , d ) and (b t , d´ ) is 0. In other words, equation (12-2) expresses the constraint that N d <N t If N t For each of the tracking information, N d It is shown that two or more pieces of detected object information cannot be associated as information representing the same object. t It contains equality constraints.

[0139] Equation (13-1) is N d ≦N t If N d For each of the detected object information, the corresponding N t This represents the constraint that there is only one decision variable among the N decision variables whose value is 1. In other words, equation (13-1) d ≦N t If N d For each of the tracking information, N t It is to be noted that any one of the tracking object information is associated as information representing the same object. d It contains equality constraints.

[0140] Equation (13-2) is N d >N t If N d For each of the detected object information, the corresponding N t Any two of the decision variables (b t ,d ) and (b t´ , d ) is 0. In other words, equation (13-2) expresses the constraint that N d >N t If N d For each of the detected object information, N t This means that two or more of the tracking object information cannot be associated as information representing the same object. d It contains equality constraints.

[0141] Equations (12-1), (12-2), (13-1) and (13-2) are t ×N d The decision variables are N t Row N d When expressed as a column matrix, this means the following:

[0142] N d =N t In the case of , the sum of the decision variable values ​​for each row is 1. Also, N d =N t If , the sum of the decision variables for each column is 1.

[0143] N t <N d In the case of , the sum of the decision variable values ​​for each row is 1. Also, N t <N d If , the sum of the decision variables for each column is either 0 or 1. t <N d , the number of columns that sum to zero is (N d -N t ) pieces.

[0144] N t >N d In the case of , the sum of the decision variable values ​​for each column is 1. Also, N t >N d If , the sum of the decision variables for each row is either 0 or 1. t >N d , the number of rows that sum to zero is (N t -N d ) pieces.

[0145] In this way, the constraint conditions are expressed by one or more constraint equations. Each of the one or more constraint equations is expressed by an equation or inequality including one or more decision variables among the multiple decision variables. In this embodiment, each of the one or more constraint equations is expressed by an equation including a linear or quadratic term of the decision variable.

[0146] From the above, the multiple constraint equations expressed by equations (12-1), (12-2), (13-1) and (13-2) imply the following (first condition), (second condition) and (third condition) constraint conditions.

[0147] (First condition) N d =N t If N d Detected object information and N t In other words, the N tracking object information currently being tracked by the object tracking unit 30 is associated with each other on a one-to-one basis. t The tracking objects are N d Each of the detected objects is associated with a one-to-one correspondence.

[0148] (Second condition) N t <N d If N d Of the detected object information (N d -N t ) pieces of detection object information are detection object information representing new detection objects. On the other hand, N d N out of the detected object information t The detected object information is N t In other words, the N d Of the detected objects (N d -N t ) detected objects are objects that have newly entered the image data and are not included in the currently tracked N t On the other hand, the N d N out of 1 detected objects t The detections are the N tIt is associated one-to-one with each tracked object.

[0149] (Third condition) N t >N d If N t Of the tracking information (N t -N d ) tracked object information does not have corresponding detected object information. On the other hand, N t N of tracking information d The tracking information is N d In other words, N t Of the traces (N t -N d ) tracked objects are objects that have gone out of frame and are currently included in the image data. d On the other hand, N t N out of 1 detected objects d The number of tracking objects is currently N d Each of the detected objects is associated with a one-to-one correspondence.

[0150] Therefore, the object tracking unit 30 can search for a combination of values ​​of multiple decision variables that minimizes the cost function under the constraints described above, and thereby associate each of one or more pieces of tracking object information with any of the one or more pieces of detected object information that is determined to represent the same object.

[0151] In this embodiment, the formulation unit 36 ​​generates a combinatorial optimization problem in a form that can be solved by a solver that solves combinatorial optimization problems without constraints, by incorporating the above-mentioned constraint conditions into the cost function. This allows the solver unit 22 to output the solution to the combinatorial optimization problem supplied from the formulation unit 36, even if the solver unit 22 is a solver that solves combinatorial optimization problems without constraints.

[0152] In this embodiment, the formulation unit 36 ​​generates a combinatorial optimization problem for minimizing a total cost function that is the sum of a cost function and a penalty function, in a form that can be solved by a solver that solves unconstrained combinatorial optimization problems.

[0153] The penalty function is a function that includes at least some of the decision variables and has a minimum value when the solution satisfies the constraint conditions. Specifically, the penalty function has a minimum value (e.g., 0) when the solution satisfies one or more constraint equations that represent the constraint conditions, and is greater than the minimum value when the values ​​of the decision variables do not satisfy one or more constraint equations.

[0154] In this embodiment, the formulation unit 36 ​​generates the total cost function shown in equation (14).

number

[0155] (CH penalty1 +C2H penalty2 ) represents the penalty function. H penalty1 represents the first penalty function. H penalty2 represents the second penalty function.

[0156] The first penalty function (H penalty1 ) is expressed by equations (15-1) and (15-2).

number

[0157] The first penalty function (H penalty1 ) is expressed by a quadratic function of the decision variables. t If all of the equality constraints are satisfied, the minimum value (0 in this example) is obtained. t The larger the violation number, which is the number of constraint equations that are satisfied among the constraint equations of equality, the larger the value becomes.

[0158] The first penalty function (H penalty1) is expressed by a quadratic function of the decision variables. t If all of the equality constraints are satisfied, the minimum value (0 in this example) is obtained. t The larger the violation number, which is the number of constraint equations that are satisfied among the constraint equations of equality, the larger the value becomes.

[0159] The second penalty function (H penalty2 ) is expressed by equations (16-1) and (16-2).

number

[0160] The second penalty function (H penalty2 ) is expressed by a quadratic function of the decision variables. d If all of the equality constraints are satisfied, the minimum value (0 in this example) is obtained. d The larger the violation number, which is the number of constraint equations that are satisfied among the constraint equations of equality, the larger the value becomes.

[0161] The second penalty function (H penalty2 Equation (16-2) of the above equations is expressed by a quadratic function of the decision variables. Equation (16-2) is a function of N d If all of the equality constraints are satisfied, the minimum value (0 in this example) is obtained. d The larger the violation count, which is the number of constraint equations that are not satisfied among the equations, the larger the value.

[0162] In equation (14), C1 is a first coupling coefficient by which the first penalty function is multiplied, and is a positive real number greater than 0. C2 is a second coupling coefficient by which the second penalty function is multiplied, and is a positive real number greater than 0.

[0163] C1 and C2 are coefficients that determine the weight of the penalty function with respect to the cost function. When C1 and C2 are sufficiently large, the values ​​of the multiple decision variables that minimize the total cost function are the values ​​of the multiple decision variables that minimize the cost function among the values ​​of the multiple decision variables that satisfy the constraint conditions (constraint satisfying solution). When C1 and C2 are relatively small, the values ​​of the multiple decision variables that minimize the total cost function may be the values ​​of the multiple decision variables that do not satisfy the constraint conditions (constraint violating solution). Therefore, C1 and C2 are appropriately adjusted based on both the viewpoints of improving the accuracy of object tracking in normal times and improving occlusion response capability.

[0164] The total cost function shown in the formula (14) is expressed by a quadratic function of multiple decision variables. Therefore, if the solver unit 22 is a solver capable of solving a quadratic unconstrained binary optimization problem, it can output a solution to the problem that minimizes the total cost function shown in the formula (14).

[0165] The formulation unit 36 ​​may generate a combinatorial optimization problem for minimizing a total cost function different from the total cost function shown in Equation 14. For example, the formulation unit 36 ​​may generate a combinatorial optimization problem for minimizing a total cost function obtained by adding a penalty function to a cost function expressed as a quadratic function of multiple decision variables.

[0166] The formulation unit 36 ​​may also generate a higher-order unconstrained binary optimization (HUBO) problem that minimizes a total cost function obtained by adding a cost function, which is a cubic or higher function of multiple decision variables, and a penalty function. However, when the formulation unit 36 ​​generates a HUBO problem, the solver unit 22 must be a solver capable of outputting a solution to the HUBO problem. A solver capable of outputting a solution to the HUBO problem can be realized by executing a simulated bifurcation algorithm, for example, as shown in Patent Document 3.

[0167] In the present embodiment, the constraint equation is expressed as an equality. However, the constraint equation may be an inequality. In addition, when the constraint condition is expressed as a plurality of constraint equations, some of the plurality of constraint equations may be equalities and the other may be inequalities.

[0168] Fig. 12 is a flowchart showing the flow of processing of the object tracking device 14. Fig. 13 is a diagram showing a data flow graph of the object tracking device 14. The processing and data flow of the object tracking device 14 will be described below with reference to the flowchart of Fig. 12 and the data flow graph of Fig. 13. Note that in the description of Figs. 12 and 13, it is assumed that one or more pieces of detection unit information are detected in every frame, and one or more pieces of tracked object information are included in the internal state information in every frame.

[0169] The object tracking device 14 repeatedly executes the processes of S32 to S41 for each frame (repeated processes between S31 and S42).

[0170] In S32, the object detection unit 26 of the object tracking device 14 executes an object detection process on the image data of the kth frame (k is an integer) acquired from the camera 12, and generates one or more pieces of detection information (detections) representing one or more detected objects included in the kth image data. Then, the acquisition unit 32 of the object tracking device 14 acquires the one or more pieces of detection information (detections) of the generated kth frame.

[0171] In parallel with S32, in S33, the prediction unit 34 of the object tracking device 14 predicts one or more pieces of tracking object information representing one or more tracking objects in the k-th frame based on one or more pieces of tracking object information (trackers) representing one or more tracking objects in the (k-1)-th frame, which are included in the internal state information of the storage unit 28 (state memory). Then, the prediction unit 34 stores the predicted one or more pieces of tracking object information in the k-th frame in the storage unit 28. In this case, the prediction unit 34 increases the age included in each of the one or more pieces of tracking object information in the k-th frame by a predetermined value (for example, 1).

[0172] Next, in S34, the formulation unit 36 ​​of the object tracking device 14 generates a combinatorial optimization problem for the k-th frame based on one or more pieces of tracked object information in the k-th frame and one or more pieces of detected object information in the k-th frame.

[0173] Next, in S35, the solver unit 22 of the object tracking device 14 obtains a combinatorial optimization problem for the k-th frame. Then, the solver unit 22 solves the obtained combinatorial optimization problem while allowing constraint conditions not to be satisfied.

[0174] Next, in S36, the arbitration unit 38 of the object tracking device 14 executes an arbitration process. Specifically, the arbitration unit 38 acquires a solution to the combinatorial optimization problem for the k-th frame from the solver unit 22. Then, the arbitration unit 38 evaluates the acquired solution. If the acquired solution satisfies the constraint condition, the arbitration unit 38 outputs a constraint satisfaction control signal based on the solution that satisfies the constraint condition. The constraint satisfaction control signal includes a match signal (matched), an unmatched detection signal (unmatched_detections), and an unmatched trackers signal (unmatched_trackers).

[0175] Furthermore, when the acquired solution does not satisfy the constraint conditions and is a valid constraint violating solution close to a constraint satisfying solution, the arbitration unit 38 identifies a constraint satisfying part of the acquired solution that satisfies the constraint conditions and a constraint violating part of the solution that does not satisfy the constraint conditions. Then, when the acquired solution does not satisfy the constraint conditions and is a valid constraint violating solution, the arbitration unit 38 generates a constraint satisfaction control signal based on the constraint satisfaction part and a constraint violation control signal based on the constraint violation part. The constraint violation control signal includes a potentially matched signal (potentially_matched).

[0176] Furthermore, the arbitration unit 38 outputs an exception signal when the acquired solution does not satisfy the constraint conditions and is not a valid constraint-violating solution.

[0177] Next, in S37, the modifying unit 40 of the object tracking device 14 acquires a match signal included in the constraint satisfaction control signal. Based on the match signal, the modifying unit 40 modifies the tracking object information associated with one or more pieces of detected object information among one or more pieces of tracking object information of the k-th frame included in the internal state information stored in the storage unit 28, based on the associated detected object information. Furthermore, the modifying unit 40 modifies the age included in the associated tracking object information to an initial value (e.g., 0).

[0178] Next, in S38, the adding unit 42 of the object tracking device 14 acquires an unmatched detected object signal included in the constraint satisfaction control signal. Based on the unmatched detected object signal, the adding unit 42 adds the detected object information that is not associated with any of the tracked object information among the one or more pieces of detected object information as new tracked object information to one or more pieces of tracked object information of the k-th frame included in the internal state information stored in the storage unit 28. Furthermore, the modifying unit 40 sets the age included in the newly added tracked object information to an initial value (e.g., 0).

[0179] Next, in S39, the life extension unit 44 of the object tracking device 14 acquires a potential match signal included in the constraint violation control signal. Based on the potential match signal, the life extension unit 44 decreases the age in the tracking object information corresponding to the constraint violation portion among one or more pieces of tracking object information in the k-th frame included in the internal state information stored in the storage unit 28 by a predetermined value (for example, 1).

[0180] Next, in S40, the deletion unit 46 of the object tracking device 14 acquires an unmatched tracking object signal included in the constraint satisfaction control signal. Based on the unmatched tracking object signal, the deletion unit 46 deletes tracking object information whose age exceeds a predetermined threshold (e.g., 1) from among one or more pieces of tracking object information of the k-th frame included in the internal state information stored in the storage unit 28. Note that the threshold is set to a value such that tracking object information whose age has been reduced by a predetermined value in the process of S39 of the k-th frame is not deleted at least in S40 of the k-th frame.

[0181] Next, in S41, the output unit 48 of the object tracking device 14 outputs one or more pieces of tracking object information of the k-th frame included in the internal state information stored in the storage unit 28 to the planning device 16. Note that the output unit 48 may execute the output process for all frames, or may execute the output process for every predetermined number of frames or every time there is a request from the planning device 16.

[0182] The object tracking device 14 executes the above process for each frame (repeated process between S31 and S42).

[0183] 14 is a flowchart showing the flow of processing by the arbitration unit 38. When the arbitration unit 38 acquires a solution from the solver unit 22, the arbitration unit 38 executes arbitration processing according to the flow shown in the flowchart of FIG.

[0184] First, in S51, the arbitration unit 38 acquires the solution output from the solver unit 22.

[0185] Next, in S52, the arbitration unit 38 evaluates the acquired solution. Specifically, first, the arbitration unit 38 evaluates whether or not the solution satisfies the constraint conditions. If the solution does not satisfy the constraint conditions, the arbitration unit 38 further evaluates whether or not the solution is a valid constraint-violating solution.

[0186] For example, the constraint conditions are expressed by one or more constraint equations, each of which is an equality or inequality including multiple decision variables as arguments. If the acquired solution satisfies all of the one or more constraint equations, the arbitration unit 38 evaluates that the solution satisfies the constraint conditions. In other words, if the acquired solution satisfies all of the one or more constraint equations, the arbitration unit 38 evaluates that the acquired solution is a constraint satisfying solution.

[0187] Furthermore, when the acquired solution does not satisfy at least one of one or more constraint equations, the arbitration unit 38 evaluates the acquired solution as a constraint violating solution that does not satisfy the constraint conditions.

[0188] If the obtained solution is a constraint violating solution, the arbitration unit 38 further evaluates whether the obtained solution is a valid constraint violating solution close to a constraint satisfying solution. The arbitration unit 38 evaluates whether the obtained solution is a valid constraint violating solution based on a predetermined evaluation criterion.

[0189] For example, the arbitration unit 38 assigns the obtained solution to each of one or more constraint equations, and calculates the number of violations, which is the number of constraint equations that do not satisfy the equation when the obtained solution is assigned among the one or more constraint equations. If the number of violations is equal to or less than a preset value, the arbitration unit 38 evaluates the solution as a valid constraint violation solution.

[0190] Also, for example, the arbitration unit 38 substitutes the obtained solution into a penalty function and calculates a penalty value, which is the value of the penalty function when the obtained solution is substituted. The arbitration unit 38 may evaluate a solution that violates the constraint as valid when the penalty value is equal to or less than a preset value.

[0191] In S53, the arbitration unit 38 judges whether the acquired solution satisfies the constraint conditions based on the evaluation result. If the acquired solution satisfies the constraint conditions (Yes in S53), the arbitration unit 38 advances the process to S54. Then, in S54, the arbitration unit 38 generates and outputs a constraint satisfaction control signal based on the acquired solution. When the arbitration unit 38 finishes the process of S54, it ends this flow.

[0192] If the acquired solution does not satisfy the constraint condition (No in S53), the arbitration unit 38 advances the process to S55. In S55, the arbitration unit 38 judges whether the acquired solution is a valid constraint-violating solution based on the evaluation result. If the acquired solution is not a valid constraint-violating solution (No in S55), the arbitration unit 38 outputs an exception signal that causes exception processing to be executed, and ends this flow.

[0193] If the acquired solution is a valid constraint violating solution (Yes in S55), the arbitration unit 38 advances the process to S56.

[0194] In S56, the arbitration unit 38 identifies constraint satisfying parts and constraint violating parts in the valid constraint violating solution.

[0195] For example, the arbitration unit 38 identifies a decision variable that causes the constraint condition not to be satisfied among the multiple decision variables based on the obtained solution. Then, the arbitration unit 38 changes the value of the decision variable that causes the constraint condition not to be satisfied to identify a constraint satisfaction part. In addition, the arbitration unit 38 extracts the value of the decision variable that causes the constraint condition not to be satisfied and identifies it as a constraint violation part.

[0196] For example, if the value of the decision variable that causes the constraint condition not to be satisfied is 0, the arbitration unit 38 changes it to 1, and if the value is 1, the arbitration unit 38 changes it to 0. Then, for example, if the solution satisfies the constraint condition after changing the value of the decision variable that causes the constraint condition not to be satisfied, the arbitration unit 38 identifies the solution after changing the value of the decision variable that causes the constraint condition not to be satisfied as a constraint satisfying part. Also, for example, if the solution satisfies the constraint condition after changing the value of the decision variable that causes the constraint condition not to be satisfied, the arbitration unit 38 identifies the decision variable whose value has been changed as the decision variable that causes the constraint condition not to be satisfied.

[0197] For example, the arbitration unit 38 judges whether or not the values ​​of a plurality of decision variables arranged in a matrix as shown in FIG. 9 satisfy the constraint condition for each column (or row). Then, the arbitration unit 38 identifies a column (or row) that does not satisfy the constraint condition. Next, the arbitration unit 38 inverts the value of any of the plurality of decision variables included in the column (or row) that is judged not to satisfy the constraint condition so as to satisfy the constraint condition. If the constraint condition is satisfied as a result of the inversion, the arbitration unit 38 identifies the decision variable whose value has been inverted as a factor that does not satisfy the constraint condition. If there are two or more decision variables in the column (or row) that is judged not to satisfy the constraint condition, which change to satisfy the constraint condition by inversion, the arbitration unit 38 may determine the decision variable that is a factor that does not satisfy the constraint condition based on the corresponding coefficient. For example, the arbitration unit 38 may select the decision variable that is a factor that does not satisfy the constraint condition, giving priority to the decision variable with the smallest corresponding IOU.

[0198] Next, in S57, the arbitration unit 38 generates and outputs a constraint satisfaction control signal based on the constraint satisfaction portion and a constraint violation control signal based on the constraint violation portion. After completing the process of S57, the arbitration unit 38 ends this flow.

[0199] Next, the arbitration process, the modification process, the addition process, the life extension process, the deletion process, and the occlusion countermeasure function will be described with reference to specific examples shown in FIGS.

[0200] Fig. 15 is a diagram showing an example of a boundary box indicated by each of the detected object information and the tracked object information, and a solution in the (k-1)th frame. Fig. 16 is a diagram showing an example of a boundary box indicated by each of the detected object information and the tracked object information, and a solution in the kth frame. Fig. 17 is a diagram showing an example of a boundary box indicated by each of the detected object information and the tracked object information, and a solution in the (k+1)th frame. Fig. 18 is a diagram showing an example of a constraint satisfying portion identified in the kth frame.

[0201] In addition, in Figs. 15 to 18, S C represents the sum of the decision variables in the column direction. R represents the sum of the values ​​of the decision variables in the row direction.

[0202] First, the arbitration process will be described.

[0203] In the k-th frame, as shown in Figure 16, d <N t N d <N t If so, then the solution may be a valid constraint-violating solution. If the solution is a valid constraint-violating solution, the arbitration unit 38 may identify portions of the solution that violate the constraints.

[0204] First, in the kth frame, N d <N t Therefore, the arbitration unit 38 judges whether the solution of the k-th frame satisfies the constraint condition based on the equations (12-2) and (13-1). As shown in FIG. 16, the solution of the k-th frame is the column-wise sum (S C ) is 2, which does not satisfy equation (13-1).

[0205] Next, the arbitration unit 38 judges whether the solution of the kth frame is a valid constraint violating solution. For example, when the number of violations of the solution of the kth frame is equal to or less than a preset value (max_violation), the arbitration unit 38 judges it to be a valid constraint violating solution. In this example, the preset value (max_violation) is 1. Note that the preset value may be 1 or more. The solution of the kth frame has a violation number of 1. Therefore, the arbitration unit 38 judges that the solution of the kth frame is a valid constraint violating solution.

[0206] Next, the arbitration unit 38 identifies constraint violating parts and constraint satisfying parts from the solution of the kth frame. The solution of the kth frame is determined by the column-wise sum (S C) violates the constraint condition. In the fourth column (d4), both the decision variable of (t2, d4) and the decision variable of (t5, d4) are 1, and it is not possible to specify which tracked object information of t2 or t5 corresponds to the detected object information shown in d4. Therefore, the solution of the kth frame satisfies the constraint condition by inverting the value of either the decision variable of (t2, d4) or the decision variable of (t5, d4). For example, the arbitration unit 38 compares IOU(t2, d4) corresponding to the decision variable of (t2, d4) with IOU(t5, d4) corresponding to the decision variable of (t5, d4), and identifies the smaller decision variable as the constraint violation part. In this example, it is assumed that IOU(t2, d4)>IOU(t5, d4). Therefore, in this example, the arbitration unit 38 identifies the decision variable of (t5, d4) as the constraint violation part. Furthermore, the arbitration unit 38 identifies the solution after the value of the decision variable of (t5, d4) is inverted from 1 to 0 as shown in FIG. 18, as the constraint satisfying portion.

[0207] The arbitration unit 38 then generates and outputs a potentially matched signal (potentially_matched) included in the constraint violation control signal based on the tracking object information indicated in the constraint violation portion. The arbitration unit 38 also generates and outputs a matched signal (matched) included in the constraint satisfaction control signal based on the detected object information and tracking object information associated one-to-one in the constraint satisfaction portion. The arbitration unit 38 also generates and outputs an unmatched tracking object signal (unmatched_trackers) included in the constraint satisfaction control signal based on the tracking object information not associated with the detected object information in the constraint satisfaction portion. The arbitration unit 38 also generates and outputs an unmatched detection object signal (unmatched_detections) included in the constraint satisfaction control signal based on the detected object information not associated with the tracking object information in the constraint satisfaction portion.

[0208] In the k-th frame shown in Fig. 18, the arbitration unit 38 generates a match signal indicating the detected object information and the tracked object information corresponding to the decision variables indicated by (t1, d3), (t2, d4), (t3, d1), and (t4, d2), respectively. In addition, in the k-th frame shown in Fig. 18, the arbitration unit 38 generates an unmatched tracked object signal indicating the tracked object information (t5) corresponding to the decision variable (t5, d4). Note that in the k-th frame shown in Fig. 18, since there is no detected object that has newly entered the image data, the arbitration unit 38 does not output an unmatched detected object signal.

[0209] 18, the arbitration unit 38 generates a potential match signal indicating the tracking object information corresponding to the decision variables of (t5, d4). That is, the tracking object information of t5 is identified as being in a state in which there is no corresponding detected object information and at the same time, there is no detected object information that is temporarily associated with the tracking object information due to occlusion.

[0210] Next, the modification process, the addition process, the life extension process, and the deletion process will be described.

[0211] At the start of the k-th frame, the internal state information includes five pieces of tracking object information (t1 to t5) as shown in FIG. 16. Each of the five pieces of tracking object information (t1 to t5) includes an age ( i ) is 0. Then, for each of the five pieces of tracking object information (t1 to t5), the age ( i ) is incremented by 1.

[0212] The correction unit 40 corrects each of the four pieces of tracking object information (t1, t2, t3, t4) indicated in the match signal among the five pieces of tracking object information (t1 to t5) included in the internal state information, based on the Kalman filter theory, using the corresponding detected object information (d1, d2, d3, d4). More specifically, the correction unit 40 corrects the tracking object information indicated in t1 using the corresponding detected object information indicated in d3. The correction unit 40 also corrects the tracking object information indicated in t2 using the corresponding detected object information indicated in d4. The correction unit 40 also corrects the tracking object information indicated in t3 using the corresponding detected object information indicated in d1. The correction unit 40 also corrects the tracking object information indicated in t4 using the corresponding detected object information indicated in d2.

[0213] Furthermore, the correction unit 40 corrects the ages (ages) of the four pieces of tracking object information (t1, t2, t3, t4) indicated in the match signal. i ) to its initial value of 0.

[0214] The adding unit 42 is N d >N t In the case where the detected object is a newly frame-in detected object, the adding unit 42 acquires an unmatched detected object signal indicating tracking object information indicating a detected object that has newly frame-into the image data. Then, the adding unit 42 adds the tracking object information indicated in the unmatched detected object signal to the internal state information as new tracking object information. The adding unit 42 also adds the age (age i ) is set to an initial value of 0. However, in the k-th frame shown in FIG. 16, since there is no detected object that has newly framed in to the image data, the adding unit 42 does not receive the unmatched detected object signal and does not add new tracking object information to the internal state information.

[0215] The life extension unit 44 reduces the age of the tracking object information (t5) indicated in the potential match signal, among the five tracking object information (t1 to t5) included in the internal state information, by a predetermined value (anti-aging). In this example, the predetermined value (anti-aging) is 1. In the k-th frame shown in FIG. 16, the life extension unit 44 reduces the age (age) of the tracking object information indicated in t5. i) is subtracted by 1 to make it 0.

[0216] The deletion unit 46 deletes the tracking object information (t5) indicated in the unmatched tracking object signal, of the five tracking object information (t1 to t5) included in the internal state information, whose age is greater than a preset threshold (max_age). In this example, the threshold is assumed to be 1. In the k-th frame shown in FIG. 16, the unmatched tracking object signal indicates the tracking object information of t5. However, the age of the tracking object information indicated at t5 is set to 0 by the life extension unit 44, which is smaller than the threshold. Therefore, the deletion unit 46 does not delete the tracking object information indicated at t5.

[0217] For example, unlike the example shown in FIG. 16, suppose that in the k-th frame, the arbitration unit 38 acquires the solution shown in FIG. 18 from the solver unit 22. In this case, the arbitration unit 38 does not output a potential match signal in the k-th frame. Therefore, the age of the tracking object information shown at t5 is not decreased by the life extension unit 44. Therefore, in this case, the deletion unit 46 deletes the tracking object information shown at t5.

[0218] Next, the occlusion countermeasure capability of the object tracking unit 30 will be described.

[0219] As shown by the boundary boxes in Figures 15, 16, and 17, the track object corresponding to the track object information of t2 moves from left to right, and the track object corresponding to the track object information of t5 moves from the bottom right to the top left.

[0220] As shown in FIG. 16, in the k-th frame, the tracked object of t2 and the tracked object of t5 intersect. As a result, the tracked object of t5 is hidden behind the tracked object of t2, and occlusion occurs. Therefore, in the k-th frame, the detected object of d4 corresponding to the tracked object of t2 is included in the image data. However, in the k-th frame, the detected object corresponding to the tracked object of t5 is not included in the image data. However, the object tracking unit 30 does not delete the tracked object information indicated by t5 in the k-th frame by detecting a valid constraint violation solution and decreasing the age of the tracked object information indicated by t5 by a predetermined value.

[0221] Subsequently, as shown in FIG. 17, in the k+1th frame, the occlusion of the tracked object at t5 is resolved. As a result, in the k+1th frame, the image data includes five detected objects. Therefore, the five pieces of tracked object information included in the internal state information and the five pieces of detected object information detected from the kth frame can have a one-to-one correspondence. The tracked object information at t5 in the k-1th frame and the tracked object information at t5 in the k+1th frame are the same identification information. Therefore, the object tracking unit 30 can identify the tracked object information at t5 in the k-1th frame of FIG. 15 and the tracked object information at t5 in the k+1th frame of FIG. 16 as information indicating the same object, regardless of the occlusion that occurred in the kth frame. Therefore, the object tracking unit 30 can continue to track the tracked object indicated in the tracked object information at t5 in the k+1th frame, following the k-1th frame, regardless of the occlusion that occurred in the kth frame. In this way, the object tracking unit 30 can have an occlusion countermeasure capability.

[0222] On the other hand, unlike the above example, suppose that in the k-th frame, the solver unit 22 outputs a solution as shown in FIG. 18, and the object tracking unit 30 does not detect a valid constraint violation solution. In this case, the object tracking unit 30 deletes the tracked object information indicated at t5 in the k-th frame. As a result, at the start of the k+1-th frame, the internal state information includes four pieces of tracked object information. Therefore, the object tracking unit 30 adds new tracked object information corresponding to the detected object information indicated at d5 in the k+1-th frame to the internal state information with new identification information (for example, t6). For this reason, the object tracking unit 30 cannot identify the tracked object information indicated at t5 in the k-1-th frame and the tracked object information indicated at t6 in the k+1-th frame as information indicating the same object due to the influence of the occlusion that occurred in the k-th frame. Therefore, if a valid constraint violation solution is not detected, the object tracking unit 30 will discontinue tracking of the tracked object indicated in the tracked object information at t5 due to the influence of the occlusion that occurred in the k-th frame. In this way, if a valid constraint violation solution is not detected, the object tracking unit 30 will not have the ability to deal with occlusion.

[0223] As described above, the object tracking unit 30 according to the present embodiment extends the lifespan of not all tracking object information included in the internal state information, but only tracking object information corresponding to a tracking object for which an event requiring occlusion countermeasures has occurred. This allows the object tracking unit 30 according to the present embodiment to have an occlusion countermeasure capability while suppressing an unnecessary increase in the number of tracking object information. Therefore, the object tracking unit 30 according to the present embodiment suppresses the expansion of prediction and matching processes, and does not cause a deterioration in the processing performance of object tracking.

[0224] Specifically, the object tracking unit 30 calculates the total cost function (H total) is minimized by solving the unconstrained combinatorial optimization problem using a heuristic solution method, and the tracking object information for which occlusion countermeasures should be taken is identified. For example, in the constraint violation solution shown in FIG. 16, the decision variable of (t5, d4) is 1 because IOU(t5, d4) is smaller than the second penalty function (H penalty2 ) (the product of Eq. (16-2) and the coupling coefficient (C2)) or the second penalty function (H penalty2 ) is greater than the value of (t2, d4) as shown in FIG. 16. A constraint violation solution in which the decision variables for (t2, d4) and (t5, d4) are both 1 corresponds to a judgment that it is appropriate to interpret that both the tracked object corresponding to the tracked object information at t2 and the tracked object corresponding to the tracked object information at t4 match the detected object corresponding to the tracked object information at d4 from the viewpoint of the total cost function. Such a constraint violation solution corresponds to a situation in which tracked objects for which occlusion countermeasures should be implemented intersect with each other. Therefore, in this way, the object tracking unit 30 uses the total cost function (H total By solving the unconstrained combinatorial optimization problem that minimizes θ(θ), using a heuristic solution method, it is possible to identify the tracking object information that should be treated to prevent occlusion.

[0225] As described above, according to the object tracking device 14 of this embodiment, it is possible to improve the performance or efficiency of the information processing function by using a constraint violation solution obtained from a combinatorial optimization problem.

[0226] In the above example, the arbitration unit 38 determines that the solution is a valid constraint-violating solution when the violation count is 1 or less. Alternatively, the arbitration unit 38 may determine that the solution is a valid constraint-violating solution when the violation count is equal to or less than a predetermined value greater than 2. This allows the object tracking unit 30 to have an occlusion countermeasure capability for multiple tracking objects.

[0227] Also, in the above example, the life extension unit 44 decreases the age of the tracking object information indicated in the potential match signal by 1. Alternatively, the life extension unit 44 may decrease the age of the tracking object information indicated in the potential match signal by 2 or more. This allows the object tracking unit 30 to have a long-term occlusion countermeasure capability.

[0228] In the above example, the deletion unit 46 is set to 1 as the threshold (max_age). Alternatively, the deletion unit 46 may be set to a value of 2 or more as the threshold (max_age). In this case, the object tracking unit 30 can have a certain occlusion countermeasure capability for all tracking object information. Furthermore, the object tracking unit 30 has a longer occlusion countermeasure capability for tracking object information that is identified by a constraint violation solution and for which occlusion countermeasures should be taken. However, if the threshold (max_age) is set too large, the object tracking unit 30 will leave tracking object information that should essentially be deleted in the internal state information. For this reason, if the threshold (max_age) is set large, the object tracking unit 30 will increase the number of tracking object information, which will lead to bloating of prediction and matching processes, and as a result, this may lead to deterioration of the processing performance of object tracking.

[0229] In addition, each of one or more pieces of tracking information has an individual threshold (max_age i ). When one or more pieces of tracking object information are added or modified, the individual threshold is set to an initial value (for example, 0). In this case, the deletion unit 46 compares the age included in the tracking object information indicated in the unmatched tracking object signal with the individual threshold, and deletes the tracking object information if the age exceeds the threshold. Also, in this case, the life extension unit 44 increases the threshold included in the tracking object information indicated in the potential match signal by a predetermined value, instead of decreasing the age included in the tracking object information indicated in the potential match signal by a predetermined value.

[0230] (First Modification) FIG. 19 is a diagram showing the process of the formulation unit 36 ​​according to the first modified example.

[0231] The formulation unit 36 ​​calculates the total cost function (H total 19 may be executed before transmitting the combinatorial optimization problem for minimizing θ(θ) to the solver unit 22.

[0232] First, in S61, the formulation unit 36 ​​calculates N d is the number of one or more pieces of tracking object information included in the internal state information, N t Determine whether it is greater than N d <N t If so (Yes in S61), the formulation unit 36 ​​advances the process to S62.

[0233] In S62, the formulation unit 36 ​​calculates a total cost function (H total ) contains the second penalty function (H penalty2 After completing the process of S62, the formulation unit 36 ​​reduces the coupling coefficient (C2) of the total cost function (H total ) is transmitted to the solver unit 22.

[0234] Also, N d <N t If not (No in S61), the formulation unit 36 ​​does not change the coupling coefficient (C2) and instead changes the total cost function (H total ) is transmitted to the solver unit 22.

[0235] In this way, the formulation unit 36 ​​calculates N t >N d If so, then N t ≦N d In this way, the formulation unit 36 ​​can reduce C2 when the occlusion does not occur, i.e., when N t ≦N d In the case where there is a possibility that occlusion has occurred and the lifetime of the tracking object information is extended using a valid constraint violation solution, that is, in the case where N t <N dIf so, the influence of the second penalty function can be reduced to make it easier to generate a solution that violates the constraints.

[0236] In addition, the formulation unit 36 ​​is d <N t If not (No in S61), increase the coupling coefficient (C2) and N d <N t 19 in place of the formulation unit 36. In this case, when converting the combinatorial optimization problem received from the formulation unit 36 ​​into an Ising problem, the formulation unit 36 ​​may convert N d <N t Determine whether or not N d <N t If so, decrease C2.

[0237] (Second Modification) FIG. 20 is a diagram showing the process of the arbitration unit 38 according to the second modified example.

[0238] The formulation unit 36 ​​calculates a total cost function (H total ) the second penalty function (H penalty2 ) with the standard value of the coupling coefficient (C2) and the total cost function (H total ) the second penalty function (H penalty2 ) may be set as a relaxed value. The relaxed value is smaller than the standard value. That is, the first combinatorial optimization problem has a larger ratio of the penalty function to the cost function than the second combinatorial optimization problem.

[0239] Then, the formulation unit 36 ​​supplies the first combinatorial optimization problem and the second combinatorial optimization problem to the solver unit 22.

[0240] The solver unit 22 solves each of the first and second combinatorial optimization problems. Then, the solver unit 22 supplies the solutions of the first and second combinatorial optimization problems to the arbitration unit .

[0241] In the second modified example, the arbitration unit 38 executes the process according to the flow shown in FIG.

[0242] First, in S71, the arbitration unit 38 acquires the solution to the first combinatorial optimization problem output from the solver unit 22. Next, in S72, the arbitration unit 38 evaluates the acquired solution to the first combinatorial optimization problem. Specifically, the arbitration unit 38 evaluates whether or not the acquired solution to the first combinatorial optimization problem satisfies the constraint conditions.

[0243] Next, in S73, the arbitration unit 38 determines whether or not the solution to the first combinatorial optimization problem satisfies the constraint conditions based on the evaluation result. If the solution to the first combinatorial optimization problem does not satisfy the constraint conditions (No in S73), the arbitration unit 38 outputs an exception signal and ends this flow.

[0244] If the solution to the first combinatorial optimization problem satisfies the constraint conditions (Yes in S73), the arbitration unit 38 advances the process to S74.

[0245] In S74, the arbitration unit 38 obtains the solution to the second combinatorial optimization problem output from the solver unit 22. Then, in S75, the arbitration unit 38 evaluates the solution to the second combinatorial optimization problem. The arbitration unit 38 performs, for example, an evaluation process similar to S52 in Fig. 14. Note that the arbitration unit 38 may perform the processes of S74 and S75 in parallel with the processes of S71 and S72.

[0246] Next, in S76, the arbitration unit 38 determines whether or not the solution of the second combinatorial optimization problem satisfies the constraint conditions based on the evaluation result. If the solution of the second combinatorial optimization problem satisfies the constraint conditions (Yes in S76), an exception signal is output and this flow ends.

[0247] If the solution to the second combinatorial optimization problem does not satisfy the constraint conditions (No in S76), the arbitration unit 38 proceeds to S78. In S78, the arbitration unit 38 determines whether or not the solution to the second combinatorial optimization problem is a valid solution that violates the constraints, based on the evaluation result. If the solution to the second combinatorial optimization problem is not a valid solution that violates the constraints (No in S78), the arbitration unit 38 outputs an exception signal and ends this flow.

[0248] If the solution to the second combinatorial optimization problem is a valid constraint violating solution (Yes in S78), the arbitration unit 38 advances the process to S79.

[0249] In S79, the arbitration unit 38 identifies a constraint satisfying portion and a constraint violating portion in the valid constraint violating solution. Specifically, the arbitration unit 38 identifies the constraint violating portion based on the difference between the solution of the first combinatorial optimization problem and the solution of the second combinatorial optimization problem. Furthermore, the arbitration unit 38 identifies the solution of the first combinatorial optimization problem as the constraint satisfying portion.

[0250] Next, in S80, the arbitration unit 38 generates and outputs a constraint satisfaction control signal based on the constraint satisfaction portion and a constraint violation control signal based on the constraint violation portion. After completing the process of S80, the arbitration unit 38 ends this flow.

[0251] Fig. 21 is a diagram showing an example of a solution to the first combinatorial optimization problem. Fig. 22 is a diagram showing an example of a solution to the second combinatorial optimization problem. Fig. 23 is a diagram showing an example of a boundary box when the solutions in Figs. 21 and 22 are obtained.

[0252] The object tracking unit 30 according to the second modification obtains solutions to two combinatorial optimization problems with different coupling coefficients (C2) of penalty functions. A combinatorial optimization problem with a large coupling coefficient (C2) is less likely to have a constraint violating solution than a combinatorial optimization problem with a small coupling coefficient (C2). The object tracking unit 30 identifies a constraint satisfying portion using a solution to the first combinatorial optimization problem with a large coupling coefficient (C2). The object tracking unit 30 also identifies a constraint violating portion using a difference between a solution to the first combinatorial optimization problem with a large coupling coefficient (C2) and a solution to the second combinatorial optimization problem with a small coupling coefficient (C2). The object tracking unit 30 according to the second modification can easily identify a constraint violating portion and a constraint satisfying solution even when a valid constraint violating solution includes decision variables that are the cause of two or more constraint violations.

[0253] For example, as shown in Fig. 21 and Fig. 22, assume that five pieces of tracked object information are included in the internal state information, and three pieces of detected object information are detected based on the image data. In this case, as shown in Fig. 23, the tracked object at t2 intersects with the tracked object at t5, and the tracked object at t1 intersects with the tracked object at t3. The solution to the second combinatorial optimization problem shown in Fig. 22 is the column-wise sum (S C ) and the column-wise sum in the third column (d3) (S C ) are 2, which does not satisfy formula (13-1). Even in such a case, the arbitration unit 38 can determine that the decision variables of (t3, d2) and (t5, d3) are the causes of the constraint violation by detecting the difference between the solution of the first combinatorial optimization problem shown in FIG. 21 and the solution of the second combinatorial optimization problem shown in FIG. 22. Therefore, the arbitration unit 38 can output a constraint violation control signal that includes a potential match signal (potentially_matched) indicating the tracked object information of t3 and the tracked object information of t5.

[0254] In the second modified example, the solver unit 22 may include, for example, two Ising machines 52, and may simultaneously solve the first combinatorial optimization problem and the second combinatorial optimization problem in parallel. This allows the solver unit 22 to execute two solution-finding processes in parallel and output a solution in a short time. The solver unit 22 may also execute two solution-finding processes sequentially using one Ising machine 52. Such a solver unit 22 can reduce the size of the circuit configuration and suppress manufacturing costs, etc.

[0255] (Application to information processing systems) As described above, the object tracking device 14 according to this embodiment executes the following series of processes.

[0256] First, the formulation unit 36 ​​of the object tracking device 14 according to this embodiment generates a combinatorial optimization problem based on data under predetermined constraint conditions. For example, the object tracking device 14 generates a combinatorial optimization problem that minimizes a total cost function that adds a cost function and a penalty function. Furthermore, the solver unit 22 of the object tracking device 14 according to this embodiment acquires the generated combinatorial optimization problem, solves the acquired combinatorial optimization problem by allowing the constraint conditions not to be satisfied, and calculates a solution to the combinatorial optimization problem obtained by solving it.

[0257] Furthermore, the object tracking device 14 according to this embodiment generates a constraint violation control signal when the solution does not satisfy the constraint conditions. Then, the object tracking device 14 according to this embodiment executes a predetermined first process on the data based on the constraint violation control signal, and outputs the data on which the first process has been executed. Furthermore, the object tracking device 14 according to this embodiment executes a predetermined second process on the data based on the solution when the solution satisfies the constraint conditions, and outputs the data on which the second process has been executed.

[0258] Furthermore, when a solution does not satisfy the constraint conditions, the arbitration unit 38 of the object tracking device 14 according to this embodiment identifies a constraint violating portion of the solution that does not satisfy the constraint conditions and a constraint satisfying portion of the solution that satisfies the constraint conditions, and outputs a constraint violation control signal based on the constraint violating portion and a constraint satisfaction control signal based on the constraint satisfaction portion. When a solution does not satisfy the constraint conditions, the object tracking device 14 according to this embodiment performs a first process on the data based on the constraint violation control signal, outputs the data on which the first process has been performed, performs a second process on the data based on the constraint satisfaction control signal, and outputs the data on which the second process has been performed.

[0259] Furthermore, if the solution satisfies the constraint condition, the object tracking device 14 according to this embodiment further executes a predetermined second process on the data based on the solution, and outputs the data on which the second process has been executed.

[0260] An information processing system that includes a process for solving a combinatorial optimization problem based on data under predetermined constraint conditions while processing data is being performed can execute a series of processes as described above.

[0261] For example, a control device that analyzes large-scale point cloud data output from a LiDAR by solving a constrained combinatorial optimization problem and controls an object to be controlled based on the analysis results can execute the above series of processes.

[0262] Furthermore, for example, a wireless base station that solves a constrained combinatorial optimization problem for allocating wireless communication resources such as frequency channels and time slots to a large number of terminal devices and allocates wireless resources to the large number of terminal devices based on the solution of the constrained combinatorial optimization problem can execute the above series of processes.

[0263] Furthermore, for example, a control device in a wireless communication system that has multiple transmitting antennas and multiple receiving antennas and can use multiple frequency channels as wireless communication paths and selects an optimal wireless communication path by solving a constrained combinatorial optimization problem can execute the series of processes described above.

[0264] In addition, for example, a financial market analysis device that extracts the maximum independent set of one or more stocks that are weakly correlated with each other based on a graph consisting of nodes corresponding to multiple stocks and edges corresponding to the correlation between two stocks can execute the above series of processes.

[0265] Furthermore, for example, a control device for a large-scale computer system having a large number of computing nodes that processes a large number of jobs simultaneously, which dynamically allocates computing resources to jobs by solving a constrained combinatorial optimization problem, can execute a series of processes as described above.

[0266] (Hardware configuration) Fig. 24 is a diagram showing an example of a hardware configuration of the host unit 24. The host unit 24 is realized by, for example, a computer having a hardware configuration as shown in Fig. 24. The host unit 24 includes a CPU (Central Processing Unit) 301, a RAM (Random Access Memory) 302, a ROM (Read Only Memory) 303, a storage device 304, and a communication interface device 305. These units are connected by a bus.

[0267] The CPU 301 is a processor that executes arithmetic processing, control processing, etc. according to a program. The CPU 301 uses a predetermined area of ​​the RAM 302 as a working area and executes various processes in cooperation with programs stored in the ROM 303, the storage device 304, etc.

[0268] The RAM 302 is a memory such as a Synchronous Dynamic Random Access Memory (SDRAM), and functions as a work area for the CPU 301. The ROM 303 is a memory that stores programs and various information in a non-rewritable manner.

[0269] The storage device 304 is a device that writes and reads data to a semiconductor storage medium such as a flash memory, or a magnetically or optically recordable storage medium, etc. The storage device 304 writes and reads data to the storage medium in response to control from the CPU 301. The communication interface device 305 communicates with external devices via a network in response to control from the CPU 301.

[0270] A program executed by the computer causes the computer to function as the host unit 24. This program is deployed on the RAM 302 by the CPU 301 (processor) and executed.

[0271] In addition, the program executed by the computer is provided as a file in a format that can be installed on the computer or in a format that can be executed by the computer, and is recorded on a computer-readable recording medium such as a CD-ROM, a flexible disk, a CD-R, or a DVD (Digital Versatile Disk).

[0272] The program may be stored on a computer connected to a network such as the Internet and provided by being downloaded via the network. The program may be provided or distributed via a network such as the Internet. The program executed by the host unit 24 may be provided by being pre-installed in the ROM 303 or the like.

[0273] The solver unit 22 is realized by a reconfigurable semiconductor device such as a field-programmable gate array (FPGA). The solver unit 22 may be realized by a CPU, a microprocessor, a graphics processing unit (GPU), an application specific integrated circuit (ASIC), or an electronic circuit including these circuits. The solver unit 22 may be realized by an information processing device such as a computer, a computer system in which multiple computers or servers communicate with each other via a network, or a PC cluster in which multiple computers cooperate to execute information processing.

[0274] Furthermore, when the solver unit 22 is realized by a reconfigurable semiconductor device such as an FPGA, the circuit information (configuration data) written in the reconfigurable semiconductor device to operate the reconfigurable semiconductor device as the solver unit 22 may be stored on a computer connected to a network such as the Internet and provided by being downloaded via the network. Furthermore, the circuit information (configuration data) written in the reconfigurable semiconductor device to operate the reconfigurable semiconductor device as the solver unit 22 may be provided by being recorded on a computer-readable recording medium.

[0275] Furthermore, when the solver unit 22 is realized by a semiconductor device such as an ASIC, the circuit information representing the configuration of the circuit described in a hardware description language may be stored on a computer connected to a network such as the Internet and provided by being downloaded via the network in order to operate the semiconductor device such as an ASIC as the solver unit 22. Furthermore, in order to operate the semiconductor device such as an ASIC as the solver unit 22, the circuit information representing the configuration of the circuit described in a hardware description language may be provided by being recorded on a computer-readable recording medium.

[0276] The solver unit 22 may be incorporated in the host unit 24. For example, the solver unit 22 may be incorporated as an accelerator in part of the host unit 24. The solver unit 22 may also include some of the functions of the host unit 24, such as the formulation unit 36 ​​or the arbitration unit 38.

[0277] Although some embodiments of the present invention have been described, these embodiments are presented as examples and are not intended to limit the scope of the invention. These novel embodiments can be implemented in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their modifications are included in the scope and spirit of the invention, and are included in the scope of the invention and its equivalents described in the claims.

[0278] (Additional Note) The above-described embodiments can be summarized as the following technical solutions.

[0279] [Technical proposal 1] An information processing device that executes processing on data, generating a combinatorial optimization problem based on the data under predetermined constraints; solving the generated combinatorial optimization problem while allowing the constraint condition not to be satisfied, and calculating a solution to the combinatorial optimization problem obtained by solving the combinatorial optimization problem; generating a constraint violation control signal if the solution does not satisfy the constraints; performing a first predetermined process on the data based on the constraint violation control signal; Outputting the data on which the first process has been performed Information processing device.

[0280] [Technical proposal 2] In generating the combinatorial optimization problem, a combinatorial optimization problem is generated that minimizes a total cost function that is an addition of a cost function and a penalty function; the cost function includes a plurality of decision variables corresponding to the data as arguments, and is a linear or higher function of the plurality of decision variables; The penalty function is a function that includes at least a portion of the plurality of decision variables and that is minimized when the solution satisfies the constraints. An information processing device according to technical proposal 1.

[0281] [Technical proposal 3] In generating the constraint violation control signal, determining whether the solution to the combinatorial optimization problem satisfies the constraints; If the solution does not satisfy the constraint, output the constraint violation control signal. An information processing device according to Technical Proposal 2.

[0282] [Technical proposal 4] the constraint condition is expressed by one or more constraint equations, each of which is an equality or an inequality; In generating the constraint violation control signal, determining whether the solution satisfies the one or more constraints; If the solution does not satisfy the one or more constraints, outputting the constraint violation control signal. An information processing device according to Technical Proposal 2.

[0283] [Technical proposal 5] the constraint condition is expressed by one or more constraint equations, each of which is an equality or an inequality; In generating the constraint violation control signal, if the solution does not satisfy the one or more constraint equations and the number of the one or more constraint equations that do not satisfy the equations is equal to or less than a preset value, the constraint violation control signal is output. An information processing device according to Technical Proposal 2.

[0284] [Technical proposal 6] If a value obtained by substituting the solution into the penalty function is greater than the minimum value and is equal to or less than a preset value, the constraint violation control signal is output. An information processing device according to Technical Proposal 2.

[0285] [Technical proposal 7] moreover, If the solution satisfies the constraints, performing a second predetermined process on the data based on the solution; Outputting the data on which the second process has been performed An information processing device described in any one of technical proposals 2 to 6.

[0286] [Technical proposal 8] If the solution does not satisfy the constraints, identifying a constraint-violating portion of the solution that does not satisfy the constraint conditions and a constraint-satisfying portion of the solution that satisfies the constraint conditions; outputting the constraint violation control signal based on the constraint violation portion and a constraint satisfaction control signal based on the constraint satisfaction portion; performing the second operation on the data based on the constraint satisfaction control signal; executes the first process on data corresponding to the portion of the data that violates the constraint based on the constraint violation control signal; An information processing device described in Technical Proposal 7.

[0287] [Technical proposal 9] In identifying the constraint violating portion and the constraint satisfying portion, identifying a decision variable among the plurality of decision variables that is a cause of not satisfying the constraint condition based on the obtained solution, and changing a value of the decision variable that is a cause of not satisfying the constraint condition to identify the constraint satisfying portion; Extracting the values ​​of decision variables that cause the constraint conditions not to be satisfied and identifying the constraint violation portion An information processing device according to Technical Proposal 8.

[0288] [Technical proposal 10] In generating the combinatorial optimization problem, a first combinatorial optimization problem and a second combinatorial optimization problem are generated; The first combinatorial optimization problem has a larger ratio of the penalty function to the cost function than the second combinatorial optimization problem. In solving the solution, the solution is solved for each of the first combinatorial optimization problem and the second combinatorial optimization problem; In identifying the constraint violating portion and the constraint satisfying portion, identifying the solution to the first combinatorial optimization problem as the constraint satisfying portion; Identifying the constraint violation portion based on a difference between the solution of the first combinatorial optimization problem and the solution of the second combinatorial optimization problem. An information processing device according to Technical Proposal 8 or 9.

[0289] [Technical proposal 11] In generating the combinatorial optimization problem, a ratio of the penalty function to the cost function is changed according to the data on which the combinatorial optimization problem is based. An information processing device according to any one of technical proposals 2 to 10.

[0290] [Technical proposal 12] The information processing device is a device that tracks an object included in image data based on the image data, further storing one or more pieces of tracking object information representing one or more tracking objects being tracked prior to the first time; further acquiring one or more pieces of detected object information at the first time, the detected object information representing one or more detected objects included in the image data at the first time; generating the combinatorial optimization problem, generating a combinatorial optimization problem that associates, under the constraint condition, each of the one or more pieces of tracking object information at a first time with any of the one or more pieces of detected object information at the first time that is determined to represent the same object; In the execution of the first process, a threshold value for deleting the tracking object information specified based on the constraint violation control signal from among one or more pieces of tracking object information at the first time is increased by a predetermined value, when the elapsed time from the addition or modification is decreased by a predetermined value, or the threshold value for deleting the tracking object information specified based on the constraint violation control signal is exceeded, In the output of the data, the one or more pieces of tracking object information at the first time are output. An information processing device according to any one of technical proposals 1 to 11.

[0291] [Technical proposal 13] In generating the constraint violation control signal, a constraint satisfaction portion of the solution that satisfies the constraint condition and a constraint violation portion of the solution that does not satisfy the constraint condition are identified, and a constraint satisfaction control signal indicating the constraint satisfaction portion and a constraint violation control signal indicating the constraint violation portion are generated; Further, based on the constraint satisfaction control signal, the tracking object information corresponding to any of the one or more detected object information among the one or more tracking object information at the first time is corrected by the corresponding detected object information; Further, based on the constraint satisfaction control signal, adding detection object information that is not associated with any of the one or more tracking object information among the one or more detection object information at the first time to the one or more tracking object information at the first time as new tracking object information; Furthermore, based on the constraint satisfaction control signal, delete tracking object information whose elapsed time exceeds the threshold from among the one or more tracking object information at the first time. An information processing device according to technical proposal 12.

[0292] [Technical proposal 14] An information processing method for executing processing on data by an information processing device, comprising: The information processing device, generating a combinatorial optimization problem based on the data under predetermined constraints; solving the generated combinatorial optimization problem while allowing the constraint condition not to be satisfied, and calculating a solution to the combinatorial optimization problem obtained by solving the combinatorial optimization problem; generating a constraint violation control signal if the solution does not satisfy the constraints; performing a first predetermined process on the data based on the constraint violation control signal; Outputting the data on which the first process has been performed Information processing methods.

[0293] [Technical proposal 15] A program for causing an information processing device to function as an information processing device that executes processing on data, The information processing device, generating a combinatorial optimization problem for optimizing the data under predetermined constraints; solving the generated combinatorial optimization problem while allowing the constraint condition not to be satisfied, and calculating a solution to the combinatorial optimization problem obtained by solving the combinatorial optimization problem; generating a constraint violation control signal if the solution does not satisfy the constraints; performing a first predetermined process on the data based on the constraint violation control signal; Outputting the data on which the first process has been performed A program to make it function like this. [Explanation of symbols]

[0294] 10. Object Tracking System 12 Camera 14 Object Tracking Device 16 Planning Device 18 Control device 22 Solver section 24 Host Club 26 Object detection unit 28 Memory section 30 Object Tracking Unit 32 Acquisition Department 34 Prediction Department 36 Formulation part 38 Mediation Department 40 Correction section 42 Additional Part 44 Life extension section 46 Deleted section 48 Output section 50 Exception handling section 52 Ising Machine 54 Pretreatment section 56 Post-processing section

Claims

1. An information processing device that executes processing on data, generating a combinatorial optimization problem based on the data under predetermined constraints; solving the generated combinatorial optimization problem while allowing the constraint condition not to be satisfied, and calculating a solution to the combinatorial optimization problem obtained by solving the combinatorial optimization problem; generating a constraint violation control signal based on a portion of the solution that does not satisfy the constraint; performing a first predetermined process on the data based on the constraint violation control signal; Outputting the data on which the first process has been performed Information processing device.

2. Identifying the constraint violating portion and the constraint satisfying portion of the solution that satisfies the constraint conditions; outputting the constraint violation control signal based on the constraint violation portion and a constraint satisfaction control signal based on the constraint satisfaction portion; performing a second predetermined operation on the data based on the constraint satisfaction control signal; performing the first processing on the data based on the constraint violation control signal; The information processing device according to claim 1 .

3. In generating the combinatorial optimization problem, a combinatorial optimization problem is generated that minimizes a total cost function that is an addition of a cost function and a penalty function; the cost function includes a plurality of decision variables corresponding to the data as arguments, and is a linear or higher function of the plurality of decision variables; The penalty function is a function that includes at least a portion of the plurality of decision variables and that is minimized when the solution satisfies the constraints. The information processing device according to claim 2 .

4. In generating the constraint violation control signal, determining whether the solution to the combinatorial optimization problem satisfies the constraints; If the solution does not satisfy the constraint, output the constraint violation control signal. The information processing device according to claim 3 .

5. the constraints are expressed by one or more constraint equations, each of which is an equality or an inequality; In generating the constraint violation control signal, determining whether the solution satisfies the one or more constraints; If the solution does not satisfy the one or more constraints, outputting the constraint violation control signal. The information processing device according to claim 3 .

6. the constraints are expressed by one or more constraint equations, each of which is an equality or an inequality; In generating the constraint violation control signal, if the solution does not satisfy the one or more constraint equations and the number of the one or more constraint equations that do not satisfy the equations is equal to or less than a preset value, the constraint violation control signal is output. The information processing device according to claim 3 .

7. If a value obtained by substituting the solution into the penalty function is greater than the minimum value and is equal to or less than a preset value, the constraint violation control signal is output. The information processing device according to claim 3 .

8. In identifying the constraint violating portion and the constraint satisfying portion, identifying a decision variable among the plurality of decision variables that is a cause of not satisfying the constraint condition based on the obtained solution, and changing a value of the decision variable that is a cause of not satisfying the constraint condition to identify the constraint satisfying portion; Extracting the values ​​of decision variables that cause the constraint conditions not to be satisfied and identifying the constraint violation portion The information processing device according to claim 3 .

9. In generating the combinatorial optimization problem, a first combinatorial optimization problem and a second combinatorial optimization problem are generated; The first combinatorial optimization problem has a larger ratio of the penalty function to the cost function than the second combinatorial optimization problem. In solving the solution, the solution is solved for each of the first combinatorial optimization problem and the second combinatorial optimization problem; In identifying the constraint violating portion and the constraint satisfying portion, identifying the solution to the first combinatorial optimization problem as the constraint satisfying portion; Identifying the constraint violation portion based on a difference between the solution of the first combinatorial optimization problem and the solution of the second combinatorial optimization problem. The information processing device according to claim 3 .

10. In generating the combinatorial optimization problem, a ratio of the penalty function to the cost function is changed according to the data on which the combinatorial optimization problem is based. The information processing device according to claim 3 .

11. The information processing device is a device that tracks an object included in image data based on the image data, further storing one or more pieces of tracking object information representative of the one or more tracking objects being tracked prior to the first time; moreover, predicting the one or more pieces of tracking object information at the first time based on the one or more pieces of tracking object information prior to the first time; rewriting the one or more pieces of tracking object information stored before the first time with the one or more pieces of tracking object information predicted at the first time; acquiring one or more pieces of detected object information at the first time, the detected object information representing one or more detected objects included in the image data at the first time; generating the combinatorial optimization problem, generating the combinatorial optimization problem for associating, under the constraint condition, with each of the one or more pieces of tracked object information stored at the first time, any detected object information among the one or more pieces of detected object information at the first time that is determined to represent the same object; In the execution of the first process, a threshold value for a deletion when the elapsed time from the addition or modification of the tracking object information specified based on the constraint violation control signal among the one or more pieces of tracking object information stored at the first time is increased by a predetermined value, or the threshold value for a deletion when the elapsed time of the tracking object information specified based on the constraint violation control signal is exceeded is increased by a predetermined value, In the output of the data, the one or more pieces of tracking object information stored at the first time are output. The information processing device according to claim 2 .

12. Based on the constraint satisfaction control signal, among the one or more pieces of tracked object information stored at the first time, a piece of tracked object information corresponding to any one of the one or more pieces of detected object information is modified by the corresponding detected object information; adding, as new tracking object information, detection object information that is not associated with any of the one or more tracking object information among the one or more detection object information at the first time, to the one or more stored tracking object information at the first time, based on the constraint satisfaction control signal; deleting, from the one or more stored tracking object information at the first time, tracking object information whose elapsed time has exceeded the threshold, based on the constraint satisfaction control signal; based on the constraint satisfaction control signal, modifying tracking object information associated with any of the one or more detected object information among the one or more stored tracking object information at the first time by the associated detected object information; based on the constraint satisfaction control signal, adding detected object information that is not associated with any of the one or more tracking object information among the one or more detected object information at the first time to the one or more stored tracking object information at the first time as new tracking object information; Based on the constraint satisfaction control signal, delete tracking object information whose elapsed time exceeds the threshold from among the one or more tracking object information stored at the first time. The information processing device according to claim 11.

13. An information processing method for executing processing on data by an information processing device, comprising: The information processing device, generating a combinatorial optimization problem based on the data under predetermined constraints; solving the generated combinatorial optimization problem while allowing the constraint condition not to be satisfied, and calculating a solution to the combinatorial optimization problem obtained by solving the combinatorial optimization problem; generating a constraint violation control signal based on a portion of the solution that does not satisfy the constraint; performing a first predetermined process on the data based on the constraint violation control signal; Outputting the data on which the first process has been performed Information processing methods.

14. A program for causing an information processing device to function as an information processing device that executes processing on data, The information processing device, generating a combinatorial optimization problem for optimizing the data under predetermined constraints; solving the generated combinatorial optimization problem while allowing the constraint condition not to be satisfied, and calculating a solution to the combinatorial optimization problem obtained by solving the combinatorial optimization problem; generating a constraint violation control signal based on a portion of the solution that does not satisfy the constraint; performing a first predetermined process on the data based on the constraint violation control signal; Outputting the data on which the first process has been performed A program to make it function like this.

15. Circuit information describing a circuit configuration, the circuit information being written in a hardware description language, causing the circuit to function as an information processing device that performs processing on data; The information processing device includes: generating a combinatorial optimization problem based on the data under predetermined constraints; solving the generated combinatorial optimization problem while allowing the constraint condition not to be satisfied, and calculating a solution to the combinatorial optimization problem obtained by solving the combinatorial optimization problem; generating a constraint violation control signal based on a portion of the solution that does not satisfy the constraint; performing a first predetermined process on the data based on the constraint violation control signal; Outputting the data on which the first process has been performed Circuit information.

16. Circuit information to be written into a reconfigurable semiconductor device in order to operate the reconfigurable semiconductor device, comprising: causing the reconfigurable semiconductor device to function as an information processing device that executes processing on data; The information processing device includes: generating a combinatorial optimization problem based on the data under predetermined constraints; solving the generated combinatorial optimization problem while allowing the constraint condition not to be satisfied, and calculating a solution to the combinatorial optimization problem obtained by solving the combinatorial optimization problem; generating a constraint violation control signal based on a portion of the solution that does not satisfy the constraint; performing a first predetermined process on the data based on the constraint violation control signal; Outputting the data on which the first process has been performed Circuit information.

Images