Object verification system, object verification method and object verification program

Abstract

To provide an object verification system which can perform object verification so as to maintain precision while improving calculation speed.SOLUTION: An input unit accepts input of a plurality of maximum matching solutions between a first node in which each target object detected in a first frame is considered as a node of one group of a bipartite graph, and a second node in which each target object detected on the basis of the frame up to a second frame adjacent to the first frame is considered as a node of the other group of the bipartite graph. An integration unit integrates a second maximum matching solution into a first maximum matching solution selected from among the plurality of maximum matching solutions to generate a portion graph, and when the node with an order of 2 exists in at least one type of node among the first node and the second node in the generated portion graph, generates the graph in which an edge is replaced so that the order becomes 1, according to the weight of the edge of the portion graph, as an integrated solution.SELECTED DRAWING: Figure 18

Description

[Technical Field]

[0001] The present disclosure relates to an object matching system, an object matching method, and an object matching program that match objects present in multiple images. [Background technology]

[0002] It is common to track multiple objects, such as vehicles and people, from images captured by a single camera, or from images captured by multiple cameras. Since the computational complexity of object matching is polynomial time, it is known that the computation time increases as the number of objects to be detected increases. Therefore, it is desirable to be able to perform object matching that can maintain accuracy while improving computation speed.

[0003] For example, Patent Document 1 describes a method for performing object matching using a quantum computer. In the method described in Patent Document 1, the similarity between the position of a detected target object and the position of a target object predicted based on past matching results is transmitted, and a matching process is performed between the past matching results and the target object detected from an image. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] International Publication No. 2023 / 112658 Summary of the Invention [Problem to be solved by the invention]

[0005] In the method described in Patent Document 1, in order to resolve the instability of the quantum annealing solution, the first node corresponding to the second node is determined by majority vote of candidates for the first node corresponding to the second node. On the other hand, when multiple object matchings are performed, it is expected that integration by majority vote will not be able to improve accuracy. Therefore, even when multiple object matchings are performed, it is preferable to be able to maintain accuracy while improving calculation speed.

[0006] Therefore, an object of the present disclosure is to provide an object matching system, an object matching method, and an object matching program that can perform object matching so as to improve calculation speed while maintaining accuracy. [Means for solving the problem]

[0007] The object matching system according to the present disclosure comprises an input unit that accepts input of multiple maximal matching solutions between a first node, in which each target object detected in a first frame is a node in one group of a bipartite graph, and a second node, in which each target object detected based on frames up to a second frame adjacent to the first frame is a node in the other group of the bipartite graph; an integration unit that generates an integrated solution by integrating the multiple maximal matching solutions; and an output unit that outputs the results of the target object matching process based on the integrated solution, wherein the integration unit generates a subgraph by integrating a first maximal matching solution selected from the multiple maximal matching solutions with a second maximal matching solution selected from the multiple maximal matching solutions excluding the first maximal matching solution, and if there is a node with a degree of 2, which represents the number of edges set for one node, among at least one type of nodes among the first and second nodes in the generated subgraph, the integration unit generates a graph in which the edges are replaced so that the degree is 1 according to the weight of the edges of the subgraph.

[0008] The object matching method according to the present disclosure accepts input of multiple maximal matching solutions between a first node, in which each target object detected in a first frame is a node in one group of a bipartite graph, and a second node, in which each target object detected based on frames up to a second frame adjacent to the first frame is a node in the other group of the bipartite graph; generates an integrated solution by integrating the multiple maximal matching solutions; and outputs the results of the target object matching process based on the integrated solution.When generating the integrated solution, a subgraph is generated by integrating a first maximal matching solution selected from the multiple maximal matching solutions with a second maximal matching solution selected from the multiple maximal matching solutions excluding the first maximal matching solution; and if, among at least one type of nodes among the first and second nodes in the generated subgraph, there is a node with a degree of 2, which represents the number of edges set for one node, the edges are replaced so that the degree is 1 according to the weight of the edges in the subgraph, and the integrated solution is generated as a graph.

[0009] The object matching program according to the present disclosure causes a computer to execute an input process for accepting input of multiple maximal matching solutions between a first node, in which each target object detected in a first frame is a node in one group of a bipartite graph, and a second node, in which each target object detected based on frames up to a second frame adjacent to the first frame is a node in the other group of the bipartite graph; an integration process for generating an integrated solution by integrating the multiple maximal matching solutions; and an output process for outputting the results of the target object matching process based on the integrated solution; the integration process generates a subgraph by integrating a first maximal matching solution selected from the multiple maximal matching solutions with a second maximal matching solution selected from the multiple maximal matching solutions excluding the first maximal matching solution; and if, among at least one type of nodes among the first and second nodes in the generated subgraph, there is a node with a degree of 2, which represents the number of edges set for one node, the program generates a graph as the integrated solution in which the edges are replaced so that the degree is 1 according to the weight of the edges of the subgraph. [Effects of the Invention]

[0010] According to the present disclosure, object matching can be performed so as to improve calculation speed while maintaining accuracy. [Brief explanation of the drawings]

[0011] [Figure 1] 1 is a block diagram illustrating a configuration example of an embodiment of an object matching system according to the present disclosure. [Figure 2] FIG. 1 is an explanatory diagram showing an example of a situation in which an object matching system is used. [Figure 3] FIG. 1 is an explanatory diagram showing an example of a bipartite graph. [Figure 4] FIG. 10 is an explanatory diagram illustrating an example of multiplex matching. [Figure 5] FIG. 10 is an explanatory diagram showing an example of multiple maximal matching solutions obtained for each tracker. [Figure 6] FIG. 10 is an explanatory diagram illustrating an example of a process for generating an integrated solution. [Figure 7] 10 is a flowchart illustrating an example of the operation of the object matching system. [Figure 8] 10 is a flowchart illustrating an example of the operation of an integration unit. [Figure 9] FIG. 10 is a block diagram illustrating a configuration example of another embodiment of an object matching system according to the present disclosure. [Figure 10] FIG. 10 is an explanatory diagram showing an example of an image in which a position code is set. [Figure 11] FIG. 10 is an explanatory diagram showing an example of a subgraph including the results of matching between some nodes. [Figure 12] 10 is a flowchart illustrating an example of the operation of an execution instruction unit. [Figure 13] FIG. 10 is an explanatory diagram showing an example of a method for integrating by majority vote. [Figure 14] FIG. 10 is an explanatory diagram showing the effect of integrating multiple matchings by majority vote. [Figure 15] FIG. 10 is an explanatory diagram illustrating the effect of integration by the object matching system according to the present disclosure. [Figure 16] FIG. 10 is an explanatory diagram showing the effect of TTS. [Figure 17] FIG. 1 is an explanatory diagram showing the effect of reverse annealing. [Figure 18] FIG. 1 is a block diagram illustrating an overview of an object matching system according to the present disclosure. [Figure 19] FIG. 1 is a schematic block diagram illustrating the configuration of a computer according to at least one embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0012] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings.

[0013] Embodiment 1. 1 is a block diagram showing an example configuration of a first embodiment of an object matching system according to the present disclosure. The object matching system 100 of this embodiment includes an image capture device 10, an object matching device 20, a quantum computer 30, and a viewing terminal 40. The object matching device 20 is connected to the image capture device 10, the quantum computer 30, and the viewing terminal 40.

[0014] The quantum computer 30 is a computer that realizes parallel computation using quantum mechanical phenomena. For example, a quantum annealing quantum computer (hereinafter referred to as a quantum annealing machine) is a dedicated device that finds the ground state of the Hamiltonian of the Ising model and performs annealing based on the Ising model. More specifically, the quantum annealing machine is a device that probabilistically finds the value of a binary variable that minimizes or maximizes the objective function (i.e., Hamiltonian) of the Ising model, which takes binary variables as arguments. The binary variables may be realized by classical bits or quantum bits.

[0015] The quantum computer 30 of this embodiment may take any form. The quantum computer 30 may be configured with any hardware that probabilistically determines the value of a binary variable that minimizes or maximizes an objective function that takes binary variables as arguments. The quantum computer 30 may be, for example, a non-von Neumann computer in which the objective function is implemented by hardware in the form of an Ising model. Furthermore, the quantum computer 30 may be a quantum annealing machine or a general annealing machine.

[0016] The imaging device 10 is installed at a predetermined position and periodically captures an image of an object of a type to be matched (hereinafter referred to as a target object). The imaging device 10 then transmits the captured images to the object matching device 20. The imaging device 10 may transmit, for example, images captured as a video in units of frames to the object matching device 20.

[0017] In this embodiment, a case where one imaging device 10 is installed will be described. However, the number of imaging devices 10 is not limited to one, and may be two or more. The imaging device 10 may be connected to a switching hub having a PoE (Power over Ethernet) function, for example, and may be supplied with power from a LAN (Local Area Network) cable.

[0018] In this embodiment, a vehicle will be mainly used as an example of the target object. Fig. 2 is an explanatory diagram showing an example of a situation in which the object matching system 100 of this embodiment is used. Fig. 2 illustrates a situation in which an image capture device 10 is installed on a road on which a plurality of vehicles travel, and captures images of a plurality of vehicles that are target objects. The object matching system 100 then compares a vehicle in an image T1 captured at time t with a vehicle in an image T2 captured at time t+1, and determines whether they are the same vehicle.

[0019] However, the target object is not limited to a vehicle as long as it is a moving object, and may be, for example, a human being. Furthermore, the target object may be an object invisible to the human eye, such as a particle, as long as it can be captured as an image. If the target object is a particle, it may be possible to estimate the particle's trajectory, for example, in radiation therapy or experiments using accelerators.

[0020] The object matching device 20 of this embodiment includes a storage unit 21, an input unit 22, an object detection unit 23, an execution instruction unit 24, an integration unit 25, and an output unit .

[0021] The storage unit 21 stores various types of information used for processing by the object matching device 20. The storage unit 21 may store, for example, input images. The storage unit 21 is realized by, for example, a magnetic disk or the like.

[0022] The input unit 22 receives input of a plurality of images of a target object captured from the imaging device 10. The input unit 22 may store the received images in the storage unit .

[0023] The object detection unit 23 detects a target object from an image. Note that the method by which the object detection unit 23 detects a target object from an image is arbitrary, and a known method such as YOLO (You Only Look Once) v5 may be used. For example, the object detection unit 23 may output, as the detection result, the coordinates of the upper left vertex and the lower right vertex of a rectangle (i.e., a bounding box) surrounding the detected target object in the image, or the coordinates of the center of the bounding box.

[0024] The execution instruction unit 24 instructs the quantum computer 30 to execute a matching process for multiple target objects (hereinafter, sometimes referred to as multi-object tracking). In particular, in this embodiment, the multi-object tracking problem is formulated as a minimum-weight maximal matching problem in a bipartite graph (hereinafter, sometimes simply referred to as maximal matching). In a multi-object tracking situation, the degrees of each node in the bipartite graph are often unequal, but maximal matching is more likely to yield stable results, and maximal matching can be expressed by a QUBO (Quadratic Unconstrained Binary Optimization) formula.

[0025] Below, we will explain an example of how to express the maximal matching problem using the Ising model. U={u(i)} Ni t=1 Let V be the set of detected objects based on frames up to time t-1, and V = {v(j)} Mj t=1 Let be the set of target objects detected in the frame at time t. Time t is counted for each frame and takes an integer value. Then, we introduce a bipartite graph G = (U ∪ V, E) with edges only between u ∈ U and v ∈ V. Here, U ∩ V = φ (empty set), and E is the set of edges E = {(u(i),v(j))|u(i)∈U,v(j)∈V}.

[0026] Figure 3 is an explanatory diagram showing an example of a bipartite graph, in which the target object detected based on frames up to time t-1 is represented by u(i) in the upper part, and the target object detected in the frame at time t is represented by v(j) in the lower part.

[0027] In the multi-object tracking problem, when the correspondence between the same object between U and V is represented as M(⊆E), different e1, e2∈M (e1≠e2) do not share the same node, so the multi-object tracking problem can be considered as a type of matching problem in a bipartite graph G.

[0028] Furthermore, to quantitatively evaluate the quality of matching, we introduce a function w on the edge E, as shown in the example below. Assume that the similarity between objects u(i) and v(j) in adjacent frames is measured by some index, such as the distance between the objects or the similarity of image features. This index is w(u(i),v(j)) (the value range is 0≦w(u(i),v(j))≦1, and the higher the similarity, the larger the value of w). In this case, the multi-object tracking problem can be thought of as a maximum weight maximal matching problem that searches for a matching M, as shown in the example of Equation 1 below.

[0029]

number

[0030] Whether (u,v)∈E is included in the matching M is determined by a binary variable x u,v This is expressed as the following formula 2.

[0031]

number

[0032] In this case, the optimization problem shown in Equation 1 above is solved by solving the cost function F on the edge E. w Using the above, the optimization problem can be rewritten as shown in the example of Equation 4 under the constraint condition shown in Equation 3 below, which represents the matching constraint condition (the edges of the matching do not share the same node).

[0033]

number

[0034]

number

[0035] Here, the variable set x={x u,vEach |(u,v)∈E} can be considered as one of the sets of variables that minimizes Equation 5 shown below.

[0036]

number

[0037] A binary variable x is always x 2 = x, if constant terms are ignored, the above-exemplified formula 5 can be simplified to the following formula 6.

number

[0038] By using Lagrange's method of undetermined multipliers, under the constraint conditions shown in the above equation 3, the optimization problem shown in the above equation 4 can be summarized as shown in the following example equation 7 using the constraint equation shown in the above equation 6.

[0039]

number

[0040] In Equation 7, λ (>0) is a hyperparameter that adjusts the strength of the constraint equation, and is set to, for example, λ = 0.7. Since F(x) is a QUBO equation, it can be implemented in a quantum computer 30 to solve combinatorial optimization problems.

[0041] Furthermore, in this embodiment, P kinds of trackers {Tp}, each of which can be formulated as a maximal matching of a bipartite graph, are used. P p=1 Then, the set of target objects in the frame at time t-1 is U={u(i)} N i=1 , the set of target objects in the frame at time t is V={v(j)} M j=1 P-fold copies of each of these are given as an extended bipartite graph G^=(∪ P p=1 U P ,∪P p=1 V P ,∪ P p=1 E P ) Consider E P is U P ,V P It is a set of edges set only between E P The weight of w P is the tracker T P The weight w P is the tracker T P may be the same regardless of

[0042] In this case, we consider combining P types of tracking problems into one QUBO equation and solving it using quantum annealing. The process of combining P types of tracking problems can be called multiplexed matching. Multiplexed matching can be expressed as Equation 8, shown below.

[0043]

number

[0044] Fig. 4 is an explanatory diagram showing an example of multiplexed matching. In the example shown in Fig. 4, the bipartite graph shown in Fig. 3 is grouped into groups corresponding to the number of types of trackers (i.e., P types).

[0045] In this case, the target object v(j)∈V={v(j)} at time t M j=1 For the optimization problem exemplified in Equation 8 above, the optimal solution (x1, x2, ..., x P ) * Therefore, v(j)=v p The set of candidate objects shown in the following formula 9 associated with (j) is aggregated into one as exemplified in the following formula 10, and the target object u(i * ) with the target object v(j) at time t.

[0046]

number

[0047]

number

[0048] The execution instruction unit 24 transmits to the quantum computer 30 the first node, the second node, the similarity between the first node and the second node, and an Ising model that represents an optimization problem of maximal matching in a bipartite graph, and instructs the quantum computer 30 to execute a matching process for target objects. Specifically, in this embodiment, the execution instruction unit 24 transmits to the quantum computer 30 an Ising model that represents multiplexed matching of target objects performed by multiple types of trackers, and instructs the quantum computer 30 to execute multi-object tracking. This allows the execution instruction unit 24 to obtain a maximal matching solution for the bipartite graph.

[0049] That is, the execution instruction unit 24 calculates the distance between each target object (V={v(j)} Mj t=1 ) is one group of the bipartite graph G (hereinafter referred to as the first node), and each target object (U={u(i)} Ni t=1 ) is the other group of the bipartite graph G (hereinafter referred to as the second node), i (V i ,E i ) for each tracker i.

[0050] Fig. 5 is an explanatory diagram showing an example of multiple maximal matching solutions obtained for each tracker. In the example shown in Fig. 5, candidate edges between nodes are shown with dashed lines, and matched edges between nodes are shown with solid lines. The numbers written between nodes indicate examples of weights between nodes.

[0051] In this embodiment, multiple detectors each capable of formulating a bipartite graph are assumed, but the number of detectors may be one type. When one type of detector is used, when predicting the position of a target object based on past matching results, the period of the frame images used as the target may be changed (for example, by gradually increasing the period going back from time t-1) to obtain multiple maximal matching solutions. Note that the method described in Patent Document 1, for example, may be used as a method for predicting the position of a target object.

[0052] The past matching result refers to the result of identifying a target object by a matching process performed before the current time. In this case, "identification" does not necessarily mean identifying each individual target object, but refers to the process of associating target objects that are assumed to match between multiple images.

[0053] The integrating unit 25 receives input of multiple maximal matching solutions obtained by the execution instruction unit 24, and generates an integrated solution by integrating the received multiple maximal matching solutions. First, the integrating unit 25 selects one maximal matching solution (hereinafter referred to as the first maximal matching solution) from the obtained maximal matching solutions. Next, the integrating unit 25 selects another maximal matching solution (hereinafter referred to as the second maximal matching solution) from the maximal matching solutions excluding the first maximal matching solution.

[0054] The integrating unit 25 integrates the two selected maximal matching solutions (i.e., the first maximal matching solution and the second maximal matching solution) to generate a subgraph H of the bipartite graph as an integrated solution K. In this case, integration means integrating all solutions included in the two maximal matching solutions. Specifically, integration means integrating all edges of the bipartite graph represented by the maximal matching solutions.

[0055] In this case, there is a possibility that there is a node in the subgraph H whose degree, which indicates the number of edges set for one node, is 2. In this case, the integration unit 25 calculates the weight W of the edge of the subgraph. iDepending on the edge, replace the edge so that its degree becomes 1.

[0056] The integration unit 25 may replace the edges with an alternating path that minimizes the sum of the weights, in the same manner as searching for a maximum matching solution by, for example, searching for an increasing path, and set this as a new integrated solution K.

[0057] Next, the integration unit 25 integrates the unselected maximal matching solutions into the obtained integrated solution K to generate a new integrated solution K. That is, the integration unit 25 generates a subgraph so that the maximal matching solutions to be integrated do not overlap. Note that if the new integrated solution K also has a node with degree 2, the integration unit 25 calculates the edge weight W i According to the above, the edge is replaced so that the degree becomes 1. The replacement method is the same as the above method.

[0058] The integrating unit 25 performs the process of generating the integrated solution K on the remaining unselected maximal matching solutions in the same manner. When there are no more maximal matching solutions to integrate, the integrating unit 25 may perform a process of integrating the first maximal matching solutions that were initially selected to generate the integrated solution K (i.e., a cyclic process).

[0059] Fig. 6 is an explanatory diagram showing an example of the process of generating an integrated solution K. In the example shown in Fig. 6, the maximal matching solution shown as solution1 is first selected as the first maximal matching solution, and the maximal matching solution shown as solution2 is selected as the second maximal matching solution. Then, as a result of integrating these two maximal matching solutions, a node with degree 2 exists, and so the edges are replaced.

[0060] Furthermore, in the example shown in Figure 6, a new integrated solution is generated by integrating the maximal matching solution shown as solution3 into the integrated solution in which this edge has been replaced, and then the maximal matching solution shown as solution1 is cyclically integrated into this integrated solution to generate the final integrated solution.

[0061] The output unit 26 outputs the result of the matching process of the target object based on the integrated solution. In this embodiment, the output unit 26 transmits the result of the matching process to the viewing terminal 40 for display. Note that the output unit 26 may also output the result of the matching process to another device (not shown) or the like.

[0062] Specifically, output unit 26 may transmit to viewing terminal 40 the position information of the detected target object in association with information identifying the target object (e.g., an index). The position information at this time may be information converted into coordinates on a planar map such as that shown in Fig. 2. Furthermore, output unit 26 may output not only the position information and the index, but also information indicating the attributes of the target object obtained as a result of image recognition (e.g., the type of vehicle, such as a large vehicle, a compact vehicle, or a motorcycle).

[0063] The input unit 22, object detection unit 23, execution instruction unit 24, integration unit 25, and output unit 26 are realized by a computer processor (e.g., a CPU (Central Processing Unit), a GPU (Graphics Processing Unit)) that operates according to a program (object matching program).

[0064] For example, the program may be stored in storage unit 21 of object matching device 20, and the processor may read the program and operate as input unit 22, object detection unit 23, execution instruction unit 24, integration unit 25, and output unit 26 in accordance with the program. Furthermore, the functions of object matching device 20 may be provided in the form of SaaS (Software as a Service).

[0065] Furthermore, the input unit 22, object detection unit 23, execution instruction unit 24, integration unit 25, and output unit 26 may each be realized by dedicated hardware. Furthermore, some or all of the components of each device may be realized by general-purpose or dedicated circuits, processors, etc., or a combination thereof. These may be configured by a single chip, or by multiple chips connected via a bus. Some or all of the components of each device may be realized by a combination of the above-mentioned circuits, etc., and a program.

[0066] Furthermore, when some or all of the components of the object matching device 20 are realized by a plurality of information processing devices, circuits, etc., the plurality of information processing devices, circuits, etc. may be centrally or decentralized. For example, the information processing devices, circuits, etc. may be realized as a client-server system, a cloud computing system, or the like, in a form in which each is connected via a communication network.

[0067] Viewing terminal 40 visualizes the matching result based on the information transmitted from object matching device 20 (more specifically, output unit 26). More specifically, viewing terminal 40 visualizes the movement status of the target object. For example, if the target object is a vehicle, viewing terminal 40 may visualize the traffic flow status.

[0068] Next, the operation of the object matching system 100 of this embodiment will be described. FIG. 7 is a flowchart showing an example of the operation of the object matching system 100 of this embodiment. The input unit 22 accepts input of an image captured by the imaging device 10 (step S11). The object detection unit 23 detects a target object from the image (step S12). The execution instruction unit 24 transmits an Ising model representing multiplexed matching of target objects performed by multiple types of trackers to the quantum computer 30, causing it to perform multi-object tracking and obtain multiple maximal matching solutions (step S13). The integration unit 25 generates an integrated solution by integrating the multiple obtained maximal matching solutions (step S14). The output unit 26 outputs the result of the matching process indicated by the integrated solution (step S15).

[0069] 8 is a flowchart showing an example of the operation of the integrating unit 25 of this embodiment. The integrating unit 25 receives input of multiple maximal matching solutions between a first node, which has each target object detected in a first frame as one group of a bipartite graph, and a second node, which has each target object detected based on frames up to a second frame adjacent to the first frame as the other group of the bipartite graph (step S141). Here, it is assumed that input of P types of maximal matching solutions has been received.

[0070] First, the integration unit 25 selects one maximal matching solution M1 from the plurality of maximal matching solutions and sets it as an integrated solution K (step S142). Next, the integration unit 25 selects one maximal matching solution M1 from the P types of maximal matching solutions M i For (1≦i≦P), the following processes from step S143 to step S146 are repeated to obtain the final integrated solution K.

[0071] First, the integration unit 25 calculates the integrated solution K=(V K ,E K ) and the maximal matching solution M j (j:=i+1 mod P) and integrate them into a subgraph H of bipartite graph G. i =(V K ∪V j ,E K ∪E j ) is generated (step S143). Here, the subgraph H i If there is an edge including a node of degree 2 in K (Yes in step S144), the integration unit 25 replaces the edges so that the sum of the weights is minimized, and sets the replaced graph as a new integrated solution K (step S145).

[0072] On the other hand, the subgraph H i If there is no edge including a node of degree 2 in the subgraph H i is set as a new integrated solution K (step S146).

[0073] As described above, in this embodiment, the integrating unit 25 receives input of multiple maximal matching solutions between the first node and the second node, and generates an integrated solution by integrating the multiple maximal matching solutions. The output unit 26 then outputs the results of the target object matching process based on the integrated solution. At this time, the integrating unit 25 integrates the second maximal matching solution with the first maximal matching solution to generate a subgraph, and generates a graph in which nodes in the generated subgraph with a degree of 2 are replaced with edges in accordance with the weights of the edges of the subgraph so that they have a degree of 1. This makes it possible to perform object matching while improving calculation speed and maintaining accuracy.

[0074] Embodiment 2. Next, a second embodiment of the object matching system of the present disclosure will be described. In multi-object tracking, tracking processing is generally performed for each frame. However, because the position of an object changes gradually, depending on the situation, the tracking result is often the same as the result in the previous frame. Taking advantage of this property, this embodiment describes a method of further improving the efficiency of multi-object tracking processing by using reverse annealing. Note that, to make it easier to distinguish from reverse annealing, in this embodiment, the quantum annealing in the first embodiment will be referred to as forward annealing.

[0075] 9 is a block diagram showing a configuration example of a second embodiment of the object matching system of this embodiment. The object matching system 200 of this embodiment includes an image capturing device 10, an object matching device 120, a quantum computer 30, and a viewing terminal 40. The contents of the image capturing device 10, the quantum computer 30, and the viewing terminal 40 are the same as those of the first embodiment.

[0076] The object matching device 120 of this embodiment includes a storage unit 21, an input unit 22, an object detection unit 23, an execution instruction unit 124, an integration unit 25, and an output unit 26. That is, the object matching device 120 of this embodiment differs from the first embodiment in that it includes the execution instruction unit 124 instead of the execution instruction unit 24 of the first embodiment. Other configurations are the same as those of the first embodiment.

[0077] The execution instruction unit 124 instructs the quantum computer 30 to execute a matching process for a target object by reverse annealing. In this embodiment, the execution instruction unit 124 instructs the quantum computer 30 to execute a matching process for a plurality of target objects (i.e., multi-object tracking) by reverse annealing.

[0078] Since reverse annealing requires setting initial values, the execution instruction unit 124 sets the optimal solution candidate as the initial value. Specifically, the execution instruction unit 124 encodes each node u indicating the target object detected based on the frames up to time t-1 and each node v indicating the target object detected in the frame at time t as follows:

[0079] The execution instruction unit 124 divides the image of the frame at each time into regions. The region division method is arbitrary, and the execution instruction unit 124 may, for example, divide the image into regions vertically and horizontally. Then, the execution instruction unit 124 sets a position code for each divided region.

[0080] Fig. 10 is an explanatory diagram showing an example of an image to which a position code has been set. In the example shown in Fig. 10, the image is divided into four regions vertically and horizontally, and a position code is set for each region using a bit of 0 or 1 so that the position code is unique for each region.

[0081] Next, for each node u representing a target object, the execution instruction unit 124 predicts the position of the target object in the frame at time t based on the detection results based on the frames up to time t-1. Then, the execution instruction unit 124 assigns a position code c(u) according to the predicted position of the target object. For example, when the predicted position is represented by a bounding box using a Kalman filter or the like, the execution instruction unit 124 may use the center position of the bounding box as the predicted position.

[0082] Similarly, the execution instruction unit 124 assigns a position code c(v) to a node v indicating an object detected in the frame at time t, according to the position of the detected object.

[0083] Then, the execution instruction unit 124 constructs a subgraph of the bipartite graph including the result of matching nodes with matching position codes, and sets this subgraph as the initial value. That is, the execution instruction unit 124 constructs a subgraph in which u and v are matched only when c(u)=c(v), and sets this as the initial value.

[0084] Fig. 11 is an explanatory diagram showing an example of a subgraph including the results of matching some nodes. Fig. 11 illustrates a case where there are nodes u(1), u(2), and u(3) representing target objects detected based on frames up to time t-1, and nodes v(1), v(2), and v(3) representing target objects detected in the frame at time t. Fig. 11 also illustrates a case where nodes u(1), u(2), and u(3) are assigned location codes "1011," "1010," and "0011," respectively, and nodes v(1), v(2), and v(3) are assigned location codes "1010," "1100," and "1011," respectively.

[0085] At this time, the execution instruction unit 124 creates a subgraph including the result of matching node u(1) and node v(3), which have matching position codes, and node u(2) and node v(1). In Fig. 11, the edges between the matched nodes are shown by solid lines.

[0086] The execution instruction unit 124 provides the obtained subgraph as an initial value to the quantum computer 30, and instructs it to execute matching processing of multiple target objects (i.e., multi-object tracking) by reverse annealing. When using the above-mentioned formulas 7 and 8, it is sufficient to include formula 11 shown below.

[0087]

number

[0088] The input unit 22, the object detection unit 23, the execution instruction unit 124, the integration unit 25, and the output unit 26 are realized by a processor of a computer that operates according to a program (object matching program).

[0089] Next, the operation of the execution instruction unit 124 of this embodiment will be described. Fig. 12 is a flowchart showing an example of the operation of the execution instruction unit 124 of this embodiment. The execution instruction unit 124 predicts the position of the target object that is estimated at the time of the first frame from the position of the target object indicated by the second node (step S21). The execution instruction unit 124 assigns a position code to each region obtained by dividing the frame, according to the predicted position of the target object and the position of the target object indicated by the first node (step S22).

[0090] The execution instruction unit 124 constructs a subgraph of the bipartite graph including the result of matching the first node and the second node whose position codes match (step S23).Then, the execution instruction unit 24 instructs the quantum computer to execute reverse annealing using the constructed subgraph as an initial value (step S24).

[0091] As described above, in this embodiment, the execution instruction unit 124 predicts the position of the target object at the time of the first frame and assigns a position code according to the predicted position of the target object and the position of the target object indicated by the first node.The execution instruction unit 124 then constructs a subgraph of a bipartite graph including the results of matching first nodes and second nodes whose position codes match, and instructs the quantum computer to perform reverse annealing using this subgraph as an initial value.Therefore, in addition to the effects of the first embodiment, it is possible to increase the search speed while improving the accuracy of the solution with an annealing schedule that is shorter than that of forward annealing. [Example]

[0092] The present disclosure will be described below using specific examples, but the scope of the present disclosure is not limited to the contents described below.

[0093] This example explains the difference between a method for increasing the accuracy of a solution by integrating multiple pattern solutions of a combinatorial optimization problem by majority voting, and a method for replacing solutions according to edge weights as in the object matching system 100 of the present disclosure. Fig. 13 is an explanatory diagram showing an example of the method for integrating solutions by majority voting.

[0094] In the majority vote integration method, only the edge connectivity of each maximal matching solution is used, so when multiple maximal matching solutions shown in Figure 5 are integrated, the result shown in Figure 13 is obtained.

[0095] To compare the two, we artificially generated an NxN bipartite graph in which the order of all nodes was 4, with 10 sample patterns for each N, and verified the effectiveness of integrating multiple matchings by majority voting based on the error rate of the solution. Note that N is the number of target objects to be detected.

[0096] Fig. 14 is an explanatory diagram showing the effect of integrating multiple matchings by majority voting. The graph shown in Fig. 14 is the result of comparing the error rates for SA (Simulated Annealing) and QA (Quantum Annealing), where SA-k and QA-k respectively represent the integration of k-multiplex matching solutions. As shown in Fig. 14, when the matching multiplicity is 3 or more, the effect of integration is about the same as when the multiplicity is 2.

[0097] Fig. 15 is an explanatory diagram showing the effect of integration by the object matching system according to the present disclosure. The graph shown in Fig. 15 is a result of comparing error rates using the same samples as the experimental results shown in Fig. 14. In the case of double matching, the accuracy is similar to the results shown in Fig. 14, but it is shown that the effect of integration becomes apparent as the degree of matching multiplicity increases.

[0098] Furthermore, we introduced TTS (Time-to-solution) as an index to measure the efficiency of the algorithm for finding the optimal solution using quantum annealing, and measured the effectiveness of the object matching system disclosed herein. TTS is the ratio of the annealing time t to the target probability P of obtaining the true optimal solution at least once. R represents the average calculation time required to obtain the true optimal solution when is predetermined. The probability that one annealing at time t will be successful is P s When (t), TTS is defined by the following equation 12.

[0099]

number

[0100] Figure 16 is an explanatory diagram showing the effect of TTS. As shown in Figure 16, even when verified with TTS, it was shown that the effect of multiple matching is remarkable when using a method using quantum annealing, which does not increase the QPU (Quantum Processing Unit) access time.

[0101] Next, the effect of reverse annealing shown in the second embodiment will be explained. To explain the effect of reverse annealing, quantitative evaluation was performed using the same sample as the experimental results shown in FIG. 14, based on the error rate and TTS. FIG. 17 is an explanatory diagram showing the effect of reverse annealing. Note that RA-(1) and RA-(2) shown in FIG. 17 are the results when the annealing times per trial were 10 μs and 3 μs, respectively.

[0102] In this experiment, trial calculations were repeated 250 times per graph. We predicted that the initial value of reverse annealing would be incorrect by 5% on a quantum bit basis, and used a bit-flipping algorithm with a 5% probability as the correct answer for maximal matching.

[0103] As shown in Figure 17, if the initial values ​​can be predicted almost accurately, the accuracy rate of matching by reverse annealing is expected to be higher than that of forward annealing. Furthermore, by integrating the results of multiple matching, it is expected that an even better accuracy rate can be obtained.

[0104] Furthermore, the evaluation results using TTS show that the solution accuracy per unit time of reverse annealing is better than that of forward annealing, and that the correct solution can be obtained with fewer annealing calculation trials.

[0105] Next, an overview of the present disclosure will be described. Fig. 18 is a block diagram showing an overview of an object matching system according to the present disclosure. An object matching system 80 (e.g., object matching system 100) according to the present disclosure includes an input unit 81 (e.g., integrating unit 25) that receives input of multiple maximal matching solutions between a first node in which each target object detected in a first frame (e.g., the frame at time t) is a node of one group of a bipartite graph, and a second node in which each target object detected based on frames up to a second frame adjacent to the first frame (e.g., the frame at time t-1) is a node of the other group of the bipartite graph, an integrating unit 82 (e.g., integrating unit 25) that generates an integrated solution by integrating the multiple maximal matching solutions, and an output unit 83 (e.g., output unit 26) that outputs the result of the target object matching process based on the integrated solution.

[0106] The integration unit 82 generates a subgraph by integrating a first maximal matching solution selected from the multiple maximal matching solutions with a second maximal matching solution selected from the multiple maximal matching solutions excluding the first maximal matching solution, and if there is a node with a degree of 2, which represents the number of edges set for one node, among at least one type of nodes among the first nodes and second nodes in the generated subgraph, the integration unit 82 generates a graph as the integrated solution in which the edges are replaced so that the degree becomes 1, depending on the weight of the edges in the subgraph.

[0107] Such a configuration allows object matching to be performed in a manner that improves calculation speed while maintaining accuracy.

[0108] Furthermore, the object matching system 80 may include an execution instruction unit (e.g., execution instruction unit 24) that transmits to a quantum computer (e.g., quantum computer 30) the first node, the second node, the similarity between the first node and the second node, and an Ising model that represents an optimization problem of maximal matching in a bipartite graph, instructs the quantum computer to execute a matching process for the target object, and acquires a maximal matching solution of the bipartite graph.The input unit 81 may then receive an input of the maximal matching solution acquired by the execution instruction unit.

[0109] Specifically, the execution instruction unit may instruct the quantum computer to transmit an Ising model that represents multiplexed matching of target objects performed by multiple types of trackers and execute a matching process, thereby obtaining a maximal matching solution of the bipartite graph for each tracker.

[0110] In addition, the execution instruction unit (e.g., execution instruction unit 124) may predict the position of the target object estimated at the time of the first frame from the position of the target object indicated by the second node, assign a predetermined position code to each region obtained by dividing the frame according to the predicted position of the target object and the position of the target object indicated by the first node, construct a subgraph of a bipartite graph including the results of matching first nodes and second nodes whose position codes match, and instruct the quantum computer to perform reverse annealing using the subgraph as an initial value.

[0111] Furthermore, the integrating unit 82 may integrate an unintegrated maximal matching solution into the generated integrated solution to generate a new integrated solution.

[0112] Furthermore, the integration unit 82 may generate, as an integrated solution, a graph in which edges are replaced so that the sum of edge weights is minimized (for example, by searching for an increasing path).

[0113] Furthermore, the edge may be weighted based on the degree of similarity (for example, the distance between the objects or image features) between the target object indicated by the first node and the target object indicated by the second node.

[0114] 19 is a schematic block diagram showing the configuration of a computer according to at least one embodiment. The computer 1000 includes a processor 1001, a main memory device 1002, an auxiliary memory device 1003, and an interface 1004. The computer 1000 may also be connected to a computer that executes a mathematical programming solver, an annealing machine, a simulator, or the like.

[0115] The above-described object matching system 80 is implemented in a computer 1000. The operations of the above-described processing units are stored in the form of a program (object matching program) in an auxiliary storage device 1003. The processor 1001 reads the program from the auxiliary storage device 1003, loads it into the main storage device 1002, and executes the above-described processing in accordance with the program.

[0116] In at least one embodiment, the auxiliary storage device 1003 is an example of a non-transitory tangible medium. Other examples of non-transitory tangible media include a magnetic disk, a magneto-optical disk, a CD-ROM (Compact Disc Read-only memory), a DVD-ROM (Read-only memory), and a semiconductor memory connected via the interface 1004. In addition, when this program is distributed to the computer 1000 via a communication line, the computer 1000 that receives the program may load the program into the main storage device 1002 and execute the above processing.

[0117] The program may also be a program for realizing part of the above-described functions. Furthermore, the program may be a so-called differential file (differential program) that realizes the above-described functions in combination with another program already stored in the auxiliary storage device 1003.

[0118] A part or all of the above-described embodiments can be described as, but not limited to, the following supplementary notes.

[0119] (Supplementary Note 1) An input unit that receives input of a plurality of maximal matching solutions between a first node having each object detected in a first frame as a node of one group of a bipartite graph, and a second node having each object detected based on frames up to a second frame adjacent to the first frame as a node of the other group of the bipartite graph; an integration unit that generates an integrated solution by integrating the plurality of maximal matching solutions; an output unit that outputs a result of the matching process of the target object based on the integrated solution, The integration unit generates a subgraph by integrating a first maximal matching solution selected from the plurality of maximal matching solutions with a second maximal matching solution selected from the plurality of maximal matching solutions excluding the first maximal matching solution, and if there is a node with a degree of 2, which indicates the number of edges set for one node, among at least one type of nodes of the first node and the second node in the generated subgraph, generates a graph as an integrated solution in which the edges are replaced so that the degree becomes 1 according to the weight of the edges of the subgraph. 1. An object matching system comprising:

[0120] (Supplementary Note 2) An execution instruction unit is provided that transmits to a quantum computer a first node, a second node, a similarity between the first node and the second node, and an Ising model that represents an optimization problem of maximal matching in a bipartite graph, and instructs the quantum computer to execute a matching process of a target object, and obtains a maximal matching solution of the bipartite graph; The input unit receives an input of the maximal matching solution obtained by the execution instruction unit. 10. The object matching system of claim 1.

[0121] (Supplementary Note 3) The execution instruction unit instructs the quantum computer to transmit an Ising model representing multiplexed matching of target objects performed by multiple types of trackers and execute a matching process, and obtains a maximal matching solution of the bipartite graph for each tracker. 2. The object matching system of claim 1.

[0122] (Appendix 4) The execution instruction unit predicts the position of the target object estimated at the time of the first frame from the position of the target object indicated by the second node, assigns a preset position code to each region obtained by dividing the frame according to the predicted position of the target object and the position of the target object indicated by the first node, constructs a subgraph of the bipartite graph including the results of matching the first node and the second node whose position codes match, and instructs the quantum computer to perform reverse annealing using the subgraph as an initial value. 4. The object matching system of claim 2 or 3.

[0123] (Note 5) The integration unit integrates the unintegrated maximal matching solutions into the generated integrated solution to generate a new integrated solution. 5. The object matching system of any one of appendices 1 to 4.

[0124] (Note 6) The integration part generates a graph as an integrated solution by replacing edges so that the sum of edge weights is minimized. 6. The object matching system of any one of appendices 1 to 5.

[0125] (Note 7) The edge is weighted based on the similarity between the target object indicated by the first node and the target object indicated by the second node. 7. The object matching system of any one of Supplementary Note 1 to Supplementary Note 6.

[0126] (Supplementary Note 8) Accepting input of a plurality of maximal matching solutions between a first node having each object detected in a first frame as a node of one group of a bipartite graph and a second node having each object detected based on frames up to a second frame adjacent to the first frame as a node of the other group of the bipartite graph; generating an integrated solution by integrating the plurality of maximal matching solutions; outputting a result of the matching process of the target object based on the integrated solution; When generating the integrated solution, a subgraph is generated by integrating a first maximal matching solution selected from the plurality of maximal matching solutions with a second maximal matching solution selected from the plurality of maximal matching solutions excluding the first maximal matching solution, and if there is a node with a degree of 2, which indicates the number of edges set for one node, among at least one type of nodes of the first node and the second node in the generated subgraph, the edges are replaced in accordance with the weights of the edges of the subgraph so that the degree becomes 1, and a graph is generated as the integrated solution. 1. An object matching method comprising:

[0127] (Appendix 9) A quantum computer is instructed to execute a matching process for the target object by transmitting the first node, the second node, the similarity between the first node and the second node, and an Ising model representing an optimization problem of a maximal matching in a bipartite graph, and a maximal matching solution for the bipartite graph is obtained; Accepts the input of the obtained maximal matching solution. 9. The object matching method according to claim 8.

[0128] (Appendix 10) To the computer, an input process for receiving input of a plurality of maximal matching solutions between a first node having each object detected in a first frame as a node of one group of a bipartite graph, and a second node having each object detected based on frames up to a second frame adjacent to the first frame as a node of the other group of the bipartite graph; an integration process for generating an integrated solution by integrating the plurality of maximal matching solutions; and executing an output process for outputting a result of the matching process of the target object based on the integrated solution; In the integration process, a subgraph is generated by integrating a first maximal matching solution selected from the plurality of maximal matching solutions with a second maximal matching solution selected from the plurality of maximal matching solutions excluding the first maximal matching solution, and if there is a node with a degree of 2, which indicates the number of edges set for one node, among at least one type of nodes of the first node and the second node in the generated subgraph, a graph in which the edges are replaced so that the degree becomes 1 according to the weight of the edges of the subgraph is generated as an integrated solution. An object matching program for.

[0129] (Appendix 11) To the computer, transmits to the quantum computer the first node, the second node, the similarity between the first node and the second node, and an Ising model representing the optimization problem of maximal matching in the bipartite graph, instructs the quantum computer to execute a matching process for the target object, and executes an execution instruction process to obtain a maximal matching solution for the bipartite graph; In the input process, the input of the maximal matching solution obtained in the execution instruction process is accepted. 11. The object matching program according to claim 10. [Industrial Applicability]

[0130] The present disclosure is preferably applied to an object matching system that matches objects present in multiple images, and also to a system that utilizes a quantum computer to grasp real-time traffic volume and perform traffic control accordingly. [Explanation of symbols]

[0131] 10. Imaging device 20,120 Object matching device 21 Memory section 22 Input section 23 Object detection unit 24,124 Execution instruction section 25 Output section 30 Quantum Computer 40 Viewing terminals 100 Object Matching System

Claims

1. an input unit that receives input of a plurality of maximal matching solutions between a first node, which has each object detected in a first frame as a node of one group of a bipartite graph, and a second node, which has each object detected based on frames up to a second frame adjacent to the first frame as a node of the other group of the bipartite graph; an integration unit that generates an integrated solution by integrating the plurality of maximal matching solutions; an output unit that outputs a result of the matching process of the target object based on the integrated solution, The integration unit generates a subgraph by integrating a first maximal matching solution selected from the plurality of maximal matching solutions with a second maximal matching solution selected from the plurality of maximal matching solutions excluding the first maximal matching solution, and if there is a node with a degree of 2, which indicates the number of edges set for one node, among at least one type of nodes of the first node and the second node in the generated subgraph, generates a graph as an integrated solution in which the edges are replaced so that the degree becomes 1 according to the weight of the edges of the subgraph.

1. An object matching system comprising:

2. an execution instruction unit that transmits to a quantum computer a first node, a second node, a similarity between the first node and the second node, and an Ising model that represents an optimization problem of maximal matching in a bipartite graph, and instructs the quantum computer to execute a matching process of a target object, and obtains a maximal matching solution of the bipartite graph; The input unit receives an input of the maximal matching solution obtained by the execution instruction unit. The object matching system of claim 1 .

3. The execution instruction unit instructs the quantum computer to transmit an Ising model that represents multiplexed matching of the target object performed by a plurality of types of trackers and execute a matching process, and obtains a maximal matching solution of the bipartite graph for each of the trackers. The object matching system of claim 2 .

4. The execution instruction unit predicts the position of the target object estimated at the time of the first frame from the position of the target object indicated by the second node, assigns a preset position code to each region obtained by dividing the frame according to the predicted position of the target object and the position of the target object indicated by the first node, constructs a subgraph of the bipartite graph including the result of matching the first node and the second node whose position codes match, and instructs the quantum computer to execute reverse annealing using the subgraph as an initial value.

4. The object matching system according to claim 2 or 3.

5. The integration unit integrates the unintegrated maximal matching solutions into the generated integrated solution to generate a new integrated solution. The object matching system according to any one of claims 1 to 3.

6. The integration part generates a graph as an integrated solution by replacing edges so that the sum of edge weights is minimized. The object matching system according to any one of claims 1 to 3.

7. The edge is weighted according to the similarity between the object represented by the first node and the object represented by the second node. The object matching system according to any one of claims 1 to 3.

8. accepts input of a plurality of maximal matching solutions between a first node having each object detected in a first frame as a node of one group of a bipartite graph, and a second node having each object detected based on frames up to a second frame adjacent to the first frame as a node of the other group of the bipartite graph; generating an integrated solution by integrating the plurality of maximal matching solutions; outputting a result of the matching process of the target object based on the integrated solution; When generating the integrated solution, a subgraph is generated by integrating a first maximal matching solution selected from the plurality of maximal matching solutions with a second maximal matching solution selected from the plurality of maximal matching solutions excluding the first maximal matching solution, and if there is a node with a degree of 2, which indicates the number of edges set for one node, among at least one type of nodes of the first node and the second node in the generated subgraph, the edges are replaced in accordance with the weights of the edges of the subgraph so that the degree becomes 1, and a graph is generated as the integrated solution.

1. An object matching method comprising:

9. transmits to a quantum computer the first node, the second node, the similarity between the first node and the second node, and an Ising model representing an optimization problem of maximal matching in a bipartite graph, and instructs the quantum computer to execute a matching process for the target object, and obtains a maximal matching solution for the bipartite graph; Accepts the input of the obtained maximal matching solution. The object matching method according to claim 8.

10. On the computer, an input process for receiving input of a plurality of maximal matching solutions between a first node having each object detected in a first frame as a node of one group of a bipartite graph, and a second node having each object detected based on frames up to a second frame adjacent to the first frame as a node of the other group of the bipartite graph; an integration process for generating an integrated solution by integrating the plurality of maximal matching solutions; and executing an output process for outputting a result of the matching process of the target object based on the integrated solution; In the integration process, a subgraph is generated by integrating a first maximal matching solution selected from the plurality of maximal matching solutions with a second maximal matching solution selected from the plurality of maximal matching solutions excluding the first maximal matching solution, and if there is a node with a degree of 2, which indicates the number of edges set for one node, among at least one type of nodes of the first node and the second node in the generated subgraph, a graph in which the edges are replaced so that the degree becomes 1 according to the weight of the edges of the subgraph is generated as an integrated solution. An object matching program for.

Images