Collation device, collation method, and program
Patent Information
- Application Number
- PCT/JP2023/018944
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2023-05-22
- Publication Date
- 2024-11-28
Smart Images

Figure JP2023018944_28112024_PF_FP_ABST
Abstract
Description
Verification device, verification method, and program
[0001] The present invention relates to a matching device, a matching method, and a program for matching objects.
[0002] Techniques for matching objects contained in a plurality of images are known.
[0003] Patent Literature 1 discloses an object tracking method that sets an area in a frame where a moving object is captured as a tracking target area, and corrects the tracking target area of the next frame by pattern matching between the tracking target area set in a frame earlier than the frame to be processed and the next frame.
[0004] Japanese Patent Application Publication No. 2006-323437
[0005] The object tracking method described in Patent Document 1 does not take into account changes in the conditions under which a moving object is captured. For example, if the conditions under which a moving object is captured change between a processing target frame and the next frame, the correlation value of the moving object between frames decreases, resulting in a decrease in the accuracy of pattern matching. On the other hand, even if the correlation value is low, pattern matching results in a decrease in the recall rate. In other words, the object tracking method described in Patent Document 1 has a problem in that it is not possible to improve the recall rate while maintaining the accuracy of object matching.
[0006] One aspect of the present invention has been made in view of the above-mentioned problems, and an example of a purpose thereof is to provide a technique for improving the recall rate while maintaining the accuracy of object matching.
[0007] A matching device according to one aspect of the present invention includes an update means for updating parameters that define the frequency distribution of one or more indices relating to an object indicated by target data, a reliability calculation means for calculating the reliability of each of the indices using the parameters, and a matching means for matching the object based on the reliability.
[0008] A matching method according to one aspect of the present invention includes a matching device updating parameters that define the frequency distribution of one or more indicators related to an object indicated by target data, calculating a reliability for each of the indicators using the parameters, and matching the object based on the reliability.
[0009] A program according to one aspect of the present invention causes a computer to execute an update process that updates parameters that define the frequency distribution of one or more indicators related to an object indicated by target data, a reliability calculation process that calculates the reliability of each indicator using the parameters, and a matching process that matches the objects based on the reliability.
[0010] According to one aspect of the present invention, it is possible to improve the recall rate while maintaining the accuracy of object matching.
[0011] FIG. 1 is a block diagram showing the configuration of a matching device according to exemplary embodiment 1 of the present invention. FIG. 2 is a flow diagram showing the flow of a matching method according to exemplary embodiment 1 of the present invention. FIG. 3 is a block diagram showing the configuration of a matching device according to exemplary embodiment 2 of the present invention. FIG. 4 is a diagram showing an example of frequency distribution of indexes 1 to 3 in exemplary embodiment 2 of the present invention. FIG. 5 is a flow diagram showing the flow of a matching method according to exemplary embodiment 2 of the present invention. FIG. 6 is a flow diagram showing the flow of an object matching process in exemplary embodiment 2 of the present invention. FIG. 7 is a flow diagram showing the flow of a confidence calculation process in exemplary embodiment 2 of the present invention. FIG. 8 is a flow diagram showing the flow of a parameter update process in exemplary embodiment 2 of the present invention. FIG. 9 is a block diagram showing an example of the hardware configuration of a matching device according to each exemplary embodiment of the present invention.
[0012] [First Exemplary Embodiment] A first exemplary embodiment of the present invention will be described in detail with reference to the drawings. This exemplary embodiment is a basic form of the exemplary embodiments described below.
[0013] (Overview of the Verification Device 1) The verification device 1 according to this exemplary embodiment is a device that performs a verification process for one or more objects. Here, the verification device 1 can determine whether or not multiple objects correspond to each other. The verification device 1 can also compare each object with verification data and determine whether or not each object is an object that matches the verification data.
[0014] In this specification, when a first object and a second object are corresponding objects, these objects may be expressed as being "the same" or "matching each other." Also, when a first object and a second object are objects that should be considered to be the same, these objects may be expressed as being "the same" or "matching each other."
[0015] In the collation device 1, objects that match each other are managed by, for example, the same ID. For example, if an object detected in a frame and an object detected in the frame following the frame are determined to be the same object, the collation device 1 associates these objects with the same ID and manages them.
[0016] (Configuration of the Verification Device 1) The configuration of the verification device 1 according to this exemplary embodiment will be described with reference to Fig. 1. Fig. 1 is a block diagram showing the configuration of the verification device 1 according to this exemplary embodiment.
[0017] 1, the verification device 1 includes an update unit 11, a reliability calculation unit 12, and a verification unit 13. In this exemplary embodiment, the update unit 11, the reliability calculation unit 12, and the verification unit 13 are components that respectively realize an update means, a reliability calculation means, and a verification means.
[0018] The update unit 11 updates parameters that define the frequency distribution of one or more indices related to the object indicated by the target data, and supplies the updated parameters to the reliability calculation unit 12.
[0019] The target data is data that is the target of the matching process by the matching device 1. The target data is not limited to, but may include, for example, image data, audio data, point cloud data, and other sensing data.
[0020] Furthermore, the one or more indices relating to an object may indicate an attribute of the object, or may be a numerical evaluation of the attribute.
[0021] For example, when the target data is image data, examples of the multiple indices include the color, shape, and pattern of an object in the image represented by the image data.
[0022] The frequency distribution of an index is data that classifies the values of the index up to now into specific ranges and shows how many data fall into each range. For example, if the target data is image data containing multiple frames, data showing the relationship between the index values for objects included as subjects in each of 10 frames and the number of frames can be used.
[0023] Furthermore, the parameter that defines the frequency distribution of an index is a parameter used to calculate the reliability, which will be described later. In other words, the parameter that defines the frequency distribution of an index is a parameter that defines the variability of the index. An example of a parameter is a parameter that, when the index has a certain value, the reliability of the index becomes a predetermined value.
[0024] The reliability calculation unit 12 calculates the reliability of each index using the parameters supplied from the update unit 11. The reliability calculation unit 12 supplies the calculated reliability to the update unit 11.
[0025] The reliability is a value indicating the degree to which an index can be trusted in matching. An example of the reliability is a value indicating the difference between a certain object and a comparison target of the object. For example, assume that the matching device 1 executes a process of tracking an object included as a subject in a video. In this case, a reliability indicating the difference in a certain index between an object included as a subject in an image captured earlier in time and an object included as a subject in an image captured later in time is used in matching.
[0026] Furthermore, for example, when the matching device 1 executes a process of comparing pre-given matching data with an object, the reliability indicating the difference between the features indicated by the matching data and the features of the object is used in the matching.
[0027] Another example of a reliability is a value that indicates how similar an object is to an object to which it is compared.
[0028] The matching unit 13 matches the object based on the reliability supplied from the reliability calculation unit 12. For example, the matching unit 13 matches the object indicated by the target data by comparing the reliability with a threshold value.
[0029] As described above, the matching device 1 according to this exemplary embodiment has a configuration including an update unit 11 that updates parameters that define the frequency distribution of one or more indices related to the object indicated by the target data, a reliability calculation unit 12 that calculates the reliability of each index using the parameters supplied from the update unit 11, and a matching unit 13 that matches the object based on the reliability supplied from the reliability calculation unit 12.
[0030] In this manner, the matching device 1 according to the present exemplary embodiment updates the parameters for calculating the reliability used in matching and performs matching using the updated parameters. Therefore, the matching device 1 according to the present exemplary embodiment has the advantage of being able to improve the recall rate while maintaining the accuracy of object matching.
[0031] (Flow of Matching Method S1) The flow of matching method S1 according to this exemplary embodiment will be described with reference to Fig. 2. Fig. 2 is a flow chart showing the flow of matching method S1 according to this exemplary embodiment.
[0032] In step S11, the update unit 11 updates parameters that define the frequency distribution of one or more indices related to the object indicated by the target data. The update unit 11 supplies the updated parameters to the reliability calculation unit 12.
[0033] (Step S12) In step S12, the reliability calculation unit 12 calculates the reliability of each index using the parameters supplied from the update unit 11. The reliability calculation unit 12 supplies the calculated reliability to the update unit 11.
[0034] (Step S13 ) In step S13 , the matching unit 13 matches the objects based on the reliability supplied from the reliability calculation unit 12 .
[0035] As described above, the matching method S1 according to this exemplary embodiment employs a configuration including step S11 in which the update unit 11 updates parameters that define the frequency distribution of one or more indices related to the object indicated by the target data, step S12 in which the reliability calculation unit 12 calculates the reliability of each indices using the parameters supplied from the update unit 11, and step S13 in which the matching unit 13 matches the objects based on the reliability supplied from the reliability calculation unit 12. Therefore, according to the matching method S1 according to this exemplary embodiment, the same effects as those of the above-described matching device 1 can be obtained.
[0036]
[0033] A second exemplary embodiment of the present invention will be described in detail with reference to the drawings. Note that components having the same functions as those described in the first exemplary embodiment are denoted by the same reference numerals, and their description will be omitted as appropriate.
[0037] (Overview of the Verification Device 20) Similar to the above-described verification device 1, the verification device 20 is a device that performs verification processing on one or more objects. In this exemplary embodiment, as an example, the verification device 20 acquires image data including multiple frames as target data. The verification device 20 then determines whether one or more objects included as subjects in a certain frame (in other words, one or more objects indicated by the target data) match one or more objects included as subjects in the immediately preceding frame. Examples of objects include air bubbles included as subjects in an image of liquid in a bottle, and people included as subjects in an image of a street.
[0038] In the following, an object included as a subject in a certain frame will be referred to as a "candidate object," and an object included as a subject in a frame prior to the certain frame that is likely to correspond to the candidate object will be referred to as a "matching target object."
[0039] (Configuration of the Verification Device 20) Fig. 3 is a block diagram showing the configuration of the verification device 20 according to this exemplary embodiment. As shown in Fig. 3, the verification device 20 according to this exemplary embodiment includes a control unit 21, an input unit 22, an output unit 23, a communication unit 24, and a storage unit 25.
[0040] The input unit 22 accepts various inputs to the verification device 20. The input unit 22 supplies the control unit 21 with data indicating the accepted inputs.
[0041] Although the specific configuration of the input unit 22 does not limit the present exemplary embodiment, as an example, the input unit 22 may be configured to include input devices such as a keyboard and a touchpad. The input unit 22 may also be configured to include a data scanner that reads data via electromagnetic waves such as infrared rays or radio waves, and a sensor that senses the environmental state.
[0042] The output unit 23 is a functional block that outputs the processing result of the collation device 20. The output unit 23 outputs the data supplied from the control unit 21.
[0043] The specific configuration of the output unit 23 is not limited to this exemplary embodiment, but as an example, the output unit 23 is configured with a display, a speaker, a printer, etc. In this case, the output unit 23 displays various processing results by the verification device 20 on a screen or outputs them as voice or figures.
[0044] The communication unit 24 is an interface for connecting the verification device 20 to a network. The communication unit 24 transmits data supplied from the control unit 21 to other devices via the network, and supplies data transmitted from other devices via the network to the control unit 21.
[0045] The specific configuration of the network does not limit this exemplary embodiment, but as an example, a wireless LAN (Local Area Network), a wired LAN, a WAN (Wide Area Network), a public line network, a mobile data communication network, or a combination of these networks can be used.
[0046] The storage unit 25 stores data referenced by the control unit 21. The storage unit 25 is configured, for example, by a semiconductor memory device, etc. As an example, as shown in Fig. 3, the storage unit 25 stores image data including multiple frames fr that are target data, parameters pm for one or multiple indices related to the object, and a frequency distribution fd for one or multiple indices related to the object.
[0047] The parameters pm of the indices and the frequency distributions fd of the indices stored in the storage unit 25 are associated with each other. That is, a certain parameter pm stored in the storage unit 25 is a parameter pm that defines a certain frequency distribution fd associated with the certain parameter pm.
[0048] Furthermore, the frequency distribution fd stored in the storage unit 25 before the matching device 20 starts matching is a frequency distribution fd generated based on a plurality of frames acquired in advance and data related to objects included as subjects in each of the plurality of frames (e.g., the position, velocity, etc. of the object). The plurality of frames and data acquired in advance are also referred to as correct answer information. Furthermore, the frequency distribution fd generated from the correct answer information is also referred to as initial information. The process of generating the initial information will be described later.
[0049] Furthermore, the parameters pm stored in the storage unit 25 before the matching device 20 starts matching are parameters that define initial information. Note that the parameters pm and frequency distribution fd stored in the storage unit 25 before the matching device 20 starts matching are not limited, and may be, for example, predetermined parameters pm and predetermined frequency distribution fd.
[0050] When the matching device 20 performs matching and a predetermined condition is satisfied, the frequency distribution fd and the parameters pm that define the frequency distribution fd stored in the storage unit 25 are updated by the update unit 11, which will be described later.
[0051] (Functions of the control unit 21) The control unit 21 controls each component included in the verification device 20. As one example, the control unit 21 stores the frame fr supplied from the communication unit 24 in the storage unit 25. As another example, the control unit 21 supplies the verification result by the verification unit 13 (described later) to at least one of the output unit 23 and the communication unit 24.
[0052] Furthermore, the control unit 21 detects an object included as a subject in the frame fr. As one example, the control unit 21 may detect the object by image processing such as an existing object extraction process. As another example, the control unit 21 may detect the object using a model trained by machine learning to receive a frame as input and output an object included as a subject in the frame.
[0053] For example, object detection can be performed using the graph cut method. In the graph cut method, the boundary of the area that constitutes the foreground object image to be cut out is calculated from the color distribution and pixel color gradient of two types of images, namely, a foreground object image including the object to be cut out and a background image. Then, the image is cut out along the calculated boundary, thereby extracting the foreground object image to be cut out.
[0054] The control unit 21 assigns a rectangular frame (bounding box) to each detected object to indicate that it has been detected. The control unit 21 also assigns an identification number (hereinafter also referred to as "ID") to each object extracted from the frame fr. As an example, the control unit 21 assigns an ID in association with the bounding box.
[0055] 3, the control unit 21 includes an update unit 11, a reliability calculation unit 12, a matching unit 13, a frequency distribution generation unit 211, and a degree calculation unit 212. In this exemplary embodiment, the update unit 11, the reliability calculation unit 12, the matching unit 13, the frequency distribution generation unit 211, and the degree calculation unit 212 respectively function as an update means, a reliability calculation means, a matching means, an initial information generation means, and a degree calculation means.
[0056] The update unit 11 calculates parameters pm that define a frequency distribution fd of indices related to objects included as subjects in the frame fr. The objects, indices, frequency distribution fd, and parameters pm are as described above.
[0057] As an example, the update unit 11 calculates the parameter pm from the initial information stored in the storage unit 25. The update unit 11 stores the calculated parameter pm in the storage unit 25.
[0058] As another example, the update unit 11 updates the parameters pm stored in the storage unit 25 from parameters defining the past frequency distribution fd (including initial information) stored in the storage unit 25 to parameters pm defining the current frequency distribution fd. Furthermore, the update unit 11 updates the past frequency distribution fd stored in the storage unit 25 to the current frequency distribution fd. The past frequency distribution and the current frequency distribution will be described later.
[0059] Furthermore, when a predetermined condition is satisfied, the update unit 11 may update the parameter pm. As an example, when the matching unit 13 described later performs object matching by removing the reliability of at least one index from among the multiple indexes, the update unit 11 may update the parameter pm that defines the frequency distribution of the removed at least one index.
[0060] Furthermore, if the degree calculated by the degree calculation unit 212 described later is smaller than a threshold value, the update unit 11 may update the parameter pm from the parameter pm that determines the past frequency distribution to the parameter pm that determines the current frequency distribution.
[0061] An example of the processing executed by the update unit 11 will be described with reference to Fig. 4. Fig. 4 is a diagram showing an example of the frequency distribution fd of the indices 1 to 3 in this exemplary embodiment.
[0062] Fig. 4 shows a frequency distribution fd of indices 1 to 3, each of which has a different mean value and standard deviation. In the frequency distribution fd shown in Fig. 4, the horizontal axis represents the difference between the index value of the object to be matched and the index values of the candidate objects over multiple frames (e.g., 10 frames). The vertical axis represents the frequency (number of frames).
[0063] As an example, the update unit 11 calculates the reliability (e -kx ) becomes 0.6. In this way, the update unit 11 updates the parameter k so that the reliability becomes a predetermined value when the difference is a certain value ("x" in the above example), thereby making it possible to suppress the variation among the indexes 1 to 3, which have different frequency distributions.
[0064] The reliability calculation unit 12 calculates the reliability (first reliability and second reliability) for each index using the parameter pm stored in the storage unit 25. The reliability calculation unit 12 supplies the calculated second reliability to the matching unit 13. As shown in FIG. 3 , the reliability calculation unit 12 includes a first reliability calculation unit 121 and a second reliability calculation unit 122.
[0065] The first reliability calculation unit 121 calculates a first reliability for one or more indicators related to a candidate object included as a subject in the frame fr. For example, the first reliability calculation unit 121 calculates the difference x between the color, size, position, and speed of the indicators of the candidate object included as a subject in the frame fr and the color, size, position, and speed of a matching target object to which the candidate object is compared, and calculates a reliability (e -kx , k is the value of the parameter pm) is defined as a first reliability. The first reliability calculation unit 121 supplies the calculated first reliability to the second reliability calculation unit 122.
[0066] The second reliability calculation unit 122 calculates one or more second reliabilities from one or more first reliabilities. As an example, the second reliability calculation unit 122 calculates the second reliabilities by applying a predetermined calculation to the one or more first reliabilities. The second reliability calculation unit 122 supplies the calculated second reliabilities to the matching unit 13.
[0067] For example, the second reliability calculation unit 122 may calculate, as the second reliability, a product of some of the multiple first reliabilities calculated by the first reliability calculation unit 121. In this case, the second reliability calculation unit 122 calculates multiple second reliabilities by excluding at least one first reliabilities from the multiple first reliabilities calculated by the first reliability calculation unit 121.
[0068] For example, if the first reliability regarding the color of the object is r11, the first reliability regarding the shape of the object is r12, and the first reliability regarding the pattern of the object is r13, the second reliability calculation unit 122 calculates the following second reliability R21, R22, and R23.
[0069] R21=r11*r12, R22=r11*r13, R23=r12*r13 Furthermore, the second reliability calculation unit 122 may calculate, as the second reliability, a value obtained by multiplying the multiple first reliabilities calculated by the first reliability calculation unit 121 after calculating a predetermined exponent. In this case, the second reliability calculation unit 122 calculates multiple second reliabilities by applying multiple combinations of exponents to the multiple first reliabilities.
[0070] In addition, when the first reliability calculation unit 121 calculates one first reliability, the second reliability calculation unit 122 may use the first reliability calculated by the first reliability calculation unit 121 as the second reliability.
[0071] The matching unit 13 matches objects based on the second reliability supplied from the reliability calculation unit 12 (second reliability calculation unit 122). As an example, the matching unit 13 matches objects by comparing the second reliability with a threshold. As another example, if the second reliability is the above-mentioned second reliability R21, R22, or R23, the matching unit 13 matches objects by comparing any of the second reliability R21, R22, or R23 with the threshold.
[0072] If the second reliability is higher than a threshold, the matching unit 13 outputs a matching result indicating that the candidate object corresponds to the matching target object. As an example, the matching unit 13 outputs a matching result in which an ID is associated with a bounding box containing the object.
[0073] The frequency distribution generation unit 211 generates a frequency distribution fd from the frame fr. As an example, the frequency distribution generation unit 211 generates initial information, which is a past frequency distribution, from past frames before the matching device 20 performs matching.
[0074] For example, the frequency distribution generation unit 211 calculates values of the position error, size error, speed error, etc. of the object between each frame from correct answer information, which is data on multiple past frames and objects.The frequency distribution generation unit 211 then calculates values of the position error, size error, speed error, etc. for each index, and generates initial information, which is the average value, standard deviation, etc., and a frequency distribution fd.The frequency distribution generation unit 211 stores the generated initial information in the storage unit 25.
[0075] With this configuration, the matching device 20 can set an appropriate parameter pm as an initial value before starting matching.
[0076] As another example, after matching, the frequency distribution generation unit 211 generates a current frequency distribution fd including index values related to the candidate object. For example, the frequency distribution generation unit 211 generates, as the current frequency distribution fd, a frequency distribution fd for the most recent 10 frames including a frame that includes the candidate object as a subject. Specifically, the frequency distribution generation unit 211 adds the value of the candidate object to the frequency distribution for the nine frames immediately preceding the frame that includes the candidate object as a subject to generate the current frequency distribution fd for the 10 frames. The frequency distribution generation unit 211 supplies the generated frequency distribution fd of the target data to the update unit 11 and the degree calculation unit 212.
[0077] The degree calculation unit 212 calculates a degree indicating the strength of correlation between a past frequency distribution and a current frequency distribution for at least one index removed by the second reliability calculation unit 122. The degree calculation unit 212 calculates a degree indicating the strength of correlation between the frequency distribution fd stored in the storage unit 25 and the latest frequency distribution fd supplied from the frequency distribution generation unit 211.
[0078] As an example, the degree calculation unit 212 calculates a degree indicating the strength of correlation using the mean value and standard deviation of the frequency distribution fd stored in the storage unit 25 and the mean value and standard deviation of the latest frequency distribution fd supplied from the frequency distribution generation unit 211. The degree calculation unit 212 also calculates the effect amount of the r-family as the strength of correlation. The effect amount of the r-family is a normalized value and can be used even in situations where the number of data samples is small. The degree calculation unit 212 supplies the calculated degree to the update unit 11.
[0079] (Flow of the verification method S2 executed by the verification device 20) The flow of the verification method S2 executed by the verification device 20 will be described with reference to Fig. 5. Fig. 5 is a flow diagram showing the flow of the verification method S2 according to this exemplary embodiment.
[0080] (Step S21) In step S21, the frequency distribution generating unit 211 generates a frequency distribution fd, which is initial information for each index, and stores the initial information for each index in the storage unit 25.
[0081] (Step S22) In step S22, the update unit 11 calculates the parameter pm of each index from the initial information of each index stored in the storage unit 25. The update unit 11 stores the calculated parameter pm in the storage unit 25.
[0082] (Step S23) In step S23, the control unit 21 acquires image data including a plurality of frames fr and stores the image data in the storage unit 25.
[0083] (Step S24) In step S24, the control unit 21 sets the value of the variable t to an initial value.
[0084] (Step S25) In step S25, the control unit 21 detects an object in the t-th frame. The specific object detection process has been described above, and therefore will not be described here.
[0085] (Step S26) In step S26, the control unit 21 sets a matching target object in the t-1th frame. Here, the matching target object is an object to be matched from among the objects displayed in the image of the t-1th frame. Note that if there are multiple matching target objects in the image of the t-1th frame, the processing described below is repeatedly executed for each matching target object.
[0086] (Step S27) In step S27, the control unit 21 executes an object matching process, which will be described in detail later with reference to FIG.
[0087] (Step S28) In step S28, the control unit 21 determines whether or not the image data acquired in step S23 has a next frame.
[0088] (Step S29) If it is determined in step S28 that there is a next frame (step S28: YES), in step S29, the control unit 21 sets the value of the variable t to t + 1. Thereafter, the control unit 21 executes the process of step S25.
[0089] (Step S30) If it is determined in step S28 that there is no next frame (step S28: NO), the control unit 21 outputs the collation result.
[0090] In this way, in the matching method S2, it is determined whether or not an object included as a subject in the t-th frame corresponds to an object included as a subject in the t-1-th frame.
[0091] (Flow of Object Matching Process) The flow of the object matching process in step S26 described above will be described with reference to Fig. 6. Fig. 6 is a flow diagram showing the flow of the object matching process in this exemplary embodiment.
[0092] (Step S261) In step S261, the control unit 21 sets a candidate object from among the objects in the t-th frame. The candidate object is an object that is likely to be the same as the object to be matched detected in the processing of step S25 in FIG. 5. As an example, the control unit 21 calculates the coordinates of the position where the object to be matched, which is included as a subject in the image of the t-1-th frame, is detected. Then, the control unit 21 sets, as the candidate object, an object located within a certain distance from the coordinates in the t-th frame.
[0093] (Step S262) In step S262, the reliability calculation unit 12 executes a reliability calculation process. The reliability calculation unit 12 supplies the second reliability calculated in step S262 to the matching unit 13 as a matching reliability. Details of the reliability calculation process will be described later with reference to FIG. 7.
[0094] (Step S263) In step S263, the update unit 11 executes a parameter update process, which will be described in detail later with reference to FIG.
[0095] (Step S264) In step S264, the matching unit 13 determines whether the matching reliability is equal to or greater than a threshold value.
[0096] (Step S265) If it is determined in step S264 that the reliability for matching is not equal to or greater than the threshold value (step S264: NO), in step S265, the control unit 21 determines whether or not there is a next candidate object.
[0097] If it is determined in step S265 that there is a next candidate object (step S265: YES), the next candidate object is set in step S261.
[0098] On the other hand, if it is determined in step S265 that there is no next candidate object (step S265: NO), the control unit 21 ends the object matching process.
[0099] (Step S266) If it is determined in step S264 that the reliability for matching is equal to or greater than the threshold value (step S264: YES), in step S266, the control unit 21 assigns the same ID as the object to be matched to the candidate object.
[0100] In this way, in the object matching process, the parameters pm updated in the parameter update process are used to perform the object matching process for the next candidate object.
[0101] (Flow of Reliability Calculation Processing) FIG. 7 is a flow diagram showing the flow of reliability calculation processing in this exemplary embodiment.
[0102] (Step S2621) In step S2621, the first reliability calculation unit 121 analyzes the matching target object and the candidate object for each index. Here, the index may be the color, shape, pattern, or speed of the object. Alternatively, the index may be the position, size, acceleration, or the like of the object.
[0103] When the indicators are the color, shape, pattern, and speed of the object, the first reliability calculation unit 121 calculates, for example, the average value of pixel values of the target object and the candidate object for the indicator "color." Furthermore, for example, the first reliability calculation unit 121 obtains, for example, a shape that approximates the contours of the target object and the candidate object for the indicator "shape." Furthermore, for example, the first reliability calculation unit 121 detects edges in the target object and the candidate object for the indicator "pattern."
[0104] Furthermore, for example, with regard to the indicator "speed," the distance and direction of movement of the target object and the candidate object between one frame are identified. Note that the distance and direction of movement of the target object are identified in advance by referring to the frame of the target object and the frame immediately preceding that frame, and the speed of the target object is also calculated. Assuming that the target object is the same as the candidate object, the distance and direction of movement of the candidate object are found.
[0105] Note that information for specifying what to use as an index and how to calculate the first reliability (hereinafter also referred to as "information related to the index") may be set in advance or may be input via the communication unit 24 or the input unit 22. Alternatively, information related to the index may be provided as metadata or the like included in the image data as the target data.
[0106] As an example, index information describing information about the index is stored in the storage unit 25, and the first reliability calculation unit 121 calculates the first reliability by referring to the index information stored in the storage unit 25. The first reliability calculation unit 121 acquires information about the index, and calculates the first reliability based on the acquired information about the index.
[0107] (Step S2622) In step S2622, the first reliability calculation unit 121 calculates a first reliability for each index. For example, if the indexes are the color, shape, pattern, and speed of an object, the first reliability calculation unit 121 calculates the first reliability from the differences between the color, shape, pattern, and speed of the object to be matched and the candidate object. Note that since there are multiple indexes, multiple (e.g., n) first reliabilities are calculated in step S2622.
[0108] (Step S2623) In step S2623, the second reliability calculation unit 122 removes any m reliabilities from the n first reliabilities calculated in step S2622, where m is a natural number less than n.
[0109] (Step S2624) In step S2624, the second reliability calculation unit 122 n C m For example, if the number of indices is 4 and one reliability is removed in step S2622, n is 4 and m is 1, so 4 C 1 = 4 combinations will be obtained.
[0110] More specifically, it is assumed that first reliability a, first reliability b, first reliability c, and first reliability d are calculated for each of index A, index B, index C, and index D. In this case, the four pairs obtained in step S2624 are (a, b, c), (a, b, d), (a, c, d), and (b, c, d).
[0111] (Step S2625) In step S2625, the second reliability calculation unit 122 calculates the second reliability by calculating the product of the first reliability included in each pair obtained in step S2624. In the above example, four second reliability values are calculated: M1 = a*b*c, M2 = a*b*d, M3 = a*c*d, and M4 = b*c*d. Here, "*" indicates an operation representing a product.
[0112] In this way, the second reliability calculation unit 122 calculates the first reliability by removing any m (m is a natural number less than n) first reliabilities from n (n is a natural number equal to or greater than 2) first reliabilities. n C m For each set, a second confidence measure for the set is calculated by taking the product of all the first confidence measures contained in the set.
[0113] (Step S2626) In step S2626, the matching unit 13 sets the maximum value max(M1, M2, M3, M4) of the second reliabilities calculated in step S2625 as the matching reliability. That is, the matching unit 13 matches the objects based on the second reliabilities having the maximum value among the multiple second reliabilities.
[0114] In this way, in the reliability calculation process, the second reliability is calculated from the first reliability calculated using the updated parameters.
[0115] (Another Example of Second Reliability Calculation Method) In the example described above, it has been explained that, in step S2623, any m reliabilities are removed from the n first reliabilities calculated in step S2622. However, instead of removing any m reliabilities, the second reliability calculation unit 122 may perform an operation on any m (m is a natural number less than n) first reliabilities, using a value greater than 0 and less than 1 as an exponent.
[0116] For example, by performing a power operation with an exponent of 0, the value of the first reliability becomes 1, so when calculating the product of the first reliability, an operation equivalent to removing the first reliability with an exponent of 0 can be performed.
[0117] That is, the first reliability calculation unit 121 performs an operation to use an exponent greater than 0 and less than 1 for any m (m is a natural number less than n) first reliabilities out of n (n is a natural number equal to or greater than 2) first reliabilities, n C m For each of the pairs, the matching unit 13 calculates the second reliability for the pair by taking the product of all the first reliability values included in the pair. n C m The object may be matched based on the second confidence score having the largest value among the second confidence scores for each of the pairs.
[0118] Specifically, it is assumed that first reliability a, first reliability b, first reliability c, and first reliability d are calculated for each of index A, index B, index C, and index D. In this case, the second reliability calculation unit 122 may set (the exponent to be applied to a, the exponent to be applied to b, the exponent to be applied to c, and the exponent to be applied to d) to (1, 1, 1, α), (1, 1, α, 1), (1, α, 1, 1), and (α, 1, 1, 1).
[0119] Here, α is a real number greater than 0 and less than 1. The second reliability calculation unit 122 may then perform exponentiation using these exponents on the corresponding first reliabilities, and calculate the second reliability by taking the product of the first reliabilities after the exponentiation.
[0120] More specifically, the second reliability calculation unit 122 calculates M1=a*b*c*d α M2 = a * b * c α * d M3 = a * b α * c * d M4 = a α A plurality of second reliabilities M1 to M4 may be calculated by *b*c*d.
[0121] (Flow of Parameter Update Processing) The flow of the parameter update processing will be described with reference to Fig. 8. Fig. 8 is a flow diagram showing the flow of the parameter update processing in this exemplary embodiment.
[0122] (Step S2631) In step S2631, the degree calculation unit 212 calculates, for the index removed in step S2623 described above, a degree indicating the strength of correlation between the past frequency distribution fd stored in the storage unit 25 and the current frequency distribution fd generated by the frequency distribution generation unit 211. The degree calculation unit 212 supplies the calculated degree to the update unit 11.
[0123] (Step S2632) In step S2632, the update unit 11 determines whether the degree supplied from the degree calculation unit 212 in step S2631 is smaller than a threshold value.
[0124] (Step S2633) If it is determined in step S2632 that the degree is smaller than the threshold value (step S2632: YES), the update unit 11 updates the parameter pm of the removed index in step S2633. In other words, if the correlation between the past frequency distribution fd and the current frequency distribution fd is weak, the update unit 11 updates the parameter pm.
[0125] For example, assume that the reliability of index A is removed in step S2623 described above, and there is a weak correlation between the past frequency distribution fd and the current frequency distribution fd of index A. In this case, in step S2633, the update unit 11 updates the parameter pm of index A from the parameter pm defining the past frequency distribution to the parameter pm defining the current frequency distribution.
[0126] (Step S2634) In step S2634, the update unit 11 stores the current frequency distribution fd in place of the past frequency distribution fd stored in the storage unit 25.
[0127] On the other hand, if it is determined in step S2632 that the degree is equal to or greater than the threshold value (step S2632: NO), the control unit 21 ends the parameter update process. Note that even if it is determined that the degree is equal to or greater than the threshold value, the update unit 11 may execute the process of step S2634 (the process of storing the current frequency distribution fd instead of the past frequency distribution fd stored in the storage unit 25).
[0128] In this way, the update unit 11 updates the parameter pm that defines the frequency distribution of the indices removed in step S2623 described above. The frequency distribution of the indices removed in step S2623 described above may not match the frequency distribution of the indices of the objects included as subjects in the current frame. Therefore, with this configuration, the update unit 11 can update the parameter pm to a parameter pm that is appropriate for the objects included as subjects in the current frame. Furthermore, with this configuration, the update unit 11 can reduce the processing load compared to when updating the parameters pm of all indices.
[0129] Furthermore, when the degree indicating the strength of correlation between the past frequency distribution and the current frequency distribution is smaller than a threshold, the updating unit 11 updates the parameter pm from the parameter pm that defines the past frequency distribution to the parameter pm that defines the current frequency distribution. With this configuration, the updating unit 11 can update the parameter pm to a parameter pm that is suitable for an object included as a subject in the current frame.
[0130] (Effects of the Matching Device 20) In this manner, the matching device 20 according to this exemplary embodiment updates the parameters pm used to calculate the reliability of the index to parameters pm that define a frequency distribution suited to the object included as a subject in the current frame. As a result, the matching device 20 improves the recall rate while maintaining the accuracy of matching the object included as a subject in each of multiple frames, even for multiple frames captured under different shooting conditions.
[0131] [Example of Software Implementation] Some or all of the functions of the verification devices 1 and 20 may be implemented by hardware such as an integrated circuit (IC chip), or may be implemented by software.
[0132] In the latter case, the collation devices 1 and 20 are realized, for example, by a computer that executes instructions of a program, which is software that realizes each function. An example of such a computer (hereinafter referred to as computer C) is shown in FIG. 9. The computer C includes at least one processor C1 and at least one memory C2. The memory C2 stores a program P for operating the computer C as the collation devices 1 and 20. In the computer C, the processor C1 reads and executes the program P from the memory C2, thereby realizing each function of the collation devices 1 and 20.
[0133] The processor C1 may be, for example, a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), a micro processing unit (MPU), a floating point number processing unit (FPU), a physics processing unit (PPU), a tensor processing unit (TPU), a quantum processor, a microcontroller, or a combination thereof. The memory C2 may be, for example, a flash memory, a hard disk drive (HDD), a solid state drive (SSD), or a combination thereof.
[0134] The computer C may further include a RAM (Random Access Memory) for expanding the program P during execution and for temporarily storing various data. The computer C may also include a communication interface for transmitting and receiving data to and from other devices. The computer C may also include an input / output interface for connecting input / output devices such as a keyboard, a mouse, a display, and a printer.
[0135] The program P can also be recorded on a non-transitory, tangible recording medium M that can be read by the computer C. Such a recording medium M can be, for example, a tape, a disk, a card, a semiconductor memory, or a programmable logic circuit. The computer C can acquire the program P via such a recording medium M. The program P can also be transmitted via a transmission medium. Such a transmission medium can be, for example, a communication network or broadcast waves. The computer C can also acquire the program P via such a transmission medium.
[0136] [Additional Note 1] The present invention is not limited to the above-described embodiments, and various modifications are possible within the scope of the claims. For example, embodiments obtained by appropriately combining the technical means disclosed in the above-described embodiments are also included in the technical scope of the present invention.
[0137] [Additional Note 2] Part or all of the above-described embodiment can also be described as follows: However, the present invention is not limited to the following described aspects.
[0138] (Supplementary Note 1) A matching device comprising: an update means for updating parameters that define the frequency distribution of one or more indices related to an object indicated by target data; a reliability calculation means for calculating the reliability of each of the indices using the parameters; and a matching means for matching the object based on the reliability.
[0139] (Supplementary Note 2) The matching device according to Supplementary Note 1, wherein the matching means removes the reliability of at least one index from among a plurality of indexes and matches the object, and the updating means updates a parameter that defines a frequency distribution of the at least one index that has been removed.
[0140] (Supplementary Note 3) The matching device according to Supplementary Note 2, further comprising a degree calculation means for calculating a degree indicating a strength of correlation between a past frequency distribution and a current frequency distribution for the at least one removed index, and when the degree is smaller than a threshold, the update means updates the parameter from a parameter defining the past frequency distribution to a parameter defining the current frequency distribution.
[0141] (Supplementary Note 4) The matching device according to Supplementary Note 2 or 3, further comprising an initial information generating means for generating initial information, which is a past frequency distribution, from past target data, wherein the updating means updates the parameters from parameters defining the initial information to parameters defining a current frequency distribution.
[0142] (Supplementary Note 5) A matching method including: a matching device updating parameters that define a frequency distribution of one or more indices related to an object indicated by target data; calculating a reliability for each of the indices using the parameters; and matching the object based on the reliability.
[0143] (Supplementary Note 6) A program that causes a computer to execute an update process that updates parameters that define the frequency distribution of one or more indices related to an object indicated by target data, a reliability calculation process that calculates the reliability of each of the indices using the parameters, and a matching process that matches the objects based on the reliability.
[0144] [Additional Note 3] Part or all of the above-described embodiment can also be expressed as follows.
[0145] (Supplementary Note 1) A matching device comprising at least one processor, the at least one processor executing an update process for updating parameters that define a frequency distribution of one or more indices related to an object indicated by target data, a reliability calculation process for calculating a reliability for each of the indices using the parameters, and a matching process for matching the objects based on the reliability.
[0146] (Supplementary Note 2) In the matching device described in Supplementary Note 1, the at least one processor, in the matching process, removes the reliability of at least one index from among a plurality of indexes, and matches the objects, and in the update process, updates a parameter that defines a frequency distribution of the at least one index that has been removed.
[0147] (Supplementary Note 3) The matching device according to Supplementary Note 2, wherein the at least one processor further performs a degree calculation process to calculate a degree indicating a strength of correlation between a past frequency distribution and a current frequency distribution for the at least one removed index, and if the degree is smaller than a threshold, updates the parameter from a parameter defining the past frequency distribution to a parameter defining the current frequency distribution in the update process.
[0148] (Supplementary Note 4) The matching device according to Supplementary Note 2 or 3, wherein the at least one processor further executes an initial information generation process to generate initial information, which is a past frequency distribution, from past target data, and in the update process, updates the parameters from parameters defining the initial information to parameters defining a current frequency distribution.
[0149] The verification device may further include a memory that stores a program for causing the processor to execute the update process, the reliability calculation process, the verification process, the degree calculation process, and the initial information generation process. The program may be recorded on a computer-readable, non-transitory, tangible recording medium.
[0150] 1, 20 Matching device 11 Update unit 12 Reliability calculation unit 13 Matching unit 21 Control unit 22 Input unit 23 Output unit 24 Communication unit 25 Storage unit 121 First reliability calculation unit 122 Second reliability calculation unit 211 Frequency distribution generation unit 212 Degree calculation unit fd Frequency distribution fr Frame pm Parameter
Claims
1. A matching device comprising: an update means for updating parameters that define the frequency distribution of one or more indexes relating to an object indicated by target data; a reliability calculation means for calculating the reliability of each of the indexes using the parameters; and a matching means for matching the object based on the reliability.
2. The matching device according to claim 1, wherein the matching means removes the reliability of at least one index from among a plurality of indexes and matches the objects, and the update means updates a parameter that defines a frequency distribution of the at least one index that has been removed.
3. The matching device according to claim 2, further comprising a degree calculation means for calculating a degree indicating the strength of correlation between a past frequency distribution and a current frequency distribution for at least one of the removed indicators, and when the degree is smaller than a threshold value, the update means updates the parameter from a parameter defining the past frequency distribution to a parameter defining the current frequency distribution.
4. The matching device according to claim 2 or 3, further comprising an initial information generating means for generating initial information, which is a past frequency distribution, from past target data, wherein the updating means updates the parameters from parameters defining the initial information to parameters defining the current frequency distribution.
5. A matching method comprising: a matching device updating parameters that define the frequency distribution of one or more indicators relating to an object represented by target data; calculating a reliability for each of the indicators using the parameters; and matching the object based on the reliability.
6. A program that causes a computer to execute an update process that updates parameters that define the frequency distribution of one or more indicators related to an object indicated by target data; a reliability calculation process that calculates the reliability of each of the indicators using the parameters; and a matching process that matches the objects based on the reliability.
Citation Information
Patent Citations
Person identification device, person identification method, and person identification program
JP2010239992A
Image processing device and area tracking program
JP2017194787A
Information processing apparatus, information processing method, and program
JP2019016098A
Information processing device, information processing method, and information processing program
WO2020217368A1
