Target positioning and counting method based on point-to-point matching

Abstract

The invention provides a target positioning and counting method based on point-to-point matching, and the method comprises the steps: obtaining a to-be-processed original image; using a convolution module in the trained target positioning model to perform multiple convolution operations on the original image to obtain K feature maps; using K target point prediction modules in a trained target positioning model to process each feature map to obtain coordinates and confidence of K groups of candidate points; and according to the coordinates and the confidence of the K groups of candidate points and a preset confidence threshold, obtaining the coordinates of a prediction point which is the central point of the target to be identified. According to the target positioning method, prediction is carried out according to the central point of the target, so that sample calibration is simpler, and the finally obtained prediction point coordinates are relatively more accurate, and therefore, the method can be well adapted to the recognition and counting of multi-shielding and overlapping targets.

Description

Technical Field

[0001] The present invention relates to the technical field of image target recognition, and in particular to a target positioning and counting method based on point-to-point matching. Background Art

[0002] The identification and counting of targets are extremely important in many aspects, such as the identification and counting of individual targets in groups such as people, cattle, and fish. In the prior art, common counting methods mostly use machine vision and deep learning algorithms to first identify individual targets in a group. The output of this identification is mostly the detection frame of the individual target, and then the number of identified detection frames is counted to perform quantity statistics.

[0003] The above methods can be used to identify and count fish well in some cases, but they are not suitable for some targets that are blocked, overlapped, or have large size differences. For example, for fish, since fish have large size differences, swim quickly in the water, and often gather together, they are prone to blockage and overlap. It is difficult to accurately locate and count fish using existing solutions. Summary of the invention

[0004] In view of the above defects in the prior art, the present invention provides a target positioning and counting method based on point-to-point matching to solve the technical problem of inaccurate recognition of overlapping and occluded targets.

[0005] To achieve the above-mentioned purpose and other related purposes, the present invention provides a target positioning based on point-to-point matching, including: obtaining an original image to be processed; using a convolution module in a trained target positioning model to perform multiple convolution operations on the original image to obtain K feature maps, where K is greater than or equal to 1; using K target point prediction modules in the trained target positioning model to process each of the feature maps respectively to obtain coordinates and confidences of K groups of candidate points; according to the coordinates and confidences of the K groups of candidate points and a preset confidence threshold, obtaining the coordinates of a predicted point, where the predicted point is the center point of the target to be identified.

[0006] In one embodiment of the present invention, the kth target point prediction module outputs (M / 2^(k-1))×(N / 2^(k-1)) candidate points, the kth feature map is divided into (M / 2^(k-1))×(N / 2^(k-1)) regions, each region in the kth feature map corresponds to a candidate point, and all candidate points in the kth feature map constitute a group of candidate points, where k is equal to {1,2,…,K}.

[0007] In one embodiment of the present invention, the target point prediction module includes: a classification unit, which is used to output the probability that there is a center point of the identified target in the area where each candidate point is located, so as to obtain the confidence of the candidate point; a regression unit, which is used to output the relative coordinates of each candidate point in the corresponding area; and a coordinate transformation unit, which is used to convert the relative coordinates of the candidate point in the corresponding area into the coordinates of the candidate point in the original image, so as to obtain the coordinates of the candidate point.

[0008] In one embodiment of the present invention, K is equal to 1; the coordinates of the predicted point are obtained according to the coordinates and confidences of the K groups of candidate points and a preset confidence threshold, including: deleting candidate points whose confidences are less than the confidence threshold to obtain high-confidence candidate points, and the coordinates of the high-confidence candidate points are the coordinates of the predicted point.

[0009] In one embodiment of the present invention, K is greater than or equal to 2; the coordinates of the predicted point are obtained according to the coordinates and confidences of the K groups of candidate points and a preset confidence threshold, including: deleting candidate points whose confidences are less than the confidence threshold to obtain K groups of high-confidence candidate points; and merging the coordinates of the K groups of high-confidence candidate points to obtain the coordinates of the predicted point.

[0010] In one embodiment of the present invention, the coordinates of K groups of high-confidence candidate points are merged to obtain the coordinates of the predicted point, including: according to the distance between the candidate points, the high-confidence candidate points of the second to K groups are respectively associated with the first group of high-confidence candidate points to obtain an association relationship; according to the association relationship and the coordinates of the K groups of high-confidence candidate points, the coordinates of the predicted point are obtained.

[0011] In one embodiment of the present invention, the coordinates of the predicted point are obtained according to the association relationship and the coordinates of the K groups of high-confidence candidate points, including: if a point in the first group of high-confidence candidate points is not associated with any high-confidence candidate points in other groups, then the coordinates of the point are the coordinates of a predicted point; if a point in the first group of high-confidence candidate points is associated with high-confidence candidate points in other groups, then the coordinates of the center point of the line segment or polygon enclosed by the point and all the points associated with it are the coordinates of a predicted point.

[0012] In an embodiment of the present invention, the relative coordinates are relative coordinates between the candidate point and a center point of a corresponding region.

[0013] In one embodiment of the present invention, the trained target positioning model is obtained through the following steps: obtaining a sample image, and annotating the center point of the target to be identified in the sample image by manual annotation to obtain the coordinates of the true value point; constructing and initializing the target positioning model; inputting the sample image into the target positioning model to obtain the coordinates and confidence of the candidate point; matching the true value point and the candidate point to obtain a matching result; calculating the loss value based on the matching result, and fine-tuning the parameters in the target positioning model based on the loss value until the training is completed to obtain the trained target positioning model.

[0014] To achieve the above-mentioned purpose and other related purposes, the present invention also provides a counting method based on point-to-point matching. According to the above-mentioned target positioning method based on point-to-point matching, the coordinates of the predicted points are obtained; the number of the predicted points is counted to obtain the number of targets to be detected.

[0015] Beneficial effects of the present invention: The present invention proposes a target positioning and counting method based on point-to-point matching. The target positioning method obtains the coordinates and confidence of candidate points through a trained target positioning model, and then obtains the coordinates of the predicted points based on the relevant information of the candidate points. This method predicts based on the center point of the target, which makes sample calibration simpler, and the coordinates of the predicted points finally obtained are also relatively more accurate, so that it can well adapt to the recognition and counting of multiple occluded and overlapping targets. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0017] Figure 1 A flow chart of a target positioning method provided by an embodiment of the present invention; Figure 2 A schematic diagram of the reasoning process using a single-branch target positioning model provided by an embodiment of the present invention; Figure 3 A schematic diagram of the reasoning process using a three-branch target positioning model provided by an embodiment of the present invention; Figure 4 A structural diagram of a classification unit provided in one embodiment of the present invention; Figure 5 A structural diagram of a regression unit provided in one embodiment of the present invention; Figure 6 A flowchart for obtaining prediction point coordinates provided by an embodiment of the present invention; Figure 7 A flowchart of merging multiple sets of candidate point coordinates provided by an embodiment of the present invention; Figure 8 A training flow chart of a model provided by an embodiment of the present invention; Fig. 9 A schematic diagram of the training process of a single-branch target localization model provided by an embodiment of the present invention; Fig.10 A schematic diagram of the training process of a three-branch target localization model provided by an embodiment of the present invention; Fig.11 A flowchart of matching true value points and candidate points in a training process provided by an embodiment of the present invention; Fig.12 A flowchart of loss value calculation during training provided by an embodiment of the present invention; Fig.13 A flow chart of classification loss calculation during training provided by one embodiment of the present invention.

[0018] Fig.14 A flow chart of a target counting method provided by one embodiment of the present invention. DETAILED DESCRIPTION

[0019] The following describes the embodiments of the present invention through specific specific embodiments, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. It should be noted that the following embodiments and the features in the embodiments can be combined with each other without conflict. In addition to the specific methods, equipment, and materials used in the embodiments, according to the technical personnel in this technical field's grasp of the prior art and the records of the present invention, any methods, equipment, and materials of the prior art that are similar or equivalent to the methods, equipment, and materials described in the embodiments of the present invention can also be used to implement the present invention.

[0020] It should be understood that the terms used in the embodiments of the present invention are for describing specific embodiments rather than for limiting the scope of protection of the present invention. Unless otherwise defined, all technical and scientific terms used in the present invention have the same meaning as those generally understood by those skilled in the art.

[0021] In the following description, numerous details are discussed to provide a more thorough explanation of the embodiments of the present invention. However, it is obvious to those skilled in the art that the embodiments of the present invention can be implemented without these specific details. In some of the embodiments, well-known structures and devices are shown in the form of block diagrams rather than in detail to avoid making the embodiments of the present invention difficult to understand.

[0022] The flowchart and block diagram in the accompanying drawings illustrate the system architecture, functions and operations that may be implemented according to the methods and computer program products of various embodiments disclosed in the present invention. In this regard, each box in the flowchart or block diagram can represent a module, a program segment, or a part of a code, and the module, the program segment, or a part of the code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or the flowchart, and the combination of the boxes in the block diagram and / or the flowchart can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0023] See also Figure 1 , Figure 1 A target positioning method based on point-to-point matching provided in an embodiment of the present invention includes steps S100 to S400.

[0024] Step S100, obtaining the original image to be processed. The original image to be processed is a group image of the target to be identified, such as a school of fish. It can be understood that the image directly captured can be resized as needed to adapt to the input size of the target positioning model. The original image mentioned here is the image after the size is adapted.

[0025] Step S200: Use the convolution module in the trained target positioning model to perform multiple convolution operations on the original image to obtain K feature maps, where K is greater than or equal to 1. Figure 2 and Figure 3 The corresponding area is the foundation area. It should be noted that the multiple convolution operations mentioned here are not necessarily equal to the number of feature maps obtained in the end. For example, Figure 3 In the structure of , five convolution operations are performed, and finally only the feature maps of the last three times are selected.

[0026] Step S300: Use the K target point prediction modules in the trained target positioning model to process each feature map to obtain the coordinates and confidence of K groups of candidate points. Figure 2 and Figure 3 The corresponding area is the green background area. Here, one or more target point prediction modules are selected to process the feature numbers respectively. It can be understood that each target point prediction module will output a set of candidate point coordinates and confidences, but the number of candidate points they output is different.

[0027] See also Figure 2 and Figure 3 In a specific embodiment of the present invention, the k-th target point prediction module outputs (M / 2^(k-1))×(N / 2^(k-1)) candidate points, the k-th feature map is divided into (M / 2^(k-1))×(N / 2^(k-1)) regions, each region in the k-th feature map corresponds to a candidate point, and all candidate points in the k-th feature map constitute a group of candidate points, where k is equal to {1,2,…,K}.

[0028] exist Figure 3 In the embodiment shown, each feature map is half the size of the previous feature map. Assuming that the size of the original image is H×W, five convolution operations are performed on it, and the sizes of the five feature maps are H / 2×W / 2, H / 4×W / 4, H / 8×W / 8, H / 16×W / 16, and H / 32×W / 32, respectively. We take the last three feature maps as the first to third feature maps. When performing region division, the size of the region can be selected as a fixed size. Assuming that the first feature map is divided into M×N small regions (evenly divided horizontally and vertically), the second feature map is reduced in size by half. Therefore, when the size of the divided region is fixed, it will be divided into M / 2×N / 2 small regions. Similarly, the third feature map will be divided into M / 4×N / 4 small regions.

[0029] The reason for doing so is that the concept of the present invention is to assume that there is a candidate point in each area, and then determine the coordinates of the predicted point based on the coordinates of the candidate point, and determine whether there is really a center point of the target in this area based on the confidence of the candidate point.

[0030] See also Figure 2 and Figure 3 In a specific embodiment of the present invention, the target point prediction module includes: a classification unit, which is used to output the probability that the center point of the identified target exists in the area where each candidate point is located, so as to obtain the confidence of the candidate point; a regression unit, which is used to output the relative coordinates of each candidate point in the corresponding area; and a coordinate transformation unit, which is used to convert the relative coordinates of the candidate point in the corresponding area into the coordinates of the candidate point in the original image, so as to obtain the coordinates of the candidate point. The confidence is a number between 0 and 1. The lower the confidence, the lower the possibility that the center point of the identified target exists in the area. The relative coordinates can simplify the calculation of the model. The relative coordinates can be the coordinates of the candidate point relative to a reference point in the corresponding area. This reference point can be, for example, the upper left corner of the corresponding area, the center point of the corresponding area, and so on.

[0031] See also Figure 4In a specific embodiment of the present invention, the classification unit package consists of a convolution layer + activation layer → convolution layer + activation layer → convolution layer, and finally a Sigmoid function is added to generate the probability value. The overall structure is as follows Figure 4 shown.

[0032] See also Figure 5 In a specific embodiment of the present invention, the regression unit is composed of convolution layer + activation layer → convolution layer + activation layer → convolution layer, and the output of the last convolution layer is the relative coordinates of each candidate point in the corresponding area.

[0033] In a specific embodiment of the present invention, the center point of the corresponding area is selected as the reference point, that is, the relative coordinates are the relative coordinates between the candidate point and the center point of the corresponding area.

[0034] Since the regression unit outputs relative coordinates, and the coordinates are relative to the feature map, the coordinates need to be transformed. The coordinate transformation unit needs to perform two transformations. The first is to transform the relative coordinates of the candidate point in the corresponding area into the coordinates of the candidate point in the corresponding feature map. The second is to scale the coordinates of the candidate point in the corresponding feature map proportionally to obtain the coordinates of the candidate point in the original image, and output this coordinate as the final coordinate of the candidate point.

[0035] Step S400: According to the coordinates and confidences of the K groups of candidate points and a preset confidence threshold, the coordinates of the predicted point are obtained, and the predicted point is the center point of the target to be identified.

[0036] See also Figure 2 In a specific embodiment of the present invention, when K is equal to 1, it is equivalent to having only one target point prediction module. Step S400 includes: deleting candidate points whose confidence is less than the confidence threshold, obtaining high-confidence candidate points, and the coordinates of the high-confidence candidate points are the coordinates of the prediction points. The confidence threshold is a preset value. When the confidence is low, it means that the candidate point is unreliable and is very likely not the target center point, so it can be deleted. The retained high-confidence candidate points are the final prediction points.

[0037] It should be noted that the confidence of the high confidence candidate point is also the confidence of the predicted point. If the confidence of the predicted point needs to be output, the confidence of the high confidence candidate point can also be output in step S400. The purpose of the present invention is to identify the center point of the target and locate it, so only the coordinates of the predicted point are output.

[0038] When K is greater than or equal to 2, there will be multiple target point prediction modules, and each target point prediction module will output a set of candidate point coordinates and confidences. However, please note that multiple target point prediction modules are only target point predictions at different scales. The candidate points output by different target point prediction modules may be predictions for the same target, so they need to be merged later; and the number of prediction points output by different target point prediction modules is also different. The smaller the feature map scale, the fewer the number of prediction points. The following example takes K equal to 3 as an example.

[0039] See also Figure 3 and Figure 6 In a specific embodiment of the present invention, step S400 includes: S410, deleting candidate points whose confidence is less than the confidence threshold to obtain K groups of high-confidence candidate points; S420, merging the coordinates of the K groups of high-confidence candidate points to obtain the coordinates of the predicted point. Since there are K target point prediction modules, the coordinates and confidences of K groups of candidate points will be output. Therefore, in step S410, K groups of high-confidence candidate points will be obtained. Considering that candidate points in different groups may represent the same predicted point, it is necessary to merge them in step S420.

[0040] See also Figure 7 In a specific embodiment of the present invention, step S420 includes: S421, according to the distance between the candidate points, respectively associating the high-confidence candidate points of the second group to the Kth group with the first group of high-confidence candidate points to obtain an association relationship; S422, according to the association relationship and the coordinates of the K groups of high-confidence candidate points, obtaining the coordinates of the predicted point.

[0041] Theoretically, the larger the feature map size, the more targets can be identified. However, for some particularly large targets, the prediction of the center point position may be biased. As the feature map size decreases, the prediction of the center point position of particularly large targets will be relatively more accurate, but at the same time, some particularly small targets may be missed. Therefore, in this embodiment, when multiple branches are used for prediction, the first group of high-confidence candidate points is used as the basis, and the high-confidence candidate points of other groups are used to correct their coordinates. Figure 2 The prediction results using a single target point prediction module will be more accurate.

[0042] In this embodiment, the distance selected when associating in step S421 can be set according to actual conditions, and when associating, a one-to-one match is required, that is, each high-confidence candidate point in the second group can only be associated one-to-one with a high-confidence candidate point in the first group, and a many-to-one situation cannot occur. The same applies to other groups.

[0043] In a specific embodiment of the present invention, step S422 includes: if a point in the first group of high-confidence candidate points is not associated with any high-confidence candidate point in other groups, then the coordinates of the point are the coordinates of a predicted point; if a point in the first group of high-confidence candidate points is associated with high-confidence candidate points in other groups, then the coordinates of the center point of the line segment or polygon enclosed by the point and all the points associated with it are the coordinates of a predicted point.

[0044] For example, if point A in the first group of high-confidence candidate points is not associated with any point in other groups, then point A itself can be used as a prediction point; for example, if point B1 in the first group of high-confidence candidate points is only associated with point B2 in the second group of high-confidence candidate points, then the center point of the line segment formed by points B1 and B2 can be used as a prediction point; if point C1 in the first group of high-confidence candidate points is associated with point C2 in the second group of high-confidence candidate points and point C3 in the third group of high-confidence candidate points, then the center point of the triangle formed by points C1, C2, and C3 can be used as a prediction point; and so on.

[0045] See also Figure 8 In a specific embodiment of the present invention, a trained target positioning model is obtained by the following steps: S10, obtaining a sample image, and annotating the center point of the target to be identified in the sample image by manual annotation to obtain the coordinates of the true value point; S20, constructing and initializing the target positioning model; S30, inputting the sample image into the target positioning model to obtain the coordinates and confidence of the candidate point; S40, matching the true value point and the candidate point to obtain the matching result; S50, calculating the loss value according to the matching result, and fine-tuning the parameters in the target positioning model according to the loss value, until the training is completed to obtain the trained target positioning model.

[0046] Here, taking the positioning of fish targets as an example, the sample images can be obtained and annotated in step S10 as follows. First, place the fry of light-lipped fish in a cylindrical fish pond with a diameter of 2.5m and a height of 1.5m, and place the Boleth E3GMPCX15 high-definition underwater camera at the edge of the fish tank, and make the camera have an upward angle of 30° to record the movement of the fish school in the water. Then, the collected video is framed at intervals of 1s to obtain sample images, and the extracted images are annotated with the center points of the fish school targets through the GitHub open source project anylabeling. The annotation does not need to be very accurate, and there can be a little deviation. After the annotation is completed, the generated JSON file is used to extract the center point coordinates of each target through python to form a txt file, and 30% of the data is reserved as a verification set for model verification.

[0047] Fig. 9 and Fig.10The first picture in is the sample image. For ease of processing, the original image and sample image involved in the reasoning process in the present invention are illustrated with the same picture; the prediction points involved in the reasoning process and the true value points involved in the training process are also illustrated with the same picture. In actual situations, these pictures and points are different.

[0048] If the number of sample images is small, the sample images can be rotated, cropped, flipped, translated, and scaled to expand the data set.

[0049] See also Fig. 9 and Fig.10 In a specific embodiment of the present invention, the target positioning model in step S20 has K branches, K is greater than or equal to 1, each branch (corresponding to the blue background area in the figure) includes a cascaded convolution unit (corresponding to the foundation area in the figure) and a target point prediction module (corresponding to the green background area in the figure), the input of the convolution unit of the i-th branch is the output of the convolution unit of the i-1-th branch, i is equal to {2,…,K}; the target point prediction module of each branch outputs a set of candidate point coordinates and confidences. Fig. 9 The diagram shows the target positioning module when K=1. Fig.10 The figure shows the target positioning module when K=3. In addition, K can also be other integers.

[0050] A convolutional unit can consist of one or more convolutional layers, such as Fig. 9 and Fig.10 The convolution unit in the first branch of is composed of three cascaded first convolutional layers, second convolutional layers, and third convolutional layers; Fig.10 In the second branch and the third branch, the convolution unit can also be composed of a convolution layer. For example, the fourth convolution layer and the fifth convolution layer each constitute a convolution unit.

[0051] In each branch, the number of candidate points output by the target point prediction module is different. In the k-th branch, the target point prediction module outputs (M / 2^(k-1))×(N / 2^(k-1)) candidate points, and the k-th feature map output by the k-th convolution unit is divided into (M / 2^(k-1))×(N / 2^(k-1)) regions. Each region in the k-th feature map corresponds to a candidate point. All candidate points in the k-th feature map constitute a set of candidate points, and k is equal to {1,2,…,K}. For example Fig.10 In the above formula, the number of candidate points in the first group is M×N, the number of candidate points in the second group is (M / 2)×(N / 2), and the number of candidate points in the third group is (M / 4)×(N / 4).

[0052] The target point prediction module includes a classification unit, a regression unit, and a coordinate transformation unit ( Fig.10The last two branches in the figure are not shown), the classification unit is used to output the probability that the center point of the identified target exists in the area where each candidate point is located, that is, the confidence of the candidate point output in step S30; the regression unit is used to output the relative coordinates of each candidate point in the corresponding area; the coordinate transformation unit is used to convert the relative coordinates of the candidate point in the corresponding area into the coordinates of the candidate point in the sample image, that is, the coordinates of the candidate point output in step S30.

[0053] The reason why the true value points and candidate points need to be matched in step S40 is that the number of candidate points output by the target positioning model is fixed. For example, in one of the above embodiments, the first branch outputs M×N candidate points, and the second branch outputs (M / 2)×(N / 2) candidate points, but the number of true value points is related to the number of targets in the sample image, so there will be a difference in the number of true value points and candidate points. This is different from the common model, in which the number of true value points and predicted points is the same, so special processing is required in step S40, that is, matching operation.

[0054] See also Fig.11 In a specific embodiment of the present invention, step S40 includes: S41, generating a cost matrix according to the coordinates of the true value point and the coordinates of the candidate point, the elements of the cost matrix are the matching costs between the corresponding true value point and the candidate point; S42, obtaining a matching result according to the cost matrix.

[0055] In a specific embodiment of the present invention, the elements of the cost matrix in step S41 can be calculated according to the following formula: , Where P i represents the i-th truth point, Q j represents the jth candidate point, || || 2 represents the square of the distance between two points, exp() represents the natural exponential of a given value, α and β are preset weight factors, and σ represents the standard deviation in the normal distribution. In this formula, the weight factor α is followed by the Euclidean distance between the two points, the weight factor β is followed by the probability density function of the normal distribution, and σ is the standard deviation in the normal distribution. If it is a standard normal distribution, its value is 1. By weighting these two physical quantities, the element value of the cost matrix can be obtained.

[0056] In the specific implementation, we need to ensure that the number of candidate points is greater than the number of true value points to generate enough predictions. Candidate points that are greater than the true value points will be classified as negative samples. After having the cost matrix, for example, we can use the Hungarian matching algorithm to match it and then get the matching result. When matching, we need to pay attention to the one-to-one matching principle.

[0057] See also Fig.12In a specific embodiment of the present invention, the loss value is calculated according to the matching result, including: S51, obtaining the regression loss according to the coordinates of the true value point and the coordinates of the candidate point matching the true value point; S52, obtaining the classification loss according to the confidence of the candidate point and the matching relationship. Considering that there are a classification unit and a regression unit in the target point prediction module, these two units are independent of each other, and therefore, their losses can be calculated separately.

[0058] In a specific embodiment of the present invention, the regression loss is calculated according to the following formula: , In the formula, X is the total number of true value points, Q i Indicates that P i The candidate points to be matched, λ1 and λ2 are preset weight factors, and σ in the formula also represents the standard deviation in the normal distribution. The calculation formula of the regression loss is very similar to the calculation formula of the cost matrix in step S41, because the regression loss is also calculated based on the Euclidean distance between the two points and the probability density function of the normal distribution. The difference is that the element value of the cost matrix corresponds to the matching cost between the two points, and the regression loss is the sum of the relevant physical quantities of all the matched points.

[0059] See also Fig.13 In a specific embodiment of the present invention, step S52 includes: S52a, according to the matching relationship, obtain the confidence truth value of each candidate point according to the following rules: if the candidate point has a matching truth point, the confidence truth value of the candidate point is 1; if the candidate point does not have a matching truth point, the confidence truth value of the candidate point is 0; S52b, according to the confidence and confidence truth value of the candidate point, obtain the classification loss.

[0060] Taking the first branch as an example, it outputs M×N candidate points, so there are exactly M×N confidences of the candidate points, but since only some of the candidate points match the true value points, and some of the candidate points do not match the true value points, the confidence truth values ​​obtained according to step S52a are also M×N, but each confidence truth value is either 0 or 1. In step S52b, the classification loss can be calculated based on the M×N confidences and the M×N confidence truth values.

[0061] In a specific embodiment of the present invention, each branch is trained in sequence. During training, the parameters in the kth branch are fine-tuned according to the coordinates and confidence of the kth group of candidate points, where k is equal to {1, 2, ..., K}.

[0062] by Fig.10Taking the target localization model in as an example, the overall training steps are as follows: (1) first train the first branch, and then fix the parameters of the convolutional unit of the first branch; (2) then train the second branch. When training the second branch, it is equivalent to training the sub-module composed of the convolutional unit of the first branch, the convolutional unit of the second branch, and the target point prediction module of the second branch, and then fix the parameters of the convolutional unit of the second branch; (3) and so on, complete the training of other branches in sequence.

[0063] For each branch, the target point prediction module also has two sub-branches. The first sub-branch is the classification unit, and the second sub-branch is the regression unit and the coordinate transformation unit. During training, the first sub-branch can also be trained first, and the classification loss is used as the loss function to obtain the model parameters of the convolution unit and the classification unit; then the parameters of the convolution unit are fixed, and the regression loss is used as the loss function to train the second sub-branch.

[0064] It can be understood that in the specific training process, it is necessary to set the model's learning rate, batch size, optimizer type, regularization method, and loss function, and then use the prepared training set (a sample consists of a sample image and the marked true value point coordinates) for training.

[0065] Understandably, after the training is completed, the reserved validation set can be used to evaluate the model in combination with parameters such as mean square error and mean absolute error, and the model parameters with better evaluation results can be selected as the final trained target positioning model.

[0066] In general, the target positioning model in the present invention is a model obtained through multiple trainings, and its training process is slightly more complicated than traditional model training.

[0067] See also Fig.14 , Fig.14 A target counting method based on point-to-point matching provided in one embodiment of the present invention comprises: S1, obtaining the coordinates of the prediction points according to the target positioning method based on point-to-point matching; S2, counting the number of prediction points to obtain the number of targets to be detected. The focus of the present invention is on the positioning of the target, that is, identifying the center point (i.e., the prediction point) of each target. After obtaining the prediction point, the detection frame of the target can be further obtained, and the number of targets can also be obtained through step S2.

[0068] It should be noted that the step division of the above methods is only for clear description. When implemented, they can be combined into one step or some steps can be split and decomposed into multiple steps. As long as they contain the same logical relationship, they are all within the scope of protection of this patent; adding insignificant modifications to the algorithm or process or introducing insignificant designs without changing the core design of the algorithm and process are all within the scope of protection of this patent.

[0069] The above embodiments are merely illustrative of the principles and effects of the present invention, and are not intended to limit the present invention. Anyone familiar with the art may modify or alter the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or alterations made by a person of ordinary skill in the art without departing from the spirit and technical concept disclosed by the present invention shall still be covered by the claims of the present invention.

Claims

1. A target positioning method based on point-to-point matching, characterized in that: include: Obtain the original image to be processed; Using the convolution module in the trained target positioning model, perform multiple convolution operations on the original image to obtain K feature maps, where K is greater than or equal to 1; Using the K target point prediction modules in the trained target positioning model, each of the feature maps is processed respectively to obtain the coordinates and confidences of K groups of candidate points; According to the coordinates and confidences of the K groups of candidate points and a preset confidence threshold, the coordinates of the predicted point are obtained, and the predicted point is the center point of the target to be identified.

2. The target positioning method based on point-to-point matching according to claim 1, characterized in that: The k-th target point prediction module outputs (M / 2^(k-1))×(N / 2^(k-1)) candidate points. The k-th feature map is divided into (M / 2^(k-1))×(N / 2^(k-1)) regions. Each region in the k-th feature map corresponds to a candidate point. All candidate points in the k-th feature map constitute a set of candidate points, and k is equal to {1,2,…,K}.

3. The target positioning method based on point-to-point matching according to claim 2, characterized in that: The target point prediction module comprises: The classification unit is used to output the probability that the center point of the identified target exists in the area where each candidate point is located, so as to obtain the confidence of the candidate point; A regression unit, used to output the relative coordinates of each candidate point in the corresponding area; and The coordinate transformation unit is used to transform the relative coordinates of the candidate point in the corresponding area into the coordinates of the candidate point in the original image to obtain the coordinates of the candidate point.

4. The target positioning method based on point-to-point matching according to claim 3 is characterized in that: Said K is equal to 1; According to the coordinates and confidences of the K groups of candidate points and a preset confidence threshold, the coordinates of the predicted point are obtained, including: The candidate points whose confidence is less than the confidence threshold are deleted to obtain high-confidence candidate points, and the coordinates of the high-confidence candidate points are the coordinates of the predicted points.

5. The target positioning method based on point-to-point matching according to claim 3 is characterized in that: The K is greater than or equal to 2; According to the coordinates and confidences of the K groups of candidate points and a preset confidence threshold, the coordinates of the predicted point are obtained, including: Deleting candidate points whose confidence is less than the confidence threshold to obtain K groups of high-confidence candidate points; The coordinates of the K groups of high-confidence candidate points are merged to obtain the coordinates of the predicted point.

6. The target positioning method based on point-to-point matching according to claim 5, characterized in that: The coordinates of K groups of high-confidence candidate points are merged to obtain the coordinates of the predicted point, including: According to the distances between the candidate points, the high-confidence candidate points of the second group to the Kth group are respectively associated with the high-confidence candidate points of the first group to obtain an association relationship; The coordinates of the predicted point are obtained according to the association relationship and the coordinates of the K groups of high-confidence candidate points.

7. The target positioning method based on point-to-point matching according to claim 6, characterized in that: According to the association relationship and the coordinates of the K groups of high-confidence candidate points, the coordinates of the predicted point are obtained, including: If a point in the first group of high-confidence candidate points is not associated with any high-confidence candidate points in other groups, the coordinates of the point are the coordinates of a predicted point; If a point in the first group of high-confidence candidate points is associated with high-confidence candidate points in other groups, the coordinates of the center point of the line segment or polygon enclosed by the point and all the points associated with it are the coordinates of a predicted point.

8. The target positioning method based on point-to-point matching according to claim 3, characterized in that: The relative coordinates are the relative coordinates of the candidate point and the center point of the corresponding area.

9. The target positioning method based on point-to-point matching according to claim 1, characterized in that: The trained target positioning model is obtained by the following steps: Obtain a sample image, and manually annotate the center point of the target to be identified in the sample image to obtain the coordinates of the true value point; Build and initialize the target positioning model; Inputting the sample image into the target positioning model to obtain the coordinates and confidence of the candidate points; Matching the true value point and the candidate point to obtain a matching result; A loss value is calculated according to the matching result, and parameters in the target positioning model are fine-tuned according to the loss value until the training is completed to obtain the trained target positioning model.

10. A target counting method based on point-to-point matching, characterized in that: include: According to any one of claims 1 to 9, the target positioning method based on point-to-point matching is used to obtain the coordinates of the predicted point; The number of the predicted points is counted to obtain the number of targets to be detected.

Images