A training method for target localization model

Abstract

The present invention provides a method for training a target positioning model, including: obtaining a sample image, and annotating the center point of a target to be identified in the sample image by manual annotation to obtain the coordinates of a true value point; constructing and initializing a target positioning model; inputting the sample image into the target positioning model to obtain the coordinates and confidence of a candidate point; matching the true value point and the candidate point to obtain a matching result; calculating a loss value according to the matching result, and fine-tuning the parameters in the target positioning model according to the loss value, until the training is completed to obtain a trained target positioning model. For general model training, the estimated results and the real results are one-to-one corresponding, but the number of candidate points and true value points of the model in the present invention is not equal, so a matching mechanism is introduced here, and the loss value is calculated according to the matching result, so as to realize reliable training of the model, and in a multi-branch model structure, multiple training methods are used to complete the training of the entire model.

Description

Technical Field

[0001] The present invention relates to the technical field of image target recognition, and in particular to a training method for a target positioning model. Background Art

[0002] The identification and counting of targets are extremely important in many aspects, such as the identification and counting of individual targets in groups such as people, cattle, and fish. In the prior art, common counting methods mostly use machine vision and deep learning algorithms to first identify individual targets in a group. The output of this identification is mostly the detection frame of the individual target, and then the number of identified detection frames is counted to perform quantity statistics.

[0003] The above methods can be used to identify and count fish well in some cases, but they are not suitable for some targets that are blocked, overlapped, or have large size differences. For example, for fish, since fish have large size differences, swim quickly in the water, and often gather together, they are prone to blockage and overlap. It is difficult to accurately locate and count fish using existing solutions. Therefore, it is urgent to train a suitable target positioning model to locate and identify targets. Summary of the invention

[0004] In view of the above defects in the prior art, the present invention provides a method for training a target positioning model to solve the technical problem of lack of a suitable model for identifying overlapping and occluded targets.

[0005] To achieve the above-mentioned purpose and other related purposes, the present invention provides a method for training a target positioning model, comprising: obtaining a sample image, and annotating the center point of the target to be identified in the sample image by manual annotation to obtain the coordinates of the true value point; constructing and initializing the target positioning model; inputting the sample image into the target positioning model to obtain the coordinates and confidence of the candidate point; matching the true value point and the candidate point to obtain a matching result; calculating a loss value based on the matching result, and fine-tuning the parameters in the target positioning model based on the loss value, until the training is completed to obtain a trained target positioning model.

[0006] In one embodiment of the present invention, the true value point and the candidate point are matched to obtain a matching result, including: generating a cost matrix based on the coordinates of the true value point and the coordinates of the candidate point, the elements of the cost matrix being the matching costs between the corresponding true value point and the candidate point; and obtaining the matching result based on the cost matrix.

[0007] In one embodiment of the present invention, the elements of the cost matrix are calculated according to the following formula:

[0008] ,

[0009] Where P i represents the i-th truth point, Q j represents the jth candidate point, || || 2 It represents the square of the distance between two points, exp() represents the natural exponential of a given value, α and β are preset weight factors, and σ represents the standard deviation in the normal distribution.

[0010] In one embodiment of the present invention, the loss value is calculated based on the matching result, including: obtaining the regression loss based on the coordinates of the true value point and the coordinates of the candidate point matching the true value point; and obtaining the classification loss based on the confidence of the candidate point and the matching relationship.

[0011] In one embodiment of the present invention, the regression loss is calculated according to the following formula:

[0012] ,

[0013] In the formula, X is the total number of true value points, Q i Indicates that P i Matching candidate points, λ 1 and λ 2 is the preset weight factor.

[0014] In one embodiment of the present invention, the classification loss is obtained according to the confidence and matching relationship of the candidate points, including: according to the matching relationship, the confidence true value of each candidate point is obtained according to the following rules: if the candidate point has a matching true value point, the confidence true value of the candidate point is 1; if the candidate point does not have a matching true value point, the confidence true value of the candidate point is 0; the classification loss is obtained according to the confidence and confidence true value of the candidate point.

[0015] In one embodiment of the present invention, the target positioning model has K branches, K is greater than or equal to 1, each of the branches includes a cascaded convolution unit and a target point prediction module, the input of the convolution unit of the i-th branch is the output of the convolution unit of the i-1-th branch, i is equal to {2,…,K}; the target point prediction module of each branch outputs a set of coordinates and confidence levels of candidate points.

[0016] In one embodiment of the present invention, each branch is trained in sequence. During training, the parameters in the kth branch are fine-tuned according to the coordinates and confidence of the kth group of candidate points, where k is equal to {1, 2, ..., K}.

[0017] Beneficial effects of the present invention: The present invention proposes a method for training a target positioning model, which calibrates a sample image to obtain information about a true value point, and then inputs the sample image into the target positioning model to obtain information about a candidate point. For general model training, the estimated result and the true result are one-to-one corresponding, but the present invention is somewhat special, and the number of candidate points and true value points is likely to be unequal, so a matching mechanism is introduced here, and the loss value is calculated based on the matching result, thereby realizing reliable training of the model, and in a multi-branch model structure, multiple training methods are used to complete the training of the entire model. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0019] Figure 1 A flowchart of a model training method provided by an embodiment of the present invention;

[0020] Figure 2 A schematic diagram of the training process of a single-branch target localization model provided by an embodiment of the present invention;

[0021] Figure 3 A schematic diagram of the training process of a three-branch target localization model provided by an embodiment of the present invention;

[0022] Figure 4 A structural diagram of a classification unit provided in one embodiment of the present invention;

[0023] Figure 5 A structural diagram of a regression unit provided in one embodiment of the present invention;

[0024] Figure 6 A flowchart of matching true value points and candidate points in a training process provided by an embodiment of the present invention;

[0025] Figure 7 A flowchart of loss value calculation during training provided by an embodiment of the present invention;

[0026] Figure 8 A flow chart of classification loss calculation during training provided by one embodiment of the present invention. DETAILED DESCRIPTION

[0027] The following describes the embodiments of the present invention through specific specific embodiments, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. It should be noted that the following embodiments and the features in the embodiments can be combined with each other without conflict. In addition to the specific methods, equipment, and materials used in the embodiments, according to the technical personnel in this technical field's grasp of the prior art and the records of the present invention, any methods, equipment, and materials of the prior art that are similar or equivalent to the methods, equipment, and materials described in the embodiments of the present invention can also be used to implement the present invention.

[0028] It should be understood that the terms used in the embodiments of the present invention are for describing specific embodiments rather than for limiting the scope of protection of the present invention. Unless otherwise defined, all technical and scientific terms used in the present invention have the same meaning as those generally understood by those skilled in the art.

[0029] In the following description, numerous details are discussed to provide a more thorough explanation of the embodiments of the present invention. However, it is obvious to those skilled in the art that the embodiments of the present invention can be implemented without these specific details. In some of the embodiments, well-known structures and devices are shown in the form of block diagrams rather than in detail to avoid making the embodiments of the present invention difficult to understand.

[0030] The flowchart and block diagram in the accompanying drawings illustrate the system architecture, functions and operations that may be implemented according to the methods and computer program products of various embodiments disclosed in the present invention. In this regard, each box in the flowchart or block diagram can represent a module, a program segment, or a part of a code, and the module, the program segment, or a part of the code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or the flowchart, and the combination of the boxes in the block diagram and / or the flowchart can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0031] See also Figure 1 , Figure 1A training method for a target positioning model provided in one embodiment of the present invention includes: S10, obtaining a sample image, and annotating the center point of a target to be identified in the sample image by manual annotation to obtain the coordinates of a true value point; S20, constructing and initializing a target positioning model; S30, inputting the sample image into the target positioning model to obtain the coordinates and confidence of a candidate point; S40, matching the true value point and the candidate point to obtain a matching result; S50, calculating a loss value according to the matching result, and fine-tuning the parameters in the target positioning model according to the loss value, until the training is completed to obtain a trained target positioning model.

[0032] Here, taking the positioning of fish targets as an example, the sample images can be obtained and annotated in step S10 as follows. First, place the fry of light-lipped fish in a cylindrical fish pond with a diameter of 2.5m and a height of 1.5m, and place the Boleth E3GMPCX15 high-definition underwater camera at the edge of the fish tank, and make the camera have an upward angle of 30° to record the movement of the fish school in the water. Then, the collected video is framed at intervals of 1s to obtain sample images, and the extracted images are annotated with the center points of the fish school targets through the GitHub open source project anylabeling. The annotation does not need to be very accurate, and there can be a little deviation. After the annotation is completed, the generated JSON file is used to extract the center point coordinates of each target through python to form a txt file, and 30% of the data is reserved as a verification set for model verification. Figure 2 and Figure 3 The first picture in is the sample image.

[0033] If the number of sample images is small, the sample images can be rotated, cropped, flipped, translated, and scaled to expand the data set.

[0034] See also Figure 2 and Figure 3 In a specific embodiment of the present invention, the target positioning model in step S20 has K branches, K is greater than or equal to 1, each branch (corresponding to the blue background area in the figure) includes a cascaded convolution unit (corresponding to the foundation area in the figure) and a target point prediction module (corresponding to the green background area in the figure), the input of the convolution unit of the i-th branch is the output of the convolution unit of the i-1-th branch, i is equal to {2,…,K}; the target point prediction module of each branch outputs a set of candidate point coordinates and confidences. Figure 2 The diagram shows the target positioning module when K=1. Figure 3 The figure shows the target positioning module when K=3. In addition, K can also be other integers.

[0035] A convolutional unit can consist of one or more convolutional layers, such as Figure 2 and Figure 3The convolution unit in the first branch of is composed of three cascaded first convolutional layers, second convolutional layers, and third convolutional layers; Figure 3 In the second branch and the third branch, the convolution unit can also be composed of a convolution layer. For example, the fourth convolution layer and the fifth convolution layer each constitute a convolution unit.

[0036] In each branch, the number of candidate points output by the target point prediction module is different. In the k-th branch, the target point prediction module outputs (M / 2^(k-1))×(N / 2^(k-1)) candidate points, and the k-th feature map output by the k-th convolution unit is divided into (M / 2^(k-1))×(N / 2^(k-1)) regions. Each region in the k-th feature map corresponds to a candidate point. All candidate points in the k-th feature map constitute a set of candidate points, and k is equal to {1,2,…,K}. For example Figure 3 In the above formula, the number of candidate points in the first group is M×N, the number of candidate points in the second group is (M / 2)×(N / 2), and the number of candidate points in the third group is (M / 4)×(N / 4).

[0037] The target point prediction module includes a classification unit, a regression unit, and a coordinate transformation unit ( Figure 3 The last two branches in the figure are not shown), the classification unit is used to output the probability that the center point of the identified target exists in the area where each candidate point is located, that is, the confidence of the candidate point output in step S30; the regression unit is used to output the relative coordinates of each candidate point in the corresponding area; the coordinate transformation unit is used to convert the relative coordinates of the candidate point in the corresponding area into the coordinates of the candidate point in the sample image, that is, the coordinates of the candidate point output in step S30.

[0038] See also Figure 4 In a specific embodiment of the present invention, the classification unit package consists of a convolution layer + an activation layer Convolution layer + activation layer Convolution layer, and finally a Sigmoid function to generate probability values. The overall structure is as follows Figure 4 shown.

[0039] See also Figure 5 In a specific embodiment of the present invention, the regression unit is composed of a convolutional layer + an activation layer Convolution layer + activation layer The output of the last convolutional layer is the relative coordinates of each candidate point in the corresponding area.

[0040] The reason why the true value points and candidate points need to be matched in step S40 is that the number of candidate points output by the target positioning model is fixed. For example, in one of the above embodiments, the first branch outputs M×N candidate points, and the second branch outputs (M / 2)×(N / 2) candidate points, but the number of true value points is related to the number of targets in the sample image, so there will be a difference in the number of true value points and candidate points. This is different from the common model, in which the number of true value points and predicted points is the same, so special processing is required in step S40, that is, matching operation.

[0041] See also Figure 6 In a specific embodiment of the present invention, step S40 includes: S41, generating a cost matrix according to the coordinates of the true value point and the coordinates of the candidate point, the elements of the cost matrix are the matching costs between the corresponding true value point and the candidate point; S42, obtaining a matching result according to the cost matrix.

[0042] In a specific embodiment of the present invention, the elements of the cost matrix in step S41 can be calculated according to the following formula:

[0043] ,

[0044] Where P i represents the i-th truth point, Q j represents the jth candidate point, || || 2 represents the square of the distance between two points, exp() represents the natural exponential of a given value, α and β are preset weight factors, and σ represents the standard deviation in the normal distribution. In this formula, the weight factor α is followed by the Euclidean distance between the two points, the weight factor β is followed by the probability density function of the normal distribution, and σ is the standard deviation in the normal distribution. If it is a standard normal distribution, its value is 1. By weighting these two physical quantities, the element value of the cost matrix can be obtained.

[0045] In the specific implementation, we need to ensure that the number of candidate points is greater than the number of true value points to generate enough predictions. Candidate points that are greater than the true value points will be classified as negative samples. After having the cost matrix, for example, we can use the Hungarian matching algorithm to match it and then get the matching result. When matching, we need to pay attention to the one-to-one matching principle.

[0046] See also Figure 7 In a specific embodiment of the present invention, the loss value is calculated according to the matching result, including: S51, obtaining the regression loss according to the coordinates of the true value point and the coordinates of the candidate point matching the true value point; S52, obtaining the classification loss according to the confidence of the candidate point and the matching relationship. Considering that there are a classification unit and a regression unit in the target point prediction module, these two units are independent of each other, and therefore, their losses can be calculated separately.

[0047] In a specific embodiment of the present invention, the regression loss is calculated according to the following formula:

[0048] ,

[0049] In the formula, X is the total number of true value points, Q i Indicates that P i Matching candidate points, λ 1 and λ 2 is a preset weight factor, and σ in the formula also represents the standard deviation in the normal distribution. The calculation formula of the regression loss is very similar to the calculation formula of the cost matrix in step S41, because the regression loss is also calculated based on the Euclidean distance between two points and the probability density function of the normal distribution. The difference is that the element value of the cost matrix corresponds to the matching cost between two points, and the regression loss is the sum of the relevant physical quantities of all matching points.

[0050] See also Figure 8 In a specific embodiment of the present invention, step S52 includes: S52a, according to the matching relationship, obtain the confidence truth value of each candidate point according to the following rules: if the candidate point has a matching truth point, the confidence truth value of the candidate point is 1; if the candidate point does not have a matching truth point, the confidence truth value of the candidate point is 0; S52b, according to the confidence and confidence truth value of the candidate point, obtain the classification loss.

[0051] Taking the first branch as an example, it outputs M×N candidate points, so there are exactly M×N confidences of the candidate points, but since only some of the candidate points match the true value points, and some of the candidate points do not match the true value points, the confidence truth values ​​obtained according to step S52a are also M×N, but each confidence truth value is either 0 or 1. In step S52b, the classification loss can be calculated based on the M×N confidences and the M×N confidence truth values.

[0052] In a specific embodiment of the present invention, each branch is trained in sequence. During training, the parameters in the kth branch are fine-tuned according to the coordinates and confidence of the kth group of candidate points, where k is equal to {1, 2, ..., K}.

[0053] by Figure 3 Taking the target localization model in as an example, the overall training steps are as follows: (1) first train the first branch, and then fix the parameters of the convolutional unit of the first branch; (2) then train the second branch. When training the second branch, it is equivalent to training the sub-module composed of the convolutional unit of the first branch, the convolutional unit of the second branch, and the target point prediction module of the second branch, and then fix the parameters of the convolutional unit of the second branch; (3) and so on, complete the training of other branches in sequence.

[0054] For each branch, the target point prediction module also has two sub-branches. The first sub-branch is the classification unit, and the second sub-branch is the regression unit and the coordinate transformation unit. During training, the first sub-branch can also be trained first, and the classification loss is used as the loss function to obtain the model parameters of the convolution unit and the classification unit; then the parameters of the convolution unit are fixed, and the regression loss is used as the loss function to train the second sub-branch.

[0055] It can be understood that in the specific training process, it is necessary to set the model's learning rate, batch size, optimizer type, regularization method, and loss function, and then use the prepared training set (a sample consists of a sample image and the marked true value point coordinates) for training.

[0056] Understandably, after the training is completed, the reserved validation set can be used to evaluate the model in combination with parameters such as mean square error and mean absolute error, and the model parameters with better evaluation results can be selected as the final trained target positioning model.

[0057] In general, the target positioning model in the present invention is a model obtained through multiple trainings, and its training process is slightly more complicated than traditional model training.

[0058] It should be noted that the step division of the above methods is only for clear description. When implemented, they can be combined into one step or some steps can be split and decomposed into multiple steps. As long as they contain the same logical relationship, they are all within the protection scope of this application; adding insignificant modifications to the algorithm or process or introducing insignificant designs without changing the core design of the algorithm and process are all within the protection scope of this patent.

[0059] The above embodiments are merely illustrative of the principles and effects of the present invention, and are not intended to limit the present invention. Anyone familiar with the art may modify or alter the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or alterations made by a person of ordinary skill in the art without departing from the spirit and technical concept disclosed by the present invention shall still be covered by the claims of the present invention.

Claims

1. A method for training a target positioning model, characterized in that: include: Obtain a sample image, and manually annotate the center point of the target to be identified in the sample image to obtain the coordinates of the true value point; Build and initialize the target positioning model; Inputting the sample image into the target positioning model to obtain the coordinates and confidence of the candidate points; Matching the true value point and the candidate point to obtain a matching result; Calculating a loss value according to the matching result, and fine-tuning parameters in the target positioning model according to the loss value until the training is completed to obtain a trained target positioning model; Matching the true value point and the candidate point to obtain a matching result, including: Generate a cost matrix according to the coordinates of the true value point and the coordinates of the candidate point, wherein the elements of the cost matrix are the matching costs between the corresponding true value point and the candidate point; Obtaining the matching result according to the cost matrix; The elements of the cost matrix are calculated according to the following formula: ， Where P i represents the i-th truth point, Q j represents the jth candidate point, || || 2 represents the square of the distance between two points, exp( ) represents the natural exponential of a given value, α and β are preset weight factors, and σ represents the standard deviation in the normal distribution; Calculating a loss value according to the matching result includes: Obtaining a regression loss according to the coordinates of the true value point and the coordinates of the candidate point matching the true value point; Obtaining classification loss according to the confidence and matching relationship of the candidate points; The regression loss is calculated as follows: ， In the formula, X is the total number of true value points, Q i Indicates that P i The candidate points for matching, λ1 and λ2 are preset weight factors.

2. The training method of the target positioning model according to claim 1, characterized in that: According to the confidence of the candidate points and the matching relationship, the classification loss is obtained, including: According to the matching relationship, the confidence true value of each candidate point is obtained according to the following rules: if the candidate point has a matching true value point, the confidence true value of the candidate point is 1; if the candidate point has no matching true value point, the confidence true value of the candidate point is 0; The classification loss is obtained according to the confidence and the true value of the confidence of the candidate point.

3. The training method of the target positioning model according to claim 1, characterized in that: The target positioning model has K branches, K is greater than or equal to 1, each of the branches includes a cascaded convolution unit and a target point prediction module, the input of the convolution unit of the i-th branch is the output of the convolution unit of the i-1-th branch, i is equal to {2,…,K}; the target point prediction module of each branch outputs a set of candidate point coordinates and confidences.

4. The target positioning model training method according to claim 3, characterized in that: Each branch is trained in sequence. During training, the parameters in the kth branch are fine-tuned according to the coordinates and confidence of the kth group of candidate points, where k is equal to {1, 2, …, K}.

Images