Target tracking model training method, target tracking method, device and electronic equipment

Abstract

Embodiments of the present application provide a target tracking model training method, a target tracking method, a device and electronic equipment, and relate to the technical field of image processing. The method comprises: acquiring a first sample video image; based on each sample video frame, a plurality of sample video frame pairs are constructed; each sample video frame pair is input into a first target tracking model to be trained respectively, to obtain a predicted position of a target object in a second sample video frame in the sample video frame pair; based on the difference between the real position and the predicted position of the target object in the second sample video frame, a loss function value is calculated; the model parameters of the first target tracking model to be trained are adjusted according to the loss function value until a preset convergence condition is reached, to obtain a trained first target tracking model, which can improve the accuracy of the tracking result.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, in particular to a target tracking model training method, a target tracking method, a device and an electronic device. BACKGROUND

[0002] With the development of image processing technology, video image processing is applied in more and more scenarios. For example, a target object in a video image is tracked based on a target tracking model, so that the position of the target object in each video frame can be obtained.

[0003] In related technologies, a target tracking model is trained based on only sample images containing a target object (i.e. positive samples), and then the trained target tracking model is used to process images to be detected.

[0004] However, the target tracking model obtained based on the above method can only learn the image features of positive samples, and when the above target tracking model is used for detection, the images containing other objects similar to the target object may be missed or misdetected, resulting in low accuracy of the tracking result. SUMMARY

[0005] The purpose of the embodiments of the present application is to provide a target tracking model training method, a target tracking method, a device and an electronic device to improve the accuracy of the tracking result. The specific technical solutions are as follows:

[0006] In a first aspect of the present application, a target tracking model training method is provided, which comprises:

[0007] obtaining a first sample video image; wherein the first sample video image contains sample video frames, and the real position of a target object in each sample video frame;

[0008] based on the sample video frames, a plurality of sample video frame pairs are constructed; wherein each sample video frame pair contains a first sample video frame and a second sample video frame; the first sample video frame is a positive sample video frame containing the target object; at least one negative sample video frame pair exists in the plurality of sample video frame pairs; the second sample video frame in the negative sample video frame pair is a negative sample video frame; the target object does not exist in the negative sample video frame;

[0009] each sample video frame pair is input into a first target tracking model to be trained, and the predicted position of the target object in the second sample video frame in the sample video frame pair is obtained;

[0010] a loss function value is calculated based on the difference between the real position and the predicted position of the target object in the second sample video frame.

[0011] According to the loss function value, the model parameters of the first target tracking model to be trained are adjusted until a preset convergence condition is reached, and a trained first target tracking model is obtained.

[0012] In some embodiments, the first sample video image includes a plurality of sample video frame sequences; each sample video frame sequence includes a plurality of sample video frames that are time-sequentially adjacent;

[0013] The plurality of sample video frame pairs are constructed based on each sample video frame, including:

[0014] For each sample video frame sequence, a positive sample video frame is selected from the sample video frame sequence as a first sample video frame;

[0015] If a positive example sample video frame pair needs to be constructed at present, another sample video frame that exists the target object is selected from the sample video frame sequence as a second sample video frame; wherein the second sample video frame in the positive example sample video frame pair is a positive sample video frame;

[0016] If a negative example sample video frame pair needs to be constructed at present, a sample video frame that does not exist the target object is selected from the sample video frame sequence as a second sample video frame.

[0017] In some embodiments, the selecting another sample video frame that exists the target object from the sample video frame sequence as a second sample video frame includes:

[0018] From the sample video frame sequence, a sample video frame that has a time-sequential distance greater than a preset threshold from the first sample video frame and exists the target object is selected as a second sample video frame.

[0019] In some embodiments, the method further includes:

[0020] If a negative example sample video frame pair needs to be constructed at present, and there is no negative sample video frame in the sample video frame sequence, a negative sample video frame is selected from another sample video frame sequence that is closest in time sequence to the sample video frame sequence as a second sample video frame.

[0021] In some embodiments, the obtaining a first sample video image includes:

[0022] A second sample video image is obtained, and a real position of the target object in each key video frame included in the second sample video image is obtained; wherein the second sample video image is obtained based on an original sample video image;

[0023] obtaining, based on the pre-trained second target tracking model and the real positions of the target object in each key video frame, a predicted position of the target object in each sample video frame of the second sample video image except the key video frames as a first prediction result;

[0024] obtaining, based on the pre-trained third target tracking model and the real positions of the target object in each key video frame, a predicted position of the target object in each sample video frame of the second sample video image except the key video frames as a second prediction result;

[0025] For each sample video frame in the second sample video image, if the first prediction result of the sample video frame is inconsistent with the second prediction result, the sample video frame is determined as a difference sample video frame;

[0026] dividing the second sample video image into a plurality of sample video frame sequences; wherein a sample video frame sequence contains sample video frames belonging to one shot;

[0027] from the plurality of sample video frame sequences obtained by division, determining a sample video frame sequence containing the difference sample video frame as a sample video frame contained in the first sample video image;

[0028] obtaining the real positions of the target object labeled in the sample video frames in the first sample video image.

[0029] In some embodiments, the obtaining the second sample video image and the real positions of the target object in each key video frame contained in the second sample video image comprises:

[0030] obtaining a third sample video image and the real positions of the target object in each key video frame contained in the third sample video image;

[0031] reducing the image size of a fourth sample video image by a preset multiple to obtain a second sample video image; wherein the fourth sample video image is obtained based on the third sample video image;

[0032] reducing the real positions of the target object in each key video frame contained in the fourth sample video image by the preset multiple to obtain the real positions of the target object in each key video frame contained in the second sample video image.

[0033] In some embodiments, the calculating the loss function value based on the difference between the real position and the predicted position of the target object in the second sample video frame comprises:

[0034] compute a first sub-loss function value based on a difference between the real position and a predicted position of the target object in the second sample video frame;

[0035] compute a second sub-loss function value based on a difference between an image feature of an image region at the real position in the second sample video frame and an image feature of an image region at the predicted position in the second sample video frame;

[0036] compute the loss function value based on the first sub-loss function value and the second sub-loss function value.

[0037] In some embodiments, the first target tracking model is a twin-structured target tracking model.

[0038] In a second aspect of the embodiments of the present application, a target tracking method is further provided, and the method comprises:

[0039] obtaining a video image to be detected;

[0040] inputting the video image to be detected into a pre-trained target tracking model to obtain a predicted position of a target object in the video image to be detected; wherein the target tracking model is obtained by training according to any one of the target tracking model training methods described above.

[0041] In a third aspect of the embodiments of the present application, a target tracking model training device is further provided, and the device comprises:

[0042] a first sample video image obtaining module, configured to obtain a first sample video image; wherein the first sample video image comprises sample video frames, and a real position of a target object in each sample video frame;

[0043] a sample video frame pair constructing module, configured to construct a plurality of sample video frame pairs based on the sample video frames; wherein each sample video frame pair comprises a first sample video frame and a second sample video frame; the first sample video frame is a positive sample video frame in which the target object exists; at least one negative example sample video frame pair exists in the plurality of sample video frame pairs; the second sample video frame in the negative example sample video frame pair is a negative sample video frame; the target object does not exist in the negative sample video frame;

[0044] an input module, configured to input each sample video frame pair into a first target tracking model to be trained respectively to obtain a predicted position of the target object in the second sample video frame in the sample video frame pair;

[0045] a loss function value computing module, configured to compute a loss function value based on a difference between the real position and a predicted position of the target object in the second sample video frame;

[0046] The parameter adjustment module is configured to adjust model parameters of the first target tracking model to be trained according to the loss function value until a preset convergence condition is reached, so as to obtain a trained first target tracking model.

[0047] In some embodiments, the first sample video image includes a plurality of sample video frame sequences; each sample video frame sequence includes a plurality of time-sequentially adjacent sample video frames;

[0048] The sample video frame pair construction module includes:

[0049] The first sample video frame determination submodule is configured to, for each sample video frame sequence, select a positive sample video frame from the sample video frame sequence as a first sample video frame.

[0050] The second sample video frame determination submodule is configured to, if a positive example sample video frame pair needs to be constructed, select another sample video frame in which the target object exists from the sample video frame sequence as a second sample video frame; wherein the second sample video frame in the positive example sample video frame pair is the positive sample video frame.

[0051] The third sample video frame determination submodule is configured to, if a negative example sample video frame pair needs to be constructed, select a sample video frame in which the target object does not exist from the sample video frame sequence as a second sample video frame.

[0052] In some embodiments, the second sample video frame determination submodule is specifically configured to:

[0053] select, from the sample video frame sequence, a sample video frame in which the target object exists and whose time-sequential distance from the first sample video frame is greater than a preset threshold, as the second sample video frame.

[0054] In some embodiments, the apparatus further includes:

[0055] The fourth sample video frame determination submodule is configured to, if a negative example sample video frame pair needs to be constructed and no negative sample video frame exists in the sample video frame sequence, select a negative sample video frame from another sample video frame sequence closest in time sequence to the sample video frame sequence as a second sample video frame.

[0056] In some embodiments, the first sample video image acquisition module includes:

[0057] The second sample video image acquisition submodule is configured to acquire a second sample video image and a real position of the target object in each key video frame included in the second sample video image; wherein the second sample video image is obtained based on an original sample video image.

[0058] The first prediction result obtaining module is configured to obtain, based on the pre-trained second target tracking model and the real positions of the target object in each key video frame, a predicted position of the target object in each sample video frame of the second sample video image except the key video frames as a first prediction result.

[0059] The second prediction result obtaining module is configured to obtain, based on the pre-trained third target tracking model and the real positions of the target object in each key video frame, a predicted position of the target object in each sample video frame of the second sample video image except the key video frames as a second prediction result.

[0060] The difference sample video frame determining module is configured to, for each sample video frame in the second sample video image, determine the sample video frame as a difference sample video frame if the first prediction result of the sample video frame is inconsistent with the second prediction result.

[0061] The shot dividing module is configured to divide the second sample video image into a plurality of sample video frame sequences; wherein the sample video frames included in one sample video frame sequence belong to one shot.

[0062] The first sample video image determining module is configured to determine, from the plurality of sample video frame sequences obtained by division, a sample video frame sequence containing the difference sample video frame as a sample video frame included in the first sample video image.

[0063] The first real position obtaining module is configured to obtain the real positions of the target object obtained by labeling the sample video frames in the first sample video image.

[0064] In some embodiments, the second sample video image obtaining submodule includes:

[0065] The third sample video image obtaining unit is configured to obtain a third sample video image and real positions of the target object in each key video frame included in the third sample video image.

[0066] The first reducing unit is configured to reduce the image size of the fourth sample video image by a preset multiple to obtain a second sample video image; wherein the fourth sample video image is obtained based on the third sample video image.

[0067] The second reducing unit is configured to reduce the real positions of the target object in each key video frame included in the fourth sample video image by the preset multiple to obtain the real positions of the target object in each key video frame included in the second sample video image.

[0068] In some embodiments, the loss function value calculation module is specifically configured to:

[0069] calculate a first sub-loss function value based on a difference between the real position and the predicted position of the target object in the second sample video frame;

[0070] calculate a second sub-loss function value based on a difference between an image feature of an image region at the real position in the second sample video frame and an image feature of an image region at the predicted position in the second sample video frame;

[0071] calculate the loss function value based on the first sub-loss function value and the second sub-loss function value.

[0072] In some embodiments, the first target tracking model is a target tracking model with a twin structure.

[0073] In a fourth aspect of the embodiments of the present application, a target tracking device is further provided, and the device comprises:

[0074] a to-be-detected image acquisition module configured to acquire a to-be-detected video image;

[0075] a predicted position acquisition module configured to input the to-be-detected video image into a pre-trained target tracking model to obtain a predicted position of a target object in the to-be-detected video image, wherein the target tracking model is obtained by training according to any one of the target tracking model training methods described above.

[0076] In yet another aspect of the embodiments of the present application, an electronic device is further provided, which comprises a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory complete communication with each other through the communication bus;

[0077] the memory is configured to store a computer program;

[0078] the processor is configured to execute the program stored on the memory to implement any one of the target tracking model training methods or the target tracking methods described above.

[0079] In yet another aspect of the embodiments of the present application, a computer readable storage medium is further provided, and the computer readable storage medium stores a computer program, wherein the computer program is executed by a processor to implement any one of the target tracking model training methods or the target tracking methods described above.

[0080] In yet another aspect of the embodiments of the present application, a computer program product containing instructions is further provided, which, when executed on a computer, causes the computer to execute any one of the target tracking model training methods or the target tracking methods described above.

[0081] The embodiment of the present application provides a target tracking model training method, which comprises the following steps: obtaining a first sample video image; wherein the first sample video image comprises sample video frames, and the real position of a target object in each sample video frame; based on the sample video frames, a plurality of sample video frame pairs are constructed; wherein each sample video frame pair comprises a first sample video frame and a second sample video frame; the first sample video frame is a positive sample video frame containing the target object; at least one negative sample video frame pair exists in the plurality of sample video frame pairs; the second sample video frame in the negative sample video frame pair is a negative sample video frame; the target object does not exist in the negative sample video frame; each sample video frame pair is input into a first target tracking model to be trained, so as to obtain the predicted position of the target object in the second sample video frame in the sample video frame pair; based on the difference between the real position and the predicted position of the target object in the second sample video frame, a loss function value is calculated; the model parameters of the first target tracking model to be trained are adjusted according to the loss function value until a preset convergence condition is reached, so as to obtain a trained first target tracking model.

[0082] Based on the above processing, in the process of training the target tracking model, the positive sample video frame containing the target object and the negative sample video frame not containing the target object are used to train the target tracking model, that is, the target tracking model not only learns the image features of the positive sample video frame containing the target object, but also learns the image features of the negative sample video frame not containing the target object. Furthermore, for the sample video frame not containing the target object, a relatively accurate tracking result can also be obtained, that is, for the sample video frame not containing the target object, even if it contains other objects similar to the target object, when the above target tracking model is used for detection, the situation of missing detection and false detection can be avoided, and the accuracy of the tracking result is improved.

[0083] The interference of other objects similar to the target object on the target tracking model when tracking the target object in the video image can be effectively reduced, and the accuracy of the target tracking model detection can be improved. BRIEF DESCRIPTION OF DRAWINGS

[0084] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or the prior art description will be briefly introduced below.

[0085] Figure 1 A flowchart of a target tracking model training method provided by the embodiment of the present application;

[0086] Figure 2 A flowchart of another target tracking model training method provided by the embodiment of the present application;

[0087] Figure 3A flowchart of target tracking model training provided for an embodiment of the present application;

[0088] Figure 4 A flowchart of determining a first sample video image provided for an embodiment of the present application;

[0089] Figure 5 A flowchart of a target tracking method provided for an embodiment of the present application;

[0090] Figure 6 A structural diagram of a target tracking model training device provided for an embodiment of the present application;

[0091] Figure 7 A structural diagram of a target tracking device provided for an embodiment of the present application;

[0092] Figure 8 A structural diagram of an electronic device provided for an embodiment of the present application. DETAILED DESCRIPTION

[0093] The technical solutions in the embodiments of the present application will be described below with reference to the drawings.

[0094] With the development of image processing technology, video image processing is applied in more and more scenarios. For example, during the production of a movie, a movie producer needs to code a certain object (e.g., a semi-transparent watermark or a commodity trademark) or a person (i.e., a target object in the embodiments of the present application) appearing in a lens. Therefore, a video image to be processed can be input into a target tracking model, the position of the target object in each video frame is determined, and coding processing is performed, so that the effect of coding the target object in the video image can be achieved.

[0095] For another example, when a factory security officer observes a construction site through a video image captured by a monitoring device, the position of a target person (i.e., a target object in the embodiments of the present application) in the video image needs to be obtained. Therefore, a target tracking model can process the video image to determine the position of the target object in each video frame, and subsequently, the position of the target object can be marked by an external rectangular frame, so that the factory security officer can clearly understand the movement of the target object in the construction site.

[0096] In the related art, when a target tracking model is trained, the target tracking model is trained only based on sample video images containing target objects. Therefore, the obtained target tracking model can only learn the image features of positive sample video frames in which target objects exist. For sample video images containing other objects similar to the target objects, when the above target tracking model is used for detection, missing detection, false detection, and the like may occur, resulting in low accuracy of the detection result.

[0097] To solve the above problems, the embodiment of the present application provides a target tracking model training method. The method can be applied to an electronic device, which can be trained based on the method provided by the embodiment of the present application to obtain a target tracking model. In addition, the trained target tracking model can also be used to detect the input video image to be detected. Alternatively, the electronic device can be trained based on the method provided by the embodiment of the present application to obtain a target tracking model, and send the trained target tracking model to other devices. Correspondingly, other devices can use the trained target tracking model to detect the input video image to be detected.

[0098] Referring to Figure 1 , Figure 1 The flowchart of the target tracking model training method provided by the embodiment of the present application can include the following steps:

[0099] S101: Obtain a first sample video image.

[0100] The first sample video image contains each sample video frame, and the real position of the target object in each sample video frame.

[0101] S102: Based on each sample video frame, a plurality of sample video frame pairs are constructed.

[0102] Each sample video frame pair includes: a first sample video frame and a second sample video frame; the first sample video frame is a positive sample video frame with a target object; there is at least one negative example sample video frame pair in the plurality of sample video frame pairs; the second sample video frame in the negative example sample video frame pair is a negative sample video frame; and there is no target object in the negative sample video frame.

[0103] S103: Each sample video frame pair is input into a first target tracking model to be trained respectively, to obtain the predicted position of the target object in the second sample video frame in the sample video frame pair.

[0104] S104: Based on the difference between the real position and the predicted position of the target object in the second sample video frame, a loss function value is calculated.

[0105] S105: According to the loss function value, the model parameters of the first target tracking model to be trained are adjusted until a preset convergence condition is reached, to obtain a trained first target tracking model.

[0106] Based on the above processing, in the process of training the target tracking model, the positive sample video frames in which the target object exists and the negative sample video frames in which the target object does not exist are used to train the target tracking model, that is, the target tracking model not only learns the image features of the positive sample video frames containing the target object, but also learns the image features of the negative sample video frames not containing the target object. Furthermore, for the sample video frames not containing the target object, a relatively accurate tracking result can also be obtained, that is, for the sample video frames not containing the target object, even if they contain other objects similar to the target object, when detection is performed using the above target tracking model, the situation of missing detection, false detection, etc. can be avoided, and the accuracy of the tracking result is improved.

[0107] For step S101, the target object represents the object that needs to be tracked at present. If there is one object that needs to be tracked in the first sample video image, the object is determined as the target object, and training is performed based on the above steps S101-S105 for the target object. If there are multiple objects that need to be tracked in the first sample video image, then for each object, training is performed based on the above steps S101-S105, and the first target tracking model obtained can realize tracking of multiple objects.

[0108] For each sample video frame, if the target object exists in the sample video frame, the sample video frame can be referred to as a positive sample video frame. Correspondingly, if the target object does not exist in the sample video frame, the sample video frame can be referred to as a negative sample video frame.

[0109] The position (i.e., the real position) of the target object in each sample video frame can be artificially annotated or determined in advance based on a target tracking algorithm.

[0110] For example, the real position of the target object in the sample video frame can be represented based on the minimum bounding rectangle. For any positive sample video frame, the electronic device can obtain the position of the minimum bounding rectangle of the target object in the positive sample video frame, and represent the real position of the target object in the positive sample video frame based on the top-left corner coordinates (x, y) of the minimum bounding rectangle, and the width value (Width) and the height value (Height) of the minimum bounding rectangle.

[0111] In addition, for the negative sample video frame, a label can be set for the negative sample video frame to indicate that the target object does not exist in the negative sample video frame.

[0112] In actual application scenarios, there can be a case that a target object exists in a sample video frame, and part of the image features of the target object is missing. For example, part of the target object in the sample video frame is occluded by other objects. That is, only part of the target object exists in the sample video frame, or the target object is deformed, blurred, or the like in the sample video frame, and the sample video frame can be referred to as a difficult sample video frame. The difficult sample video frame can be a difference sample video frame in the present application, and the process of obtaining the difference sample video frame will be described in detail in subsequent embodiments. It can be understood that the difficult sample video frame is a special positive sample video frame.

[0113] For step S102, a sample video frame pair (Pairs) is composed of two sample video frames, that is, a first sample video frame (which can also be referred to as a template video frame) and a second sample video frame in the present application. For any sample video frame pair, the first sample video frame is a positive sample video frame, and the second sample video frame can be a positive sample video frame or a negative sample video frame. The second sample video frame in the positive example sample video frame pair is a positive sample video frame, and the second sample video frame in the negative example sample video frame pair is a negative sample video frame. The positive example sample video frame pair in the present application can also be referred to as a positive example, and the negative example sample video frame pair can also be referred to as a negative example. In addition, the sample video frame pair containing the second sample video frame as a difficult sample video frame can be referred to as a difficult example sample video frame pair (which can also be referred to as a difficult example), and it can be understood that the difficult example is a special positive example.

[0114] At least one negative example exists in the plurality of constructed sample video frame pairs.

[0115] For example, the number of negative examples in the plurality of constructed sample video frame pairs can be pre-set. The proportion of negative examples in the plurality of constructed sample video frame pairs is 40%, or can also be 35%, but is not limited thereto.

[0116] For step S103, for any sample video frame pair, the first target tracking model can obtain the position of the target object in the second sample video frame in the sample video frame pair (that is, the predicted position) based on the real position of the target object in the first sample video frame.

[0117] The first target tracking model can track a single target in a sample video frame pair. The input of the first target tracking model is two images, one of which is a template image, and the other of which is a to-be-tracked image. The template image contains the object to be tracked, and the principle is to match the image region occupied by the object in the template image in the to-be-tracked image to determine the position of the object in the to-be-tracked image.

[0118] In the present application, for a sample video frame pair, since the first sample video frame therein contains the target object, the first sample video frame can be taken as a template image, and then, based on the image region occupied by the target object in the first sample video frame, matching can be performed in the second sample video frame to obtain the position (i.e., the predicted position) of the target object in the second sample video frame.

[0119] For steps S104 and S105, for example, the loss function value can be calculated based on a cross-entropy loss function. For example, the model parameters of the first target tracking model can be adjusted in a gradient descent manner.

[0120] In some embodiments, the first target tracking model is a twin structure target tracking model.

[0121] For example, the first target tracking model can be a Siam FC (Fully Convolutional) model, a Siam RPN++ (Region Proposal Network-based twin network) model.

[0122] Alternatively, the first target tracking model can also be a Transformer Tracker (Transformer-based target tracking) model.

[0123] In some embodiments, the first sample video image contains a plurality of sample video frame sequences; each sample video frame sequence contains a plurality of time-sequentially adjacent sample video frames. As shown in Figure 2 Figure 2 The flowchart of another target tracking model training method provided by the embodiments of the present application is based on Figure 1 Step S102, which comprises:

[0124] S1021: For each sample video frame sequence, select a positive sample video frame from the sample video frame sequence as the first sample video frame.

[0125] S1022: If a positive example sample video frame pair needs to be constructed at present, select another sample video frame containing a target object from the sample video frame sequence as the second sample video frame.

[0126] Among them, the second sample video frame in the positive example sample video frame pair is the positive sample video frame.

[0127] S1023: If a negative example sample video frame pair needs to be constructed at present, select a sample video frame not containing a target object from the sample video frame sequence as the second sample video frame.

[0128] ​In the embodiments of the present application, the first sample video image comprises a plurality of sample video frame sequences, which can be determined based on other sample video images.

[0129] Each sample video frame sequence comprises a plurality of temporally adjacent sample video frames. The process of specifically determining the sample video frame sequence will be described in detail in subsequent embodiments.

[0130] For example, for any sample video frame sequence, the electronic device can record the type of each sample video frame in the plurality of temporally adjacent sample video frames included in the sample video frame sequence, and the type of the sample video frame includes: positive sample video frame and negative sample video frame. For example, the electronic device can record each positive sample video frame and its corresponding timestamp through the positive sample pool, and record each negative sample video frame and its corresponding timestamp through the negative sample pool. The positive sample pool and the negative sample pool can be understood as a collection of sample video frames.

[0131] For each sample video frame sequence, a plurality of sample video frame pairs can be constructed. When constructing any sample video frame pair, a positive sample video frame can be randomly determined from the plurality of sample video frames included in the sample video frame sequence as a first sample video frame.

[0132] It can be understood that when constructing any sample video frame pair, after step S1021 is performed, any one of steps S1022 or S1023 can be performed. If step S1022 is performed, it means that a sample video frame pair containing two positive sample video frames is generated.

[0133] Step S1022 can be implemented based on at least any of the following ways:

[0134] Method one: the electronic device can randomly select one positive sample video frame other than the first sample video frame as a second sample video frame.

[0135] Method two: step S1022 comprises: selecting a sample video frame with a temporal distance greater than a preset threshold from the first sample video frame and having a target object in the sample video frame sequence as a second sample video frame.

[0136] In the embodiments of the present application, for each sample video frame sequence, when a positive example needs to be determined, and after the electronic device determines the first sample video frame and the temporal sequence of the first sample video frame in the sample video frame sequence, the positive sample video frame with a temporal distance greater than a preset threshold from the first sample video frame can be determined according to the above temporal sequence to obtain the second sample video frame. Wherein, the temporal distance can be represented by the frame number of the video frame, and correspondingly, the preset threshold can be 100.

[0137] For example, when a positive example needs to be determined, the electronic device can randomly select a video frame from the positive sample pool as a first sample video frame, and then can select a sample video frame with a time sequence distance greater than a preset threshold from the first sample video frame as a second sample video frame according to the time sequence order of the sample video frames in the positive sample pool.

[0138] Based on the above processing, a certain time interval can be ensured between the determined first sample video frame and the second sample video frame, and accordingly, a certain time shift can be ensured between the target object in the first sample video frame and the target object in the second sample video frame. For example, the size of the target object changes, the environment in which the target object is located changes, and the image features of the target object partially change, etc. Further, the constructed sample video frame pair can have diversity, and the robustness of the target tracking model can be improved.

[0139] After step S1021 is performed, if step S1023 is performed, it indicates that a sample video frame pair containing one positive sample video frame and one negative sample video frame is generated.

[0140] If there is a negative sample video frame in the sample video frame sequence, the electronic device can determine any negative sample video frame in the sample video frame sequence as the second sample video frame.

[0141] For example, when a negative example needs to be determined, the electronic device can randomly select a video frame from the positive sample pool as a first sample video frame, and then can randomly select a video frame from the negative sample pool as a second sample video frame.

[0142] In one embodiment, the method further includes:

[0143] If a negative example sample video frame pair needs to be constructed at present, and there is no negative sample video frame in the sample video frame sequence, a negative sample video frame is selected from other sample video frame sequences closest in time sequence distance to the sample video frame sequence as a second sample video frame.

[0144] For each sample video frame sequence, there can be no negative sample video frame in the sample video frame sequence, i.e., each video frame in the sample video frame sequence contains a target object. Further, when a negative example needs to be constructed based on the sample video frame sequence, a negative sample video frame cannot be directly obtained from the sample video frame sequence. In order to learn the image features of the video frames contained in the sample video frame sequence, the electronic device can determine a sample video frame sequence closest in time sequence distance to the sample video frame sequence from other sample video frame sequences containing negative sample video frames. Further, a negative sample video frame is selected from the sample video frame sequence as a second sample video frame.

[0145] Based on the above processing, when constructing negative examples, the negative sample video frame corresponding to the first sample video frame can be determined. Since the negative sample video frame and the first sample video frame may belong to the same sample video frame sequence or other sample video frame sequences with similar time sequences, it is more likely to ensure that the image features contained in the determined negative sample video frame are closer to the image features contained in the first sample video frame. Further, the first sample video frame can avoid false tracking in other sample video frame sequences, and the accuracy of the target tracking model detection can be improved.

[0146] In some embodiments, step S104 comprises:

[0147] Step one: calculating a first sub-loss function value based on the difference between the real position and the predicted position of the target object in the second sample video frame.

[0148] Step two: calculating a second sub-loss function value based on the difference between the image features of the image region at the real position in the second sample video frame and the image features of the image region at the predicted position in the second sample video frame.

[0149] Step three: calculating a loss function value based on the first sub-loss function value and the second sub-loss function value.

[0150] The first sub-loss function value can be calculated based on a cross-entropy loss function.

[0151] For example, the electronic device can calculate the first sub-loss function value based on the loss function (1), which is as follows:

[0152] loss = a [- (1-scale) * label cls *log (predict cls ))-scale * (1-label_cls) log (1-predict_cls)] + (1-a) * l2_loss (label_loc, predict_loc) …… (1)

[0153] Wherein, loss represents the first sub-loss function value, scale represents a hyperparameter for balancing positive sample video frames and negative sample video frames, scale can be determined based on the proportion of positive sample video frames and negative sample video frames in the first sample video image, and the value range is a floating point number between (0, 1); a represents a hyperparameter for balancing cross-entropy loss and Euclidean distance loss, a is a floating point number between (0, 1); label_cls is one-hot encoding for representing label categories, for example, for positive sample video frames, its label cls is 1, and for negative sample video frames, its labelcls =0; predict_cls is the prediction probability vector used to represent the category of the predicted value, that is, the confidence of each pixel at the predicted position of the target object in the obtained second sample video frame. The confidence of a pixel represents the probability that the similar point is the target object; l2_loss represents the Euclidean distance loss between the predicted position and the true position.

[0154] Furthermore, for each pair of sample video frames, the electronic device can calculate a second sub-loss function value based on the difference between the image features of the image region at the actual location in the second sample video frame and the image features of the image region at the predicted location in the second sample video frame. For example, the electronic device can utilize a self-attention mechanism to obtain the mask information of the target object, and based on the mask information, obtain the image features of the image region at the actual location in the second sample video frame and the image features of the image region at the predicted location, and calculate the difference between the two image features to obtain the second sub-loss function value. For example, the electronic device can calculate a weighted sum of the first and second sub-loss function values ​​to obtain the loss function value. For example, the image features of the aforementioned image region can be represented as spatial domain features or frequency domain features of the image region.

[0155] Based on the above processing, the target tracking model can learn not only the image features of positive sample video frames containing the target object, but also the image features of negative sample video frames that do not contain the target object. Furthermore, based on the self-attention mechanism and the difference between the image features of the image region at the actual location and the image features of the image region at the predicted location, the loss function value is obtained, which can further suppress background interference and improve the accuracy of the target tracking model detection.

[0156] like Figure 3 As shown, Figure 3 This is a flowchart of a target tracking model training method provided in an embodiment of the present invention.

[0157] Figure 3 In step S301: Obtain the annotation sequence and annotation results.

[0158] That is, to obtain the first sample video image, wherein the first sample video image contains each sample video frame and the real position of the target object in each sample video frame.

[0159] Step S302: The model is trained using a pair training method, with a 40% probability of taking a negative example.

[0160] That is, a plurality of sample video frame pairs are constructed based on each sample video frame. The number of negative examples in the plurality of constructed sample video frame pairs can be 40%. Wherein, the negative example is the negative example sample video frame pair in the present application, and the negative example contains one positive sample video frame and one negative sample video frame.

[0161] Step S303: Determine whether a positive example is obtained. If yes, execute step S304; if no, execute step S305.

[0162] That is, it is determined whether the sample video frame pair currently to be constructed is a positive example. Wherein, the positive example is the positive example sample video frame pair in the present application, and the positive example contains two positive sample video frames.

[0163] Step S304: Randomly take one picture from the sequence as a template image, and then randomly take another picture at an interval of 100 to obtain a positive example.

[0164] That is, for each sample video frame sequence, if a positive example is constructed, a positive sample video frame is selected from the sample video frame sequence as a first sample video frame; and a sample video frame with a time sequence distance greater than a preset threshold from the first sample video frame and with a target object is selected from the sample video frame sequence as a second sample video frame. Wherein, the preset threshold is 100.

[0165] Step S305: Randomly take one picture from the sequence as a template image, and then randomly take one picture from the corresponding negative example sequence or other negative example sequence to obtain a negative example.

[0166] That is, for each sample video frame sequence, if a negative example is constructed, a positive sample video frame is selected from the sample video frame sequence as a first sample video frame; and a sample video frame without a target object is selected from the sample video frame sequence as a second sample video frame, or a negative sample video frame is selected from other sample video frame sequences as a second sample video frame.

[0167] Step S306: Calculate the loss function value according to the obtained Pairs.

[0168] That is, each sample video frame pair is input into the first target tracking model to be trained to obtain the predicted position of the target object in the second sample video frame in the sample video frame pair; and the loss function value is calculated based on the difference between the real position and the predicted position of the target object in the second sample video frame. Subsequently, the model parameters of the first target tracking model to be trained are adjusted according to the loss function value until a preset convergence condition is reached to obtain the trained first target tracking model.

[0169] In some embodiments, step S101 comprises:

[0170] Step 1: obtaining a second sample video image, and a real position of the target object in each key video frame included in the second sample video image.

[0171] The second sample video image is obtained based on the original sample video image.

[0172] Step 2: obtaining, based on a pre-trained second target tracking model and the real position of the target object in each key video frame, a predicted position of the target object in each sample video frame other than the key video frames in the second sample video image as a first prediction result.

[0173] Step 3: obtaining, based on a pre-trained third target tracking model and the real position of the target object in each key video frame, a predicted position of the target object in each sample video frame other than the key video frames in the second sample video image as a second prediction result.

[0174] Step 4: for each sample video frame in the second sample video image, if the first prediction result of the sample video frame is inconsistent with the second prediction result, determining the sample video frame as a difference sample video frame.

[0175] Step 5: dividing the second sample video image into a plurality of sample video frame sequences.

[0176] Each sample video frame sequence includes sample video frames belonging to one shot.

[0177] Step 6: determining, from the plurality of sample video frame sequences obtained by the division, a sample video frame sequence including a difference sample video frame as a sample video frame included in the first sample video image.

[0178] Step 7: obtaining a real position of the target object labeled in the sample video frame in the first sample video image.

[0179] In an implementation manner, the second sample video image represents the original sample video image.

[0180] In another implementation manner, the second sample video image is obtained based on the original sample video image, and a process of obtaining the second sample video image based on the original sample video image will be described in detail in subsequent embodiments.

[0181] For example, the second sample video image can be processed based on a shot detection algorithm to divide the second sample video image into a plurality of sample video frame sequences, and each sample video frame sequence includes a plurality of time-sequentially adjacent sample video frames.

[0182] For any key video frame in the second sample video image, the real position of the target object in the key video frame can be represented based on the minimum bounding rectangle. Then, based on the second target tracking model and the real position of the target object in each key video frame, the predicted position of the target object in each sample video frame in the second sample video image except the key video frame can be obtained as the first prediction result. That is, the key video frame is taken as a template video frame, and the predicted position of the target object in other video frames is calculated. Similarly, based on the third target tracking model and the real position of the target object in each key video frame, the predicted position of the target object in each sample video frame in the second sample video image except the key video frame can be obtained as the second prediction result.

[0183] The second target tracking model and the third target tracking model can be any one of a Siam FC (Fully Convolutional) model, a Siam RPN++ (Region Proposal Network-based Siamese Network) model, and a Transformer Tracker (Transformer Network-based Target Tracking) model. The second target tracking model and the third target tracking model are different models, and the first target tracking model can be the same as the second target tracking model or the third target tracking model.

[0184] For each sample video frame in the second sample video image, if the first prediction result and the second prediction result of the sample video frame are inconsistent, it indicates that the results obtained by detecting the target object in the sample video frame based on different target tracking models are different, i.e., the current sample video frame cannot be effectively detected. For example, only a part of the target object exists in the sample video frame, or the target object is deformed, blurred, or the like in the sample video frame. Therefore, the sample video frame can be labeled based on manual labeling to obtain the real position of the target object in the sample video frame. Correspondingly, the tracking model is trained based on manual labeling, so that the tracking model obtained by training can effectively detect the sample video frame. In the present application, detecting a video frame means determining the position of the target object in the video frame.

[0185] For example, if the positions represented by the first prediction result and the second prediction result are different, it can be determined that the first prediction result and the second prediction result are inconsistent; or if the Intersection of Union (IoU) between the positions represented by the first prediction result and the second prediction result is less than a preset IoU, it can be determined that the first prediction result and the second prediction result are inconsistent. Alternatively, the confidence of each pixel point at the predicted position of the target object in the first prediction result (which can be referred to as the first confidence) and the confidence of each pixel point at the predicted position of the target object in the second prediction result (which can be referred to as the second confidence) can be obtained, and then if the difference between the first confidence and the second confidence is greater than a preset difference, it can be determined that the first prediction result and the second prediction result are inconsistent.

[0186] In addition, since other sample video frames in the sequence of sample video frames to which the sample video frame belongs have a high similarity with the sample video frame, i.e., the other sample video frames cannot be effectively detected at present, the other sample video frames can also be labeled based on manual labeling to obtain the real positions of the target objects in the other sample video frames.

[0187] That is, the sample video frame can be determined as a difference sample video frame. Furthermore, the sequence of sample video frames (which can also be referred to as a split sequence) to which each difference sample video frame belongs can be determined as a first sample video image. Subsequently, each sample video frame in the first sample video image can be labeled by manual labeling to determine the real positions of the target objects in each sample video frame in the first sample video image.

[0188] For example, after each split sequence is determined, each split sequence can be divided, for example, into a group of 20 video frames, and labeled by manual labeling.

[0189] Based on the above processing, the sequence of video frames that need to be further learned in the second sample video image can be determined based on different target tracking models, and then trained based on the real positions of the target objects, so as to improve the accuracy of detection of the target tracking model.

[0190] In some embodiments, step 1 includes:

[0191] Step 8: obtaining a third sample video image, and the real positions of the target objects in each key video frame included in the third sample video image.

[0192] Step 9: reducing the image size of the fourth sample video image by a preset multiple to obtain a second sample video image.

[0193] The fourth sample video image is obtained based on the third sample video image.

[0194] Step 10: reducing the real position of the target object in each key video frame contained in the fourth sample video image by a preset multiple, to obtain the real position of the target object in each key video frame contained in the second sample video image.

[0195] For example, the third sample video image can represent an original sample video image. For example, when it is required to code a commodity trademark in a movie, the third sample video image is the original file of the movie. When it is required to mark a target person in a video image captured by a monitoring device, the third sample video image is the original file of the monitoring image.

[0196] For any key video frame in the third sample video image, the real position of the target object in the key video frame can be represented based on a minimum bounding rectangle.

[0197] The fourth sample video image is obtained based on the third sample video image. For example, the fourth sample video image can be the third sample video image. For another example, before step 7, the method further includes: frame extraction on the third sample video image to obtain the fourth sample video image.

[0198] For example, the sample video frames can be obtained from the third sample video image based on a preset sampling frequency, as the fourth sample video image. Since the number of video frames in the original sample video image is large, frame extraction on the third sample video image can reduce the number of video frames processed during training, thereby improving the training efficiency.

[0199] The electronic device can reduce the image size of the fourth sample video image by a preset multiple to obtain the second sample video image. Correspondingly, the image size corresponding to the target object in each key video frame contained in the fourth sample video image is also changed, and therefore, the size of the minimum bounding rectangle of the target object in each key video frame contained in the fourth sample video image is reduced by the preset multiple. It can be understood that, for any key video frame, the relative position of the target object in the key video frame before and after the reduction is unchanged, and the proportion of the size of the minimum bounding rectangle of the target object in the entire image size is unchanged. The preset multiple can be 1 / 4, or 1 / 8.

[0200] Based on this, the image size of the sample video frame in the second video image can be reduced, and therefore, the image size of the sample video frame in the first video image can be reduced, and correspondingly, the calculation amount can be reduced and the training efficiency can be improved when the model is trained based on the first video image.

[0201] As shown in FIG. 7, the electronic device can obtain the second sample video image based on the fourth sample video image. Figure 4 Figure 4 ​A flowchart for determining a first sample video image is provided for an embodiment of the present application.

[0202] Figure 4 In the step S401, preliminary data preparation is performed.

[0203] That is, a third sample video image is obtained, and the real positions of the target object in each key video frame contained in the third sample video image. The third sample video image is frame-extracted to obtain a fourth sample video image. The image size of the fourth sample video image is reduced by a preset multiple to obtain a second sample video image. Based on the real positions of the target object in each key video frame contained in the fourth sample video image and the preset multiple, the real positions of the target object in each key video frame contained in the second sample video image are determined.

[0204] In the step S402, a tracking result is obtained based on an existing algorithm.

[0205] That is, based on the pre-trained second target tracking model and the real positions of the target object in each key video frame, the predicted positions of the target object in each sample video frame other than the key video frames of the second sample video image are obtained as a first prediction result; and based on the pre-trained third target tracking model and the real positions of the target object in each key video frame, the predicted positions of the target object in each sample video frame other than the key video frames of the second sample video image are obtained as a second prediction result.

[0206] In the step S403, frame-level difference points, i.e., points to be optimized, are obtained based on the tracking result.

[0207] For each sample video frame in the second sample video image, if the first prediction result of the sample video frame is inconsistent with the second prediction result, the sample video frame is determined as a difference sample video frame.

[0208] In the step S404, clip information containing difference points is obtained in combination with a lens detection result.

[0209] That is, from the second sample video image, a sample video frame sequence is determined as a first sample video image to obtain the real positions of the target object obtained by manually labeling the sample video frames in the first sample video image; the sample video frames contained in one sample video frame sequence belong to one lens, and each sample video frame sequence contains a difference sample video frame.

[0210] In the step S405, based on the clip information and the frame-extracted images of the entire video, a reference image and a sequence to be labeled corresponding to each clip are obtained.

[0211] For each determined sample video frame sequence, a key video frame in the sample video frame sequence is determined to correspond to a video frame in the third sample video image as a reference image for reference during manual annotation. The sequence to be annotated contains video frames corresponding to video frames in the third sample video image except the key video frame in the sample video frame sequence.

[0212] Step S406: The sequence to be annotated is split into short sequences with 20 video frames in each group, and provided to the annotators for annotation.

[0213] In some embodiments, referring to Figure 5 , Figure 5 A flowchart of a target tracking method provided by an embodiment of the present application is shown in FIG. 6. The method includes the following steps:

[0214] S501: Obtain a video image to be detected.

[0215] S502: Input the video image to be detected into a pre-trained target tracking model to obtain a predicted position of a target object in the video image to be detected.

[0216] The target tracking model is obtained by training according to any of the target tracking model training methods described above.

[0217] Based on the same inventive concept, an embodiment of the present application further provides a target tracking model training device, referring to Figure 6 , Figure 6 A structural diagram of a target tracking model training device provided by an embodiment of the present application is shown in FIG. 7. The device includes the following components:

[0218] A first sample video image acquisition module 601 is configured to acquire a first sample video image. The first sample video image contains sample video frames, and a real position of a target object in each sample video frame.

[0219] A sample video frame pair construction module 602 is configured to construct a plurality of sample video frame pairs based on the sample video frames. Each sample video frame pair contains a first sample video frame and a second sample video frame. The first sample video frame is a positive sample video frame in which the target object exists. At least one negative example sample video frame pair exists in the plurality of sample video frame pairs. The second sample video frame in the negative example sample video frame pair is a negative sample video frame in which the target object does not exist.

[0220] An input module 603 is configured to input each sample video frame pair into a first target tracking model to be trained respectively to obtain a predicted position of the target object in the second sample video frame in the sample video frame pair.

[0221] The loss function value calculation module 604 is configured to calculate a loss function value based on a difference between the real position and the predicted position of the target object in the second sample video frame.

[0222] The parameter adjustment module 605 is configured to adjust the model parameters of the first target tracking model to be trained according to the loss function value until a preset convergence condition is reached, so as to obtain the trained first target tracking model.

[0223] In some embodiments, the first sample video image includes a plurality of sample video frame sequences; each sample video frame sequence includes a plurality of time-sequentially adjacent sample video frames.

[0224] The sample video frame pair construction module 602 includes:

[0225] The first sample video frame determination sub-module is configured to, for each sample video frame sequence, select a positive sample video frame from the sample video frame sequence as a first sample video frame.

[0226] The second sample video frame determination sub-module is configured to, if a positive example sample video frame pair needs to be constructed at present, select another sample video frame in which the target object exists from the sample video frame sequence as a second sample video frame; wherein the second sample video frame in the positive example sample video frame pair is a positive sample video frame.

[0227] The third sample video frame determination sub-module is configured to, if a negative example sample video frame pair needs to be constructed at present, select a sample video frame in which the target object does not exist from the sample video frame sequence as a second sample video frame.

[0228] In some embodiments, the second sample video frame determination sub-module is specifically configured to:

[0229] select, from the sample video frame sequence, a sample video frame in which the target object exists and whose time-sequential distance from the first sample video frame is greater than a preset threshold, as the second sample video frame.

[0230] In some embodiments, the apparatus further includes:

[0231] The fourth sample video frame determination module is configured to, if a negative example sample video frame pair needs to be constructed at present and there is no negative sample video frame in the sample video frame sequence, select a negative sample video frame from another sample video frame sequence closest in time sequence to the sample video frame sequence as a second sample video frame.

[0232] a second sample video image obtaining sub-module, configured to obtain a second sample video image, and the second sample video image containing a real position of the target object in each key video frame;

[0233] a first prediction result obtaining module, configured to obtain, based on a pre-trained second target tracking model and the real position of the target object in each key video frame, a predicted position of the target object in each sample video frame of the second sample video image except the key video frames as a first prediction result;

[0234] a second prediction result obtaining module, configured to obtain, based on a pre-trained third target tracking model and the real position of the target object in each key video frame, a predicted position of the target object in each sample video frame of the second sample video image except the key video frames as a second prediction result;

[0235] a difference sample video frame determining module, configured to, for each sample video frame in the second sample video image, if the first prediction result of the sample video frame is inconsistent with the second prediction result, determine the sample video frame as a difference sample video frame;

[0236] a shot dividing module, configured to divide the second sample video image into a plurality of sample video frame sequences; wherein the sample video frames contained in one sample video frame sequence belong to one shot;

[0237] a first sample video image determining module, configured to determine, from the plurality of sample video frame sequences obtained by division, a sample video frame sequence containing the difference sample video frame as a sample video frame contained in a first sample video image;

[0238] a first real position obtaining module, configured to obtain a real position of the target object obtained by labeling the sample video frames in the first sample video image.

[0239] In some embodiments, the second sample video image obtaining sub-module comprises:

[0240] a third sample video image obtaining unit, configured to obtain a third sample video image, and the third sample video image containing a real position of the target object in each key video frame;

[0241] a first reducing unit, configured to reduce the image size of a fourth sample video image by a preset multiple to obtain a second sample video image; wherein the fourth sample video image is obtained based on the third sample video image;

[0242] A second reducing unit is configured to reduce the real position of the target object in each key video frame contained in the fourth sample video image by the preset multiple, to obtain the real position of the target object in each key video frame contained in the second sample video image.

[0243] In some embodiments, the loss function value calculation module 604 is specifically configured to:

[0244] calculate a first sub-loss function value based on the difference between the real position and the predicted position of the target object in the second sample video frame;

[0245] calculate a second sub-loss function value based on the difference between the image features of the image region at the real position in the second sample video frame and the image features of the image region at the predicted position in the second sample video frame;

[0246] calculate the loss function value based on the first sub-loss function value and the second sub-loss function value.

[0247] In some embodiments, the first target tracking model is a twin structure target tracking model.

[0248] The embodiment of the present application also provides a target tracking device, referring to Figure 7 , Figure 7 The embodiment of the present application provides a structure diagram of a target tracking device, and the device comprises:

[0249] The image to be detected is acquired by the image to be detected acquisition module 701.

[0250] The predicted position acquisition module 702 is configured to input the image to be detected into a pre-trained target tracking model to obtain the predicted position of the target object in the image to be detected, wherein the target tracking model is obtained by training according to the target tracking model training method described above.

[0251] The embodiment of the present application also provides an electronic device, as shown in Figure 8 The processor 801, the communication interface 802 and the memory 803 can communicate with each other through the communication bus 804.

[0252] The memory 803 is configured to store a computer program.

[0253] The processor 801 is configured to execute the program stored in the memory 803, to implement the target tracking model training method or the target tracking method described above.

[0254] The communication bus mentioned above can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For the convenience of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or only one type of bus.

[0255] The communication interface is used for communication between the terminal and other devices.

[0256] The memory can include a Random Access Memory (RAM) and can also include a non-volatile memory, such as at least one disk memory. Optionally, the memory can also be at least one storage device located away from the processor.

[0257] The processor mentioned above can be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc. It can also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component.

[0258] In another embodiment provided by the application, a computer readable storage medium is also provided, and the computer readable storage medium stores a computer program. When the computer program is executed by a processor, the target tracking model training method or the target tracking method in any of the above embodiments is implemented.

[0259] In another embodiment provided by the application, a computer program product containing instructions is also provided, and when the computer program product is run on a computer, the computer executes the target tracking model training method or the target tracking method in any of the above embodiments.

[0260] In the embodiments described above, all or some of the steps can be implemented by software, hardware or firmware, or any combination thereof. When implemented by software, all or some of the steps can be implemented in the form of one or more computer programs. The computer program can be stored in any computer readable medium, and loaded into the computer system for execution. The computer readable medium includes: a computer storage medium and a computer communication medium. The computer storage medium includes: volatile media (such as random access memory (RAM) and others) and non-volatile media (such as read-only memory (ROM), floppy disks, CD-ROMs, optical disks, hard disks, etc.). The computer communication medium includes: computer networks and other media.

[0261] It should be noted that, in this document, the terms such as first and second are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. In addition, the terms "comprise", "comprise" or any other variant thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or other elements inherent to such a process, method, article or device. Without more limitations, the element defined by the statement "comprises a" does not exclude the presence of additional identical elements in the process, method, article or device that includes the element.

[0262] Each of the embodiments in the specification is described in a related manner, and the same or similar parts between each of the embodiments can be referred to each other. Each of the embodiments focuses on the difference from other embodiments. In particular, for the device, electronic device, computer readable storage medium, and computer program product embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the part of the method embodiment.

[0263] The above merely provides the preferred embodiments of the application, and not intended to limit the protection scope of the application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the application shall fall within the protection scope of the application.

Claims

1. A method for training a target tracking model, characterized in that, The method comprises: obtaining a second sample video image and real positions of a target object in each key video frame contained in the second sample video image; wherein the second sample video image is obtained based on an original sample video image; based on a pre-trained second target tracking model and the real positions of the target object in each key video frame, obtaining predicted positions of the target object in each sample video frame other than the key video frames in the second sample video image as a first prediction result; based on a pre-trained third target tracking model and the real positions of the target object in each key video frame, obtaining predicted positions of the target object in each sample video frame other than the key video frames in the second sample video image as a second prediction result; for each sample video frame in the second sample video image, if the first prediction result of the sample video frame is inconsistent with the second prediction result, the sample video frame is determined as a difference sample video frame; dividing the second sample video image into a plurality of sample video frame sequences; wherein the sample video frames contained in one sample video frame sequence belong to one shot; from the plurality of sample video frame sequences obtained by division, determining a sample video frame sequence containing the difference sample video frame as sample video frames contained in a first sample video image; obtaining real positions of the target object obtained by labeling sample video frames in the first sample video image; wherein the first sample video image contains each sample video frame and the real positions of the target object in each sample video frame; based on each sample video frame, constructing a plurality of sample video frame pairs; wherein each sample video frame pair contains: a first sample video frame and a second sample video frame; the first sample video frame is a positive sample video frame in which the target object exists; at least one negative example sample video frame pair exists in the plurality of sample video frame pairs; the second sample video frame in the negative example sample video frame pair is a negative sample video frame; the target object does not exist in the negative sample video frame; respectively inputting each sample video frame pair into a first target tracking model to be trained to obtain predicted positions of the target object in the second sample video frame in the sample video frame pair; based on the difference between the real positions and the predicted positions of the target object in the second sample video frame, calculating a loss function value; adjusting model parameters of the first target tracking model to be trained according to the loss function value until a preset convergence condition is reached to obtain a trained first target tracking model.

2. The method of claim 1, wherein, The first sample video image contains a plurality of sample video frame sequences; each sample video frame sequence contains a plurality of time-sequentially adjacent sample video frames; the method comprises: for each sample video frame sequence, selecting a positive sample video frame from the sample video frame sequence as a first sample video frame; If a positive example sample video frame pair is currently needed to be constructed, another sample video frame in which the target object exists is selected from the sample video frame sequence as a second sample video frame; wherein the second sample video frame in the positive example sample video frame pair is a positive sample video frame; If a negative example sample video frame pair is currently needed to be constructed, a sample video frame in which the target object does not exist is selected from the sample video frame sequence as a second sample video frame.

3. The method of claim 2, wherein, The selecting another sample video frame in which the target object exists from the sample video frame sequence as a second sample video frame comprises: A sample video frame in which the target object exists and whose time sequence distance from the first sample video frame is greater than a preset threshold is selected from the sample video frame sequence as a second sample video frame.

4. The method of claim 2, wherein, The method further comprises: If a negative example sample video frame pair is currently needed to be constructed and no negative sample video frame exists in the sample video frame sequence, a negative sample video frame is selected from another sample video frame sequence closest in time sequence to the sample video frame sequence as a second sample video frame.

5. The method of claim 1, wherein, The acquiring a second sample video image and the real position of the target object in each key video frame contained in the second sample video image comprises: A third sample video image is acquired and the real position of the target object in each key video frame contained in the third sample video image is acquired; An image size of a fourth sample video image is reduced by a preset multiple to obtain a second sample video image; wherein the fourth sample video image is obtained based on the third sample video image; The real position of the target object in each key video frame contained in the fourth sample video image is reduced by the preset multiple to obtain the real position of the target object in each key video frame contained in the second sample video image.

6. The method of claim 1, wherein, The calculating a loss function value based on a difference between the real position and the predicted position of the target object in the second sample video frame comprises: A first sub-loss function value is calculated based on a difference between the real position and the predicted position of the target object in the second sample video frame; A second sub-loss function value is calculated based on a difference between an image feature of an image region at the real position in the second sample video frame and an image feature of an image region at the predicted position in the second sample video frame; The loss function value is calculated based on the first sub-loss function value and the second sub-loss function value.

7. The method of claim 1, wherein, The first target tracking model is a target tracking model of a twin structure.

8. A target tracking method characterized by, The method comprises: An image to be detected is acquired; The image to be detected is input into a pre-trained target tracking model to obtain a predicted position of a target object in the image to be detected; wherein the target tracking model is obtained by training according to any one of the methods in claims 1-7. 9.A target tracking model training apparatus, characterized by comprising: The device comprises: The first sample video image obtaining module is specifically configured to obtain a second sample video image and a real position of a target object in each key video frame included in the second sample video image, wherein the second sample video image is obtained based on an original sample video image; based on a pre-trained second target tracking model and the real position of the target object in each key video frame, a predicted position of the target object in each sample video frame other than the key video frame in the second sample video image is obtained as a first prediction result; based on a pre-trained third target tracking model and the real position of the target object in each key video frame, a predicted position of the target object in each sample video frame other than the key video frame in the second sample video image is obtained as a second prediction result; for each sample video frame in the second sample video image, if the first prediction result of the sample video frame is inconsistent with the second prediction result, the sample video frame is determined as a difference sample video frame; the second sample video image is divided into a plurality of sample video frame sequences; wherein the sample video frames included in one sample video frame sequence belong to one shot; from the plurality of sample video frame sequences obtained by division, a sample video frame sequence including the difference sample video frame is determined as a sample video frame included in the first sample video image; a real position of the target object obtained by labeling the sample video frames in the first sample video image is obtained; wherein the first sample video image includes each sample video frame and the real position of the target object in each sample video frame; The sample video frame pair constructing module is configured to construct a plurality of sample video frame pairs based on each sample video frame; wherein each sample video frame pair includes: a first sample video frame and a second sample video frame; the first sample video frame is a positive sample video frame in which the target object exists; at least one negative example sample video frame pair exists in the plurality of sample video frame pairs; the second sample video frame in the negative example sample video frame pair is a negative sample video frame; the target object does not exist in the negative sample video frame; The input module is configured to input each sample video frame pair into a first target tracking model to be trained respectively to obtain a predicted position of the target object in the second sample video frame in the sample video frame pair; The loss function value calculating module is configured to calculate a loss function value based on a difference between the real position and the predicted position of the target object in the second sample video frame; The parameter adjusting module is configured to adjust model parameters of the first target tracking model to be trained according to the loss function value until a preset convergence condition is reached to obtain a trained first target tracking model.

10. A target tracking device, characterized by, The device comprises: The to-be-detected image obtaining module is configured to obtain a to-be-detected video image; The predicted position obtaining module is configured to input the to-be-detected video image into a pre-trained target tracking model to obtain a predicted position of a target object in the to-be-detected video image; wherein the target tracking model is obtained by training according to any one of the methods in claims 1-7.

11. An electronic device, comprising: The computer readable storage medium stores a computer program, and the computer program is executed by the processor to realize the method steps in any one of claims 1-7 or 8. The computer readable storage medium stores a computer program, and the computer program is executed by the processor to realize the method steps in any one of claims 1-7 or 8. The computer readable storage medium stores a computer program, and the computer program is executed by the processor to realize the method steps in any one of claims 1-7 or 8.

12. A computer-readable storage medium, characterized in that, ​

Images