Multi-target tracking method, device, equipment and medium based on twin network

Abstract

The present invention provides a multi-target tracking method based on a twin network, comprising: extracting features from multiple target detection frames in a first detection object and multiple target search frames in a second detection object, respectively, to obtain first target features and second target features; cross-correlating the first target features with the second target features to obtain multiple predicted target objects; extracting features from multiple second target features to obtain multiple third target features, and performing target detection based on the multiple third target features to obtain multiple detected target objects; matching multiple predicted target objects with multiple detected target objects; and tracking the successfully matched predicted target objects and the unmatched detected target objects as target tracking objects. The present invention adopts a twin network for online target tracking, which can solve the problems in the prior art that the multi-target tracking method based on offline graph for data association has large computational complexity, long time consumption, and is difficult to implement.

Description

Technical Field

[0001] The present application relates to the field of target tracking technology, and in particular to a multi-target tracking method, device, equipment and medium based on a twin network. Background Art

[0002] The multi-object tracking (MOT) task refers to detecting all targets on each frame in a video or sequence of images and associating them in time across frames to form motion trajectories. Some early works viewed data association as a graph optimization problem under the tracking by detection paradigm, in which a node represents a detection box and an edge encodes the possibility (or similarity) of linking two nodes together. In fact, these methods often use a combination of visual cues and motion cues to represent a node, which usually requires a relatively large amount of computation. Moreover, they usually construct a large offline graph, and it is not easy to solve based on this graph, which limits the possibility of such methods in real-time tracking. Summary of the invention

[0003] In view of the above-mentioned shortcomings of the prior art, the present invention provides a multi-target tracking method, device, equipment and medium based on a twin network to solve the above-mentioned technical problems.

[0004] The present invention provides a multi-target tracking method based on a twin network, the method comprising:

[0005] Acquire a first detection object and a second detection object, wherein the second detection object is a current frame image, and the first detection object is a previous frame image adjacent to the current frame; the first detection object includes a plurality of target detection frames, and the second detection object includes a plurality of target search frames, and the target search frames are obtained by expanding the target detection frames;

[0006] Extracting features of the regions surrounded by the multiple target detection frames and the regions surrounded by the multiple target search frames respectively to obtain multiple first target features and multiple second target features;

[0007] Cross-correlating the plurality of first target features with the plurality of second target features to obtain a plurality of predicted target objects;

[0008] Performing feature extraction on the multiple second target features to obtain multiple third target features, and performing target detection based on the multiple third target features to obtain multiple detection target objects;

[0009] Matching the multiple predicted target objects with the multiple detected target objects;

[0010] The predicted target objects that are matched successfully and the detected target objects that are not matched successfully are tracked as target tracking objects.

[0011] In one embodiment of the present invention, feature extraction is performed on the areas surrounded by the multiple target detection frames to obtain multiple first target features, including:

[0012] Using a first branch in a pre-trained twin network to extract features of the areas surrounded by the multiple target detection boxes, to obtain multiple first target features;

[0013] Performing feature extraction on the areas surrounded by the multiple target search boxes to obtain multiple second target features, including:

[0014] The second branch in the pre-trained twin network is used to extract features of the area surrounded by the multiple target search boxes to obtain multiple second target features.

[0015] In an embodiment of the present invention, the first branch and the second branch have the same structure.

[0016] In one embodiment of the present invention, the first branch in the pre-trained twin network is used to extract features of the areas surrounded by the multiple target detection boxes to obtain multiple first target features, including:

[0017] Using the backbone network in the first branch, extracting features from the areas surrounded by the multiple target detection frames to obtain multiple first features;

[0018] Performing a pooling operation on the multiple first features to obtain multiple first target features;

[0019] The method of using the second branch in the pre-trained twin network to extract features of the areas surrounded by the multiple target search boxes to obtain multiple second target features includes:

[0020] Using the backbone network in the second branch to extract features from the areas surrounded by the multiple target search boxes, to obtain multiple second features;

[0021] A pooling operation is performed on the multiple second features to obtain multiple second target features.

[0022] In one embodiment of the present invention, after performing a pooling operation on the plurality of first features and the plurality of second features, the method further includes:

[0023] Transforming the plurality of first features so that the sizes of the plurality of first features are the same;

[0024] The plurality of second features are transformed so that the plurality of second features have the same size.

[0025] In one embodiment of the present invention, matching the multiple predicted target objects with the multiple detected target objects to obtain matching results includes:

[0026] Calculating the intersection-over-union ratio / similarity between the detected target object in the target detection result and the predicted target object in the target prediction result;

[0027] Based on the intersection-over-union ratio / similarity, the intersection-over-union ratio / similarity is matched through Hungarian matching to obtain a matching result.

[0028] In one embodiment of the present invention, after obtaining the matching result, the method further includes:

[0029] Delete the predicted target objects that have not been successfully matched.

[0030] The detected target object that has not been successfully matched is used as a new target tracking object.

[0031] The present invention provides a multi-target tracking device based on a twin network, the device comprising:

[0032] An acquisition module is used to acquire a first detection object and a second detection object, wherein the second detection object is a current frame image, and the first detection object is a previous frame image adjacent to the current frame; the first detection object includes a plurality of target detection frames, and the second detection object includes a plurality of target search frames, and the target search frames are obtained by expanding the target detection frames;

[0033] A first feature extraction module is used to extract features from the areas surrounded by the multiple target detection frames and the areas surrounded by the multiple target search frames to obtain multiple first target features and multiple second target features;

[0034] A target prediction module, used for cross-correlating the plurality of first target features with the plurality of second target features to obtain a plurality of predicted target objects;

[0035] A second feature extraction module is used to extract features from the plurality of second target features to obtain a plurality of third target features, and perform target detection according to the plurality of third target features to obtain a plurality of detection target objects;

[0036] A target matching module, used for matching the plurality of predicted target objects with the plurality of detected target objects;

[0037] The target tracking module is used to track the successfully matched predicted target objects and the unmatched detected target objects as target tracking objects.

[0038] The present invention provides an electronic device, comprising:

[0039] one or more processors;

[0040] A storage device is used to store one or more programs. When the one or more programs are executed by the one or more processors, the electronic device implements the above-mentioned multi-target tracking method based on the twin network.

[0041] The present invention provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor of a computer, the computer executes the above-mentioned multi-target tracking method based on a twin network.

[0042] Beneficial effects of the present invention: A multi-target tracking method based on a twin network in the present invention includes: obtaining a first detection object and a second detection object, the second detection object is a current frame image, and the first detection object is a previous frame image adjacent to the current frame; the first detection object includes multiple target detection frames, and the second detection object includes multiple target search frames, and the target search frames are expanded on the basis of the target detection frames; feature extraction is performed on the areas surrounded by the multiple target detection frames and the areas surrounded by the multiple target search frames respectively to obtain multiple first target features and multiple second target features; the multiple first target features are cross-correlated with the multiple second target features to obtain multiple predicted target objects; feature extraction is performed on the multiple second target features to obtain multiple third target features, and target detection is performed based on the multiple third target features to obtain multiple detected target objects; the multiple predicted target objects are matched with the multiple detected target objects; and the successfully matched predicted target objects and the unmatched detected target objects are tracked as target tracking objects. The present invention adopts a twin network for online target tracking, which can solve the problems of large computational complexity, long time consumption and difficulty in implementation of the multi-target tracking method based on offline graph for data association in the prior art; at the same time, the present invention cross-correlates the multiple first target features with the multiple second target features, generates a pixel-level response map through channel-by-channel convolution, and uses the confidence of each pixel in the response map to represent the possibility that the pixel contains the target, thereby improving the detection accuracy.

[0043] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] The drawings herein are incorporated into the specification and constitute a part of the specification, showing embodiments consistent with the present application, and together with the specification, are used to explain the principles of the present application. Obviously, the drawings described below are only some embodiments of the present application, and for those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative work. In the drawings:

[0045] Figure 1 is a schematic diagram of an implementation environment of a multi-target tracking method based on a twin network shown in an exemplary embodiment of the present application;

[0046] Figure 2 is a flowchart of a multi-target tracking method based on a twin network shown in an exemplary embodiment of the present application;

[0047] Figure 3 is a flowchart of a multi-target tracking method based on a twin network shown in another exemplary embodiment of the present application;

[0048] Figure 4 A flowchart of a process for feature extraction for the first branch of an exemplary embodiment of the present application;

[0049] Figure 5 A flowchart of a process for feature extraction for the second branch of an exemplary embodiment of the present application;

[0050] Figure 6 This is a flowchart of an exemplary embodiment of step S250 of the present application;

[0051] Figure 7 A block diagram of a multi-target tracking device based on a twin network is shown as an exemplary embodiment of the present application;

[0052] Figure 8 A schematic diagram of the structure of a computer system suitable for implementing an electronic device of an embodiment of the present application is shown. DETAILED DESCRIPTION

[0053] The following will describe the embodiments of the present invention with reference to the accompanying drawings and preferred embodiments. Those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be understood that the preferred embodiments are only for illustrating the present invention, not for limiting the scope of protection of the present invention.

[0054] It should be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present invention, and thus the drawings only show components related to the present invention rather than being drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component may be changed arbitrarily, and the component layout may also be more complicated.

[0055] In the following description, numerous details are discussed to provide a more thorough explanation of the embodiments of the present invention. However, it is obvious to those skilled in the art that the embodiments of the present invention can be implemented without these specific details. In other embodiments, well-known structures and devices are shown in the form of block diagrams rather than in detail to avoid making the embodiments of the present invention difficult to understand.

[0056] Figure 1 This is a schematic diagram of an exemplary implementation environment of a multi-target tracking method based on a twin network in this application. Figure 1 , the implementation environment includes a terminal device 101 and a server 102, and the terminal device 101 and the server 102 communicate with each other through a wired or wireless network. The terminal device can obtain a first detection object and a second detection object, the second detection object is a current frame image, and the first detection object is a previous frame image adjacent to the current frame; the first detection object includes multiple target detection frames, and the second detection object includes multiple target search frames, and the target search frame is obtained by expanding the target detection frame; the regions surrounded by the multiple target detection frames and the regions surrounded by the multiple target search frames are respectively subjected to feature extraction to obtain multiple first target features and multiple second target features; the multiple first target features are cross-correlated with the multiple second target features to obtain multiple predicted target objects; the multiple second target features are subjected to feature extraction to obtain multiple third target features, and target detection is performed according to the multiple third target features to obtain multiple detected target objects; the multiple predicted target objects are matched with the multiple detected target objects; the predicted target objects that are successfully matched and the detected target objects that are not successfully matched are tracked as target tracking objects. The present invention adopts a twin network for online target tracking, which can solve the problems of large computational complexity, long time consumption and difficulty in implementation of the multi-target tracking method based on offline graph for data association in the prior art; at the same time, the present invention cross-correlates the multiple first target features with the multiple second target features, generates a pixel-level response map through channel-by-channel convolution, and uses the confidence of each pixel in the response map to represent the possibility that the pixel contains the target, thereby improving the detection accuracy.

[0057] It should be understood that Figure 1The number of terminal devices 101 and servers 102 in the embodiment is only for illustration. Any number of terminal devices 101 and servers 102 may be provided according to actual needs.

[0058] Among them, the terminal device 101 corresponds to the client, which can be any electronic device with a user input interface, including but not limited to smart phones, tablets, laptops, computers, car computers, etc., among which the user input interface includes but is not limited to touch screens, keyboards, physical buttons, audio pickup devices, etc.

[0059] Among them, server 102 corresponds to the service end, which can be a server that provides various services. It can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It can also be a cloud server that provides cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network) and basic cloud computing services such as big data and artificial intelligence platforms. This is not restricted here.

[0060] The terminal device 101 can communicate with the server 102 through wireless networks such as 3G (third generation mobile information technology), 4G (fourth generation mobile information technology), and 5G (fifth generation mobile information technology), and this is not restricted here.

[0061] Since the prior art has the problem that the accuracy of multi-target tracking based on twin networks is not high enough, in order to solve these problems, the embodiments of the present application respectively propose a multi-target tracking method based on twin networks, a multi-target tracking device based on twin networks, an electronic device, and a computer-readable storage medium. These embodiments will be described in detail below.

[0062] See also Figure 2 , 3 , Figure 2 , 3 is a flowchart of a multi-target tracking method based on a twin network shown in an exemplary embodiment of the present application. The method can be applied to Figure 1 The implementation environment shown is specifically executed by the terminal device 101 in the implementation environment. It should be understood that the method can also be applied to other exemplary implementation environments and be specifically executed by devices in other implementation environments. This embodiment does not limit the implementation environment to which the method is applicable.

[0063] See also Figure 2 , 3 , Figure 2This is a flowchart of an exemplary multi-target tracking method based on a twin network of the present application. The multi-target tracking method based on a twin network at least includes steps S210 to S260, which are described in detail as follows:

[0064] Step S210, obtaining a first detection object and a second detection object, wherein the second detection object is a current frame image, and the first detection object is a previous frame image adjacent to the current frame; the first detection object includes a plurality of target detection frames, and the second detection object includes a plurality of target search frames, and the target search frames are obtained by expanding the target detection frames;

[0065] In this embodiment, the first detection object and the second detection object are both images, wherein the first detection object is the current frame image, and the second detection object is the previous frame image adjacent to the current frame. If the second detection object is the nth frame image, the first detection object is the n-1th frame image.

[0066] After the first detection object is detected, a corresponding detection frame will be generated. Since the detection frame contains the target object, the detection frame can be defined as a target detection frame. If the first detection object includes multiple targets, multiple target detection frames will be obtained accordingly. The second detection object is the current frame image. Due to the setting of the frame, generally speaking, the second detection object and the first detection object are images of different frames in the same scene. When the second detection object is detected, the target detection frame in the first detection object will be enlarged to obtain multiple target search frames, where the target search frame also includes the target object and the image around the target object.

[0067] Step S220, performing feature extraction on the areas surrounded by the multiple target detection frames and the areas surrounded by the multiple target search frames respectively, to obtain multiple first target features and multiple second target features;

[0068] In an exemplary embodiment of the present application, feature extraction is performed on the areas surrounded by the multiple target detection frames to obtain multiple first target features, including:

[0069] The first branch in the pre-trained twin network is used to extract features of the areas surrounded by the multiple target detection boxes to obtain multiple first target features.

[0070] Specifically, the first detection object is input into the first branch of the pre-trained twin network, and the first branch is used to extract features of the areas surrounded by multiple target detection boxes to obtain multiple first target features. It should be noted that the multiple first target features are features of multiple target objects, and each target object corresponds to a first target feature.

[0071] More specifically, see Figure 4, Figure 4 The process flow chart for feature extraction for the first branch is as follows: Figure 4 As shown, the first branch in the pre-trained twin network is used to extract features of the areas surrounded by the multiple target detection frames to obtain multiple first target features, including step S410 and step S420.

[0072] S410 uses the backbone network in the first branch to extract features from the areas surrounded by the multiple target detection frames to obtain multiple first features;

[0073] The backbone network can use neural network structures such as ResNet, InceptionNet, and EfficientNet. The backbone network is used to extract the features of the area surrounded by the target detection box.

[0074] After the first detection object is subjected to feature extraction by the backbone network of the first branch, multiple target R i s={R0,R1,......,R n1}. The features of , get multiple first features, that is

[0075] S420 performs a pooling operation on the multiple first features to obtain multiple first target features;

[0076] Specifically, ROIAlign can be used to perform a pooling operation on multiple first features to obtain multiple first target features, that is,

[0077] It should be noted that if the sizes of multiple first target features are inconsistent, all features need to be transformed into a uniform size through a resize operation.

[0078] In an exemplary embodiment of the present application, feature extraction is performed on the areas surrounded by the multiple target search boxes to obtain multiple second target features, including:

[0079] The second branch in the pre-trained twin network is used to extract features of the area surrounded by the multiple target search boxes to obtain multiple second target features.

[0080] Specifically, the second detection object is input into the second branch of the pre-trained twin network, and the second branch is used to extract features of the areas surrounded by the multiple target detection boxes to obtain multiple second target features. It should be noted that the multiple second target features are features of multiple target objects, and each target object corresponds to a second target feature.

[0081] More specifically, see Figure 5 , Figure 5 The process flow chart for feature extraction for the second branch is as follows: Figure 5 As shown, the second branch in the pre-trained twin network is used to extract features of the area surrounded by the multiple target search boxes to obtain multiple second target features, including step S510 and step S520.

[0082] S510 uses the backbone network in the second branch to extract features from the areas surrounded by the multiple target search boxes to obtain multiple second features;

[0083] The backbone network can use neural network structures such as ResNet, InceptionNet, and EfficientNet. The backbone network is used to extract the features of the area surrounded by the target detection box.

[0084] After the second detection object is subjected to feature extraction by the backbone network of the second branch, multiple target S i s={S0,S1,......,S n1} features, and obtain multiple second features, namely

[0085] S420 performs a pooling operation on the plurality of second features to obtain a plurality of second target features;

[0086] Specifically, ROIAlign can be used to perform a pooling operation on multiple second features to obtain multiple second target features, that is,

[0087] It should be noted that if the sizes of multiple second target features are inconsistent, all features need to be transformed into a uniform size through a resize operation.

[0088] Step S230, cross-correlating the plurality of first target features with the plurality of second target features to obtain a plurality of predicted target objects;

[0089] The first target features, namely and multiple second target features After the cross-correlation operation, the predicted target object R is obtained it 's={R 0t ',R 1t ',......,R n2t '}.

[0090] The cross-correlation operation described in this application refers to generating a pixel-level response map through channel-by-channel convolution, and the confidence of each pixel in the response map is used to represent the possibility that the pixel contains the target.

[0091] Step S240, performing feature extraction on the plurality of second target features to obtain a plurality of third target features, and performing target detection according to the plurality of third target features to obtain a plurality of detection target objects;

[0092] Specifically, a CNN network (Convolutional Neural Network) can be used to extract the second target feature, and target detection can be performed according to the multiple third target features, which can be achieved through the corresponding head in the multi-task module.

[0093] The multi-task module, in addition to the detection head for target detection, also includes a detection head for image classification and a detection head for image segmentation. The multi-task module can realize visual tasks such as detection, classification, and segmentation. In this embodiment, the detection head for target detection is used to realize the detection of multiple targets.

[0094] After the backbone network of the second branch, a CNN network is connected. After the second detection object is feature extracted through the backbone network, a CNN network is used to further extract fine features, and then target detection is performed based on the fine features to obtain multiple detection target objects, R it s={R 0t ,R 1t ,......,R n2t}.

[0095] Step S250, matching the multiple predicted target objects with the multiple detected target objects;

[0096] See also Figure 6 , Figure 6 FIG. 2 is a schematic diagram of an exemplary embodiment of step S250 of the present application; Figure 6 As shown, matching the multiple predicted target objects with the multiple detected target objects to obtain a matching result includes step S610 and step S620;

[0097] Step S610, calculating the intersection-over-union ratio / similarity between the detected target object in the target detection result and the predicted target object in the target prediction result;

[0098] In this step, the intersection over union (IoU) is a concept used in object detection. IoU calculates the overlap ratio of the "predicted bounding box" and the "real bounding box", that is, the ratio of their intersection and union. The ideal situation is complete overlap, that is, the ratio is 1.

[0099] In this embodiment, if the intersection-over-union ratio is greater than or equal to a preset threshold, the match is considered successful, and if the intersection-over-union ratio is less than the preset threshold, the match is considered unsuccessful.

[0100] Step S620: Based on the IoU / similarity, the IoU / similarity is matched by Hungarian matching (Hungarian algorithm).

[0101] Step S260: Track the successfully matched predicted target object and the unmatched detected target object as target tracking objects.

[0102] In one embodiment, the predicted target objects that have not been successfully matched are deleted, and the detected target objects that have not been successfully matched are used as new target tracking objects.

[0103] For the predicted target objects that are not matched successfully, they are regarded as disappeared targets and removed from the tracking target list; for the detected target objects that are not matched, they are regarded as newly appeared targets and added to the tracking target list.

[0104] The present invention discloses a multi-target tracking algorithm based on a twin network, which uses a twin network for online target tracking, and can solve the problems of large computational complexity, long time consumption and difficulty in implementation of the multi-target tracking method based on offline graph for data association in the prior art; at the same time, the present invention cross-correlates the multiple first target features with the multiple second target features, generates a pixel-level response map through channel-by-channel convolution, and uses the confidence of each pixel in the response map to indicate the possibility that the pixel contains a target, thereby improving detection accuracy.

[0105] Figure 7 is a block diagram of a multi-target tracking device based on a twin network shown in an exemplary embodiment of the present application. The device can be applied to Figure 1 The device can also be applied to other exemplary implementation environments and specifically configured in other devices. This embodiment does not limit the implementation environment to which the device is applied.

[0106] like Figure 7 As shown, the present application provides a multi-target tracking device based on a twin network, the device comprising:

[0107] The acquisition module 710 is used to acquire a first detection object and a second detection object, wherein the second detection object is a current frame image, and the first detection object is a previous frame image adjacent to the current frame; the first detection object includes a plurality of target detection frames, and the second detection object includes a plurality of target search frames, and the target search frames are obtained by expanding the target detection frames;

[0108] A first feature extraction module 720 is used to extract features from the areas surrounded by the multiple target detection frames and the areas surrounded by the multiple target search frames to obtain multiple first target features and multiple second target features;

[0109] A target prediction module 730, configured to cross-correlate the plurality of first target features with the plurality of second target features to obtain a plurality of predicted target objects;

[0110] A second feature extraction module 740 is used to extract features from the plurality of second target features to obtain a plurality of third target features, and perform target detection according to the plurality of third target features to obtain a plurality of detected target objects;

[0111] A target matching module 750 is used to track the successfully matched predicted target objects and the unmatched detected target objects as target tracking objects;

[0112] The target tracking module 760 is used to track the successfully matched target prediction object as a target tracking object.

[0113] It should be noted that the multi-target tracking device based on the twin network provided in the above embodiment and the multi-target tracking method based on the twin network provided in the above embodiment belong to the same concept, and the specific manner in which each module and unit performs the operation has been described in detail in the method embodiment and will not be repeated here. In actual applications, the multi-target tracking device based on the twin network provided in the above embodiment can distribute the above functions to different functional modules as needed, that is, divide the internal structure of the device into different functional modules to complete all or part of the functions described above, and this is not limited here.

[0114] An embodiment of the present application also provides an electronic device, comprising: one or more processors; a storage device for storing one or more programs, when the one or more programs are executed by the one or more processors, the electronic device implements the multi-target tracking method based on the twin network provided in the above-mentioned embodiments.

[0115] Figure 8 The structure diagram of the computer system suitable for implementing the electronic device of the embodiment of the present application is shown. It should be noted that: Figure 8 The computer system 800 of the electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0116] like Figure 8As shown, the computer system 800 includes a central processing unit (CPU) 801, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 802 or the program loaded from the storage part 808 to the random access memory (RAM) 803, such as executing the method described in the above embodiment. In the RAM 803, various programs and data required for system operation are also stored. The CPU 801, ROM 802 and RAM 803 are connected to each other through a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0117] The following components are connected to the I / O interface 805: an input section 806 including a keyboard, a mouse, etc.; an output section 807 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker; a storage section 808 including a hard disk, etc.; and a communication section 809 including a network interface card such as a LAN (Local Area Network) card, a modem, etc. The communication section 809 performs communication processing via a network such as the Internet. The drive 77 is also connected to the I / O interface 805 as needed. A removable medium 811, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 810 as needed so that a computer program read therefrom is installed into the storage section 808 as needed.

[0118] In particular, according to an embodiment of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a computer program for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication section 809, and / or installed from a removable medium 811. When the computer program is executed by a central processing unit (CPU) 801, various functions defined in the system of the present application are executed.

[0119] It should be noted that the computer-readable medium shown in the embodiment of the present application can be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium can be, for example, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media can include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, wherein a computer-readable computer program is carried. This propagated data signal can take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. A computer program contained on a computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0120] The flowchart and block diagram in the accompanying drawings illustrate the possible architecture, functions and operations of the system, method and computer program product according to various embodiments of the present application. Wherein, each box in the flowchart or block diagram can represent a module, a program segment, or a part of the code, and the above-mentioned module, program segment, or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0121] The units involved in the embodiments described in this application may be implemented by software or hardware, and the units described may also be set in a processor. The names of these units do not, in some cases, constitute limitations on the units themselves.

[0122] Another aspect of the present application further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor of a computer, enables the computer to execute the multi-target tracking method based on a twin network as described above. The computer-readable storage medium may be included in the electronic device described in the above embodiment, or may exist independently without being assembled into the electronic device.

[0123] Another aspect of the present application also provides a computer program product or a computer program, which includes a computer instruction stored in a computer-readable storage medium. The processor of the computer device reads the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device executes the multi-target tracking method based on the twin network provided in each of the above embodiments.

[0124] The above embodiments are merely illustrative of the principles and effects of the present invention, and are not intended to limit the present invention. Anyone familiar with the technology may modify or change the above embodiments without violating the spirit and scope of the present invention. Therefore, all equivalent modifications or changes made by a person of ordinary skill in the art without departing from the spirit and technical ideas disclosed by the present invention shall still be covered by the claims of the present invention.

Claims

1. A multi-target tracking method based on a twin network, characterized in that: The method comprises: Acquire a first detection object and a second detection object, wherein the second detection object is a current frame image, and the first detection object is a previous frame image adjacent to the current frame; the first detection object includes a plurality of target detection frames, and the second detection object includes a plurality of target search frames, and the target search frames are obtained by expanding the target detection frames; Extracting features of the regions surrounded by the multiple target detection frames and the regions surrounded by the multiple target search frames respectively to obtain multiple first target features and multiple second target features; Cross-correlating the plurality of first target features with the plurality of second target features to obtain a plurality of predicted target objects; Performing feature extraction on the multiple second target features to obtain multiple third target features, and performing target detection based on the multiple third target features to obtain multiple detection target objects; Matching the multiple predicted target objects with the multiple detected target objects; Tracking the successfully matched predicted target object and the unmatched detected target object as target tracking objects; Extracting features from the areas surrounded by the multiple target detection frames to obtain multiple first target features, including: extracting features from the areas surrounded by the multiple target detection frames using a first branch in a pre-trained Siamese network to obtain multiple first target features; Performing feature extraction on the area surrounded by the multiple target search boxes to obtain multiple second target features, including: utilizing a second branch in a pre-trained twin network to perform feature extraction on the area surrounded by the multiple target search boxes to obtain multiple second target features.

2. The multi-target tracking method according to claim 1, characterized in that: The first branch and the second branch have the same structure.

3. The multi-target tracking method according to claim 1 or 2, characterized in that: The first branch in the pre-trained twin network is used to extract features of the areas surrounded by the multiple target detection frames to obtain multiple first target features, including: Using the backbone network in the first branch, extracting features from the areas surrounded by the multiple target detection frames to obtain multiple first features; Performing a pooling operation on the multiple first features to obtain multiple first target features; The method of using the second branch in the pre-trained twin network to extract features of the areas surrounded by the multiple target search boxes to obtain multiple second target features includes: Using the backbone network in the second branch to extract features from the areas surrounded by the multiple target search boxes, to obtain multiple second features; A pooling operation is performed on the multiple second features to obtain multiple second target features.

4. The multi-target tracking method according to claim 3, characterized in that: After performing a pooling operation on the plurality of first features and the plurality of second features, the method further includes: Transforming the plurality of first features so that the sizes of the plurality of first features are the same; The plurality of second features are transformed so that the plurality of second features have the same size.

5. The multi-target tracking method according to claim 1, characterized in that: The matching of the plurality of predicted target objects with the plurality of detected target objects to obtain a matching result includes: Calculate the intersection-over-union ratio between the detected target object and the predicted target object; Based on the intersection-and-union ratio, the intersection-and-union ratio is matched through Hungarian matching to obtain a matching result.

6. The multi-target tracking method according to claim 1 or 5, characterized in that: After obtaining the matching result, the method further includes: Delete the predicted target objects that have not been successfully matched; The detected target object that has not been successfully matched is used as a new target tracking object.

7. A multi-target tracking device based on a twin network, characterized in that: The device comprises: An acquisition module is used to acquire a first detection object and a second detection object, wherein the second detection object is a current frame image, and the first detection object is a previous frame image adjacent to the current frame; the first detection object includes a plurality of target detection frames, and the second detection object includes a plurality of target search frames, and the target search frames are obtained by expanding the target detection frames; A first feature extraction module is used to extract features from the areas surrounded by the multiple target detection frames and the areas surrounded by the multiple target search frames, respectively, to obtain multiple first target features and multiple second target features; the first target features are obtained by extracting features from the areas surrounded by the multiple target detection frames through the first branch in the pre-trained twin network; the second target features are obtained by extracting features from the areas surrounded by the multiple target search frames through the second branch in the pre-trained twin network; A target prediction module, used for cross-correlating the plurality of first target features with the plurality of second target features to obtain a plurality of predicted target objects; A second feature extraction module is used to extract features from the plurality of second target features to obtain a plurality of third target features, and perform target detection according to the plurality of third target features to obtain a plurality of detection target objects; A target matching module, used for matching the plurality of predicted target objects with the plurality of detected target objects; The target tracking module is used to track the successfully matched predicted target objects and the unmatched detected target objects as target tracking objects.

8. An electronic device, characterized in that: The electronic device comprises: one or more processors; A storage device for storing one or more programs, which, when executed by the one or more processors, enables the electronic device to implement the multi-target tracking method based on a twin network as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and when the computer program is executed by a processor of a computer, the computer is enabled to execute the multi-target tracking method based on a twin network as described in any one of claims 1 to 6.

Images