Target tracking method, apparatus, and device

Abstract

The present disclosure provides a target tracking method, device and equipment, relates to the field of artificial intelligence, in particular to the fields of deep learning, image processing and computer vision, and can be applied to scenarios such as automatic driving. A specific embodiment of the method comprises: acquiring a current frame image, a previous frame image and a previous frame intermediate feature matrix; based on the current frame image, the previous frame image and the previous frame intermediate feature matrix, time sequence fusion features are obtained by fusion; target detection is performed based on the time sequence fusion features to obtain a target detection result; and target tracking is performed based on the target detection result to obtain a target tracking result. This embodiment uses time sequence feature information contained in adjacent frames for supplementation, thereby improving the stability and generalization of target detection.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of artificial intelligence, in particular to the technical field of deep learning, image processing and computer vision, and can be applied to scenarios such as automatic driving. BACKGROUND

[0002] With the development and landing of automatic driving technology, large-scale, multi-dynamic and complex application scenarios have higher requirements for the perception ability of automatic driving vehicles. Target tracking, as an important research direction in computer vision, has a relatively wide application in the field of automatic driving. For example, tracking obstacles can enable an automatic driving vehicle to effectively avoid obstacles during driving.

[0003] Currently, the commonly used target tracking technology is to perform target detection on a current frame and perform target tracking based on the detection result of the current frame. SUMMARY

[0004] Embodiments of the present disclosure provide a target tracking method, device, equipment, storage medium and program product.

[0005] In a first aspect, embodiments of the present disclosure provide a target tracking method, comprising: obtaining a current frame image, a previous frame image and a previous frame intermediate feature matrix; fusing to obtain a time sequence fusion feature based on the current frame image, the previous frame image and the previous frame intermediate feature matrix; performing target detection based on the time sequence fusion feature to obtain a target detection result; and performing target tracking based on the target detection result to obtain a target tracking result.

[0006] In a second aspect, embodiments of the present disclosure provide a target tracking device, comprising: an obtaining module configured to obtain a current frame image, a previous frame image and a previous frame intermediate feature matrix; a fusion module configured to fuse to obtain a time sequence fusion feature based on the current frame image, the previous frame image and the previous frame intermediate feature matrix; a detection module configured to perform target detection based on the time sequence fusion feature to obtain a target detection result; and a tracking module configured to perform target tracking based on the target detection result to obtain a target tracking result.

[0007] In a third aspect, embodiments of the present disclosure provide an electronic device, comprising: at least one processor; and a memory connected with the at least one processor in communication; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method described in any of the implementations of the first aspect.

[0008] In a fourth aspect, embodiments of the present disclosure provide a non-transitory computer-readable storage medium storing computer instructions, the computer instructions being used to enable a computer to perform the method described in any of the implementations of the first aspect.

[0009] In a fifth aspect, the embodiments of the present disclosure provide a computer program product, comprising a computer program, wherein the computer program, when executed by a processor, implements the method according to any implementation manner of the first aspect.

[0010] The target tracking method provided by the embodiments of the present disclosure uses the time sequence feature information contained in the adjacent frames for supplement. By inputting the shallow layer encoding information of the adjacent frames, the stability and generalization of target detection are improved by associating and fusing the current frame feature information.

[0011] It should be understood that the content described in this part is not intended to identify key or important features of the embodiments of the present disclosure, nor to limit the scope of the present disclosure. Other features of the present disclosure will become apparent through the following description. BRIEF DESCRIPTION OF DRAWINGS

[0012] Other features, objects, and advantages of the present disclosure will become more apparent from the following detailed description of non-limiting embodiments with reference to the drawings. The drawings are intended to better understand the present scheme and do not constitute a limitation of the present disclosure. Among them:

[0013] Figure 1 is a flowchart of one embodiment of the target tracking method according to the present disclosure;

[0014] Figure 2 is a flowchart of another embodiment of the target tracking method according to the present disclosure;

[0015] Figure 3 is a scene diagram of the target tracking method that can implement the embodiments of the present disclosure;

[0016] Figure 4 is a structural schematic diagram of one embodiment of the target tracking device according to the present disclosure;

[0017] Figure 5 is a block diagram of an electronic device for implementing the target tracking method according to the embodiments of the present disclosure. DETAILED DESCRIPTION

[0018] The exemplary embodiments of the present disclosure are described below in conjunction with the accompanying drawings, which include various details of the embodiments of the present disclosure to help understanding, and should be considered as merely exemplary. Therefore, those of ordinary skill in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Also, in order to be clear and concise, the description below omits the description of well-known functions and structures.

[0019] It should be noted that the embodiments in the present disclosure and the features in the embodiments can be combined with each other without conflict. The present disclosure will be described in detail below with reference to the accompanying drawings and in conjunction with the embodiments.

[0020] Figure 1 A flow 100 of one embodiment of a target tracking method according to the present disclosure is shown. The target tracking method comprises the following steps:

[0021] Step 101, obtaining a current frame image, a previous frame image and a previous frame intermediate feature matrix.

[0022] In this embodiment, the execution subject of the target tracking method can obtain a current frame image, a previous frame image and a previous frame intermediate feature matrix.

[0023] Among them, the current frame image and the previous frame image are two adjacent frame images in the same video. When applied to the field of automatic driving, the video can be obtained by collecting the surrounding environment through the camera installed on the automatic driving car. The previous frame intermediate feature matrix can be the intermediate feature matrix obtained when the previous frame image is detected by the target tracking method of the present embodiment.

[0024] Step 102, fusing to obtain a time sequence fusion feature based on the current frame image, the previous frame image and the previous frame intermediate feature matrix.

[0025] In this embodiment, the execution subject described above can fuse to obtain a time sequence fusion feature based on the current frame image, the previous frame image and the previous frame intermediate feature matrix.

[0026] For example, the current frame image, the previous frame image and the previous frame intermediate feature matrix are first sampled to the same size, and then the sampled current frame image, the previous frame image and the previous frame intermediate feature matrix are concatenated to obtain the time sequence fusion feature.

[0027] At the same time of inputting the current frame image, the previous frame image and the previous frame intermediate feature matrix are also inputted. By inputting the shallow encoding information of the adjacent frames, the time sequence feature information contained in the adjacent frames is used to supplement the current frame, so as to realize the association and fusion on the time sequence feature. Moreover, combined with the information fusion of feature level and image level, the context feature information fusion is realized, which can make the time sequence fusion feature have both shallow feature information and higher resolution position semantic information, and also have deeper and more detailed semantic feature information.

[0028] Step 103, performing target detection based on the time sequence fusion feature to obtain a target detection result.

[0029] In this embodiment, the execution subject described above can perform target detection based on the time sequence fusion feature to obtain a target detection result.

[0030] For example, the time sequence fusion feature is detected by the detection network. Specifically, the detection includes multi-classification and information regression. If the classification score is high, it indicates that the target exists in the current frame image. If the classification score is low, it indicates that the target does not exist in the current frame image. The target feature can be a characteristic possessed by the target. The target detection result can be information recording whether the target exists in the current frame image. If the number of targets is multiple, multi-target detection can be realized. For example, when applied to the field of automatic driving, the target can include but is not limited to: vehicles, pedestrians, stones, trees and other obstacles.

[0031] Step 104, performing target tracking based on the target detection result to obtain a target tracking result.

[0032] In the embodiment, the execution subject can perform target tracking based on the target detection result to obtain a target tracking result.

[0033] After the target tracking method provided in the embodiment is used to track the images before the current frame image, a plurality of tracking trajectories can be formed. The target detection result is matched with each tracking trajectory, and the corresponding tracking trajectory is updated based on the target detection result that is successfully matched. Here, the Hungarian matching can be used for target association and tracking.

[0034] The target tracking method provided in the embodiment uses the time sequence feature information contained in the adjacent frames for supplement. By inputting the shallow encoding information of the adjacent frames, the current frame feature information is associated and fused, so that the stability and generalization of target detection are improved.

[0035] Continuing to refer to Figure 2 which shows a flow 200 of yet another embodiment of the target tracking method according to the present disclosure. The target tracking method includes the following steps:

[0036] Step 201, acquiring a current frame image, a previous frame image and a previous frame intermediate feature matrix.

[0037] In the embodiment, the specific operation of step 201 has been described in detail in the embodiment shown in Figure 1 The specific operation of step 101 in the embodiment shown has been described in detail, and will not be repeated here.

[0038] Step 202, fusing the current frame image, the previous frame image and the previous frame intermediate feature matrix by using a time sequence fusion model to obtain a time sequence fusion feature.

[0039] In the embodiment, the execution subject of the target tracking method can fuse the current frame image, the previous frame image and the previous frame intermediate feature matrix by using the time sequence fusion model to obtain the time sequence fusion feature.

[0040] For example, the current frame image, the previous frame image and the previous frame intermediate feature matrix are simultaneously input to three feature extraction sub-models

[0041] The time sequence fusion model can be a convolution model based on deep learning, which can realize efficient fusion of the current frame image, the previous frame image and the previous frame intermediate feature matrix. Generally, the time sequence fusion model can include three feature extraction sub-models and a fusion sub-model. The three feature extraction sub-models can be used for feature extraction of the input current frame image, the previous frame image and the previous frame intermediate feature matrix, respectively. The fusion sub-model can be used for fusion of the features extracted by the three feature sub-models.

[0042] In some embodiments, the previous frame intermediate feature matrix can be preprocessed and used as input of the time sequence fusion model. For example, the previous frame intermediate feature matrix is preprocessed to obtain a preprocessed intermediate feature matrix, the current frame image, the previous frame image and the preprocessed intermediate feature matrix are input to the three feature extraction sub-models for feature extraction, and the feature extraction results are input to the fusion sub-model for fusion to obtain the time sequence fusion feature.

[0043] In order to facilitate model deployment and simplify post-processing process, the previous frame intermediate feature matrix can be an intermediate feature matrix output by the target detection model of the embodiment when performing target detection on the previous frame image. Meanwhile, the following at least one way is introduced to preprocess the previous frame intermediate feature matrix:

[0044] First, the previous frame intermediate feature matrix is filtered by a mask, and the pixel points with pixel values less than a preset threshold are set to 0.

[0045] Specifically, the pixel value of each pixel point of the previous frame intermediate feature matrix is compared with the preset threshold, the pixel value of the pixel point not less than the preset threshold is retained, and the pixel value of the pixel point less than the preset threshold is set to 0. In this way, the area with higher response of the previous frame intermediate feature matrix is retained for target detection, and the area with lower response is avoided to interfere with target detection.

[0046] Second, the multi-channel of the pixel points of the previous frame intermediate feature matrix is compressed into a single channel, and then the inference and training process is simplified.

[0047] The prior art also has a scheme of introducing timing feature information. The intermediate feature matrix in the previous frame in the inference process is generated by the detection result of the previous frame image through Gaussian blur distribution, and its post-processing process is complex. The intermediate feature matrix in the previous frame in the training process is generated by the label through Gaussian blur. This also causes the inference and training processes to be misaligned. In the embodiment, whether in the inference process or in the training process, the intermediate feature matrix in the previous frame is obtained by using a target detection model, and the preprocessing process of the intermediate feature matrix in the previous frame is simple. The intermediate feature matrix in the previous frame obtained by using the target detection model and the preprocessing of the intermediate feature matrix in the previous frame do not depend on the label, decouple the dependence of the model input intermediate feature matrix in the previous frame on the label, and directly use the model to infer the intermediate feature matrix in the previous frame after preprocessing as the input, thereby aligning the difference in the generation of timing features in the inference and training processes.

[0048] In step 203, the timing fusion feature is input into the target detection model to obtain a target detection result.

[0049] In the embodiment, the execution subject described above can input the timing fusion feature into the target detection model to obtain a target detection result.

[0050] The input of the target detection model can be the timing fusion feature output by the timing fusion model, and the feature is extracted through a backbone network. The backbone network can be, for example, a DLA (Deep Layer Aggregation) 34 backbone network based on deep aggregation, which realizes the deep fusion and sharing of spatial features and semantic features through the mixing of iterative deep aggregation and hierarchical deep aggregation. The timing fusion feature input into the target detection model has both shallow feature information and higher resolution location semantic information, and also has deeper and more detailed semantic feature information. Therefore, the output of the target detection model can include classification information, positioning information, direction information, attribute information, tracking feature information, and classification scores. The classification information can include the category information of the target. The positioning information can include the location information and scale information of the target. The direction information can include the angle information and orientation information of the target. The attribute information can include whether the target is occluded, truncated, and in the lane. The tracking feature information can represent the two-dimensional offset of the target in the image between adjacent frames, providing additional clues for the observation of the tracker.

[0051] In practice, in a large-scale training set of the target detection model, usually only a small part of the data is continuous frame labeled data. In order to quickly flatten the scale of the original discrete frame training set, a method based on discrete frame transformation can be used to simulate continuous frames to realize data enhancement of continuous frames. Discrete frame transformation can include but is not limited to cropping, shifting, scaling, etc., to simulate the transformation of the previous frame image and the corresponding target, thereby realizing timing enhancement and quickly solving the data quantity barrier.

[0052] In step 204, the target detection result is divided into a first target frame detection result and a second target frame detection result according to the classification score.

[0053] In the embodiment, the execution subject can match the target detection result with each tracking track. Generally, Hungarian matching can be used for target association and tracking. A secondary matching method is introduced to recall low-score frames missed due to detection jitter. Specifically, the target detection result is divided into a first target frame detection result and a second target frame detection result according to the classification score, and steps 205 and 206 are performed for twice matching.

[0054] The first target frame classification score can be greater than the second target frame classification score. For example, the classification score of the target detection result is compared with a preset score threshold, and the target detection result greater than the preset score threshold is taken as the first target frame detection result, and the target detection result not greater than the preset score threshold is taken as the second target frame detection result.

[0055] In step 205, the first target frame detection result is matched with each tracking track, and the corresponding tracking track is updated based on the first target frame detection result that is successfully matched.

[0056] In the embodiment, the execution subject can match the first target frame detection result with each tracking track, and update the corresponding tracking track based on the first target frame detection result that is successfully matched.

[0057] Here, a secondary matching method is introduced, which matches based on the target detection result, considering the category, positioning, distance and tracking features. The first matching uses the first target frame detection result with a higher classification score for matching. If the matching is successful, the corresponding tracking track is updated based on the first target frame detection result that is successfully matched; if the matching fails, a new tracking track is created based on the first target frame detection result that fails to match.

[0058] In step 206, the second target frame detection result is matched with the tracking track that fails to match, and the corresponding tracking track is updated based on the second target frame detection result that is successfully matched.

[0059] In the embodiment, the execution subject can match the second target frame detection result with the tracking track that fails to match, and update the corresponding tracking track based on the second target frame detection result that is successfully matched.

[0060] Here, a secondary matching method is introduced, which matches based on the target detection result, considering the category, positioning, distance and tracking features. The second matching uses the second target box detection result with a lower classification score for matching. And, only the tracking trajectory that fails to match is matched. If the matching is successful, the corresponding tracking trajectory is updated based on the second target box detection result that matches successfully; if the matching fails, the tracking trajectory that fails to match is retained until the tracking reaches a preset number of frames (such as 30 frames), and the matching is performed again when it appears again.

[0061] From Figure 2 it can be seen that, compared with Figure 1 the corresponding embodiment, the flow 200 of the target tracking method in the embodiment highlights the feature fusion step, the target detection step and the target tracking step. Therefore, the scheme described in the embodiment can utilize the time sequence fusion model to achieve efficient feature fusion. Moreover, when performing target detection, the target detection model under different cameras can be migrated, and through the association and fusion of time sequence feature information, the false detection and missed detection problems of the target detection model in the truncation, occlusion and small target scenes can be alleviated, and the stability of the target three-dimensional attribute prediction is ensured, thereby further improving the basic perception ability of the autonomous vehicle. At the same time, the secondary matching method is introduced to recall the low-score frame missed detection caused by detection jitter.

[0062] In order to facilitate understanding, Figure 3 a scene diagram of a target tracking method that can implement the embodiment of the disclosure is shown. As Figure 3 shown, time sequence fusion is performed based on a previous frame image I t-k , a middle feature matrix M t-k of the previous frame image and a current frame image I t . The time sequence fusion result is input to a detector for classification, positioning and tracking. The classification, positioning and tracking results are input to a tracker for mapping, and then each tracking trajectory is updated.

[0063] Further referring to Figure 4 , as an implementation of the method shown in the above figures, the disclosure provides an embodiment of a target tracking device, which corresponds to the method embodiment shown in Figure 1 . The device can be specifically applied to various electronic devices.

[0064] As Figure 4As shown, the target tracking device 400 of this embodiment may include: an acquisition module 401, a fusion module 402, a detection module 403, and a tracking module 404. The acquisition module 401 is configured to acquire the current frame image, the previous frame image, and the intermediate feature matrix of the previous frame; the fusion module 402 is configured to fuse the current frame image, the previous frame image, and the intermediate feature matrix of the previous frame to obtain temporal fusion features; the detection module 403 is configured to perform target detection based on the temporal fusion features to obtain a target detection result; and the tracking module 404 is configured to perform target tracking based on the target detection result to obtain a target tracking result.

[0065] In this embodiment, the specific processing of the acquisition module 401, fusion module 402, detection module 403, and tracking module 404 in the target tracking device 400, and the resulting technical effects, can be found in references to [reference needed]. Figure 1 The relevant descriptions of steps 101-104 in the corresponding embodiments will not be repeated here.

[0066] In some optional implementations of this embodiment, the fusion module 402 includes a fusion submodule configured to fuse the current frame image, the previous frame image, and the intermediate feature matrix of the previous frame using a temporal fusion model to obtain temporal fusion features.

[0067] In some optional implementations of this embodiment, the temporal fusion model includes three feature extraction sub-models and a fusion sub-model; and the fusion sub-module includes: a preprocessing unit configured to preprocess the intermediate feature matrix of the previous frame to obtain a preprocessed intermediate feature matrix; and a fusion unit configured to input the current frame image, the previous frame image, and the preprocessed intermediate feature matrix into the three feature extraction sub-models for feature extraction, and input the feature extraction results into the fusion sub-model for fusion to obtain temporal fusion features.

[0068] In some optional implementations of this embodiment, the preprocessing unit is further configured to: filter the intermediate feature matrix of the previous frame using a mask, set pixels with pixel values ​​less than a preset threshold to 0; and / or compress the multi-channel pixels of the intermediate feature matrix of the previous frame into a single channel.

[0069] In some optional implementations of this embodiment, the detection module 403 is further configured to: input temporal fusion features into the target detection model to obtain target detection results, wherein the target detection results include classification information, positioning information, orientation information, attribute information, tracking feature information and classification score.

[0070] In some optional implementations of this embodiment, the intermediate feature matrix of the previous frame is the intermediate feature matrix output when the target detection model performs target detection on the previous frame image.

[0071] In some optional implementation forms of the embodiment, the tracking module 404 is further configured to: divide the target detection results into first target frame detection results and second target frame detection results according to the classification scores, wherein the first target frame classification scores are greater than the second target frame classification scores; match the first target frame detection results with the tracking trajectories, and update the corresponding tracking trajectories based on the first target frame detection results that are successfully matched; and match the second target frame detection results with the tracking trajectories that fail to be matched, and update the corresponding tracking trajectories based on the second target frame detection results that are successfully matched.

[0072] In some optional implementation forms of the embodiment, the tracking module 404 is further configured to: create a new tracking trajectory based on the first target frame detection results that fail to be matched.

[0073] In the technical solutions of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved in the technical solutions comply with relevant laws and regulations and do not violate public order and good customs.

[0074] According to the embodiments of the present disclosure, the present disclosure further provides an electronic device, a readable storage medium and a computer program product.

[0075] Figure 5 A schematic block diagram of an example electronic device 500 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptops, desktops, tablets, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular telephones, smart phones, wearable devices, and other similar computing devices. The components shown here, their connections and relationships, and their functions, are meant to be examples only, and are not meant to limit implementations of the present disclosure described and / or claimed in this document.

[0076] As shown in Figure 5 The device 500 includes a computing unit 501 that can perform various appropriate actions and processes in accordance with a computer program stored in a read-only memory (ROM) 502 or a computer program loaded into a random access memory (RAM) 503 from a storage unit 508. Various programs and data required for the operation of the device 500 can also be stored in the RAM 503. The computing unit 501, the ROM 502, and the RAM 503 are connected to each other through a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0077] A number of components in the device 500 are connected to the I / O interface 505, including: an input unit 506, such as a keyboard, a mouse, etc.; an output unit 507, such as various types of displays, speakers, etc.; a storage unit 508, such as a magnetic disk, an optical disk, etc.; and a communication unit 509, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 509 allows the device 500 to exchange information / data with other devices through a computer network, such as the Internet, and / or various telecommunication networks.

[0078] The computing unit 501 can be various general and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 501 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The computing unit 501 performs various methods and processes described above, such as the target tracking method. For example, in some embodiments, the target tracking method can be implemented as a computer software program, which is tangibly embodied in a machine-readable medium, such as the storage unit 508. In some embodiments, part or all of the computer program can be loaded and / or installed onto the device 500 via the ROM 502 and / or the communication unit 509. When the computer program is loaded onto the RAM 503 and executed by the computing unit 501, one or more steps of the target tracking method described above can be performed. Alternatively, in other embodiments, the computing unit 501 can be configured to perform the target tracking method by any other appropriate means, such as by means of firmware.

[0079] Various implementations of the systems and techniques described above herein can be realized in digital electronic circuitry, integrated circuitry, a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on a chip (SOC), a programmable logic device (CPLD), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0080] Program code for carrying out methods of the present disclosure can be written in any combination of one or more programming languages. The program code can be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the program code, when executed by the processor or controller, produces the functions / operations specified in the flowcharts and / or block diagrams. The program code can be executed entirely on a machine, partially on a machine, partially on a machine as a standalone software package, or entirely on a remote machine or server.

[0081] In the context of the present disclosure, a machine-readable medium can be a tangible medium that contains or stores a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include but is not limited to an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of the machine-readable storage medium will include one or more lines of electrical connections, portable computer disks, hard disk drives, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or Flash memory), optical fibers, portable compact disc read-only memories (CD-ROMs), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0082] To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0083] The systems and techniques described here can be implemented in a computing system that includes a back end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front end component (e.g., a user computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0084] The computer system can include clients and servers. This relationship can be. The servers are typically remote from the clients with the interactions between them occurring over a communication network. The relationship between client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. The servers can be cloud servers, servers of a distributed system, or servers incorporating blockchain.

[0085] It should be understood that the various forms of flow shown above can be re-ordered, added to, or deleted from without departing from the scope of the present disclosure. For example, the steps recited in the present disclosure can be performed in parallel, in series, or in a different order, as long as the desired results of the technical solutions provided by the present disclosure are achieved, and the present disclosure is not limited herein.

[0086] The specific implementation described above does not constitute a limitation on the protection scope of the present disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent replacements, and improvements made within the spirit and principles of the present disclosure should be included in the protection scope of the present disclosure.

Claims

1. A target tracking method, comprising: Obtain the current frame image, the previous frame image, and the intermediate feature matrix of the previous frame; The intermediate feature matrix of the previous frame is preprocessed to obtain a preprocessed intermediate feature matrix. In this process, a mask is used to filter the intermediate feature matrix of the previous frame, and pixels with pixel values ​​less than a preset threshold are set to 0. The multi-channel pixels of the intermediate feature matrix of the previous frame are compressed into a single channel. Based on the current frame image, the previous frame image, and the preprocessed intermediate feature matrix, temporal fusion features are obtained by fusing them together; Target detection is performed based on the aforementioned temporal fusion features to obtain target detection results; Target tracking is performed based on the target detection results to obtain the target tracking results.

2. The method according to claim 1, wherein, The process of fusing the current frame image, the previous frame image, and the preprocessed intermediate feature matrix to obtain temporal fusion features includes: The temporal fusion feature is obtained by fusing the current frame image, the previous frame image, and the preprocessed intermediate feature matrix using a temporal fusion model.

3. The method according to claim 2, wherein, The temporal fusion model includes three feature extraction sub-models and a fusion sub-model; and The process of fusing the current frame image, the previous frame image, and the preprocessed intermediate feature matrix using a temporal fusion model to obtain the temporal fusion features includes: The current frame image, the previous frame image, and the preprocessed intermediate feature matrix are respectively input into the three feature extraction sub-models for feature extraction, and the feature extraction results are input into the fusion sub-model for fusion to obtain the temporal fusion feature.

4. The method according to claim 1, wherein, The target detection based on the temporal fusion features, to obtain the target detection result, includes: The temporal fusion features are input into the target detection model to obtain the target detection result, wherein the target detection result includes classification information, localization information, orientation information, attribute information, tracking feature information, and classification score.

5. The method according to claim 4, wherein, The intermediate feature matrix of the previous frame is the intermediate feature matrix output when the target detection model performs target detection on the previous frame image.

6. The method according to claim 4, wherein, The target tracking based on the target detection result, to obtain the target tracking result, includes: The target detection results are divided into a first target box detection result and a second target box detection result based on the classification score, wherein the classification score of the first target box is greater than the classification score of the second target box. The first target box detection result is matched with each tracking trajectory, and the corresponding tracking trajectory is updated based on the successfully matched first target box detection result; The second target bounding box detection result is matched with the tracking trajectory that failed to match, and the corresponding tracking trajectory is updated based on the second target bounding box detection result that successfully matches.

7. The method according to claim 6, wherein, The step of performing target tracking based on the target detection result to obtain the target tracking result further includes: A new tracking trajectory is created based on the detection result of the first target bounding box that failed to match.

8. A target tracking device, comprising: The acquisition module is configured to acquire the current frame image, the previous frame image, and the intermediate feature matrix of the previous frame; The preprocessing module is configured to preprocess the intermediate feature matrix of the previous frame to obtain a preprocessed intermediate feature matrix, wherein the intermediate feature matrix of the previous frame is filtered using a mask, pixels with pixel values ​​less than a preset threshold are set to 0, and the multi-channel pixels of the intermediate feature matrix of the previous frame are compressed into a single channel. The fusion module is configured to fuse temporal fusion features based on the current frame image, the previous frame image, and the preprocessed intermediate feature matrix; The detection module is configured to perform target detection based on the temporal fusion features to obtain target detection results; The tracking module is configured to perform target tracking based on the target detection results to obtain target tracking results.

9. The apparatus according to claim 8, wherein, The fusion module includes: The fusion submodule is configured to use a temporal fusion model to fuse the current frame image, the previous frame image, and the preprocessed intermediate feature matrix to obtain the temporal fusion feature.

10. The apparatus according to claim 9, wherein, The temporal fusion model includes three feature extraction sub-models and a fusion sub-model; as well as The fusion submodule includes: The fusion unit is configured to input the current frame image, the previous frame image, and the preprocessed intermediate feature matrix into the three feature extraction sub-models for feature extraction, and input the feature extraction results into the fusion sub-model for fusion to obtain the temporal fusion feature.

11. The apparatus according to claim 8, wherein, The detection module is further configured to: The temporal fusion features are input into the target detection model to obtain the target detection result, wherein the target detection result includes classification information, localization information, orientation information, attribute information, tracking feature information, and classification score.

12. The apparatus according to claim 11, wherein, The intermediate feature matrix of the previous frame is the intermediate feature matrix output when the target detection model performs target detection on the previous frame image.

13. The apparatus according to claim 11, wherein, The tracking module is further configured to: The target detection results are divided into a first target box detection result and a second target box detection result based on the classification score, wherein the classification score of the first target box is greater than the classification score of the second target box. The first target box detection result is matched with each tracking trajectory, and the corresponding tracking trajectory is updated based on the successfully matched first target box detection result; The second target bounding box detection result is matched with the tracking trajectory that failed to match, and the corresponding tracking trajectory is updated based on the second target bounding box detection result that successfully matches.

14. The method according to claim 13, wherein, The tracking module is further configured to: A new tracking trajectory is created based on the detection result of the first target bounding box that failed to match.

15. An electronic device comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-7.

16. A non-transitory computer-readable storage medium storing computer instructions for causing the computer to perform the method of any one of claims 1-7.

17. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1-7.

Images