Target tracking method, device and electronic equipment

Abstract

The present invention discloses a target tracking method, device and electronic device. The method comprises: collecting a current frame image of a vehicle driving environment and a previous frame image of the current frame image; using at least one perception frame identified in the previous frame image to predict a predicted perception frame in the current frame image; matching at least one perception frame identified in the current frame image with the predicted perception frame at the corresponding position for the first time; obtaining a successfully matched target perception frame in the current frame image; calling a neural network model to extract perception frame features from the successfully matched target perception frame; based on the extracted perception frame features of the target perception frame, using the Hungarian algorithm to perform a secondary match on the target perception frame; in the case of a successful match, determining the position of the tracked target object. The present invention at least solves the problem of inaccurate target tracking due to occlusion between targets and rapid changes in target motion.

Description

Technical Field

[0001] The present invention relates to the field of autonomous driving, and in particular to a target tracking method, device and electronic equipment. Background Art

[0002] Target tracking is an important technology in the field of autonomous driving. Since there are often many obstacles on the road, autonomous vehicles need to identify and track these obstacles to facilitate subsequent path planning. Among them, target tracking technology can track the trajectory of the target, thereby providing support for subsequent path planning of autonomous vehicles.

[0003] However, existing target tracking methods often rely on large neural network models, which usually involve a large number of complex mathematical calculations, thus reducing the speed of target tracking and failing to meet the needs of autonomous vehicles for real-time target tracking, which is not conducive to the large-scale production of autonomous vehicles. In addition, in practical applications, there may be occlusions between multiple targets, and the movement of some targets changes rapidly, resulting in poor target tracking accuracy.

[0004] To address the above-mentioned problems, no effective solution has been proposed yet. Summary of the invention

[0005] The embodiments of the present invention provide a target tracking method, device and electronic device to at least solve the technical problem of inaccurate target tracking due to occlusion between targets and rapid changes in target motion.

[0006] According to one aspect of an embodiment of the present invention, a target tracking method is provided, including: during vehicle driving, collecting a current frame image of a vehicle driving environment and a previous frame image of the current frame image; using at least one perception frame identified in the previous frame image to predict a predicted perception frame in the current frame image, wherein the perception frame is used to identify a target object to be tracked in the previous frame image; performing a first match between at least one perception frame identified in the current frame image and a predicted perception frame at a corresponding position; obtaining a successfully matched target perception frame in the current frame image; calling a neural network model to extract perception frame features from the successfully matched target perception frame; based on the extracted perception frame features of the target perception frame, using the Hungarian algorithm to perform a secondary match on the target perception frame; and in the case of a successful match, determining the position of the tracked target object.

[0007] Furthermore, the target tracking method also includes: identifying a target object to be tracked from a previous frame image, and identifying at least one perception frame matching the target object in the previous frame image; based on the coordinates of at least one perception frame of the target object in the previous frame image, predicting the position of the predicted perception frame of the target object in the current frame image.

[0008] Furthermore, the target tracking method also includes: obtaining the coordinate value of the target object in the camera coordinate system from the previous frame image, wherein the camera coordinate system is a coordinate system with the camera that photographs the target object as a positioning reference; converting the coordinate value of the target object in the camera coordinate system to the object coordinate system, wherein the object coordinate system is a coordinate system with the target object as a positioning reference; based on the relative speed of the target object and the time difference before and after the displacement in two adjacent frame images, predicting the target object position in the object coordinate system of the current frame image; converting the predicted target object position to the camera coordinate system; according to the coordinate change of the camera coordinate system in two adjacent frame images, calculating the position of the target object in the image coordinate system of the current frame image; based on the position of the target object in the image coordinate system of the current frame image, obtaining the position of the predicted perception box of the target object in the current frame image.

[0009] Furthermore, the target tracking method also includes: calculating the intersection-and-union (IOU) of at least one perception frame identified in the current frame image and the predicted perception frame of the corresponding type to obtain the IOU of each target object in the current frame image; determining the target object with the largest IOU from the current frame image to obtain the tracked object; if the IOU of the tracked object meets the minimum limit, checking whether there is a coordinate jump between the perception frame of the tracked object and the corresponding predicted perception frame; if there is no coordinate jump, the perception frame of the tracked object is the successfully matched target perception frame in the current frame image.

[0010] Furthermore, the target tracking method also includes: multi-tasking scheduling of the target perception frame that has been successfully matched in the current frame image; expanding the range of the target perception frame to obtain the expanded target perception frame; calling the neural network model to classify the expanded target perception frame, determine the type of the target perception frame, and extract the perception frame features of the target perception frame that has been successfully matched in the current frame image.

[0011] Furthermore, the target tracking method also includes: performing cosine similarity calculation on at least one perception frame feature extracted from the current frame image and the historical features of the corresponding target object; determining the target perception frame with the greatest similarity from the calculation results; if the intersection over union (IOU) of the target perception frame with the greatest similarity meets the minimum threshold, obtaining the weight matrix of the target perception frame with the greatest similarity; based on the weight matrix of the target perception frame with the greatest similarity, using the Hungarian algorithm to perform secondary matching on the target perception frame with the greatest similarity.

[0012] Furthermore, the target tracking method also includes: in the case of successful matching, checking whether there is a coordinate jump between the target perception frame with the greatest similarity and the corresponding historical perception frame; if there is no coordinate jump, determining that the target object corresponding to the target perception frame with the greatest similarity is the tracked target object, and obtaining the position of the tracked target object.

[0013] Furthermore, the target tracking method also includes: obtaining a target perception frame that fails to match, and assigning new identification information to the target perception frame that fails to match.

[0014] According to another aspect of an embodiment of the present invention, a target tracking device is also provided, including: an acquisition module, used to acquire a current frame image of a vehicle driving environment and a previous frame image of the current frame image; a prediction module, used to use at least one perception frame identified in the previous frame image to predict a predicted perception frame in the current frame image, wherein the perception frame is used to identify the target object to be tracked in the previous frame image; a first matching module, used to perform a first match between at least one perception frame identified in the current frame image and the predicted perception frame at a corresponding position; an acquisition module, used to obtain a successfully matched target perception frame in the current frame image; a calling module, used to call a neural network model to extract perception frame features from the successfully matched target perception frame; a second matching module, used to perform a secondary match on the target perception frame using the Hungarian algorithm based on the extracted perception frame features of the target perception frame; and a determination module, used to determine the position of the tracked target object when the match is successful.

[0015] According to another aspect of an embodiment of the present invention, an electronic device is also provided, which includes one or more processors; a storage device for storing one or more programs, which, when the one or more programs are executed by the one or more processors, enables the one or more processors to run the programs, wherein the programs are configured to execute the above-mentioned target tracking method when running.

[0016] In an embodiment of the present invention, a method of performing two matches based on the previous frame image and the current frame image is adopted. During the driving process of the vehicle, the current frame image of the vehicle driving environment and the previous frame image of the current frame image are collected, and at least one perception frame identified in the previous frame image is used to predict the predicted perception frame in the current frame image, and then at least one perception frame identified in the current frame image is matched for the first time with the predicted perception frame at the corresponding position, and the successfully matched target perception frame in the current frame image is obtained, so as to call the neural network model to extract the perception frame features from the successfully matched target perception frame, and based on the extracted perception frame features of the target perception frame, the Hungarian algorithm is used to perform a second match on the target perception frame, and then the position of the tracked target object is determined in the case of successful matching. Among them, the perception frame is used to identify the target object to be tracked in the previous frame image.

[0017] From the above content, it can be seen that after obtaining the successfully matched target perception frame in the current frame image, the perception frame features are extracted from the successfully matched target perception frame through the neural network model. Therefore, compared with the prior art, the present application does not need to extract features from all perception frames, that is, compared with the prior art, the number of target perception frames to be processed by the neural network model in the present application is smaller, thereby reducing the overall amount of calculation, improving the speed of target tracking, and meeting the needs of mass production of autonomous driving vehicles. In addition, after the present application performs the first matching of at least one perception frame identified in the current frame image with the predicted perception frame at the corresponding position, the Hungarian algorithm is used to perform a second matching of the target perception frame based on the perception frame features of the extracted target perception frame, that is, multiple features are used for target matching in the present application, and the two matching processes can effectively avoid the problem of inaccurate target tracking caused by occlusion between multiple target objects and rapid changes in the movement of some target objects, thereby improving the accuracy of target tracking.

[0018] It can be seen that through the technical solution of the present application, the purpose of improving the target tracking speed and accuracy is achieved, thereby saving the chip computing power of the autonomous driving car and meeting the needs of mass production of autonomous driving cars, thereby solving the technical problem of inaccurate target tracking due to occlusion between targets and rapid changes in target motion. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0020] Figure 1 is a flow chart of an optional target tracking method according to an embodiment of the present invention;

[0021] Figure 2 is a flow chart of an optional target tracking method according to an embodiment of the present invention;

[0022] Figure 3 is a schematic block diagram of an optional target tracking device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0023] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0024] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0025] Example 1

[0026] According to an embodiment of the present invention, a method embodiment of a target tracking method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0027] In addition, it should be noted that the electronic device can be used as the execution subject of the target tracking method of the embodiment of the present invention. The electronic device can be installed on the self-driving car.

[0028] Figure 1 is a flow chart of an optional target tracking method according to an embodiment of the present invention. Figure 1 As shown, the method comprises the following steps:

[0029] Step S101 , during the driving process of the vehicle, a current frame image of the vehicle driving environment and a frame image previous to the current frame image are collected.

[0030] In step S101, the electronic device installed on the self-driving car is connected to an image acquisition device. The image acquisition device may be a driving recorder, a vehicle-mounted camera, or other devices, and the image acquisition device may continuously take pictures of the vehicle driving environment during the vehicle driving process. In addition, the image acquisition device may also synchronously transmit the captured image to the electronic device, so that the electronic device performs frame processing on the image to obtain the current frame image and the previous frame image of the current frame image.

[0031] Step S102: using at least one perception frame identified in the previous frame image to predict a predicted perception frame in the current frame image.

[0032] In step S102, the perception frame is used to identify the target object to be tracked in the previous frame image. The target object may be a vehicle traveling on the road. After acquiring the current frame image and the previous frame image, the electronic device may use the perception frame of the previous frame image to predict the predicted perception frame in the current frame image. Specifically, the electronic device may first identify the vehicle to be tracked from the previous frame image, and identify at least one perception frame matching the vehicle to be tracked in the previous frame image, and then calculate the coordinates of at least one perception frame of the vehicle to be tracked in the previous frame image, and predict the position of the predicted perception frame of the vehicle to be tracked in the current frame image based on the coordinates.

[0033] Step S103, performing a first match between at least one perception frame identified in the current frame image and the predicted perception frame at the corresponding position.

[0034] In step S103, the electronic device may identify at least one perception frame in the current frame image, and since the electronic device also predicts a predicted perception frame for the current frame image through the previous frame image, the electronic device may use the intersection-and-union (IOU) of each target object in the current frame image to perform a first match between the predicted perception frame and all perception frames in the current frame image. The intersection-and-union (IOU) is the ratio of the intersection and union of a perception frame and the predicted perception frame at the corresponding position of the perception frame.

[0035] Step S104, obtaining the successfully matched target perception frame in the current frame image.

[0036] In step S104, through the first matching, the electronic device can determine the successfully matched target perception frame from at least one perception frame corresponding to the current frame image according to the matching result. Specifically, the electronic device can calculate the intersection-and-union (IOU) of at least one perception frame identified in the current frame image and the corresponding type of predicted perception frame, thereby obtaining the IOU of each target object in the current frame image, and then the electronic device determines from the current frame image that the target object with the largest IOU is the tracked object. If the IOU of the tracked object meets the minimum limit, it continues to check whether there is a coordinate jump between the perception frame of the tracked object and the corresponding predicted perception frame. If there is no coordinate jump, the perception frame of the tracked object is the successfully matched target perception frame in the current frame image.

[0037] Step S105, calling the neural network model to extract perception frame features from the successfully matched target perception frame.

[0038] In step S105, the neural network model called by the electronic device is a fast neural network model. After completing the first matching, the electronic device can perform multi-task scheduling on the result of the first matching, call the fast neural network model to extract the perception frame features according to the target perception frame matched by the result of the first matching, and classify the target perception frame.

[0039] It should be noted that since the present application extracts perception frame features from the successfully matched target perception frame through a neural network model, compared with the prior art, the present application does not need to perform feature extraction on all perception frames. The neural network model in the present application only needs to process a small number of target perception frames, thereby reducing the overall amount of calculation and improving the speed of target (ie, target object) tracking, thereby realizing the demand for mass production of autonomous driving vehicles.

[0040] Step S106: Based on the extracted perception frame features of the target perception frame, a Hungarian algorithm is used to perform secondary matching on the target perception frame.

[0041] In step S106, the electronic device can use the perception frame features extracted by the fast neural network model and IOU to calculate new weights to obtain the weight matrix of the target perception frame, and then use the Hungarian matching algorithm to perform secondary matching on the target perception frame based on the weight matrix to generate the final matching result.

[0042] It should be noted that after the first match, the present application will subsequently perform a second match on the target perception frame based on the extracted perception frame features of the target perception frame using the Hungarian algorithm. Therefore, multiple features are actually used for target matching in the present application. The two matching processes can also effectively avoid the problem of inaccurate target tracking caused by occlusion between multiple target objects and rapid motion changes of some target objects, thereby improving the accuracy of target tracking.

[0043] Step S107: if the match is successful, determine the position of the tracked target object.

[0044] In step S107, if the match is successful, the electronic device will check whether there is a coordinate jump between the target perception frame of the target object and the corresponding historical perception frame. If there is no coordinate jump, the target object is determined to be the tracked target object, and the position of the tracked target object is obtained. In addition, the electronic device will also obtain the target perception frame that failed to match, and assign new identification information to the target perception frame.

[0045] Based on the contents of the above steps S101 to S107, it can be known that in the embodiment of the present invention, a method of performing two matches based on the previous frame image and the current frame image is adopted. During the driving process of the vehicle, the current frame image of the vehicle driving environment and the previous frame image of the current frame image are collected, and at least one perception frame identified in the previous frame image is used to predict the predicted perception frame in the current frame image, and then at least one perception frame identified in the current frame image is matched for the first time with the predicted perception frame at the corresponding position, and the successfully matched target perception frame in the current frame image is obtained, so as to call the neural network model to extract the perception frame features from the successfully matched target perception frame, and based on the extracted perception frame features of the target perception frame, the Hungarian algorithm is used to perform a second match on the target perception frame, and then the position of the tracked target object is determined in the case of successful matching. Among them, the perception frame is used to identify the target object to be tracked in the previous frame image.

[0046] It can be seen from the above content that after obtaining the successfully matched target perception frame in the current frame image, the perception frame features are extracted from the successfully matched target perception frame through the neural network model. Therefore, compared with the prior art, the present application does not need to perform feature extraction on all perception frames, that is, compared with the prior art, the number of target perception frames to be processed by the neural network model in the present application is smaller, thereby reducing the overall amount of calculation, improving the speed of target tracking, and meeting the needs of mass production of autonomous driving vehicles. In addition, after the present application performs the first matching of at least one perception frame identified in the current frame image with the predicted perception frame at the corresponding position, the Hungarian algorithm is used to perform a second matching of the target perception frame based on the perception frame features of the extracted target perception frame, that is, multiple features are used for target matching in the present application, and the two matching processes can effectively avoid the problem of inaccurate target tracking caused by occlusion between targets and rapid changes in the movement of some targets, thereby improving the accuracy of target tracking.

[0047] It can be seen that through the technical solution of the present application, the purpose of improving the target tracking speed and accuracy is achieved, thereby saving the chip computing power of the autonomous driving car and meeting the needs of mass production of autonomous driving cars, thereby solving the technical problem of inaccurate target tracking due to occlusion between targets and rapid changes in target motion.

[0048] In an optional embodiment, the electronic device may identify the target object to be tracked from the previous frame image, and identify at least one perception frame matching the target object in the previous frame image, thereby predicting the position of the predicted perception frame of the target object in the current frame image based on the coordinates of at least one perception frame of the target object in the previous frame image.

[0049] Optionally, after acquiring the previous frame image and the current frame image, the electronic device may identify the target object to be tracked from the previous frame image. For example, the target object may be a moving object such as a vehicle or pedestrian on the road. At the same time, the electronic device will also identify at least one perception frame matching the target object in the previous frame image. For example, if there are multiple target objects in the previous frame image, the electronic device identifies the perception frame corresponding to each target in the previous frame image, where the perception frame corresponding to each target may be one or more.

[0050] In addition, the electronic device can also obtain the coordinate value of the target object in the camera coordinate system from the previous frame image, and convert the coordinate value of the target object in the camera coordinate system to the object coordinate system, and then the electronic device predicts the target object position in the object coordinate system of the current frame image based on the relative speed of the target object and the time difference before and after the displacement occurs in two adjacent frame images, and converts the predicted target object position to the camera coordinate system, so as to calculate the position of the target object in the image coordinate system of the current frame image according to the coordinate change of the camera coordinate system in two adjacent frame images, and then obtain the position of the predicted perception frame of the target object in the current frame image based on the position of the target object in the image coordinate system of the current frame image. Among them, the camera coordinate system is a coordinate system with the camera shooting the target object as the positioning reference, and the object coordinate system is a coordinate system with the target object as the positioning reference.

[0051] Optionally, taking the target object as a vehicle, the electronic device can use the height of the vehicle in the previous frame image, according to the pinhole imaging principle, to calculate the coordinate value of the vehicle in the camera coordinate system, and then convert the coordinate value from the camera coordinate system to the vehicle coordinate system (i.e., the object coordinate system with the vehicle as the positioning reference). Subsequently, the electronic device will combine the relative speed of the vehicle and the time difference before and after the displacement of the vehicle in two adjacent frame images (i.e., the previous frame image and the current frame image) in the vehicle coordinate system to predict the vehicle position (i.e., the target object position) of the vehicle in the vehicle coordinate system of the current frame image, and the electronic device will also convert the predicted vehicle position to the camera coordinate system.

[0052] Furthermore, after converting the predicted vehicle position to the camera coordinate system, the electronic device can calculate the position of the vehicle in the image coordinate system of the current frame image based on the coordinate changes of the camera coordinate system in two adjacent frame images. Specifically, the electronic device can calculate the change ratio of the X coordinate (horizontal coordinate), and then calculate the width and height of the vehicle in the image coordinate system of the current frame image based on the change ratio, where the width and height can be used to characterize the position of the vehicle. Finally, based on the position of the vehicle in the image coordinate system of the current frame image, the electronic device can obtain the position of the predicted perception box of the vehicle in the current frame image according to the pinhole imaging principle.

[0053] In the above process, the position of the target object in the image coordinate system of the current frame image is obtained by calculating the coordinate change of the camera coordinate system in two adjacent frame images, and based on the position of the target object in the image coordinate system of the current frame image, the position of the predicted perception box of the target object in the current frame image is obtained, thereby achieving the effect of obtaining the predicted perception box of the current frame image based on the perception box prediction of the previous frame image.

[0054] In an optional embodiment, the electronic device may perform an IOU calculation on at least one perception frame identified in the current frame image and a corresponding type of predicted perception frame, and obtain the IOU of each target object in the current frame image, and then determine the target object with the largest IOU from the current frame image to obtain the tracked object. At the same time, if the IOU of the tracked object meets the minimum limit, check whether there is a coordinate jump between the perception frame of the tracked object and the corresponding predicted perception frame; if there is no coordinate jump, the perception frame of the tracked object is the successfully matched target perception frame in the current frame image.

[0055] Optionally, still taking the target object as a vehicle as an example, when there is at least one vehicle in the current frame image, the electronic device can calculate the interaction ratio IOU for at least one perception frame corresponding to each vehicle and the corresponding type of predicted perception frame, and based on the calculation results, select the perception frame with the largest IOU for each vehicle, thereby determining the vehicle with the largest interaction ratio IOU from the current frame image as the tracking object.

[0056] Furthermore, the electronic device will also determine whether the interaction ratio IOU of the tracked object meets the minimum limit. If so, it will check whether there is a coordinate jump between the perception frame of the tracked object and the corresponding predicted perception frame. If there is no coordinate jump, the perception frame of the tracked object is the successfully matched target perception frame in the current frame image, and the electronic device will save the successfully matched target perception frame.

[0057] It should be noted that by calculating the intersection-over-union (IOU), the electronic device can perform a first match between the predicted perception box corresponding to each object and the perception box in the current frame image, thereby determining a successfully matched target perception box. Through the first match, some perception boxes that do not meet the matching features can be removed, thereby reducing the amount of computation of the subsequent neural network model.

[0058] In an optional embodiment, the electronic device performs multi-tasking scheduling on the matching result of the first match to execute the call of the neural network model to extract the perception frame features from the successfully matched target perception frame. Specifically, the electronic device can perform multi-tasking scheduling on the successfully matched target perception frame in the current frame image, and expand the range of the target perception frame to obtain the expanded target perception frame, thereby calling the neural network model to classify the expanded target perception frame, determine the type of the target perception frame, and extract the perception frame features of the successfully matched target perception frame in the current frame image.

[0059] Optionally, the electronic device may select the successfully matched target perception frame in the current frame image for multi-task scheduling, and expand the range of the target perception frame (i.e., the width and height of the target perception frame) by a certain coefficient to obtain the expanded target perception frame. The certain coefficient may be custom set by the operator.

[0060] Optionally, the electronic device may also call a fast neural network model to classify the target perception frame after expansion, thereby determining the type of the target perception frame and extracting the perception frame features of the target perception frame for a subsequent second matching.

[0061] It should be noted that the target perception frame is multi-tasked scheduled according to the result of the first matching, and the features of the perception frame are extracted using a fast neural network. There is no need to extract features for all the perception frames, thereby reducing the computational workload of the neural network model and improving the speed of target tracking.

[0062] In an optional embodiment, the electronic device may perform secondary matching on the target perception frame based on the perception frame features of the extracted target perception frame using the Hungarian algorithm. Specifically, the electronic device performs cosine similarity calculation on at least one perception frame feature extracted from the current frame image and the historical features of the corresponding target object, and determines the target perception frame with the greatest similarity from the calculation results. If the intersection over union (IOU) of the target perception frame with the greatest similarity meets the minimum threshold, the weight matrix of the target perception frame with the greatest similarity is obtained, and based on the weight matrix of the target perception frame with the greatest similarity, the Hungarian algorithm is used to perform secondary matching on the target perception frame with the greatest similarity.

[0063] Optionally, the electronic device will first check each perception frame extracted from the current frame image to see whether the perception frame has a perception frame feature. If so, the electronic device will perform cosine similarity calculation on at least one perception frame feature extracted from the current frame image and the historical features of the corresponding target object, and determine the target perception frame with the greatest similarity from at least one perception frame based on the calculation result.

[0064] Furthermore, after determining the target perception frame with the greatest similarity, the electronic device will also determine whether the intersection over union (IOU) of the target perception frame meets the minimum threshold. If so, the IOU value w is set to IOU And cosine similarity w f According to a certain combination of weight coefficients, the weight matrix of the target perception frame is calculated. Based on the weight matrix, the electronic device can perform secondary matching on the target perception frame through the Hungarian algorithm. The formula for calculating the weight matrix of the target perception frame is as follows:

[0065] w new =α*w f +β*w IOU

[0066] Among them, w in the above formula new is the weight matrix, α and β are the cosine similarity w f And the IOU value w IOU The coefficient of .

[0067] It should be noted that in the above process, the two matches based on different strategies are combined with a fast neural network model, so that the fast neural network can be used to extract features for target matching, reduce the amount of calculation, save the computing power of the autonomous driving chip, and thus improve the target tracking speed and accuracy. In addition, the two matches can also use multiple features for target matching, thereby further improving the accuracy of target tracking.

[0068] In an optional embodiment, when the match is successful, the electronic device checks whether there is a coordinate jump between the target perception frame with the greatest similarity and the corresponding historical perception frame. If there is no coordinate jump, it determines that the target object corresponding to the target perception frame with the greatest similarity is the tracked target object, and obtains the position of the tracked target object.

[0069] Optionally, after obtaining the result of the second match, if the target perception frame with the greatest similarity matches successfully, the electronic device will check whether there is a coordinate jump between the target perception frame and the corresponding historical perception frame. If no coordinate jump occurs, it is determined that the target object corresponding to the target sensing frame is the tracked target object. At the same time, the electronic device will obtain the position of the tracked target object, thereby realizing target tracking.

[0070] Furthermore, in the second matching process, there may be multiple target perception frames with the same similarity. If the IOU values ​​of some target perception frames do not meet the minimum threshold or there is a coordinate jump, these target perception frames will fail to match. On this basis, the electronic device will obtain these target perception frames that failed to match and assign new identification information to these target perception frames.

[0071] In an optional embodiment, Figure 2 FIG. 4 shows a flow chart of an optional target tracking method according to an embodiment of the present invention. Figure 2 As shown, the electronic device first uses the perception frame of the previous frame image to predict the perception frame in the current frame image, that is, to obtain the predicted perception frame, and then the electronic device performs a first match between the predicted perception frame and the perception frame of the current frame image (which can be understood as the actual perception frame of the current frame image) according to the intersection and union (IOU) of each target object in the current frame image, and saves the matching result. Furthermore, the electronic device also calls the fast neural network model for the target perception frame that successfully matches in the first matching process, so as to classify and extract features of the target perception frame. In addition, the electronic device also calculates the weight matrix of the target perception frame with the greatest similarity based on the perception frame features extracted by the fast neural network model and the corresponding IOU value, so as to perform a secondary match on the target perception frame with the greatest similarity using the Hungarian algorithm according to the weight matrix, generate the final matching result, and achieve target tracking.

[0072] It can be seen from the above content that after obtaining the successfully matched target perception frame in the current frame image, the perception frame features are extracted from the successfully matched target perception frame through the neural network model. Therefore, compared with the prior art, the present application does not need to perform feature extraction on all perception frames, that is, compared with the prior art, the number of target perception frames to be processed by the neural network model in the present application is smaller, thereby reducing the overall amount of calculation, improving the speed of target tracking, and meeting the needs of mass production of autonomous driving vehicles. In addition, after the present application performs the first matching of at least one perception frame identified in the current frame image with the predicted perception frame at the corresponding position, the Hungarian algorithm is used to perform a second matching of the target perception frame based on the perception frame features of the extracted target perception frame, that is, multiple features are used for target matching in the present application, and the two matching processes can effectively avoid the problem of inaccurate target tracking caused by occlusion between targets and rapid changes in the movement of some targets, thereby improving the accuracy of target tracking.

[0073] It can be seen that through the technical solution of the present application, the purpose of improving the target tracking speed and accuracy is achieved, thereby saving the chip computing power of the autonomous driving car and meeting the needs of mass production of autonomous driving cars, thereby solving the technical problem of inaccurate target tracking due to occlusion between targets and rapid changes in target motion.

[0074] Example 2

[0075] According to another aspect of the present invention, a target tracking device embodiment is provided, wherein: Figure 3 is a schematic block diagram of an optional target tracking device according to an embodiment of the present invention. Figure 3 As shown, the device includes: a collection module 301, a prediction module 302, a first matching module 303, an acquisition module 304, a calling module 305, a second matching module 306 and a determination module 307.

[0076] Among them, the acquisition module 301 is used to acquire the current frame image of the vehicle driving environment and the previous frame image of the current frame image; the prediction module 302 is used to use at least one perception frame identified in the previous frame image to predict the predicted perception frame in the current frame image, wherein the perception frame is used to identify the target object to be tracked in the previous frame image; the first matching module 303 is used to perform a first match between at least one perception frame identified in the current frame image and the predicted perception frame at the corresponding position; the acquisition module 304 is used to obtain the successfully matched target perception frame in the current frame image; the calling module 305 is used to call the neural network model to extract the perception frame features from the successfully matched target perception frame; the second matching module 306 is used to perform a secondary match on the target perception frame based on the extracted perception frame features of the target perception frame using the Hungarian algorithm; the determination module 307 is used to determine the position of the tracked target object when the match is successful.

[0077] It should be noted that the above-mentioned acquisition module 301, prediction module 302, first matching module 303, acquisition module 304, calling module 305, second matching module 306 and determination module 307 correspond to steps S101 to S107 in the above-mentioned embodiment, and the examples and application scenarios implemented by the seven modules are the same as the corresponding steps, but are not limited to the contents disclosed in the above-mentioned embodiment 1.

[0078] Optionally, the prediction module further includes: an identification module and a first prediction module. The identification module is used to identify the target object to be tracked from the previous frame image and identify at least one perception frame matching the target object in the previous frame image; the first prediction module is used to predict the position of the predicted perception frame of the target object in the current frame image based on the coordinates of at least one perception frame of the target object in the previous frame image.

[0079] Optionally, the first prediction module further includes: a first acquisition module, a first conversion module, a second prediction module, a second conversion module, a calculation module and a second acquisition module. The first acquisition module is used to acquire the coordinate value of the target object in the camera coordinate system from the previous frame image, wherein the camera coordinate system is a coordinate system based on the positioning reference of the camera that photographs the target object; the first conversion module is used to convert the coordinate value of the target object in the camera coordinate system to the object coordinate system, wherein the object coordinate system is a coordinate system based on the positioning reference of the target object; the second prediction module is used to predict the target object position in the object coordinate system of the current frame image based on the relative speed of the target object and the time difference before and after the displacement occurs in two adjacent frame images; the second conversion module is used to convert the predicted target object position to the camera coordinate system; the calculation module is used to calculate the position of the target object in the image coordinate system of the current frame image according to the coordinate change of the camera coordinate system in two adjacent frame images; the second acquisition module is used to obtain the position of the predicted perception box of the target object in the current frame image based on the position of the target object in the image coordinate system of the current frame image.

[0080] Optionally, the first matching module further includes: a first calculation module, a first determination module, a verification module, and a second determination module. The first calculation module is used to calculate the intersection-and-union (IOU) of at least one perception frame identified in the current frame image and the predicted perception frame of the corresponding type to obtain the IOU of each target object in the current frame image; the first determination module is used to determine the target object with the largest IOU from the current frame image to obtain the tracked object; the verification module is used to check whether there is a coordinate jump between the perception frame of the tracked object and the corresponding predicted perception frame if the IOU of the tracked object meets the minimum limit; the second determination module is used to determine that the perception frame of the tracked object is the target perception frame that has been successfully matched in the current frame image if there is no coordinate jump.

[0081] Optionally, the above-mentioned calling module also includes: a scheduling module, an expansion module and a first calling module. Among them, the scheduling module is used to perform multi-task scheduling on the successfully matched target perception frame in the current frame image; the expansion module is used to expand the range of the target perception frame to obtain the expanded target perception frame; the first calling module is used to call the neural network model to classify the expanded target perception frame, determine the type of the target perception frame, and extract the perception frame features of the successfully matched target perception frame in the current frame image.

[0082] Optionally, the second matching module further includes: a similarity meter module, a third determination module, a third acquisition module and a secondary matching module. Among them, the similarity calculation module is used to perform cosine similarity calculation on at least one perception frame feature extracted from the current frame image and the historical features of the corresponding target object; the third determination module is used to determine the target perception frame with the greatest similarity from the calculation results; the third acquisition module is used to obtain the weight matrix of the target perception frame with the greatest similarity if the intersection-over-union ratio (IOU) of the target perception frame with the greatest similarity meets the minimum threshold; the secondary matching module is used to perform secondary matching on the target perception frame with the greatest similarity using the Hungarian algorithm based on the weight matrix of the target perception frame with the greatest similarity.

[0083] Optionally, the determination module further includes: a first verification module and a fourth determination module. The first verification module is used to verify whether there is a coordinate jump between the target perception frame with the greatest similarity and the corresponding historical perception frame when the match is successful; the fourth determination module is used to determine that the target object corresponding to the target perception frame with the greatest similarity is the tracked target object if there is no coordinate jump, and obtain the position of the tracked target object.

[0084] Optionally, the target tracking device further includes: a fourth acquisition module, configured to acquire a target perception frame that fails to match, and assign new identification information to the target perception frame that fails to match.

[0085] Example 3

[0086] According to another aspect of an embodiment of the present invention, an electronic device is also provided, which includes one or more processors; a storage device for storing one or more programs, so that when the one or more programs are executed by the one or more processors, the one or more processors are implemented to run the programs, wherein the programs are configured to execute the target tracking method in the above-mentioned embodiment 1 when running.

[0087] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0088] In the above embodiments of the present invention, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0089] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of units can be a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0090] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed over multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0091] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0092] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the methods of each embodiment of the present invention. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, magnetic disk or optical disk and other media that can store program codes.

[0093] The above are only preferred embodiments of the present invention. It should be pointed out that, for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A target tracking method, characterized in that: include: During the driving of the vehicle, a current frame image of the driving environment of the vehicle and a previous frame image of the current frame image are collected; Using at least one perception frame identified in the previous frame image, predicting a predicted perception frame in the current frame image, wherein the perception frame is used to identify a target object to be tracked in the previous frame image; Performing a first match between at least one perception frame identified in the current frame image and the predicted perception frame at the corresponding position; Obtaining a successfully matched target perception frame in the current frame image; Calling a neural network model to extract perception frame features from the successfully matched target perception frame; Based on the extracted perception frame features of the target perception frame, a Hungarian algorithm is used to perform secondary matching on the target perception frame; In case of successful matching, determining the position of the tracked target object; The at least one perception frame identified in the current frame image is matched for the first time with the predicted perception frame at the corresponding position, including: Calculate the intersection-and-union (IOU) of at least one perception frame identified in the current frame image and the predicted perception frame of the corresponding type to obtain the IOU of each target object in the current frame image; Determine the target object with the largest intersection-over-union (IOU) ratio from the current frame image to obtain the tracking object; If the IOU of the tracked object meets the minimum limit, check whether there is a coordinate jump between the perception frame of the tracked object and the corresponding predicted perception frame; If the coordinate jump does not exist, the perception frame of the tracked object is the successfully matched target perception frame in the current frame image.

2. The method according to claim 1, characterized in that Using at least one perception frame identified in the previous frame image to predict a predicted perception frame in the current frame image includes: Identifying the target object to be tracked from the previous frame image, and identifying at least one perception frame matching the target object in the previous frame image; Based on the coordinates of at least one perception box of the target object in the previous frame image, a position of a predicted perception box of the target object in the current frame image is predicted.

3. The method according to claim 2, characterized in that Predicting a position of a predicted perception frame of the target object in the current frame image based on the coordinates of at least one perception frame of the target object in the previous frame image includes: Acquire the coordinate value of the target object in the camera coordinate system from the previous frame image, wherein the camera coordinate system is a coordinate system based on the positioning reference of the camera that photographs the target object; Converting the coordinate value of the target object in the camera coordinate system to an object coordinate system, wherein the object coordinate system is a coordinate system based on the target object as a positioning reference; Predicting a target object position of the target object in the object coordinate system of the current frame image based on a relative speed of the target object and a time difference before and after the displacement occurs in two adjacent frame images; Convert the predicted target object position to the camera coordinate system; Calculating the position of the target object in the image coordinate system of the current frame image according to the coordinate change of the camera coordinate system in the two adjacent frame images; Based on the position of the target object in the image coordinate system of the current frame image, the position of the predicted perception box of the target object in the current frame image is obtained.

4. The method according to claim 1, characterized in that Calling a neural network model to extract perception frame features from the successfully matched target perception frame includes: Perform multi-task scheduling on the successfully matched target perception frame in the current frame image; Expanding the range of the target perception frame to obtain an expanded target perception frame; The neural network model is called to classify the expanded target perception frame, determine the type of the target perception frame, and extract the perception frame features of the successfully matched target perception frame in the current frame image.

5. The method according to claim 4, characterized in that Based on the extracted perception frame features of the target perception frame, a Hungarian algorithm is used to perform secondary matching on the target perception frame, including: Performing cosine similarity calculation on at least one perception frame feature extracted from the current frame image and the corresponding historical feature of the target object; Determine the target perception box with the greatest similarity from the calculation results; If the intersection-over-union (IOU) ratio of the target perception frame with the greatest similarity meets the minimum threshold, obtaining the weight matrix of the target perception frame with the greatest similarity; Based on the weight matrix of the target perception frame with the greatest similarity, the Hungarian algorithm is used to perform secondary matching on the target perception frame with the greatest similarity.

6. The method according to claim 5, characterized in that In the case of successful matching, determining the position of the tracked target object includes: If the match is successful, check whether there is a coordinate jump between the target perception frame with the greatest similarity and the corresponding historical perception frame; If the coordinate jump does not exist, the target object corresponding to the target perception frame with the greatest similarity is determined to be the tracked target object, and the position of the tracked target object is obtained.

7. The method according to claim 6, characterized in that The method further comprises: A target perception frame that fails to match is obtained, and new identification information is assigned to the target perception frame that fails to match.

8. A target tracking device, characterized in that: include: An acquisition module, used for acquiring a current frame image of a vehicle driving environment and a frame image previous to the current frame image; A prediction module, configured to use at least one perception frame identified in the previous frame image to predict a predicted perception frame in the current frame image, wherein the perception frame is used to identify a target object to be tracked in the previous frame image; A first matching module, configured to perform a first matching of at least one perception frame identified in the current frame image with a predicted perception frame at a corresponding position; An acquisition module, used to acquire a successfully matched target perception frame in the current frame image; A calling module, used for calling a neural network model to extract perception frame features from the successfully matched target perception frame; A second matching module is used to perform secondary matching on the target perception frame using the Hungarian algorithm based on the extracted perception frame features of the target perception frame; A determination module, used to determine the position of the tracked target object when the match is successful; The first matching module includes: A first calculation module is used to calculate the intersection-and-union ratio (IOU) of at least one perception frame identified in the current frame image and the predicted perception frame of the corresponding type to obtain the IOU of each target object in the current frame image; A first determination module is used to determine a target object with a maximum intersection-over-union (IOU) ratio from the current frame image to obtain a tracked object; A verification module, used to verify whether there is a coordinate jump between the perception frame of the tracked object and the corresponding predicted perception frame if the intersection-over-union (IOU) ratio of the tracked object meets the minimum limit; The second determination module is used for determining that, if the coordinate jump does not exist, the perception frame of the tracked object is the successfully matched target perception frame in the current frame image.

9. An electronic device, characterized in that: The electronic device includes one or more processors; a storage device for storing one or more programs, which, when the one or more programs are executed by the one or more processors, enables the one or more processors to run the programs, wherein the program is configured to execute the target tracking method described in any one of claims 1 to 7 when running.

Images