Obstacle cross-frame tracking method, device, equipment and storage medium

Abstract

The application relates to the field of image recognition and discloses an obstacle cross-frame tracking method, device and equipment and a storage medium, which comprises the following steps: acquiring an obstacle recognition result in a first detection image, and generating a tracking query based on the obstacle recognition result; extracting an image feature of an obstacle in a second detection image, and obtaining a new obstacle query based on the image feature; inputting the tracking query and the new obstacle query into a preset obstacle tracking model, tracking and identifying the obstacle in the second detection image through the obstacle tracking model, and obtaining a tracking and identifying result. Through the method, the query value of the obstacle feature between two continuous frame images can be compared based on image information, the tracking process for consecutive images in time sequence is directly completed implicitly in the obstacle tracking model, the complexity of the algorithm architecture is reduced, and the optimizability and analyzability of the whole tracking process are improved.

Description

Technical Field

[0001] This invention relates to the field of image recognition, and more particularly to a method, apparatus, device, and storage medium for cross-frame obstacle tracking. Background Technology

[0002] In existing autonomous driving tasks, obstacle detection boxes and obstacle categories can be detected through single-frame image processing. However, to determine the dynamic behavior of the identified obstacles, it is necessary to identify the same obstacle across multiple frames, i.e., to track the identified obstacle across several frames. Current technology detects obstacles within a single frame and then obtains the association relationship frame by frame using a one-to-one matching algorithm. However, obstacle similarity calculation is easily affected by background content, and the matching algorithm leads to a complex tracking process, excessive processing time, and low recognition accuracy. Summary of the Invention

[0003] The main objective of this invention is to solve the technical problems in existing obstacle cross-frame tracking methods, such as the obstacle similarity calculation being easily affected by background content, the matching algorithm leading to a complex tracking process, excessive processing time, and low recognition accuracy.

[0004] The first aspect of the present invention provides an obstacle cross-frame tracking method, the obstacle cross-frame tracking method comprising: acquiring obstacle recognition results in a first detection image, and generating a tracking query based on the obstacle recognition results; extracting image features of obstacles in a second detection image, and obtaining a new obstacle query based on the image features; inputting the tracking query and the new obstacle query into a preset obstacle tracking model, and tracking and recognizing obstacles in the second detection image through the obstacle tracking model to obtain a tracking and recognition result.

[0005] Optionally, in a first implementation of the first aspect of the present invention, before obtaining the obstacle recognition result in the first detection image and generating a tracking query based on the obstacle recognition result, the method further includes: obtaining a video to be tracked; performing frame segmentation processing on the video to be tracked to obtain a plurality of temporally consecutive frame images; constructing a second detection image based on the last frame image in the temporal sequence; constructing a first detection image based on other frame images in the temporal sequence corresponding to the frame image corresponding to the second detection image; and performing obstacle detection on each frame image using a preset obstacle detector to obtain the obstacle recognition result of each frame image.

[0006] Optionally, in a second implementation of the first aspect of the present invention, the obstacle features include at least an interrogation value. The step of obtaining obstacle recognition results in the first detection image and generating a tracking interrogation based on the obstacle recognition results includes: extracting image features from the first detection image; obtaining obstacle recognition results in the first detection image and extracting obstacle features of each obstacle from the image features based on the obstacle recognition results; determining the predicted score of each obstacle and selecting obstacles whose predicted scores meet preset conditions as target obstacles; and selecting an interrogation value from the obstacle features of the target obstacles as a tracking interrogation for the second detection image.

[0007] Optionally, in a third implementation of the first aspect of the present invention, the step of extracting image features of obstacles in the second detection image and obtaining new obstacle queries based on the image features includes: extracting image features from the second detection image; obtaining obstacle recognition results in the second detection image and extracting obstacle features of each obstacle from the image features based on the obstacle recognition results; and extracting query values ​​from the obstacle features as new obstacle queries for the second detection image.

[0008] Optionally, in a fourth implementation of the first aspect of the present invention, the step of inputting the tracking query and the new obstacle query into a preset obstacle tracking model, and tracking and identifying obstacles in the second detection image through the obstacle tracking model to obtain a tracking and identification result includes: using the preset obstacle tracking model to traverse and calculate the similarity between the tracking query and the new obstacle query; selecting tracking queries and new obstacle queries with similarity higher than a preset threshold; and associating the obstacle identification result in the first detection image corresponding to the tracking query and the obstacle identification result in the second detection image corresponding to the new obstacle query to obtain a tracking and identification result.

[0009] Optionally, in the fifth implementation of the first aspect of the present invention, before acquiring the video to be tracked and obtaining a number of temporally consecutive frame images by performing frame-segmentation processing on the video to be tracked, the method further includes: constructing a cross-frame obstacle tracking model to be trained; collecting preprocessed sample tracking queries, sample new obstacle queries, and sample tracking recognition results; inputting the sample tracking queries and sample new obstacle queries into the cross-frame obstacle tracking model to be trained to obtain tracking recognition results based on the sample tracking queries and sample new obstacle queries; calculating a loss function using the Hungarian algorithm based on the sample tracking queries and the sample new obstacle queries, wherein the sample new obstacle queries contain the ground truth value corresponding to the sample tracking queries; determining whether the loss function value is less than a preset threshold; and optimizing the cross-frame obstacle tracking model to be trained based on the loss function value to obtain the cross-frame obstacle tracking model.

[0010] Optionally, in a sixth implementation of the first aspect of the present invention, before determining whether the loss function value is less than a preset threshold, the method further includes: calculating the distance between the sample tracking query and the sample new obstacle query; calculating several classification loss values, center position regression loss values, and size regression loss values ​​based on the Hungarian algorithm to obtain calculation results; and selecting the smallest term among the calculation results of the sample tracking query and the sample new obstacle query as the preset threshold of the loss function value.

[0011] A second aspect of the present invention provides an obstacle cross-frame tracking device, the obstacle cross-frame tracking device comprising: a tracking interrogation generation module, configured to acquire obstacle recognition results in a first detection image and generate a tracking interrogation based on the obstacle recognition results; a feature extraction module, configured to extract image features of obstacles in a second detection image and obtain a new obstacle interrogation based on the image features; and a tracking recognition acquisition module, configured to input the tracking interrogation and the new obstacle interrogation into a preset obstacle tracking model, and perform tracking recognition on the obstacles in the second detection image through the obstacle tracking model to obtain a tracking recognition result.

[0012] Optionally, in the first implementation of the second aspect of the present invention, the obstacle cross-frame tracking device further includes a video preprocessing module, which is specifically used for: acquiring a video to be tracked; performing frame-by-frame processing on the video to be tracked to obtain a number of temporally consecutive frame images; constructing a second detection image based on the last frame image in the time sequence; constructing a first detection image based on other frame images in the time sequence corresponding to the frame image corresponding to the second detection image; and performing obstacle detection on each frame image using a preset obstacle detector to obtain the obstacle recognition result of each frame image.

[0013] Optionally, in a second implementation of the second aspect of the present invention, the tracking query generation module is specifically used for: extracting image features from the first detection image; obtaining obstacle recognition results in the first detection image, and extracting obstacle features of each obstacle from the image features based on the obstacle recognition results; determining the predicted score of each obstacle, and selecting obstacles whose predicted scores meet preset conditions as target obstacles; and selecting query values ​​from the obstacle features of the target obstacles as tracking queries for the second detection image.

[0014] Optionally, in a third implementation of the second aspect of the present invention, the feature extraction module is specifically used to: extract image features from the second detection image; obtain obstacle recognition results in the second detection image, and extract obstacle features of each obstacle from the image features based on the obstacle recognition results; and extract query values ​​from the obstacle features as new obstacle queries for the second detection image.

[0015] Optionally, in a fourth implementation of the second aspect of the present invention, the tracking and identification acquisition module is specifically used to: calculate the similarity between the tracking query and the new obstacle query using a preset obstacle tracking model; select tracking queries and the new obstacle query with similarity higher than a preset threshold; and associate the obstacle identification result in the first detection image corresponding to the tracking query with the obstacle identification result in the second detection image corresponding to the new obstacle query to obtain the tracking and identification result.

[0016] Optionally, in a fifth implementation of the second aspect of the present invention, the obstacle cross-frame tracking device further includes a model training module, which is specifically used for: a model building unit for building an obstacle cross-frame tracking model to be trained; a sample training unit for collecting preprocessed sample tracking queries, sample new obstacle queries, and sample tracking recognition results, and inputting the sample tracking queries and sample new obstacle queries into the obstacle cross-frame tracking model to be trained to obtain tracking recognition results based on the sample tracking queries and sample new obstacle queries; a loss function calculation unit for calculating a loss function based on the sample tracking queries and the sample new obstacle queries using the Hungarian algorithm, wherein the sample new obstacle queries contain the true value corresponding to the sample tracking queries; a loss function judgment unit for judging whether the loss function value is less than a preset threshold; and a model optimization unit for optimizing the obstacle cross-frame tracking model to be trained based on the loss function value to obtain the obstacle cross-frame tracking model.

[0017] Optionally, in a sixth implementation of the second aspect of the present invention, the model training module further includes a preset threshold selection unit, which is specifically used for: calculating the distance between the sample tracking query and the sample new obstacle query; calculating several classification loss values, center position regression loss values, and size regression loss values ​​based on the Hungarian algorithm to obtain the calculation results; and selecting the smallest term of the calculation results between the sample tracking query and the sample new obstacle query as the preset threshold of the loss function value.

[0018] A third aspect of the present invention provides an obstacle cross-frame tracking device, comprising: a memory and at least one processor, wherein the memory stores a request, and the memory and the at least one processor are interconnected via a line; the at least one processor invokes the request in the memory to cause the obstacle cross-frame tracking device to perform the steps of the obstacle cross-frame tracking method described above.

[0019] A fourth aspect of the present invention provides a computer-readable storage medium storing a request that, when executed on a computer, causes the computer to perform the steps of the obstacle cross-frame tracking method described above.

[0020] In the technical solution of this invention, obstacle recognition results are obtained from a first detection image, and tracking queries are generated based on these results. Image features of obstacles in a second detection image are extracted, and new obstacle queries are obtained based on these features. The tracking queries and the new obstacle queries are input into a preset obstacle tracking model, and the obstacle tracking model tracks and identifies obstacles in the second detection image to obtain tracking and identification results. This method can compare query values ​​regarding obstacle features between two consecutive frames based on image information, implicitly completing the tracking process for temporally continuous images directly within the obstacle tracking model. This reduces the complexity of the algorithm architecture and improves the optimizability and analyzability of the entire tracking process. Attached Figure Description

[0021] Figure 1 This is a schematic diagram of the first embodiment of the obstacle cross-frame tracking method in this invention; Figure 2 This is a schematic diagram of a second embodiment of the obstacle cross-frame tracking method in this invention; Figure 3 This is a schematic diagram of a third embodiment of the obstacle cross-frame tracking method in this invention; Figure 4 This is a schematic diagram of one embodiment of the obstacle cross-frame tracking device in this invention; Figure 5This is a schematic diagram of another embodiment of the obstacle cross-frame tracking device in this invention; Figure 6 This is a schematic diagram of one embodiment of the obstacle cross-frame tracking device in this invention. Detailed Implementation

[0022] In the technical solution of this invention, obstacle recognition results are obtained from a first detection image, and tracking queries are generated based on these results. Image features of obstacles in a second detection image are extracted, and new obstacle queries are obtained based on these features. The tracking queries and the new obstacle queries are input into a preset obstacle tracking model, and the obstacle tracking model tracks and identifies obstacles in the second detection image to obtain tracking and identification results. This method can compare query values ​​regarding obstacle features between two consecutive frames based on image information, implicitly completing the tracking process for temporally continuous images directly within the obstacle tracking model. This reduces the complexity of the algorithm architecture and improves the optimizability and analyzability of the entire tracking process.

[0023] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification, claims, and accompanying drawings of this invention are used to distinguish similar constructions and are not necessarily used to describe a particular order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” or “having,” and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0024] For ease of understanding, the specific process of the embodiments of the present invention is described below. Please refer to [link / reference]. Figure 1 The first embodiment of the obstacle cross-frame tracking method in this invention includes: 101. Obtain the obstacle recognition results in the first detection image, and generate a tracking query based on the obstacle recognition results; In this embodiment, obstacle recognition results contained in the first detection image are obtained, wherein the first detection image contains several recognized obstacle recognition results, several query values ​​are extracted from the first detection image, and a tracking query corresponding to the second detection image is generated.

[0025] Specifically, image features are extracted from obstacles in the obstacle recognition results using a convolutional neural network, resulting in image features corresponding to several obstacles. Each image feature contains a query value, key value, and value corresponding to the obstacle. The query values ​​in the first detection image are then used as tracking queries. For the first detection image, there are no tracking queries, only new obstacle queries. Subsequently, the new obstacle queries in the first detection image are used as tracking queries for the second frame.

[0026] 102. Extract the image features of obstacles in the second detected image, and obtain new obstacle queries based on the image features; In this embodiment, image features of all obstacles in the second detection image are extracted, and query values ​​in the image features are extracted as new obstacle queries in the second detection image.

[0027] Specifically, multi-scale image features are extracted using convolutional neural networks such as residual neural networks and feature pyramid networks. These image features include query values, key values, and values ​​corresponding to obstacles. The query values ​​in the second detection image are then used as new obstacle queries in the second detection image.

[0028] 103. Input the tracking query and the new obstacle query into the preset obstacle tracking model, and use the obstacle tracking model to track and identify obstacles in the second detection image to obtain the tracking and identification results.

[0029] In this embodiment, by inputting the new obstacle query of the second detection image and the tracking query extracted from the first detection image corresponding to the second detection image into a preset obstacle tracking model, the obstacle tracking model will track and identify the obstacles in the second detection image to obtain the tracking and identification results.

[0030] Specifically, the tracking and identification results include obstacles that appear simultaneously in both the first and second detection images.

[0031] In this embodiment, obstacle recognition results are obtained from the first detection image, and tracking queries are generated based on these results. Image features of obstacles in the second detection image are extracted, and new obstacle queries are obtained based on these features. The tracking queries and the new obstacle queries are input into a preset obstacle tracking model, which tracks and identifies obstacles in the second detection image to obtain tracking and identification results. This method can compare query values ​​regarding obstacle features between two consecutive frames based on image information, implicitly completing the tracking process for temporally continuous images within the obstacle tracking model. This reduces the complexity of the algorithm architecture and improves the optimizability and analyzability of the entire tracking process.

[0032] Please see Figure 2 A second embodiment of the obstacle cross-frame tracking method in this invention includes: 201. Construct a cross-frame obstacle tracking model to be trained; In this embodiment, a cross-frame obstacle tracking model to be trained is constructed by employing convolutional neural networks, such as residual neural networks and pyramid networks.

[0033] Specifically, for the input training samples, the feature size is Nx2xHxWxC, where N is the size of the batch, M is the number of images in each sample, H is the height, W is the width, and C is the number of channels.

[0034] 202. Collect the preprocessed sample tracking query, sample new obstacle query, and sample tracking recognition results. Input the sample tracking query and sample new obstacle query into the obstacle cross-frame tracking model to be trained to obtain the tracking and recognition results based on the sample tracking query and sample new obstacle query. In this embodiment, feature extraction can be performed on the image in advance, and the extracted sample tracking query and sample new obstacle query can be used as training samples for the obstacle cross-frame tracking model to be trained.

[0035] On the other hand, the feature extraction step can also be integrated into the obstacle cross-frame tracking model to be trained. However, preferably, in order to decouple the various functional modules and facilitate subsequent modular management, the feature extraction function of the extracted image can be separated for easy maintenance.

[0036] 203. Based on sample tracking queries and sample new obstacle queries, the loss function is calculated using the Hungarian algorithm; In this embodiment, based on the input sample tracking queries and sample new obstacle queries, the new obstacle queries are treated as one set, and similarly, the sample tracking queries are also treated as one set. Simultaneously, the sample tracking identification results are treated as the ground truth set. The distance between the new obstacle query set and the sample tracking query set is calculated, and this distance becomes the loss function value.

[0037] Specifically, the Hungarian algorithm is a combinatorial optimization algorithm that solves the task assignment problem in polynomial time. Widely used in operations research, it addresses the task assignment problem, also known as the task allocation problem, where n tasks are assigned to n people to maximize efficiency. The goal is to find the n distinct rows and columns of zero elements in the transformation coefficient matrix to find the optimal solution to the assignment problem.

[0038] 204. Calculate the distance between sample tracking inquiry and sample new obstacle inquiry; 205. Based on the Hungarian algorithm, calculate several classification loss values, center position regression loss values, and size regression loss values ​​respectively, and obtain the calculation results; In this embodiment, for the calculation of the loss function belonging to the first detection image, the distance between the query value belonging to the first detection image (i.e., the sample tracking query and the sample tracking recognition result) is used as the loss function. In the second detection image, for the already matched sample tracking queries and sample new obstacle queries, the correctness of the match can be determined based on the information of the corresponding sample tracking recognition results. For valid tracking queries belonging to the first detection image, a one-to-one match is made with sample new obstacle queries in the second frame, and the distance between the matched sample tracking queries and sample new obstacle queries is calculated to obtain the second loss function. The two loss functions are accumulated as the final loss function training process.

[0039] 206. Select the minimum term calculated from the sample tracking query and the sample new obstacle query as the preset threshold for the loss function value; In this embodiment, the loss function includes classification loss (lcls), center position regression loss (lcenter), and size regression loss (lsize). The minimum loss among all possible pairings is calculated using the Hungarian algorithm and used as the preset threshold for the final loss function.

[0040] 207. Determine whether the loss function value is less than a preset threshold; 208. Optimize the obstacle cross-frame tracking model to be trained based on the loss function value to obtain the obstacle cross-frame tracking model; 209. Obtain the video to be tracked. By performing frame segmentation on the video to be tracked, obtain several temporally consecutive frame images. On the other hand, it is also possible to achieve real-time obstacle cross-frame tracking by processing the real-time images captured by the vehicle-mounted camera in real time and performing frame segmentation on the real-time images.

[0041] 210. Construct a second detection image based on the last frame image in the time sequence, and construct a first detection image based on other frame images that are in the time sequence earlier than the frame image corresponding to the second detection image; In this embodiment, the last frame image in the time sequence is used as the second detection image, and other frame images earlier than the last frame image are used to construct the first detection image.

[0042] Specifically, when acquiring real-time images, the latest frame image is used as the second detection image. By continuously updating the second detection image, the number of tracking queries is accumulated, and new obstacle queries are updated to continuously identify obstacles in the last frame image.

[0043] 211. Obstacle detection is performed on each frame image using a preset obstacle detector to obtain the obstacle recognition results for each frame image; 212. Obtain the obstacle recognition results in the first detection image, and generate a tracking query based on the obstacle recognition results; 213. Extract the image features of obstacles in the second detected image, and obtain new obstacle queries based on the image features; 214. Input the tracking query and the new obstacle query into the preset obstacle tracking model, and use the obstacle tracking model to track and identify obstacles in the second detection image to obtain the tracking and identification results.

[0044] This embodiment, based on the previous embodiment, describes in detail the process of extracting image features from the first detection image; obtaining obstacle recognition results in the first detection image, and extracting obstacle features of each obstacle from the image features based on the obstacle recognition results; determining the predicted score of each obstacle, and selecting obstacles whose predicted scores meet preset conditions as target obstacles; and selecting query values ​​from the obstacle features of the target obstacles as the tracking query for the second detection image. Compared with traditional methods, this embodiment clarifies the specific process from obstacle recognition to obstacle feature extraction and the application of query values ​​in the first and second detection images. By determining the extraction process of target obstacles in the first and second detection images, the tracking relationship between the first and second detection images can be implicitly represented, realizing the obstacle tracking process between the two frames.

[0045] Please see Figure 3 The third embodiment of the obstacle cross-frame tracking method in this invention includes: 301. Extract image features from the first detection image; In this embodiment, obstacle recognition results contained in the first detection image are obtained, wherein the first detection image contains several recognized obstacle recognition results, several query values ​​are extracted from the first detection image, and a tracking query corresponding to the second detection image is generated.

[0046] Specifically, image features are extracted from obstacles in the obstacle recognition results using a convolutional neural network, resulting in image features corresponding to several obstacles. Each image feature contains a query value, key value, and value corresponding to the obstacle. The query values ​​in the first detection image are then used as tracking queries. For the first detection image, there are no tracking queries, only new obstacle queries. Subsequently, the new obstacle queries in the first detection image are... 302. Obtain the obstacle recognition results in the first detection image, and extract the obstacle features of each obstacle from the image features based on the obstacle recognition results; In this embodiment, each frame image is first processed by obstacle recognition to obtain a first detection image or a second detection image containing obstacle recognition results. The obstacle recognition results are several detection boxes in the frame image that are identified as obstacles, and the obstacle features in the image within the detection boxes are extracted.

[0047] 303. Determine the predicted score of each obstacle, and select the obstacle whose predicted score meets the preset conditions as the target obstacle; In this embodiment, each frame of the video to be detected and the frame between the two frames can be used as the first detection image and the second detection image. For the first frame of the video to be detected, there is no tracking query value. In any first detection image other than the first frame, there are several identified obstacles. Obstacles in the first detection image whose predicted scores meet the preset conditions are used as target obstacles.

[0048] 304. Based on the obstacle features of the target obstacle, select the query value as the tracking query of the second detection image; In this embodiment, obstacle features of the target obstacle in the first detection image are extracted, wherein the obstacle features include query value, key value and value, and the query value is selected as the tracking query of the second detection image.

[0049] Specifically, the follow-up inquiry includes valid follow-up inquiries and invalid follow-up inquiries. The valid follow-up inquiries determined in step 303 are applied to the second detection image.

[0050] Specifically, a valid tracking query can be understood as a tracking query that is determined to be a valid obstacle, that is, an obstacle is confirmed to exist within the detection box, while an invalid obstacle means that the detection box is empty, which is a recognition error and there is no obstacle.

[0051] 305. Extract image features from the second detection image; In this embodiment, obstacle recognition results contained in the first detection image are obtained, wherein the first detection image contains several recognized obstacle recognition results, several query values ​​are extracted from the first detection image, and a tracking query corresponding to the second detection image is generated.

[0052] Specifically, image features are extracted from obstacles in the obstacle recognition results using a convolutional neural network, resulting in image features corresponding to several obstacles. Each image feature contains a query value, key value, and value corresponding to the obstacle. The query values ​​in the first detection image are then used as tracking queries. For the first detection image, there are no tracking queries, only new obstacle queries. Subsequently, the new obstacle queries in the first detection image are used as tracking queries for the second frame.

[0053] 306. Obtain the obstacle recognition results in the second detection image, and extract the obstacle features of each obstacle from the image features based on the obstacle recognition results; In this embodiment, both the first and second detection images contain obstacle recognition results. Similarly, the obstacle features of the second detection image are obtained by extracting features from the detection box contained in the second detection image.

[0054] 307. Extract the query value from the obstacle features and use it as a new obstacle query in the second detection image; 308. Calculate the similarity between tracking queries and new obstacle queries using a pre-defined obstacle tracking model. In this embodiment, each valid tracking query is first matched one-to-one with the obstacle detection box that should be predicted in the second detection image. The loss function still includes classification loss, center position regression loss, size regression loss, etc. The remaining tracking queries in the second detection image detection boxes that are not matched are then compared with the new obstacle query set in the second detection image by distance calculation, where the distance calculation is used to calculate the similarity between the two.

[0055] 309. Select tracking queries and new obstacle queries with similarity higher than a preset threshold; 310. Associate the obstacle recognition results in the first detection image corresponding to the tracking inquiry with the obstacle recognition results in the second detection image corresponding to the new obstacle inquiry to obtain the tracking recognition results.

[0056] This embodiment, based on the previous embodiment, details the process of extracting image features from the first detection image; obtaining obstacle recognition results in the first detection image, and extracting obstacle features of each obstacle from the image features based on the obstacle recognition results; determining the predicted score of each obstacle, and selecting obstacles whose predicted scores meet preset conditions as target obstacles; and selecting query values ​​from the obstacle features of the target obstacles as the tracking query for the second detection image. Compared with traditional methods, this embodiment refines the process of extracting tracking queries from the first detection image. By using a convolutional neural network to extract image features from the obstacles in the obstacle recognition results, several image features corresponding to obstacles are obtained. Each image feature contains a query value, key value, and value corresponding to the obstacle, and the query values ​​in the first detection image are uniformly used as tracking queries. For the first detection image, there are no tracking queries, only new obstacle queries. Subsequently, the new obstacle queries in the first detection image are used as the tracking queries for the second frame.

[0057] The obstacle cross-frame tracking method in the embodiments of the present invention has been described above. The obstacle cross-frame tracking device in the embodiments of the present invention is described below. Please refer to [link / reference]. Figure 4 One embodiment of the obstacle cross-frame tracking device in this invention includes: The tracking query generation module 401 is used to obtain the obstacle recognition result in the first detection image and generate a tracking query based on the obstacle recognition result; The feature extraction module 402 is used to extract image features of obstacles in the second detection image and obtain new obstacle queries based on the image features; The tracking and recognition acquisition module 403 is used to input the tracking query and the new obstacle query into a preset obstacle tracking model, and to track and recognize the obstacles in the second detection image through the obstacle tracking model to obtain the tracking and recognition result.

[0058] In this embodiment of the invention, the obstacle cross-frame tracking device operates the aforementioned obstacle cross-frame tracking method, including: acquiring obstacle recognition results in a first detection image and generating a tracking query based on the obstacle recognition results; extracting image features of obstacles in a second detection image and obtaining a new obstacle query based on the image features; inputting the tracking query and the new obstacle query into a preset obstacle tracking model, and using the obstacle tracking model to track and identify obstacles in the second detection image to obtain a tracking and identification result. This method can compare query values ​​regarding obstacle features between two consecutive frames based on image information, implicitly completing the tracking process for temporally continuous images directly within the obstacle tracking model, reducing the complexity of the algorithm architecture, and improving the optimizability and analyzability of the entire tracking process.

[0059] Please see Figure 5 A second embodiment of the obstacle cross-frame tracking device in this invention includes: The tracking query generation module 401 is used to obtain the obstacle recognition result in the first detection image and generate a tracking query based on the obstacle recognition result; The feature extraction module 402 is used to extract image features of obstacles in the second detection image and obtain new obstacle queries based on the image features; The tracking and recognition acquisition module 403 is used to input the tracking query and the new obstacle query into a preset obstacle tracking model, and to track and recognize the obstacles in the second detection image through the obstacle tracking model to obtain the tracking and recognition result.

[0060] In this embodiment, the obstacle cross-frame tracking device further includes a video preprocessing module 404, which is specifically used for: The video to be tracked is acquired, and the video is divided into frames to obtain several temporally consecutive frame images. A second detection image is constructed based on the last frame image in the time sequence, and a first detection image is constructed based on other frame images that are temporally earlier than the frame image corresponding to the second detection image. An obstacle detection is performed on each frame image using a preset obstacle detector to obtain the obstacle recognition result of each frame image.

[0061] In this embodiment, the tracking query generation module 401 is specifically used for: Extract image features from the first detection image; obtain obstacle recognition results from the first detection image, and extract obstacle features of each obstacle from the image features based on the obstacle recognition results; determine the predicted score of each obstacle, and select the obstacle whose predicted score meets the preset conditions as the target obstacle; select the query value of the target obstacle as the tracking query of the second detection image based on the obstacle features of the target obstacle.

[0062] In this embodiment, the feature extraction module 402 is specifically used for: Extract image features from the second detection image; obtain obstacle recognition results from the second detection image, and extract obstacle features of each obstacle from the image features based on the obstacle recognition results; extract query values ​​from the obstacle features as new obstacle queries for the second detection image.

[0063] In this embodiment, the tracking and identification acquisition module 403 is specifically used for: The similarity between the tracking query and the new obstacle query is calculated by traversing through the preset obstacle tracking model; tracking queries and new obstacle queries with similarity higher than a preset threshold are selected; the obstacle recognition result in the first detection image corresponding to the tracking query and the obstacle recognition result in the second detection image corresponding to the new obstacle query are associated to obtain the tracking recognition result.

[0064] In this embodiment, the obstacle cross-frame tracking device further includes a model training module 405, which is specifically used for: The model building unit 4051 constructs a cross-frame obstacle tracking model to be trained; the sample training unit 4052 collects preprocessed sample tracking queries, sample new obstacle queries, and sample tracking recognition results, and inputs the sample tracking queries and sample new obstacle queries into the cross-frame obstacle tracking model to be trained to obtain tracking and recognition results based on the sample tracking queries and sample new obstacle queries; the loss function calculation unit 4053 calculates a loss function using the Hungarian algorithm based on the sample tracking queries and the sample new obstacle queries, wherein the sample new obstacle queries contain the ground truth value corresponding to the sample tracking queries; the loss function judgment unit 4054 determines whether the loss function value is less than a preset threshold; and the model optimization unit 4055 optimizes the cross-frame obstacle tracking model to be trained based on the loss function value to obtain the cross-frame obstacle tracking model.

[0065] In this embodiment, the model training module 405 further includes a preset threshold selection unit 4056, which is specifically used for: Calculate the distance between the sample tracking query and the sample new obstacle query; calculate several classification loss values, center position regression loss values, and size regression loss values ​​based on the Hungarian algorithm to obtain the calculation results; select the smallest term among the calculation results of the sample tracking query and the sample new obstacle query as the preset threshold of the loss function value.

[0066] Based on the previous embodiment, this embodiment describes in detail the specific functions of each module and the unit composition of some modules. Through the above modules, the specific functions of the original modules are refined, the operation of the obstacle cross-frame tracking device is improved, the reliability of its operation is enhanced, and the actual logic between each step is clarified, thereby improving the practicality of the device.

[0067] above Figure 4 and Figure 5 The obstacle cross-frame tracking device in the embodiments of the present invention will be described in detail from the perspective of modular functional entities. The obstacle cross-frame tracking device in the embodiments of the present invention will be described in detail from the perspective of hardware processing.

[0068] Figure 6 This is a schematic diagram of the structure of an obstacle cross-frame tracking device 600 provided in an embodiment of the present invention. The obstacle cross-frame tracking device 600 can vary significantly due to different configurations or performance. It may include one or more central processing units (CPUs) 610 (e.g., one or more processors) and a memory 620, and one or more storage media 630 (e.g., one or more mass storage devices) storing application programs 633 or data 632. The memory 620 and storage media 630 can be temporary or persistent storage. The program stored in the storage media 630 may include one or more modules (not shown in the diagram), each module may include a series of request operations on the obstacle cross-frame tracking device 600. Furthermore, the processor 610 may be configured to communicate with the storage media 630 and execute the series of request operations in the storage media 630 on the obstacle cross-frame tracking device 600 to implement the steps of the above-described obstacle cross-frame tracking method.

[0069] The obstacle cross-frame tracking device 600 may also include one or more power supplies 640, one or more wired or wireless network interfaces 650, one or more input / output interfaces 660, and / or one or more operating systems 631, such as Windows Server, MacOSX, Unix, Linux, FreeBSD, etc. Those skilled in the art will understand that... Figure 6The obstacle cross-frame tracking device structure shown does not constitute a limitation on the obstacle cross-frame tracking device provided in this application. It may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0070] The present invention also provides a computer-readable storage medium, which may be a non-volatile computer-readable storage medium or a volatile computer-readable storage medium, wherein the computer-readable storage medium stores a request that, when the request is executed on a computer, causes the computer to perform the steps of the obstacle cross-frame tracking method described above.

[0071] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the system, device, or unit described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0072] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several requests to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0073] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for obstacle tracking across frames, characterized in that, The obstacle cross-frame tracking method includes: Obtain obstacle recognition results from the first detection image, and generate a tracking query based on the obstacle recognition results; Image features of obstacles are extracted from the second detected image, and new obstacle queries are obtained based on the image features; the image features include query values, key values, and values ​​corresponding to the obstacles; The tracking query and the new obstacle query are input into a preset obstacle tracking model. The obstacle tracking model is used to track and identify obstacles in the second detected image to obtain tracking and identification results. The step of inputting the tracking query and the new obstacle query into a preset obstacle tracking model, and using the obstacle tracking model to track and identify obstacles in the second detected image to obtain tracking and identification results includes: The similarity between the tracking query and the new obstacle query is calculated by traversing through the pre-defined obstacle tracking model. Select tracking queries and new obstacle queries that have a similarity higher than a preset threshold; The obstacle identification results in the first detection image corresponding to the tracking query and the obstacle identification results in the second detection image corresponding to the new obstacle query are correlated to obtain the tracking identification results; Constructing a cross-frame obstacle tracking model specifically includes: Construct a cross-frame obstacle tracking model to be trained; Collect preprocessed sample tracking queries, sample new obstacle queries, and sample tracking recognition results. Input the sample tracking queries and sample new obstacle queries into the obstacle cross-frame tracking model to be trained to obtain tracking and recognition results based on the sample tracking queries and sample new obstacle queries. Based on the sample tracking query and the sample new obstacle query, a loss function is calculated using the Hungarian algorithm, wherein the sample new obstacle query contains the truth value corresponding to the sample tracking query; Determine whether the value of the loss function is less than a preset threshold; The obstacle cross-frame tracking model to be trained is optimized based on the loss function value to obtain the obstacle cross-frame tracking model.

2. The obstacle cross-frame tracking method according to claim 1, characterized in that, Before acquiring the obstacle recognition result in the first detection image and generating a tracking interrogation based on the obstacle recognition result, the method further includes: The video to be tracked is acquired, and the video to be tracked is processed into frames to obtain several frames that are consecutive in time. A second detection image is constructed based on the last frame image in the time sequence, and a first detection image is constructed based on other frame images whose time sequence is earlier than the frame image corresponding to the second detection image. Obstacle detection is performed on each frame image using a preset obstacle detector to obtain the obstacle recognition results for each frame image.

3. The obstacle cross-frame tracking method according to claim 2, characterized in that, The obstacle features include at least the query value. The step of acquiring obstacle recognition results in the first detection image and generating a tracking query based on the obstacle recognition results includes: Extract image features from the first detected image; Obtain obstacle recognition results in the first detection image, and extract obstacle features of each obstacle from the image features based on the obstacle recognition results; Determine the predicted score for each obstacle, and select the obstacle whose predicted score meets the preset conditions as the target obstacle; Based on the obstacle features of the target obstacle, the query value is selected as the tracking query for the second detection image.

4. The obstacle cross-frame tracking method according to claim 2, characterized in that, The step of extracting image features of obstacles in the second detected image and obtaining new obstacle queries based on the image features includes: Extract image features from the second detected image; Obtain the obstacle recognition results in the second detection image, and extract the obstacle features of each obstacle from the image features based on the obstacle recognition results; The query value is extracted from the obstacle features and used as a new obstacle query in the second detected image.

5. The obstacle cross-frame tracking method according to claim 1, characterized in that, Before determining whether the loss function value is less than a preset threshold, the method further includes: Calculate the distance between the sample tracking query and the sample new obstacle query; Based on the Hungarian algorithm, several classification loss values, center position regression loss values, and size regression loss values ​​were calculated respectively to obtain the calculation results; The minimum term calculated from the sample tracking query and the sample new obstacle query is selected as the preset threshold for the loss function value.

6. An obstacle cross-frame tracking device, characterized in that, The obstacle cross-frame tracking device includes: The tracking query generation module is used to obtain the obstacle recognition results in the first detection image and generate tracking queries based on the obstacle recognition results; The feature extraction module is used to extract image features of obstacles in the second detection image and obtain new obstacle queries based on the image features; the image features include query values, key values, and values ​​corresponding to the obstacles; The tracking and recognition acquisition module is used to input the tracking query and the new obstacle query into a preset obstacle tracking model, and to track and recognize the obstacles in the second detection image through the obstacle tracking model to obtain the tracking and recognition result; The tracking and identification module is used for: The similarity between the tracking query and the new obstacle query is calculated by traversing through the pre-defined obstacle tracking model. Select tracking queries and new obstacle queries that have a similarity higher than a preset threshold; The obstacle identification results in the first detection image corresponding to the tracking query and the obstacle identification results in the second detection image corresponding to the new obstacle query are correlated to obtain the tracking identification results; The obstacle cross-frame tracking device further includes a model training module, which is specifically used for: The model building unit is used to build a cross-frame obstacle tracking model to be trained. The sample training unit is used to collect preprocessed sample tracking queries, sample new obstacle queries, and sample tracking recognition results. The sample tracking queries and sample new obstacle queries are input into the obstacle cross-frame tracking model to be trained to obtain tracking and recognition results based on the sample tracking queries and sample new obstacle queries. The loss function calculation unit is used to calculate the loss function using the Hungarian algorithm based on the sample tracking query and the sample new obstacle query, wherein the sample new obstacle query contains the truth value corresponding to the sample tracking query; A loss function determination unit is used to determine whether the loss function value is less than a preset threshold. The model optimization unit is used to optimize the obstacle cross-frame tracking model to be trained based on the loss function value, so as to obtain the obstacle cross-frame tracking model.

7. An obstacle cross-frame tracking device, characterized in that, The obstacle cross-frame tracking device includes: a memory and at least one processor, wherein the memory stores requests, and the memory and the at least one processor are interconnected via a line; The at least one processor invokes the request in the memory to cause the obstacle cross-frame tracking device to perform the various steps of the obstacle cross-frame tracking method as described in any one of claims 1-5.

8. A computer-readable storage medium storing a computer program thereon, characterized in that, When the computer program is executed by the processor, it implements the various steps of the obstacle cross-frame tracking method as described in any one of claims 1-5.

Images