Identification and tracking network training method, tracking method, system, device and medium

Abstract

The present application discloses a recognition and tracking network training method, recognition and tracking method, system, device and medium, the method comprising: obtaining the current frame image, the frame image before the current frame and the frame image after the current frame of the visual sensor; inputting the target frame image into the target detection network to obtain the target frame image corresponding to the target frame image; performing a first processing on the current frame image and the historical frame image to obtain the image of the current frame image and the historical frame image; inputting the image of the current frame image and the historical frame image into the twin neural network to obtain the first feature matrix and the second feature matrix; and obtaining the fusion matrix; inputting the fusion matrix into the reinforcement learning model to obtain the predicted image of the frame after the current frame; determining the loss function according to the predicted image and the target frame image; adjusting the parameters of the twin neural sub-network and the reinforcement learning model until the loss function is minimized, and obtaining a trained recognition and tracking network. The present application can be widely used in the field of artificial intelligence technology.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a recognition and tracking network training method, a tracking method, a system, a device and a storage medium. Background Art

[0002] With the rapid development of science and technology today, information processing and target recognition technology has gradually become the core of various industries. Especially in security-sensitive fields such as military and civilian security, high-precision target recognition and tracking technology is crucial.

[0003] Since the birth of radar and satellite technology, both the military and civilian fields have been working hard to optimize the accuracy of target identification and tracking. Although traditional technologies have been proven in multiple application scenarios, they still show obvious limitations in certain situations, such as tracking high-speed targets or in environments with high environmental complexity. In military applications, since the enemy may use highly complex camouflage or stealth technology, traditional methods still have the defects of low detection accuracy and untimely tracking in target detection and tracking methods. Therefore, there are still technical problems that need to be solved in related technologies. Summary of the invention

[0004] The purpose of this application is to solve one of the technical problems existing in the prior art to at least a certain extent.

[0005] To this end, an object of an embodiment of the present application is to provide a recognition and tracking network training method, a tracking method, a system, a device and a storage medium, which can improve the recognition accuracy and the timeliness of tracking.

[0006] In order to achieve the above technical objectives, the technical solutions adopted by the embodiments of the present application include: a recognition and tracking network training method, the network includes a first twin neural sub-network, a second twin neural sub-network and a reinforcement learning model, the training method includes: obtaining continuous frame image data of a visual sensor; the continuous frame image data includes a current frame image, a historical frame image of a frame before the current frame, and a target frame image of a frame after the current frame; inputting the target frame image into a target detection network to obtain a target frame RGB image corresponding to the target frame image; performing a first processing on the current frame image and the historical frame image to obtain a first RGB image of the current frame image and a second RGB image of the historical frame image; the first RG The B image is input into the first twin neural sub-network to obtain a first feature matrix, and the second RGB image is input into the second twin neural network to obtain a second feature matrix; a fusion matrix is ​​obtained based on the second feature matrix and the first feature matrix; the fusion matrix is ​​input into the reinforcement learning model to obtain a predicted RGB image of the frame after the current frame; a loss function is determined based on the predicted RGB image and the target frame RGB image; the parameters of the first twin neural sub-network, the second twin neural sub-network and the reinforcement learning model are adjusted until the loss function is minimized to obtain a trained intelligent recognition and tracking network.

[0007] In addition, the method for identifying and tracking network training according to the above embodiment of the present invention may also have the following additional technical features:

[0008] Further, in an embodiment of the present application, the predicted RGB image includes a first target recognition rectangular frame and the target frame RGB image includes a second target recognition rectangular frame, and the step of determining the loss function according to the predicted RGB image and the target frame RGB image specifically includes:

[0009] Extracting an overlapping area of ​​an overlapping portion between the second target recognition rectangular frame and the first target recognition rectangular frame;

[0010] Determine the ratio of the overlap area to the total area of ​​the second target recognition rectangular frame and the first target recognition rectangular frame as an overlap ratio;

[0011] The difference between the constant 1 and the overlap ratio is used as the loss function.

[0012] Further, in the embodiment of the present application, the step of determining the ratio of the overlap area to the total area of ​​the second target recognition rectangular frame and the first target recognition rectangular frame as the overlap ratio specifically includes:

[0013] Calculating the total area of ​​the second target recognition rectangular frame and the first target recognition rectangular frame;

[0014] A quotient operation is performed on the overlap area and the total area to obtain a first quotient, and the first quotient is used as the overlap ratio.

[0015] Further, in the embodiment of the present application, the step of performing a first processing on the current frame image and the historical frame image to obtain a first RGB image of the current frame image and a second RGB image of the historical frame image specifically includes:

[0016] The current frame image and the historical frame image are input into a target detection network to obtain a first image corresponding to the current frame image and a second image corresponding to the historical frame image;

[0017] The first image and the second image are spatially transformed and cropped to obtain a first RGB image corresponding to the current frame image and a second RGB image corresponding to the historical frame image, wherein the first RGB image and the second RGB image are both images with a resolution of 64x64.

[0018] Further, in the embodiment of the present application, the step of performing a quotient operation on the overlap area and the total area to obtain a first quotient, and using the first quotient as the overlap ratio specifically includes: extracting the overlap area and the total area; inputting the overlap area and the total area into a ratio calculation formula to obtain a first quotient, wherein the ratio calculation formula is:

[0019] IoU(A,B)=Area(A∩B) / Area(A∪B)

[0020] Among them, Area(A∩B) is the overlapping area, Area(A∪B) is the total area, and IoU(A,B) is the first quotient.

[0021] Further, in the embodiment of the present application, the step of using the difference between the constant 1 and the overlap ratio as the loss function specifically includes: inputting the overlap ratio into a loss function determination formula to obtain a loss function;

[0022] The loss function determination formula is:

[0023] IoU Loss = 1-IoU(A,B)

[0024] Among them, IoU Loss is the loss function, and IoU(A,B) is the overlap ratio.

[0025] On the other hand, an embodiment of the present application also provides an intelligent recognition and tracking method, which includes: acquiring at least two consecutive frames of image data from a visual sensor; inputting the image data of the two consecutive frames into the trained intelligent recognition and tracking network described above, and determining the target image data of the next frame after the two consecutive frames of image data.

[0026] On the other hand, an embodiment of the present application further provides a recognition and tracking network training system, wherein the network includes a twin neural sub-network and a reinforcement learning model, and the system includes:

[0027] An acquisition unit, used to acquire continuous frame image data of a visual sensor; the continuous frame image data includes a current frame image, a historical frame image of a frame before the current frame, and a target frame image of a frame after the current frame;

[0028] A first processing unit is used to input the target frame image into a target detection network to obtain a target frame RGB image corresponding to the target frame image;

[0029] A second processing unit, configured to perform a first processing on the current frame image and the historical frame image to obtain a first RGB image of the current frame image and a second RGB image of the historical frame image;

[0030] A third processing unit is used to input the first RGB image into the twin neural sub-network to obtain a first feature matrix, and input the second RGB image into the twin neural network to obtain a second feature matrix;

[0031] a fourth processing unit, configured to obtain a fusion matrix according to the second feature matrix and the first feature matrix;

[0032] A fifth processing unit, configured to input the fusion matrix into a reinforcement learning model to obtain a predicted RGB image of a frame after the current frame;

[0033] A sixth processing unit, configured to determine a loss function according to the predicted RGB image and the target frame RGB image;

[0034] The seventh processing unit is used to adjust the parameters of the first twin neural sub-network, the second twin neural sub-network and the reinforcement learning model until the loss function is minimized to obtain a trained intelligent recognition and tracking network.

[0035] On the other hand, the present application also provides a recognition and tracking network training device, comprising:

[0036] at least one processor;

[0037] at least one memory for storing at least one program;

[0038] When the at least one program is executed by the at least one processor, the at least one processor implements the recognition and tracking network training method as described above.

[0039] In addition, the present application also provides a storage medium, in which processor-executable instructions are stored. When the processor-executable instructions are executed by the processor, they are used to execute a recognition and tracking network training method as described above.

[0040] The advantages and benefits of the present application will be partially given in the following description, and partially become apparent from the following description, or be understood through the practice of the present application:

[0041] The present application can obtain the continuous frame image data of the visual sensor including the current frame image, the historical frame image of the previous frame of the current frame, and the target frame image of the next frame of the current frame; input the target frame image into the target detection network to obtain the target frame RGB image corresponding to the target frame image; perform the first processing on the current frame image and the historical frame image to obtain the first RGB image of the current frame image and the second RGB image of the historical frame image; input the first RGB image into the twin neural network to obtain the first feature matrix, and input the second RGB image into the twin neural network to obtain the second feature matrix; obtain the fusion matrix according to the second feature matrix and the first feature matrix; input the fusion matrix into the reinforcement learning model to obtain the predicted RGB image of the next frame after the current frame; determine the loss function according to the predicted RGB image and the target frame RGB image; adjust the parameters of the twin neural network and the reinforcement learning model until the loss function is minimized to obtain a trained intelligent recognition and tracking network. The present application can use the continuous frame image data as a training sample to train the recognition and tracking network, and obtain a training model that can be used to predict the next frame image data of the continuous frame image, which has better detection progress and better tracking timeliness than the existing traditional target detection and tracking models or methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 A schematic diagram of the steps of a recognition and tracking network training method in a specific embodiment of the present invention;

[0043] Figure 2 A schematic diagram of another identification and tracking network training method in a specific embodiment of the present invention;

[0044] Figure 3 A schematic diagram of target detection and tracking inference model training in a specific embodiment of the present invention;

[0045] Figure 4 A schematic diagram of model reasoning of a target detection and tracking reasoning model in a specific embodiment of the present invention;

[0046] Figure 5 It is a structural schematic diagram of a recognition and tracking network training system in a specific embodiment of the present invention;

[0047] Figure 6 It is a structural schematic diagram of a recognition and tracking network training device in a specific embodiment of the present invention. DETAILED DESCRIPTION

[0048] The embodiments of the present invention are described in detail below with reference to the accompanying drawings to illustrate the principles and processes of the identification and tracking network training method, system, device and storage medium in the embodiments of the present invention.

[0049] Reference Figure 1 The present invention provides a recognition and tracking network training method, the network may include a first twin neural sub-network, a second twin neural sub-network and a reinforcement learning model, and the training method may include but is not limited to steps S101-S108.

[0050] S101, acquiring continuous frame image data of a visual sensor; the continuous frame image data includes a current frame image, a historical frame image of a frame before the current frame, and a target frame image of a frame after the current frame.

[0051] It is understandable that the continuous frame image data may include the current frame image, the image of the frame after the current frame, and the image of the frame before the current frame. The current frame may be the second frame in the continuous frame image arranged in time sequence, or the frame image after the second frame. For example, if the continuous frame image data has a total of 10 frames, the current frame may be any frame starting from the second frame.

[0052] In some feasible embodiments of the present application, the processor can establish a wired or wireless connection with the visual sensor. Through data communication between the processor and the acquisition device, the processor can acquire continuous frame image data of the visual sensor; wherein the continuous frame image data may include a current frame image, a historical frame image of a frame before the current frame, and a target frame image of a frame after the current frame.

[0053] It should be noted that the visual sensor can be one or more combinations of visible light devices, infrared devices or ultraviolet devices. The above-mentioned limited connection methods may include the connection between the mobile device and the host computer, and may also include the connection between the host computer and other wired connections between devices and the host computer that are currently known or will be developed in the future; and the above-mentioned wireless connection methods may include but are not limited to 3G / 4G / 5G connection, WiFi connection, Bluetooth connection, WiMAX connection, Zigbee connection, UWB (Ultra Wide Band) connection and other wireless connection methods that are currently known or will be developed in the future.

[0054] S102, input the target frame image into the target detection network to obtain a target frame RG B image corresponding to the target frame image.

[0055] It is understandable that the target frame image can be the image of the frame after the current frame, and the target detection network can be a trained target detection network, and the target detection network can use the pre-trained Detection Transformer (DETR) as the target detection algorithm to obtain the initial rectangular frame of the target. DETR, as the core target detection technology in the algorithm, represents an innovative direction in the field of target detection. Unlike traditional target detection methods, DETR abandons the concepts of prior boxes and non-maximum suppression (NMS), and can directly output the category and location information of the target from the original image, realizing end-to-end training.

[0056] In some feasible embodiments of the present application, the processor may input the target frame image into the target detection network to obtain a target frame RGB image corresponding to the target frame image.

[0057] S103 , performing a first process on the current frame image and the historical frame image to obtain a first RGB image of the current frame image and a second RGB image of the historical frame image.

[0058] It can be understood that the first processing can include rectangular box prediction processing of target recognition in the current frame image and spatial transformation processing and cropping processing of the rectangular box of target recognition. The first RGB image is the image of the current frame image after the first processing, and the second RGB image is the RGB image of the historical frame image after the first processing.

[0059] In some feasible embodiments of the present application, the processor can perform rectangular frame prediction processing for target recognition and spatial transformation processing and cropping processing on the rectangular frame of target recognition on the current frame image and the historical frame image to obtain a first RGB image of the current frame image and a second RGB image of the historical frame image.

[0060] S104. Input the first RGB image into the first twin neural sub-network to obtain a first feature matrix, and input the second RGB image into the second twin neural network to obtain a second feature matrix.

[0061] It is understandable that the first twin neural sub-network and the second twin neural network can be two completely identical networks. The first feature matrix and the second feature matrix can be two two-dimensional matrices of the same size with different elements in the matrix.

[0062] In some feasible embodiments of the present application, the processor can input the first RGB image into the first twin neural sub-network to obtain a first feature matrix, and input the second RGB image into the second twin neural network to obtain a second feature matrix. The first feature matrix and the second feature matrix can be used as input for subsequent feature fusion.

[0063] S105. Obtain a fusion matrix according to the second feature matrix and the first feature matrix.

[0064] In some feasible embodiments of the present application, the processor may fuse the second feature matrix and the first feature matrix to obtain a fused matrix.

[0065] It should be noted that the fusion matrix can be used as input to the reinforcement learning model.

[0066] S106: Input the fusion matrix into the reinforcement learning model to obtain a predicted RGB image of the frame after the current frame.

[0067] It can be understood that the predicted RGB image of the frame after the current frame is different from the target frame RGB image.

[0068] In some feasible embodiments of the present application, the processor may input the fusion matrix into the reinforcement learning model to obtain a predicted RGB image of a frame after the current frame.

[0069] S107. Determine a loss function according to the predicted RGB image and the target frame RGB image.

[0070] In some feasible embodiments of the present application, the processor may have a built-in calculation algorithm for the loss function, and then determine the loss function based on the predicted RGB image and the target frame RGB image.

[0071] S108. Adjust the parameters of the first twin neural sub-network, the second twin neural sub-network and the reinforcement learning model until the loss function is minimized to obtain a trained intelligent recognition and tracking network.

[0072] In some feasible embodiments of the present application, the processor can adjust the parameters of the first twin neural sub-network, the second twin neural sub-network and the reinforcement learning model until the loss function is minimized to obtain a trained intelligent recognition and tracking network.

[0073] Furthermore, in some feasible embodiments of the present application, the predicted RGB image includes a first target recognition rectangular box and the target frame RGB image includes a second target recognition rectangular box. The step of determining the loss function based on the predicted RGB image and the target frame RGB image may specifically include but is not limited to steps S201-S203.

[0074] S201, extracting the overlapping area of ​​the overlapping portion between the second target recognition rectangular frame and the first target recognition rectangular frame;

[0075] S202, determining that the ratio of the overlapped area to the total area of ​​the second target recognition rectangular frame and the first target recognition rectangular frame is an overlapped ratio;

[0076] S203, taking the difference between the constant 1 and the overlap ratio as the loss function.

[0077] Further, in some feasible embodiments of the present application, the step of determining the ratio of the overlapping area to the total area of ​​the second target recognition rectangular frame and the first target recognition rectangular frame as the overlapping ratio may specifically include but is not limited to steps S301-S302.

[0078] S301, calculating the total area of ​​the second target recognition rectangular frame and the first target recognition rectangular frame;

[0079] S302, performing a quotient operation on the overlap area and the total area to obtain a first quotient, and taking the first quotient as the overlap ratio.

[0080] Furthermore, in some feasible embodiments of the present application, the step of performing a first processing on the current frame image and the historical frame image to obtain a first RGB image of the current frame image and a second RGB image of the historical frame image may specifically include but is not limited to steps S401-S402.

[0081] The current frame image and the historical frame image are input into the target detection network to obtain a first image corresponding to the current frame image and a second image corresponding to the historical frame image;

[0082] The first image and the second image are spatially transformed and cropped to obtain a first RGB image corresponding to the current frame image and a second RGB image corresponding to the historical frame image. Both the first RGB image and the second RGB image are images with a resolution of 64x64.

[0083] Further, in some feasible embodiments of the present application, the step of performing a quotient operation on the overlap area and the total area to obtain a first quotient, and using the first quotient as the overlap ratio specifically includes:

[0084] Extract the overlapping area and total area;

[0085] Input the overlap area and the total area into the ratio calculation formula to obtain the first quotient, where the ratio calculation formula is:

[0086] IoU(A,B)=Area(A∩B) / Area(A∪B)

[0087] Among them, Area(A∩B) is the overlapping area, Area(A∪B) is the total area, and IoU(A,B) is the first quotient.

[0088] Further, in some feasible embodiments of the present application, the step of using the difference between the constant 1 and the overlap ratio as the loss function specifically includes:

[0089] Input the overlap ratio into the loss function determination formula to obtain the loss function;

[0090] The loss function is determined as follows:

[0091] IoU Loss = 1-IoU(A,B)

[0092] Among them, IoU Loss is the loss function, and IoU(A,B) is the overlap ratio.

[0093] In addition, refer to Figure 2 The present application also provides an intelligent identification and tracking method. The method may include steps S111 to S112.

[0094] S111, acquiring at least two consecutive frames of image data from a visual sensor;

[0095] S112, inputting the image data of two consecutive frames into the intelligent recognition and tracking network trained in any of the preceding items, and determining the target image data of the next frame after the two consecutive frames of image data.

[0096] In some feasible embodiments of the present application, after the intelligent recognition and tracking network is trained through the above steps, the processor can input the latest image data of at least two consecutive frames of the visual sensor, where the latest image data refers to the image data of the current frame and the image data of the frame before the current frame, into the trained intelligent recognition and tracking network, and finally obtain the target image data of the frame after the latest frame of image data.

[0097] The specific implementation principle of this application is described below with reference to the accompanying drawings:

[0098] In this embodiment, the specific implementation method of the present application may include steps S1 to S4.

[0099] S1. Collect continuous frame image data from various visual sensors (such as visible light, infrared, ultraviolet, etc.);

[0100] S2, use target detection algorithms to accurately locate and identify targets;

[0101] S3, combined with reinforcement learning methods, information fusion of historical frames and current frames is performed to achieve accurate target tracking;

[0102] S4. Use the timeline-level target detection and tracking inference model for comprehensive analysis.

[0103] In various application scenarios, visual sensors collect different types of image data, such as visible light, infrared, ultraviolet, etc. The algorithm of the present invention can identify and perform special training on these different types of data, thereby enhancing its performance under various lighting conditions, improving its reasoning ability in multiple scenarios, and ensuring efficient and accurate target detection and tracking even in light-limited environments.

[0104] Furthermore, the specific method of step S2 is: using a pre-trained Detection Transformer (DETR) as the target detection algorithm to obtain the initial rectangular box of the target. DETR, as the core target detection technology in the algorithm, can represent an innovative direction in the field of target detection. Unlike traditional target detection methods, DETR can abandon the concepts of prior boxes and non-maximum suppression (NMS), can directly output the category and location information of the target from the original image, and can achieve end-to-end training.

[0105] The basic building block of DETR is the Transformer model, which was originally designed for natural language processing and has performed well in models such as BERT and GPT. However, in the context of object detection, the Transformer's self-attention mechanism and position encoding can enable it to capture long-range dependencies in images, thereby analyzing image content more deeply.

[0106] In practical applications, the original image is first fed into a convolutional neural network (such as ResNet) or a Transformer network with similar functions (such as Swin Transformer) to obtain feature maps, which are then fed into the Transformer part of DETR to further extract context-rich feature information. Among them, the decoder of DETR can use a fixed number of queries to directly predict the bounding boxes and categories of all objects in the image without complex post-processing steps, which can ensure that the output detection results are globally optimal.

[0107] In practice, in order to make DETR better adapt to specific tasks or datasets, model fine-tuning is usually required. First, load the pre-trained DETR model, which is usually trained on a large-scale general dataset such as COCO. Next, further train the model on a specific task dataset to adapt its parameters to the new data distribution. This process is called model fine-tuning. This pre-training plus fine-tuning strategy can make full use of the general features in the pre-trained model and provide customized optimization for specific tasks on this basis.

[0108] Although DETR can greatly simplify the target detection process, some simple post-processing, such as threshold screening, may still be required to ensure the quality and accuracy of detection, depending on the specific application scenario. In general, DETR can not only provide an efficient and accurate target detection method, but also simplify multiple complex steps in traditional methods, which can meet the requirements of fast and accurate detection in the algorithm.

[0109] Furthermore, the specific method of step S3 is as follows: after successfully determining the target and obtaining its initial rectangular frame using DETR, the in-depth step of target tracking is then carried out. In order to ensure that the target is continuously and accurately tracked in continuous video frames, the algorithm of the present application can adopt a deep fusion strategy for historical frame and current frame information, and optimize it in combination with a reinforcement learning algorithm.

[0110] First, the initial rectangular box obtained from the historical frame and the current frame by DETR needs to be spatially transformed and cropped to ensure the continuity and consistency of the target area on the timeline. The processed image data is embedded and fused through a twin convolutional neural network to further refine the target features. The fused data is then input into a reinforcement learning network designed for target tracking, which aims to effectively utilize historical information to optimize the prediction of the target's position in consecutive frames. In order to evaluate the accuracy of the prediction, the network will receive corresponding rewards or penalties after each decision based on the degree of match between the predicted rectangular box and the actual target position. The evaluation of the degree of matching mainly depends on the calculation of IoU (Intersection Over Union), and its formula is:

[0111] IoU(A,B)=Area(A∩B) / Area(A∪B)

[0112] In the above formula, A represents the box predicted by reinforcement learning, and B is the real box detected by DETR. IoU can describe the ratio of the overlapping part of two boxes to their total area, and its value is between 0 and 1, where 0 means no overlap at all and 1 represents a complete match. In order to more intuitively reflect the prediction error, the algorithm of the present invention can use the loss function IoULoss, which is defined as:

[0113] IoU Loss = 1-IoU(A,B)

[0114] In contrast to IoU, the smaller the IoU Loss, the more accurate the prediction. During model training, optimizing this loss function can help the model predict the location and shape of the target more accurately, thereby improving the accuracy of target detection and tracking.

[0115] The reinforcement learning network in the present invention consists of three sub-networks: a policy network, a scoring network, and an action network. The policy network is responsible for generating a behavior strategy based on the current state of the environment, and determining which behavior is most likely to successfully track the target in the current state. The scoring network evaluates the expected rewards of these behaviors and provides a score for each possible behavior, which helps the policy network make better decisions. The action network is responsible for executing the behavior selected by the policy network, such as adjusting the size of the target box or changing its position to track the target more accurately. These three sub-networks work together to ensure accurate tracking decisions in various scenarios.

[0116] Thanks to continuous learning and real-time feedback, the network is able to gradually improve its tracking strategy. For example, when the target moves quickly, the policy network may tend to choose a more aggressive action to keep up with the target's speed; when the target is partially occluded, it may use the information provided by the scoring network to choose a more conservative strategy to avoid misjudgment. In this way, high tracking accuracy can be maintained regardless of the situation.

[0117] In addition to accurately tracking the target in the current frame, the reinforcement learning network finally outputs a prediction of the target's potential position in the future frame. This prediction mechanism is based on the target's motion trends in the past and current frames and the target's dynamic patterns learned from multiple frames of data. When the network processes the current frame, it generates a prediction model that takes into account the target's velocity, acceleration, and direction of motion in the past frames, thereby providing a preliminary estimate of the target's position for the future frame.

[0118] This prediction capability can bring significant advantages to target tracking, especially in scenes where the target moves fast or changes suddenly. The prediction mechanism can provide a reference baseline for actual tracking, ensuring that the tracking algorithm does not easily lose the target, while also improving the system's reaction speed. This can further ensure stable and accurate tracking of the target in a continuous video stream, and DETR's precise target detection can provide a reliable benchmark for this process.

[0119] Further, refer to Figure 3 , the method of step S4 is:

[0120] On the model training side:

[0121] After the raw image data collected by the video sensor is processed in step S2, a preliminary rectangular frame of the target can be obtained. This data is further spatially transformed and cropped to form two 64x64 resolution RGB three-channel images of the current frame (t) and the historical frame (t-1). Both images are subjected to feature extraction by the same twin neural network, followed by feature fusion through the concatenation layer and input into the reinforcement learning network. In step S3, the reinforcement learning network is trained to predict future frame images (t+

[0122] Next, the predicted target box of the future frame is compared with the real frame target box obtained by the DETR target detection algorithm to calculate the error (IOU Loss). This error is then applied to the neural network through the back propagation mechanism to adjust the weight parameters in the network. In the method of the present invention, the target box of the actual frame is directly obtained through DETR target detection, which can completely avoid the need for manual annotation of data.

[0123] On the model inference side:

[0124] Reference Figure 4 In the inference phase, the image captured by the visual sensor is first processed by step S2 to determine the preliminary rectangular frame of the target. After precise spatial transformation and cropping, the current frame and its previous frame are sent to the twin convolutional neural network in parallel to extract their feature information. After feature fusion, the image data is sent to the fully connected neural network that has been pre-trained by reinforcement learning. The network outputs predictions for future frames, thereby achieving rapid and accurate detection and tracking of high-speed moving targets.

[0125] In summary, the beneficial effects of the present application are as follows: the timeline-level intelligent recognition and tracking algorithm proposed in the present invention can integrate DETR and reinforcement learning, and can provide new ideas for target detection and tracking of continuous frame images. This algorithm is capable of processing data collected from a variety of visual sensors, and has been carefully optimized to ensure that it can exhibit excellent performance in various lighting conditions (including low visibility, low pixel or low frame rate environments), ensuring the accuracy and efficiency of target detection and tracking. Thanks to its efficient design, the algorithm has extremely low latency, which is particularly suitable for scenarios with strong demand for real-time response speed. At the same time, using DETR, the algorithm of the present application can abandon the reliance on a large amount of manually annotated data, thereby simplifying the training process of the model. In addition, its unique reinforcement learning structure can give it an excellent ability to predict the position of the target in the future frame, which is particularly valuable for tracking high-speed moving targets. Overall, the present application can open up a new path combining accuracy, real-time and adaptability in the field of target recognition and tracking technology, and has broad application prospects.

[0126] In addition, refer to Figure 5 ,and Figure 1 Corresponding to the method, an embodiment of the present application also provides a recognition and tracking network training system. The system may include an acquisition unit 1001, a first processing unit 1002, a second processing unit 1003, a third processing unit 1004, a fourth processing unit 1005, a fifth processing unit 1006, a sixth processing unit 1007 and a seventh processing unit 1008. Among them, the acquisition unit 1001 can be used to acquire continuous frame image data of a visual sensor; the continuous frame image data includes a current frame image, a historical frame image of a frame before the current frame, and a target frame image of a frame after the current frame; the first processing unit 1002 can be used to input the target frame image into a target detection network to obtain a target frame RGB image corresponding to the target frame image; the second processing unit 1003 can be used to perform a first processing on the current frame image and the historical frame image to obtain a first RGB image of the current frame image and a second RGB image of the historical frame image. GB image; the third processing unit 1004 can be used to input the first RGB image into the first twin neural sub-network to obtain a first feature matrix, and input the second RGB image into the second twin neural network to obtain a second feature matrix; the fourth processing unit 1005 can be used to obtain a fusion matrix according to the second feature matrix and the first feature matrix; the fifth processing unit 1006 can be used to input the fusion matrix into the reinforcement learning model to obtain a predicted RGB image of the frame after the current frame; the sixth processing unit 1007 can be used to determine the loss function according to the predicted RGB image and the target frame RGB image; the seventh processing unit 1008 can be used to adjust the parameters of the first twin neural sub-network, the second twin neural sub-network and the reinforcement learning model until the loss function is minimized to obtain a trained intelligent recognition and tracking network.

[0127] It should be noted that the acquisition unit 1001 may be any integrated circuit module or microprocessor module obtained by integrating a chip having a processing function and its peripheral circuits through existing integration technology. The first processing unit 1002 may also be any integrated circuit module or microprocessor module obtained by integrating a chip having a processing function and its peripheral circuits through existing integration technology. The first processing unit 1002 may also include one or more memories.

[0128] It should be noted that the contents of the above-mentioned recognition and tracking network training method embodiment are all applicable to the present recognition and tracking network training system embodiment. The functions specifically implemented by the present recognition and tracking network training system embodiment are the same as those of the above-mentioned recognition and tracking network training method embodiment, and the beneficial effects achieved are also the same as those achieved by the above-mentioned recognition and tracking network training method embodiment.

[0129] and Figure 1 Corresponding to the method, the present application embodiment also provides a recognition tracking network training device, the specific structure of which can be referred to Figure 6 ,include:

[0130] at least one processor 1011;

[0131] At least one memory 1012, used to store at least one program;

[0132] When the at least one program is executed by the at least one processor, the at least one processor implements the identification and tracking network training method.

[0133] It should be noted that the contents of the above method embodiments are all applicable to the present device embodiments, the functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0134] and Figure 1 Corresponding to the method, an embodiment of the present application further provides a storage medium, in which processor-executable instructions are stored, and the processor-executable instructions are used to execute the identification and tracking network training method when executed by the processor.

[0135] It should be noted that the contents of the above-mentioned recognition and tracking network training method embodiment are all applicable to the present storage medium embodiment, and the functions specifically implemented by the present storage medium embodiment are the same as those of the above-mentioned recognition and tracking network training method embodiment, and the beneficial effects achieved are also the same as those achieved by the above-mentioned recognition and tracking network training method embodiment.

[0136] In some optional embodiments, the function / operation mentioned in the block diagram may not occur in the order mentioned in the operation diagram. For example, depending on the function / operation involved, the two boxes shown in succession can actually be executed substantially simultaneously or the boxes can sometimes be executed in reverse order. In addition, the embodiment presented and described in the flow chart of the application is provided by way of example, for the purpose of providing a more comprehensive understanding of technology. The disclosed method is not limited to the operation and logic flow presented herein. Optional embodiments are expected, wherein the order of various operations is changed and the sub-operation described as a part of a larger operation is performed independently.

[0137] In addition, although the present application is described in the context of functional modules, it should be understood that, unless otherwise specified, one or more of the functions and / or features can be integrated into a single physical device and / or software module, or one or more functions and / or features can be implemented in a separate physical device or software module. It is also understood that a detailed discussion of the actual implementation of each module is unnecessary for understanding the present application. More specifically, in view of the properties, functions, and internal relationships of the various functional modules in the device disclosed herein, the actual implementation of the module will be understood within the conventional techniques of the engineer. Therefore, those skilled in the art can implement the present application set forth in the claims without excessive experimentation using ordinary techniques. It is also understood that the specific concepts disclosed are merely illustrative and are not intended to limit the scope of the present application, which is determined by the full scope of the attached claims and their equivalents.

[0138] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several programs to enable a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0139] The logic and / or steps represented in the flowchart or otherwise described herein, for example, may be considered as an ordered list of executable programs for implementing the logical functions, and may be embodied in any computer-readable medium for use by a program execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch and execute a program from a program execution system, device or apparatus), or in conjunction with such program execution systems, devices or apparatuses. For purposes of this specification, a "computer-readable medium" may be any device that can contain, store, communicate, propagate or transmit a program for use by a program execution system, device or apparatus, or in conjunction with such program execution systems, devices or apparatuses.

[0140] More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk case (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be a paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering or, if necessary, processing in another suitable manner, and then stored in a computer memory.

[0141] It should be understood that the various parts of the present application can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable program execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0142] In the above description of this specification, the description with reference to the terms "one embodiment / example", "another embodiment / example" or "certain embodiments / examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0143] Although the embodiments of the present application have been shown and described, those skilled in the art will appreciate that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present application, and that the scope of the present application is defined by the claims and their equivalents.

[0144] The above is a specific description of the preferred implementation of the present application, but the present application is not limited to the described embodiments. Technical personnel familiar with the field may make various equivalent modifications or substitutions without violating the spirit of the present application. These equivalent modifications or substitutions are all included in the scope defined by the claims of the present application.

Claims

1. A recognition and tracking network training method, characterized in that: The network includes a first twin neural sub-network, a second twin neural sub-network and a reinforcement learning model, and the training method includes: Acquire continuous frame image data of the visual sensor; the continuous frame image data includes a current frame image, a historical frame image of a frame before the current frame, and a target frame image of a frame after the current frame; Inputting the target frame image into a target detection network to obtain a target frame RGB image corresponding to the target frame image; Performing a first processing on the current frame image and the historical frame image to obtain a first RGB image of the current frame image and a second RGB image of the historical frame image; Input the first RGB image into the first twin neural sub-network to obtain a first feature matrix, and input the second RGB image into the second twin neural network to obtain a second feature matrix; Obtaining a fusion matrix according to the second feature matrix and the first feature matrix; Input the fusion matrix into the reinforcement learning model to obtain a predicted RGB image of a frame after the current frame; Determining a loss function according to the predicted RGB image and the target frame RGB image; Adjusting the parameters of the first twin neural sub-network, the second twin neural sub-network, and the reinforcement learning model until the loss function is minimized, thereby obtaining a trained intelligent recognition and tracking network; The step of performing a first processing on the current frame image and the historical frame image to obtain a first RGB image of the current frame image and a second RGB image of the historical frame image specifically includes: The current frame image and the historical frame image are input into a target detection network to obtain a first image corresponding to the current frame image and a second image corresponding to the historical frame image; The first image and the second image are spatially transformed and cropped to obtain a first RGB image corresponding to the current frame image and a second RGB image corresponding to the historical frame image, wherein the first RGB image and the second RGB image are both images with a resolution of 64x64.

2. According to claim 1, a recognition and tracking network training method is characterized in that: The predicted RGB image includes a first target recognition rectangular frame and the target frame RGB image includes a second target recognition rectangular frame, and the step of determining the loss function according to the predicted RGB image and the target frame RGB image specifically includes: Extracting an overlapping area of ​​an overlapping portion between the second target recognition rectangular frame and the first target recognition rectangular frame; Determine the ratio of the overlap area to the total area of ​​the second target recognition rectangular frame and the first target recognition rectangular frame as an overlap ratio; The difference between the constant 1 and the overlap ratio is used as the loss function.

3. According to claim 2, a recognition and tracking network training method is characterized in that: The step of determining that the ratio of the overlap area to the total area of ​​the second target recognition rectangular frame and the first target recognition rectangular frame is the overlap ratio specifically includes: Calculating the total area of ​​the second target recognition rectangular frame and the first target recognition rectangular frame; A quotient operation is performed on the overlap area and the total area to obtain a first quotient, and the first quotient is used as the overlap ratio.

4. According to claim 3, a recognition and tracking network training method is characterized in that: The step of performing a quotient operation on the overlap area and the total area to obtain a first quotient, and using the first quotient as the overlap ratio specifically includes: extracting the overlapping area and the total area; The overlap area and the total area are input into a ratio calculation formula to obtain a first quotient, wherein the ratio calculation formula is: IoU(A,B)=Area(A∩B) / Area(A∪B) Among them, Area(A∩B) is the overlapping area, Area(A∪B) is the total area, and IoU(A,B) is the first quotient.

5. According to claim 2, a recognition and tracking network training method is characterized in that: The step of using the difference between the constant 1 and the overlap ratio as the loss function specifically includes: Inputting the overlap ratio into a loss function determination formula to obtain a loss function; The loss function determination formula is: IoU Loss = 1-IoU(A,B) Among them, IoU Loss is the loss function, and IoU(A,B) is the overlap ratio.

6. An intelligent identification and tracking method, characterized in that: include: Acquire at least two consecutive frames of image data from the visual sensor; The image data of two consecutive frames are input into the trained intelligent recognition and tracking network described in any one of claims 1 to 5 to determine the target image data of the next frame of the two consecutive frames of image data.

7. A recognition and tracking network training system, characterized in that: The recognition and tracking network includes a first twin neural sub-network, a second twin neural sub-network and a reinforcement learning model, and the training system includes: An acquisition unit, used to acquire continuous frame image data of a visual sensor; the continuous frame image data includes a current frame image, a historical frame image of a frame before the current frame, and a target frame image of a frame after the current frame; A first processing unit is used to input the target frame image into a target detection network to obtain a target frame RGB image corresponding to the target frame image; A second processing unit, configured to perform a first processing on the current frame image and the historical frame image to obtain a first RGB image of the current frame image and a second RGB image of the historical frame image; The step of performing a first processing on the current frame image and the historical frame image to obtain a first RGB image of the current frame image and a second RGB image of the historical frame image specifically includes: The current frame image and the historical frame image are input into a target detection network to obtain a first image corresponding to the current frame image and a second image corresponding to the historical frame image; Performing spatial transformation and cropping processing on the first image and the second image to obtain a first RGB image corresponding to the current frame image and a second RGB image corresponding to the historical frame image, wherein the first RGB image and the second RGB image are both images with a resolution of 64x64; A third processing unit, configured to input the first RGB image into the first twin neural sub-network to obtain a first feature matrix, and input the second RGB image into the second twin neural network to obtain a second feature matrix; a fourth processing unit, configured to obtain a fusion matrix according to the second feature matrix and the first feature matrix; A fifth processing unit, configured to input the fusion matrix into a reinforcement learning model to obtain a predicted RGB image of a frame after the current frame; A sixth processing unit, configured to determine a loss function according to the predicted RGB image and the target frame RGB image; The seventh processing unit is used to adjust the parameters of the first twin neural sub-network, the second twin neural sub-network and the reinforcement learning model until the loss function is minimized to obtain a trained intelligent recognition and tracking network.

8. A recognition and tracking network training device, characterized in that include: at least one processor; at least one memory for storing at least one program; When the at least one program is executed by the at least one processor, the at least one processor implements the recognition and tracking network training method as described in any one of claims 1 to 5.

9. A computer-readable storage medium storing instructions executable by a processor, characterized in that: The processor-executable instructions are used to execute a recognition and tracking network training method as described in any one of claims 1-5 when executed by the processor.

Images