Method and apparatus for training and testing an object detection network for detecting objects on images

Abstract

The present application provides a method of training an object detection network by utilizing an attention map. The method comprises the following steps: a) an on-device learning device inputs a training image to a feature extraction network, inputs an output of the feature extraction network to an attention network and a concatenation layer, and inputs an output of the attention network to the concatenation layer; b) the on-device learning device inputs an output of the concatenation layer to a region candidate network and an ROI pooling layer, inputs an output of the region candidate network to a binary converter and the ROI pooling layer, and inputs an output of the ROI pooling layer to a detection network, thereby outputting object detection data; and c) the on-device learning device trains at least one of the feature extraction network, the detection network, the region candidate network and the attention network by using back propagation with an object detection loss, a region candidate network loss and a cross-entropy loss.

Description

[0001] Related cross-references

[0002] This application claims priority to U.S. Patent Application No. 17 / 112,413, filed December 4, 2020, and U.S. Patent Application No. 63 / 024,018, filed May 13, 2020, the disclosures of which are incorporated herein by reference. Technical Field

[0003] This invention relates to a method for training and testing an object detection network, and more specifically, to a method for training and testing an object detection network, as well as an up-learning device and a testing device using the method, wherein the object detection network detects objects in an image by utilizing an attention map. Background Technology

[0004] Typically, autonomous vehicles achieve autonomous driving by analyzing the driving environment based on machine learning. They perform autonomous driving through a learning network that is pre-trained using various learning data corresponding to the driving environment.

[0005] However, pre-trained learning networks cannot cope with the various driving environments when a vehicle is in motion.

[0006] Another solution is to transmit the information collected by the device to a cloud server for analysis and then retransmit it to the device to train the learning network installed on the device to cope with various driving environments. However, this method is difficult to use when cloud communication is limited.

[0007] Therefore, in recent years, on-device learning methods have been proposed, which can cope with various driving environments and be trained under conditions where cloud communication is limited.

[0008] In other words, on-device learning is a method in which a device collects information on its own and uses that information to train a learning network without accessing a cloud server.

[0009] However, on-device learning struggles with positive / negative sampling due to the inability to utilize teacher networks capable of knowledge distillation, and end-to-end backpropagation is difficult due to the limited computing power of embedded systems in devices such as autonomous vehicles.

[0010] Therefore, a new technology is needed that can overcome the limitations of on-device learning and improve the performance of learning networks. Summary of the Invention

[0011] Technical issues

[0012] The purpose of this invention is to solve all the problems mentioned above.

[0013] Another objective of this invention is to improve the performance of object detection networks without performing additional training operations.

[0014] Another objective of this invention is to improve the performance of object detection networks that utilize attention maps to detect objects in images.

[0015] Another objective of this invention is to prevent catastrophic forgetting by utilizing a cross-distilled loss layer.

[0016] Technical solution

[0017] In order to achieve the above-mentioned objectives of the present invention and to realize the characteristic effects of the present invention described later, the characteristic structure of the present invention is as follows.

[0018] According to an embodiment of the present invention, a method for training an object detection network is provided, the object detection network detecting objects on an image using an attention map, comprising: a) an on-device learning device i) inputting a training image into a feature extraction network, such that the feature extraction network performs at least one convolution operation on the training image to output a training feature map; ii) inputting the training feature map into an attention network, such that the attention network outputs ii-1) a training attention map corresponding to the training object density of the training feature map, and ii-2) a training prediction map corresponding to the training feature map; iii) inputting the training feature map and the training attention map into a stitching layer, such that the stitching layer stitches the training feature map and the training attention map to output a training attention feature map; b) the on-device learning device i) inputting the training attention feature map into a region candidate network, such that the region candidate network outputs training candidate boxes corresponding to training objects on the training attention feature map; ii) inputting the training candidate boxes into a binary converter, such that the binary converter converts the candidate boxes into binary images; iii ... training candidate boxes into binary images; iii) inputting the training attention feature map into a region candidate network, such that the region candidate network outputs training candidate boxes corresponding to training objects on the training attention feature map; ii) inputting the training candidate boxes into a binary converter, The training attention feature map and the training candidate boxes are input into a ROI pooling layer, so that the ROI pooling layer performs pooling on the regions in the training attention feature map corresponding to each training candidate box, thereby generating a training pooled feature map; iv) the training pooled feature map is input into a detection network, so that the detection network performs a training operation on the training pooled feature map, thereby generating training object detection information corresponding to each training candidate box; and c) the learning device on the device performs at least one of the following processes: i) a process of training at least one of the detection network and the feature extraction network by backpropagation object loss, wherein the object loss is generated by referring to the training object detection information and object ground truth; ii) a process of training the region candidate network by backpropagation RPN loss, wherein the RPN loss is generated by referring to the training candidate boxes and RPN ground truth; iii) a process of training the attention network by backpropagation cross-entropy loss, wherein the cross-entropy loss is generated by referring to the training prediction map and the training binary map, wherein the training binary map is generated by binarizing the training candidate boxes.

[0019] In step a), the learning device on the device causes the attention network to perform a second convolution operation on the training feature map to generate a first sub-feature map for training; a 1×1 convolution operation is then performed on the first sub-feature map to generate a second sub-feature map for training, wherein the second sub-feature map for training has the same channels as the training feature map; and a sigmoid function is applied to the second sub-feature map for training to generate the attention map for training.

[0020] In step a), the learning device on the device causes the attention network to perform the second convolution operation on the training feature map to generate the first sub-feature map for training; the first sub-feature map for training is then subjected to the 1×1 convolution operation to generate the second sub-feature map for training, wherein the second sub-feature map for training has the same channels as the training feature map; the ReLU function is applied to the second sub-feature map for training to generate the second sub-feature map for training; the second sub-feature map for training is then subjected to the 1×1 convolution operation to generate the third sub-feature map for training with one channel; and the sigmoid function is applied to the third sub-feature map for training to generate the prediction map for training.

[0021] In step a), the learning device on the device causes the stitching layer to generate an intermediate attention map for training by performing element-wise multiplication of the training feature map and the training attention map, and to generate the training attention feature map by performing element-wise addition of the training feature map and the intermediate attention map for training.

[0022] In step b), the learning device on the device inputs the training candidate boxes to the binary converter, so that the binary converter generates an intermediate binary image for training by binarizing the training candidate boxes, and then adjusts it to the size of the training feature map to generate the training binary image.

[0023] According to another embodiment of the present invention, a method for training an object detection network is provided, the object detection network detecting objects on an image using an attention map, comprising: a) performing on a learning device using an on-device approach; I) inputting a training image into a feature extraction network, such that the feature extraction network performs at least one convolution operation on the training image to output a training feature map; inputting the training feature map into an attention network, such that the attention network outputs a training prediction map corresponding to the training feature map and a training attention map corresponding to the training object density of the training feature map; and inputting the training feature map and the training attention map into an attention network. The image is input to a stitching layer, which stitches the training feature map and the training attention map together to output a training attention feature map. II) The training attention feature map is input to a region candidate network, which outputs training candidate boxes corresponding to the training objects on the training attention feature map. The training candidate boxes are input to a binary converter, which converts the candidate boxes into binary images. The training attention feature map and the training candidate boxes are input to a ROI pooling layer, which pools the training attention feature map for each training object. The regions of the training candidate boxes are pooled to generate training pooled feature maps; the training pooled feature maps are input into the detection network to train the detection network on the training pooled feature maps to generate training object detection information corresponding to each training candidate box; III) at least one of the following processes is performed: training at least one of the detection network and the feature extraction network by backpropagation object loss, wherein the object loss is generated by referring to the training object detection information and object ground truth; training the region candidate network by backpropagation RPN loss, wherein the RPN loss... The cross-entropy loss is generated by referring to the training candidate boxes and the ground truth of the RPN; the attention network is trained by backpropagation cross-entropy loss, wherein the training cross-entropy loss is generated by referring to the training prediction map and the training binary map, and the training binary map is generated by binarizing the training candidate boxes; in this case, when acquiring a test video, the test device inputs the current image corresponding to the current frame in the test video into the feature extraction network, so that the feature extraction network performs at least one convolution operation on the current image, thereby outputting the current feature map and storing the current feature map in memory;b) The test device i) inputs a previous feature map of a previous image corresponding to a previous frame in the test video stored in the memory and the current feature map into the attention network, so that the attention network outputs an updated previous soft prediction map corresponding to the previous feature map, a current soft prediction map and a current prediction map corresponding to the current feature map, and a current attention map corresponding to the current object density in the current feature map; ii) stores the current soft prediction map in the memory; iii) inputs the current feature map and the current attention map into the stitching layer, so that the stitching layer stitches the current feature map and the current attention map to output a current attention feature map; iv) inputs the current attention feature map into the region candidate network, so that the region candidate network outputs a current candidate box corresponding to the current object on the current attention feature map; v) inputs the current training candidate box into the binary converter, so that the binary converter changes the current candidate box to the current object density. The process involves: vi) inputting the current attention feature map and the current candidate box into the ROI pooling layer, causing the ROI pooling layer to perform pooling on the regions in the current attention feature map corresponding to each current candidate box, thereby generating a current pooled feature map; vii) inputting the current pooled feature map into the detection network, causing the detection network to perform training operations on the current pooled feature map, thereby generating current object detection information corresponding to each current candidate box; and c) the test device utilizing i) the current cross-entropy loss, which is generated by referring to the current prediction map and the current binary map, the current binary map being generated by binarizing the current candidate box; and ii) the distillation loss, which is generated by referring to the updated previous soft prediction map and the current soft prediction map, the previous image corresponding to the previous frame in the test video stored in the memory; and iii) training the attention network by backpropagating the current cross-entropy loss and the distillation loss.

[0024] In step b), the testing device causes the attention network to perform a second convolution operation on the current feature map to generate a current first sub-feature map; then performs a 1×1 convolution operation on the current first sub-feature map to generate a current second sub-feature map, wherein the current second sub-feature map has the same channels as the current feature map; and applies a sigmoid function to the current second sub-feature map to generate the current attention map. In step b), the testing device causes the attention network to perform the second convolution operation on the current feature map to generate the current first sub-feature map; then performs a 1×1 convolution operation on the current first sub-feature map... The product operation is performed to generate the current second sub-feature map, wherein the current second sub-feature map has the same channels as the current feature map; the ReLU function is applied to the current second sub-feature map to generate the current second sub-feature map; the 1×1 convolution operation is performed on the current second sub-feature map to generate the current third sub-feature map with one channel; the sigmoid function is applied to the current third sub-feature map to generate the current prediction map. In step b), the testing device causes the attention network to perform the second convolution operation on each of the previous feature map and the current feature map to generate the previous first sub-feature map and the current sub-feature map, respectively. The current first sub-feature map; the previous first sub-feature map and the current first sub-feature map are respectively subjected to the 1×1 convolution operation to generate the previous second sub-feature map and the current second sub-feature map, wherein the previous second sub-feature map has the same number of channels as the current feature map; the ReLU function is applied to the current second sub-feature map and the previous second sub-feature map respectively to generate the current second sub-feature map and the previous second sub-feature map; the current second sub-feature map and the previous second sub-feature map are respectively subjected to the 1×1 convolution operation to generate the current third sub-feature map and the previous third sub-feature map with one channel. Feature maps; applying a soft sigmoid function to the current third sub-feature map and the previous third sub-feature map to generate the current soft prediction attention map; applying a ReLU function to the previous second sub-feature map and the current second sub-feature map respectively, followed by a 1×1 convolution operation to generate a previous third sub-feature map and the current third sub-feature map with one channel respectively; applying a sigmoid function to the current third sub-feature map to generate the current prediction map; applying a soft sigmoid function to the previous third sub-feature map and the current third sub-feature map respectively to generate the current soft prediction map and the updated previous soft prediction map.

[0025] The soft sigmoid function is the activation function of the sigmoid function, which takes the input value and divides it by a preset hyperparameter.

[0026] In step b), the testing device causes the splicing layer to generate a current intermediate attention map by performing element-wise multiplication of the current feature map and the current attention map, and to generate a current attention feature map by performing element-wise addition of the current feature map and the current intermediate attention map.

[0027] The testing device inputs the current candidate box into a binary converter, so that the binary converter binarizes the current candidate box and adjusts it to the size of the current feature map, thereby generating the current binary map.

[0028] According to another embodiment of the present invention, a method for training an object detection network is provided, the object detection network detecting objects on an image using an attention map, comprising: at least one memory storing instructions; and at least one processor configured to execute the instructions, wherein a learning device on the device performs the following processes: I) the processor performs i) inputting a training image into a feature extraction network, such that the feature extraction network performs at least one convolution operation on the training image to output a training feature map; ii) inputting the training feature map into an attention network, such that the attention network outputs ii-1) a feature map corresponding to the training image. The training attention map and ii-2) corresponding to the training prediction map of the training feature map; iii) inputting the training feature map and the training attention map into a concatenation layer to concatenate the training feature map and the training attention map, thereby outputting a training attention feature map; II) the processor executes i) inputting the training attention feature map into a region candidate network to output training candidate boxes corresponding to the training objects on the training attention feature map; ii) inputting the training candidate boxes into a binary converter to output the binary... The converter converts the candidate boxes into binary images; iii) the training attention feature map and the training candidate boxes are input into an ROI pooling layer, so that the ROI pooling layer performs pooling on the regions in the training attention feature map corresponding to each training candidate box, thereby generating a training pooled feature map; iv) the training pooled feature map is input into a detection network, so that the detection network performs training operations on the training pooled feature map, thereby generating training object detection information corresponding to each training candidate box; and III) the processor executes at least one of the following processes, i) by reverse... The process of training at least one of the detection network and the feature extraction network by propagating object loss, wherein the object loss is generated by referring to the training object detection information and the object ground truth; ii) the process of training the region candidate network by backpropagating RPN loss, wherein the RPN loss is generated by referring to the training candidate boxes and the RPN ground truth; iii) the process of training the attention network by backpropagating cross-entropy loss, wherein the cross-entropy loss is generated by referring to the training prediction graph and the training binary graph, wherein the training binary graph is generated by binarizing the training candidate boxes.

[0029] In step I), the processor causes the attention network to perform a second convolution operation on the training feature map to generate a first sub-feature map for training; then, it performs a 1×1 convolution operation on the first sub-feature map to generate a second sub-feature map for training, wherein the second sub-feature map has the same channels as the training feature map; finally, it applies a sigmoid function to the second sub-feature map to generate the attention map for training.

[0030] In step I), the processor causes the attention network to perform the second convolution operation on the training feature map to generate the first training sub-feature map; the processor then performs the 1×1 convolution operation on the first training sub-feature map to generate the second training sub-feature map, wherein the second training sub-feature map has the same channels as the training feature map; the processor applies the ReLU function to the second training sub-feature map to generate the second training sub-feature map; the processor then performs the 1×1 convolution operation on the second training sub-feature map to generate the third training sub-feature map with one channel; and the processor applies the sigmoid function to the third training sub-feature map to generate the training prediction map.

[0031] In step I), the processor causes the splicing layer to generate an intermediate attention map for training by performing element-wise multiplication of the training feature map and the training attention map, and to generate the training attention feature map by performing element-wise addition of the training feature map and the intermediate attention map.

[0032] In step II), the processor inputs the training candidate boxes to the binary converter, so that the binary converter generates an intermediate binary image for training by binarizing the training candidate boxes, and then adjusts it to the size of the training feature map to generate the training binary image.

[0033] According to another embodiment of the present invention, a method for training an object detection network is provided, the object detection network detecting objects on an image using an attention map, comprising: at least one memory storing instructions; and at least one processor configured to execute the instructions, wherein an on-device test device performs the following process: I) inputting a training image into a feature extraction network, wherein the feature extraction network performs at least one convolution operation on the training image to output a training feature map, provided that the operation has been performed by means of an on-device learning device; and inputting the training feature map into an attention network, wherein the attention network outputs a training prediction map corresponding to the training feature map. The training attention map corresponds to the training object density of the training feature map; the training feature map and the training attention map are input into a concatenation layer to concatenate them, thereby outputting a training attention feature map; the training attention feature map is input into a region candidate network to output training candidate boxes corresponding to the training objects on the training attention feature map; the training candidate boxes are input into a binary converter to convert them into binary images; the training attention feature map and the training candidate boxes are input into the ROI pool. The ROI pooling layer pools the regions corresponding to each training candidate box in the training attention feature map to generate a training pooled feature map; the training pooled feature map is input into a detection network to generate training object detection information corresponding to each training candidate box; at least one of the following processes has been performed: training at least one of the detection network and the feature extraction network using object loss, wherein the object loss is generated by referring to the training object detection information and object ground truth; training the region candidate network using RPN loss. The process includes: the RPN loss being generated by referencing the training candidate boxes and the RPN ground truth; the process of training the attention network using cross-entropy loss, wherein the training cross-entropy loss is generated by referencing the training prediction map and the training binary map, the training binary map being generated by binarizing the training candidate boxes; in this case, when acquiring a test video, the processor inputs the current image corresponding to the current frame in the test video into the feature extraction network, so that the feature extraction network performs at least one convolution operation on the current image, thereby outputting the current feature map and storing the current feature map in memory;II) The processor i) inputs a previous feature map of the previous image corresponding to a previous frame in the test video stored in the memory and the current feature map into the attention network, so that the attention network outputs an updated previous soft prediction map corresponding to the previous feature map, a current soft prediction map and a current prediction map corresponding to the current feature map, and a current attention map corresponding to the current object density in the current feature map; ii) stores the current soft prediction map in the memory; iii) inputs the current feature map and the current attention map into the stitching layer, so that the stitching layer stitches the current feature map and the current attention map to output a current attention feature map; iv) inputs the current attention feature map into the region candidate network, so that the region candidate network outputs a current candidate box corresponding to the current object on the current attention feature map; v) inputs the current training candidate box into the binary converter, so that the binary converter changes the current candidate box into a current binary box. Figure 1, vi) Inputting the current attention feature map and the current candidate box into the ROI pooling layer, so that the ROI pooling layer performs pooling on the regions in the current attention feature map corresponding to each current candidate box, thereby generating a current pooled feature map; vii) Inputting the current pooled feature map into the detection network, so that the detection network performs training operations on the current pooled feature map, thereby generating current object detection information corresponding to each current candidate box; and III) The processor utilizes i) the current cross-entropy loss, which is generated by referring to the current prediction map and the current binary map, the current binary map being generated by binarizing the current candidate box; and ii) the distillation loss, which is generated by referring to the updated previous soft prediction map and the current soft prediction map, the previous image corresponding to the previous frame in the test video stored in the memory; iii) The process of training the attention network by backpropagating the current cross-entropy loss and the distillation loss.

[0034] In step II), the processor causes the attention network to perform a second convolution operation on the current feature map to generate a current first sub-feature map; then, it performs a 1×1 convolution operation on the current first sub-feature map to generate a current second sub-feature map, wherein the current second sub-feature map has the same channels as the current feature map; finally, it applies the sigmoid function to the current second sub-feature map to generate the current attention map.

[0035] In step II), the processor causes the attention network to perform the second convolution operation on the current feature map to generate the current first sub-feature map; performs the 1×1 convolution operation on the current first sub-feature map to generate the current second sub-feature map, wherein the current second sub-feature map has the same number of channels as the current feature map; applies the ReLU function to the current second sub-feature map to generate the current second sub-feature map; performs the 1×1 convolution operation on the current second sub-feature map to generate the current third sub-feature map with one channel; and applies the sigmoid function to the current third sub-feature map to generate the current prediction map.

[0036] In step II), the processor causes the attention network to perform the second convolution operation on each of the previous feature map and the current feature map, thereby generating a previous first sub-feature map and the current first sub-feature map respectively; the processor then performs the 1×1 convolution operation on each of the previous first sub-feature map and the current first sub-feature map, thereby generating a previous second sub-feature map and a current second sub-feature map respectively, wherein the previous second sub-feature map has the same number of channels as the current feature map; the processor then applies the ReLU function to each of the current second sub-feature map and the previous second sub-feature map, thereby generating a current second sub-feature map and a previous second sub-feature map; and finally performs the 1×1 convolution operation on each of the current second sub-feature map and the previous second sub-feature map. A 1×1 convolution operation is performed to generate a current third sub-feature map and a previous third sub-feature map with one channel. A soft sigmoid function is applied to the current third sub-feature map and the previous third sub-feature map to generate the current soft prediction attention map. A ReLU function is applied to the previous second sub-feature map and the current second sub-feature map, followed by a 1×1 convolution operation to generate a previous third sub-feature map and a current third sub-feature map with one channel. A sigmoid function is applied to the current third sub-feature map to generate the current prediction map. A soft sigmoid function is applied to the previous third sub-feature map and the current third sub-feature map to generate the current soft prediction map and the updated previous soft prediction map.

[0037] The soft sigmoid function is the activation function of the sigmoid function, which takes the input value and divides it by a preset hyperparameter.

[0038] In step II), the processor causes the splicing layer to generate the current intermediate attention map by performing element-wise multiplication of the current feature map and the current attention map, and to generate the current attention feature map by performing element-wise addition of the current feature map and the current intermediate attention map.

[0039] The processor inputs the current candidate box into a binary converter, so that the binary converter binarizes the current candidate box and adjusts it to the size of the current feature map, thereby generating the current binary map.

[0040] Technical effect

[0041] This invention provides a method to improve the performance of object detection networks without requiring additional training operations.

[0042] This invention provides a method to improve the performance of object detection networks that detect objects on images by utilizing attention maps.

[0043] In addition, the present invention can provide a method for preventing catastrophic amnesia by utilizing a cross-distillation loss layer. Attached Figure Description

[0044] The following drawings, used to illustrate embodiments of the present invention, are only a part of the embodiments of the present invention. Those skilled in the art (hereinafter referred to as "the skill") can obtain other drawings based on the following drawings without making creative work.

[0045] Figure 1 This is a schematic diagram illustrating a training device for training an object detection network that uses attention maps to detect objects on an image, according to an embodiment of the present invention.

[0046] Figure 2 This is a schematic diagram illustrating a method for training an object detection network that uses attention maps to detect objects in an image according to an embodiment of the present invention.

[0047] Figure 3 This is a schematic diagram illustrating the process of generating attention maps and cross-entropy loss during the training of an object detection network that uses attention maps to detect objects on an image, according to an embodiment of the present invention.

[0048] Figure 4 This is a schematic diagram illustrating the process of generating an attention feature map using a feature map and an attention map according to an embodiment of the present invention.

[0049] Figure 5 This is a schematic diagram of a test device for testing an object detection network that uses attention maps to detect objects on an image, according to an embodiment of the present invention.

[0050] Figure 6 This is a schematic diagram illustrating a method for testing an object detection network that uses attention maps to detect objects on an image according to an embodiment of the present invention.

[0051] Figure 7This is a schematic diagram illustrating the process of continuously training an attention network during testing of an object detection network that uses attention maps to detect objects in an image, according to an embodiment of the present invention. Detailed Implementation

[0052] The objectives, technical solutions, and advantages of the present invention will now be described in detail with reference to the accompanying drawings, which exemplarily illustrate specific embodiments in which the present invention can be implemented. These embodiments will be described in detail below to enable those skilled in the art to fully implement the present invention.

[0053] Furthermore, throughout the description and claims of this invention, the term "comprising" and its variations are not intended to exclude other technical features, additions, components, or steps. Other objects, advantages, and features of the invention will become apparent to those skilled in the art through this specification, while others will become apparent through embodiments of the invention. The following examples and figures are illustrative only and are not intended to limit the scope of the invention.

[0054] Furthermore, this invention encompasses all combinations of the embodiments described in this specification. It should be understood that while the various embodiments of the invention differ, they are not mutually exclusive. For example, specific shapes, structures, and characteristics relating to one embodiment of this specification may be implemented as other embodiments without departing from the technical concept and scope of the invention. Additionally, it should be understood that the position or configuration of the various components in each disclosed embodiment may be changed without departing from the technical concept and scope of the invention. Therefore, the following detailed description is not intended to be limiting, and if properly described, the scope of the invention is defined only by the full scope of the claims and their equivalents. Similar reference numerals in the drawings refer to the same or similar functions in several respects.

[0055] Preferred embodiments of the present invention will now be described in detail with reference to the accompanying drawings, so that those skilled in the art can readily implement the present invention.

[0056] Figure 1 This is a schematic diagram illustrating a training device for training an object detection network according to an embodiment of the present invention, wherein the object detection network utilizes attention maps to detect objects in an image. (Reference) Figure 1 The learning device 1000 on the device may include a memory 1100 and a processor 1200. The memory 1100 stores instructions for training an object detection network that uses an attention network to detect objects in an image. The processor 1200 executes the operation of training the object detection network for detecting objects in an image using the attention network according to the instructions stored in the memory 1100.

[0057] Specifically, the learning device 1000 on the device can typically utilize a combination of computing devices and computer software (i.e., instructions that enable the computing devices to function in a specific way) to achieve the required system performance. For example, the computing device may be a device that includes a computer processor, memory, storage, input devices and output devices and other existing computing device components; electronic communication devices such as routers and switches; electronic information storage systems such as network attached storage (NAS) and storage area network (SAN).

[0058] In addition, the processor of a computing device may include hardware configurations such as an MPU (Micro Processing Unit) or CPU (Central Processing Unit), cache memory, and data bus. Furthermore, the computing device may include an operating system and software components that execute applications for specific purposes.

[0059] However, this does not preclude the computing device from including the integrated form of the medium, processor, and memory used to implement the present invention, i.e., an integrated processor.

[0060] The following is for reference. Figure 2 This invention describes a method for training an object detection network that uses an attention map to detect objects on a training image using an on-device learning device configured as described above, according to an embodiment of the present invention.

[0061] First, after acquiring the training images, the learning device 1000 on the device inputs the training images into the feature extraction network 1210, so that the feature extraction network 1210 outputs a training feature map by performing convolution operations on the training images.

[0062] At this time, the feature extraction network 1210 may include at least one convolutional layer, and a training feature map may be generated by performing at least one convolution operation on the training image. Furthermore, the size of the training feature map may be... H is the height of the training feature map, W is the width of the training feature map, and K is the number of channels in the training feature map.

[0063] Then, the learning device 1000 on the device inputs the training feature map into the attention network 1220, so that the attention network 1220 outputs a training prediction map and a training attention map, wherein the training prediction map corresponds to the training feature map and the training attention map corresponds to the training object density of the training feature map.

[0064] At this time, refer to Figure 3The attention network 1220 generates a first sub-feature map for training by performing at least one convolution operation on the training feature map through at least one convolutional layer 1221, and generates a second sub-feature map for training by performing a 1×1 convolution operation on the first sub-feature map for training through a first 1×1 convolutional layer 1222. At this time, the first 1×1 convolutional layer 1222 performs a 1×1 convolution operation on the first sub-feature map for training using k convolutional kernels, so that the second sub-feature map for training has k channels, and k can be the same as the number of channels in the training feature map, so that the number of channels in the second sub-feature map for training is the same as the number of channels in the training feature map. That is, the size of the second sub-feature map for training can be... .

[0065] Additionally, the attention network 1220 can output a training attention map corresponding to the training second sub-feature map by applying the sigmoid function to the training second sub-feature map.

[0066] At this time, the training attention map may be a map showing the density of the training objects located on the training image, or it may be a map focusing on a specific region on the training image that contains more information.

[0067] Additionally, the attention network 1220 can apply the ReLU function to the training second sub-feature map through the linear correction unit (ReLU) 1223 to generate a characterized training second sub-feature map, and then perform a 1×1 convolution operation on the characterized training second sub-feature map through the second 1×1 convolution layer 1224 to generate a training third sub-feature map.

[0068] At this point, the second 1×1 convolutional layer 1224 can perform a 1×1 convolution operation on the training second sub-feature map using a convolution kernel, so that the training third sub-feature map has one channel. That is, the size of the training third sub-feature map can be... .

[0069] Then, the attention network 1220 can generate a training prediction map corresponding to the training feature map by applying the sigmoid function to the training third sub-feature map.

[0070] Re-reference Figure 2 The learning device 1000 on the device inputs the training feature map and the training attention map into the concatenation layer 1230, so that the concatenation layer 1230 outputs the training attention feature map by concatenating the training feature map and the training attention map.

[0071] At this time, refer to Figure 4 The splicing layer 1230 can generate an intermediate attention map for training by performing element-wise multiplication 1231 on the training attention map and the training feature map, and then generate the training attention feature map by performing element-wise sum 1232 on the intermediate attention map and the training feature map. At this time, the training attention feature map is a map of a specific region with more information on the training feature map within the reference attention map set, thus improving the performance of subsequent object detection.

[0072] Then, the learning device 1000 on the device can input the training attention feature map into the region proposal network (RPN) 1240 so that the region proposal network 1240 outputs training candidate boxes related to the training candidate objects on the training attention feature map.

[0073] At this time, the region candidate network 1240 generates the training candidate boxes using the training attention feature map, thus generating more accurate candidate boxes than when simply using the training feature map. The training attention feature map is formed by concatenating the training feature map and the training attention map, and it is a map that concentrates the regions with more information on the training feature map.

[0074] Additionally, the on-device learning device 1000 can input the training candidate boxes into the binary converter 1270, causing the binary converter 1270 to binarize the training candidate boxes, thereby converting them into a training binary image. At this time, the on-device learning device 1000 can adjust the size of the training binary image to match the size of the training feature map, so that the training binary image can be used as the target value for training the attention network 1220.

[0075] Additionally, the learning device 1000 on the device can input the training attention feature map and the training candidate boxes into the ROI (Region of Interest) pooling layer 1250, so that the ROI pooling layer 1250 performs pooling on the regions in the training attention feature map corresponding to each training candidate box, thereby generating a training pooled feature map; the training pooled feature map can be input into the detection network 1260, so that the detection network 1260 performs training operations on the training pooled feature map (e.g., performs fully connected operations) to generate training object detection information corresponding to each training candidate box.

[0076] At this time, the detection network 1260 performs object detection using the training pooling feature map, thus enabling it to perform object detection with higher performance than when simply using the training feature map. The training pooling feature map is generated by using the training attention feature map, which is a map that focuses on regions with more information on the training feature map.

[0077] Then, the on-device learning device 1000 can input the object detection information and its corresponding ground truths into the object loss layer 1261 to enable the object loss layer 1261 to acquire object loss; input the training candidate boxes and their corresponding RPN ground truths into the RPN loss layer 1241 to enable the RPN loss layer 1241 to acquire RPN loss; and input the training prediction map and the training binary map into the cross-distillation loss layer 1280 to enable the cross-distillation loss layer 1280 to acquire training cross-entropy loss. Then, the on-device learning device 1000 can perform at least a portion of the following processes: training at least one of the feature extraction network 1210 and the detection network 1260 by backpropagating the object loss; training the region candidate network 1240 by backpropagating the RPN loss; and training the attention network 1220 by backpropagating the training cross-entropy loss.

[0078] At this point, the cross-entropy loss can be expressed as the following mathematical formula.

[0079]

[0080] in, This can be interpreted as applying the sigmoid function to the training prediction map of the third sub-feature map used for training. The training binary graph can be represented.

[0081] Figure 5 This is a schematic diagram of a test device illustrating an object detection network that uses attention maps to detect objects in an image, according to an embodiment of the present invention. The object detection network has been trained using the method described above. (Reference) Figure 5 The on-device testing device 2000 may include a memory 2100 and a processor 2200. The memory 2100 stores instructions for testing an object detection network that uses an attention network to detect objects in an image. The processor 2200 executes the operations of the object detection network for testing the detection of objects in the image according to the instructions stored in the memory 2100. In this case, the memory 2100 may be an on-device storage device, but is not limited to this.

[0082] Specifically, the on-device test device 2000 can typically utilize a combination of computing devices and computer software (i.e., instructions that enable the computing devices to function in a specific manner) to achieve the required system performance. For example, the computing device may be a device including a computer processor, memory, storage, input devices and output devices, and other existing computing device components; electronic communication devices such as routers and switches; and electronic information storage systems such as network attached storage (NAS) and storage area networks (SAN).

[0083] In addition, the processor of a computing device may include hardware configurations such as an MPU (Micro Processing Unit) or CPU (Central Processing Unit), cache memory, and data bus. Furthermore, the computing device may include an operating system and software components that execute applications for specific purposes.

[0084] However, this does not preclude the computing device from including the integrated form of the medium, processor, and memory used to implement the present invention, i.e., an integrated processor.

[0085] The following is for reference. Figure 6 A method for testing an object detection network on an on-device test device 2000 according to an embodiment of the present invention, configured as described above, is described, wherein the object detection network utilizes attention maps to detect objects on an image.

[0086] In the following description, details that can be obtained from the reference will not be repeated. Figures 2 to 3 The parts of the description that are easy to understand.

[0087] Additionally, the on-board testing equipment and object detection network described below can be mounted on a device that performs object detection, such as an autonomous vehicle, autonomous aircraft, or autonomous robot, and can be configured to perform on-device learning within the device.

[0088] First, an object detection network that uses attention maps to detect objects in an image can be one that has been described with reference to the above. Figure 2 and Figure 3 The training method in the training state is the state being trained.

[0089] That is, with the help of the learning device 1000 on the device, the following operations can be performed: the training image is input into the feature extraction network 1210, and the feature extraction network 1210 performs a convolution operation on the training image to output the training feature map; the training feature map is input into the attention network 1220, and the attention network 1220 outputs the training prediction map corresponding to the training feature map and the training attention map corresponding to the training object density of the training feature map; The training feature map and the training attention map are input to the concatenation layer 1230, where they are concatenated to output the training attention feature map. The training attention feature map is input to the region candidate network (RPN) 1240, which outputs training candidate boxes related to the training candidate objects on the training attention feature map. The training candidate boxes are input to the binary converter 1270, which converts them into the training binary map. The training candidate boxes and the training attention feature map are input to the ROI pooling layer 1250, where the regions in the training attention feature map corresponding to each training candidate box are pooled to generate the training pooled feature map. The training pooled feature map is input to the detection network 1260, and the training pooled feature map is trained by the detection network 1260 to generate training object detection information corresponding to each training candidate box; at least a portion of the following processes have been performed: training at least one of the feature extraction network 1210 and the detection network 1260 by backpropagating the object loss, wherein the object loss is generated by referring to the training object detection information and its corresponding object ground truth; training the region candidate network 1240 by backpropagating the RPN loss, wherein the RPN loss is generated by referring to the training candidate box and its corresponding RPN ground truth; and training the attention network 1220 by backpropagating the training cross-entropy loss, wherein the training cross-entropy loss is generated by referring to the training binary map and the training prediction map.

[0090] With the object detection network that uses attention maps to detect objects in an image trained as described above, when a test video is acquired, the test device 2000 on the device can input the current image corresponding to the current frame in the test video into the feature extraction network 1210, so that the feature extraction network 1210 performs at least one convolution operation on the current image to output the current feature map. At this time, the size of the current feature map can be... K is the number of channels in the current feature map, H is the height of the current feature map, and W is the width of the current feature map.

[0091] Then, the test device 2000 on the device can store the current feature map in the memory 2100 for continuous training of the attention network 1200 associated with subsequent frames in the test video; input the previous feature map corresponding to the previous frame and the current feature map stored in the memory 2100 to the attention network 1220, so that the attention network 1220 outputs an updated previous soft prediction map corresponding to the previous feature map, a current prediction map and a current soft prediction map corresponding to the current feature map, and a current attention map corresponding to the current object density in the current feature map, for lifelong learning (i.e., continuous training) of the attention network 1220 associated with the current frame; and store the current soft prediction map in the memory 2100 for continuous training of the attention network 1220 associated with subsequent frames.

[0092] At this time, refer to Figure 7 The attention network 1220 performs at least one convolution operation on each of the previous and current feature maps using at least one convolutional layer 1221 to generate a previous first sub-feature map and a current first sub-feature map, respectively. It then performs a 1×1 convolution operation on each of the previous and current first sub-feature maps using a first 1×1 convolutional layer 1222 to generate a previous second sub-feature map and a current second sub-feature map, respectively. The first 1×1 convolutional layer 1222 can use k convolutional kernels to perform 1×1 convolution operations on each of the previous and current first sub-feature maps, so that each of the previous and current second sub-feature maps has k channels. By making k the same as the number of channels in the previous and current feature maps, the number of channels in the previous and current second sub-feature maps can be the same as the number of channels in the previous and current feature maps. That is, the size of the previous and current second sub-feature maps can be... .

[0093] Additionally, the attention network 1220 can output the current attention map corresponding to the current second sub-feature map by applying the sigmoid function to the current second sub-feature map.

[0094] At this time, the current attention map can be a map showing the density of the current object located on the current image, or it can be a map focusing on a specific region with more information on the current image.

[0095] Furthermore, the attention network 1220 applies the ReLU function to the previous second sub-feature map and the current second sub-feature map respectively using a rectified linear unit (ReLU), thereby generating a characterized previous second sub-feature map and a characterized current second sub-feature map respectively. A second 1×1 convolutional layer 1224 performs a 1×1 convolution operation on the characterized previous second sub-feature map and the characterized current second sub-feature map respectively, thereby generating a previous third sub-feature map and a current third sub-feature map respectively. At this time, the second 1×1 convolutional layer 1224 can use a single convolution kernel to perform a 1×1 convolution operation on the previous second sub-feature map and the current second sub-feature map respectively, so that each of the previous third sub-feature map and the current third sub-feature map has one channel. That is, the size of the current third sub-feature map and the previous third sub-feature map can be... .

[0096] Then, the attention network 1220 can apply the soft sigmoid function to the previous third sub-feature map to generate the current soft prediction map; apply the sigmoid function to the current third sub-feature map to generate the current prediction map; apply the soft sigmoid function to the current third sub-feature map to generate the current soft prediction map and store it in the memory 2100.

[0097] At this point, the soft sigmoid function can be represented by the following mathematical expression.

[0098]

[0099] in, The input can be a sigmoid function, Z, and T, which is the temperature hyperparameter. Furthermore, the temperature hyperparameter can be used to calibrate prediction confidence in classification work, and when applied to out-of-distribution detection problems, it can further amplify the difference in class scores between in-distribution and out-of-distribution samples, thus helping to more easily distinguish out-of-distribution samples.

[0100] Then refer to again Figure 6 The testing device 2000 on the device inputs the current feature map and the current attention map into the stitching layer 1230, so that the stitching layer 1230 outputs the current attention feature map by stitching the current feature map and the current attention map.

[0101] Then, the test device 2000 on the device can input the current attention feature map into the region candidate network 1240 so that the region candidate network 1240 outputs the current candidate box related to the current candidate object on the current attention feature map.

[0102] Additionally, the test device 2000 on the device can input the current candidate box to the binary converter 1270 so that the binary converter 1270 converts the current candidate box into the current binary image.

[0103] Additionally, the test device 2000 on the device can input the current attention feature map and the current candidate box into the ROI pooling layer 1250, so that the ROI pooling layer 1250 performs pooling on the regions in the current attention feature map corresponding to each current candidate box, thereby generating a current pooled feature map; the current pooled feature map can be input into the detection network 1260, so that the detection network 1260 performs training operations on the current pooled feature map (e.g., performs fully connected operations) to generate current object detection information corresponding to each current candidate box.

[0104] Then refer to Figure 7 The test device 2000 on the device can input the current prediction map and the current binary map into the cross-distillation loss layer 1280 so that the cross-distillation loss layer 1280 can obtain the current cross-entropy loss; input the previous soft prediction map corresponding to the previous frame stored in the memory 2100 and the updated previous soft prediction map into the cross-distillation loss layer 1280 so that the cross-distillation loss layer 1280 can obtain the distillation loss for continuous training of the attention network 1220 associated with the current frame; and perform continuous training of the attention network 1220 by backpropagating the current cross-entropy loss and the distillation loss.

[0105] At this point, the total loss used to perform continuous training of the attention network 1220 can be expressed as the following mathematical formula.

[0106]

[0107] in, It can represent the current prediction graph, This indicates the previously updated soft prediction plot. Indicates the current binary graph, Indicates the previous soft prediction plot, Represents cross-entropy loss, This indicates distillation loss.

[0108] At this time, while the attention network 1220 is continuously trained on the distillation loss device, it can be used to prevent catastrophic forgetting.

[0109] In addition, when using the object detection network trained by the method described above to detect objects in an image, the attention network 1220 performs object detection and on-device continuous training simultaneously, thus enabling it to be updated in every frame of the video.

[0110] When an object detection network that detects objects on images using attention maps is trained and tested according to the method described above, it is possible to improve the performance of the detection network by focusing on the driving environments frequently encountered by autonomous vehicles, which have limited computing power of embedded systems (e.g., autonomous vehicles), and also to prevent catastrophic forgetting.

[0111] Furthermore, the embodiments of the present invention described above can be implemented and recorded in the form of program commands executed by various computer components on a computer-readable recording medium. The computer-readable recording medium may include program commands, data files, data structures, etc., individually or in combination. The program commands recorded on the computer-readable recording medium may be specifically designed or configured for the present invention, or may be program commands known and used by those skilled in the art of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes; optical recording media such as CD-ROMs and DVDs; magneto-optical media such as floppy disks; and hardware devices such as ROMs, RAMs, and flash memory specifically configured to store and execute program commands. Examples of program commands include not only machine language code generated by a compiler, but also executable high-level language code such as that executed by a computer using an interpreter. The hardware device may be configured to operate as one or more software modules to perform the processing according to the present invention, and vice versa.

[0112] The present invention has been described above with reference to specific components and other specific matters, limited embodiments, and accompanying drawings. However, these descriptions are intended to help to provide a more comprehensive understanding of the present invention. The present invention is not limited to the described embodiments, and those skilled in the art can attempt various modifications and variations based on these descriptions.

[0113] Therefore, the technical concept of the present invention should not be limited to the embodiments described above, and all contents of the claims of the present invention and their equivalent or equivalent transformations are within the technical concept of the present invention.

Claims

1. A method for testing an object detection network, wherein the object detection network uses attention maps to detect objects in an image, wherein, Includes the following steps: a) Learning device execution with the aid of the device: I) Inputting a training image into a feature extraction network, causing the feature extraction network to perform at least one convolution operation on the training image, thereby outputting a training feature map; inputting the training feature map into an attention network, causing the attention network to output a training prediction map corresponding to the training feature map and a training attention map corresponding to the training object density of the training feature map; inputting the training feature map and the training attention map into a concatenation layer, causing the concatenation layer to concatenate the training feature map and the training attention map, thereby outputting a training image. II) Input the training attention feature map into a region candidate network so that the region candidate network outputs training candidate boxes corresponding to the training objects on the training attention feature map; input the training candidate boxes into a binary converter so that the binary converter converts the candidate boxes into binary maps; input the training attention feature map and the training candidate boxes into a ROI pooling layer so that the ROI pooling layer pools the regions in the training attention feature map corresponding to each training candidate box, thereby generating a training pooled feature map; The training pooling feature map is input into the detection network, so that the detection network performs a training operation on the training pooling feature map to generate training object detection information corresponding to each training candidate box; III) At least one of the following processes is performed: training at least one of the detection network and the feature extraction network by backpropagation object loss, wherein the object loss is generated by referring to the training object detection information and object ground truth; training the region candidate network by backpropagation RPN loss, wherein the RPN loss is generated by referring to the training candidate box and RPN ground truth; training the attention network by backpropagation cross-entropy loss, wherein the training cross-entropy loss is generated by referring to the training prediction map and the training binary map, wherein the training binary map is generated by binarizing the training candidate boxes; In this case, when acquiring a test video, the test device inputs the current image corresponding to the current frame in the test video into the feature extraction network, so that the feature extraction network performs at least one convolution operation on the current image to output a current feature map and store the current feature map in memory; b) The test device i) inputs a previous feature map of a previous image corresponding to a previous frame in the test video stored in the memory and the current feature map into the attention network, so that the attention network outputs an updated previous soft prediction map corresponding to the previous feature map, a current soft prediction map and a current prediction map corresponding to the current feature map, and a current attention map corresponding to the current object density in the current feature map; ii) stores the current soft prediction map in the memory; iii) inputs the current feature map and the current attention map into the stitching layer, so that the stitching layer stitches the current feature map and the current attention map to output a current attention feature map; iv) inputs the current attention feature map into the region The region candidate network outputs a current candidate box corresponding to the current object on the current attention feature map; v) the current training candidate box is input to the binary converter, so that the binary converter changes the current candidate box into the current binary map; vi) the current attention feature map and the current candidate box are input to the ROI pooling layer, so that the ROI pooling layer performs pooling on the region corresponding to each current candidate box in the current attention feature map, thereby generating a current pooled feature map; vii) the current pooled feature map is input to the detection network, so that the detection network performs training operation on the current pooled feature map, thereby generating current object detection information corresponding to each current candidate box; and c) The test device utilizes i) a current cross-entropy loss, generated by referring to the current prediction map and the current binary map, the current binary map being generated by binarizing the current candidate box, and ii) a distillation loss, generated by referring to the updated previous soft prediction map and the current soft prediction map, the previous image corresponding to the previous frame in the test video stored in the memory, and iii) training the attention network by backpropagating the current cross-entropy loss and the distillation loss. In step b), The testing device causes the attention network to perform a second convolution operation on the current feature map to generate a current first sub-feature map; then performs a 1×1 convolution operation on the current first sub-feature map to generate a current second sub-feature map, wherein the current second sub-feature map has the same number of channels as the current feature map; finally, the sigmoid function is applied to the current second sub-feature map to generate the current attention map. In step b), The testing device causes the attention network to perform the second convolution operation on the current feature map to generate the current first sub-feature map; to perform the 1×1 convolution operation on the current first sub-feature map to generate the current second sub-feature map, wherein the current second sub-feature map has the same number of channels as the current feature map; to apply the ReLU function to the current second sub-feature map to generate the current second sub-feature map; to perform the 1×1 convolution operation on the current second sub-feature map to generate the current third sub-feature map with one channel; and to apply the sigmoid function to the current third sub-feature map to generate the current prediction map. In step b), The testing device causes the attention network to perform the second convolution operation on each of the previous and current feature maps, thereby generating a previous first sub-feature map and the current first sub-feature map, respectively; it then performs the 1×1 convolution operation on each of the previous and current first sub-feature maps, thereby generating a previous second sub-feature map and a current second sub-feature map, wherein the previous second sub-feature map has the same number of channels as the current feature map; the ReLU function is applied to both the current second sub-feature map and the previous second sub-feature map, thereby generating a current second sub-feature map and a previous second sub-feature map; and finally, the 1×1 convolution operation is performed on both the current second sub-feature map and the previous second sub-feature map. The process involves several steps: first, generating a current third sub-feature map and a previous third sub-feature map with one channel; second, applying a soft sigmoid function to both the current and previous third sub-feature maps to generate a current soft prediction attention map; third, applying a ReLU function to both the previous and current second sub-feature maps and performing a 1×1 convolution operation to generate a previous and current third sub-feature map with one channel; fourth, applying a sigmoid function to the current third sub-feature map to generate a current prediction map; and finally, applying a soft sigmoid function to both the previous and current third sub-feature maps to generate the current soft prediction map and the updated previous soft prediction map. The soft sigmoid function is the activation function of the sigmoid function, which takes the input value and divides it by a preset hyperparameter.

2. The method according to claim 1, wherein: In step b), The testing device enables the splicing layer to generate a current intermediate attention map by performing element-wise multiplication of the current feature map and the current attention map, and to generate a current attention feature map by performing element-wise addition of the current feature map and the current intermediate attention map.

3. The method according to claim 1, wherein: The testing device inputs the current candidate box into a binary converter, so that the binary converter binarizes the current candidate box and adjusts it to the size of the current feature map, thereby generating the current binary map.

4. An on-device testing device for testing an object detection network, said object detection network utilizing attention maps to detect objects in an image, wherein, include: At least one memory that stores instructions; as well as At least one processor configured to execute the instructions, The testing device on the device performs the following process: I) With the assistance of the learning device on the device, the following operations are performed: A training image is input into a feature extraction network, which performs at least one convolution operation on the training image to output a training feature map; The training feature map is input into an attention network, which outputs a training prediction map corresponding to the training feature map and a training attention map corresponding to the training object density of the training feature map; The training feature map and the training attention map are input into a concatenation layer, which concatenates the training feature map and the training attention map to output a training attention feature map; The training attention feature map is input into a region candidate network, which outputs training candidate boxes corresponding to the training objects on the training attention feature map; The training candidate boxes are input into a binary converter, which converts the candidate boxes into binary images; The training attention feature map and the training candidate boxes are input into a ROI pooling layer, which performs a convolution operation on the regions in the training attention feature map corresponding to each training candidate box. Pooling is performed to generate a training pooled feature map; the training pooled feature map is then input into a detection network to train the detection network on the training pooled feature map, thereby generating training object detection information corresponding to each training candidate box; at least one of the following processes has been performed: training at least one of the detection network and the feature extraction network using object loss, wherein the object loss is generated by referring to the training object detection information and object ground truth; training the region candidate network using RPN loss, wherein the RPN loss is generated by referring to the training pooled feature map. The process involves generating candidate boxes and RPN ground truth values; training the attention network using cross-entropy loss, wherein the training cross-entropy loss is generated by referencing the training prediction map and the training binary map, and the training binary map is generated by binarizing the training candidate boxes; in this case, when acquiring a test video, the processor inputs the current image corresponding to the current frame in the test video into the feature extraction network, so that the feature extraction network performs at least one convolution operation on the current image, thereby outputting a current feature map and storing the current feature map in memory;II) The processor i) inputs a previous feature map of the previous image corresponding to a previous frame in the test video stored in the memory and the current feature map into the attention network, so that the attention network outputs an updated previous soft prediction map corresponding to the previous feature map, a current soft prediction map and a current prediction map corresponding to the current feature map, and a current attention map corresponding to the current object density in the current feature map; ii) stores the current soft prediction map in the memory; iii) inputs the current feature map and the current attention map into the stitching layer, so that the stitching layer stitches the current feature map and the current attention map to output a current attention feature map; iv) inputs the current attention feature map into the region candidate network, so that the region candidate network outputs a current candidate box corresponding to the current object on the current attention feature map; v) inputs the current training candidate box into the binary converter, so that the binary converter changes the current candidate box into a current binary box. Figure 1, vi) inputting the current attention feature map and the current candidate box into the ROI pooling layer, so that the ROI pooling layer performs pooling on the regions in the current attention feature map corresponding to each current candidate box, thereby generating a current pooled feature map; vii) inputting the current pooled feature map into the detection network, so that the detection network performs training operations on the current pooled feature map, thereby generating current object detection information corresponding to each current candidate box; and III) the processor utilizing i) the current cross-entropy loss, which is generated by referring to the current prediction map and the current binary map, the current binary map being generated by binarizing the current candidate box; and ii) the distillation loss, which is generated by referring to the updated previous soft prediction map and the current soft prediction map, the previous image corresponding to the previous frame in the test video stored in the memory; iii) the process of training the attention network by backpropagating the current cross-entropy loss and the distillation loss; In the process described in II), The processor instructs the attention network to perform a second convolution operation on the current feature map to generate a current first sub-feature map; it then performs a 1×1 convolution operation on the current first sub-feature map to generate a current second sub-feature map, wherein the current second sub-feature map has the same number of channels as the current feature map; finally, it applies a sigmoid function to the current second sub-feature map to generate the current attention map. In the process described in II), The processor causes the attention network to perform the second convolution operation on the current feature map to generate the current first sub-feature map; performs the 1×1 convolution operation on the current first sub-feature map to generate the current second sub-feature map, wherein the current second sub-feature map has the same number of channels as the current feature map; applies the ReLU function to the current second sub-feature map to generate the current second sub-feature map; performs the 1×1 convolution operation on the current second sub-feature map to generate the current third sub-feature map with one channel; and applies the sigmoid function to the current third sub-feature map to generate the current prediction map. In the process described in II), The processor causes the attention network to perform the second convolution operation on each of the previous feature maps and the current feature map, thereby generating a previous first sub-feature map and the current first sub-feature map, respectively; it then performs the 1×1 convolution operation on each of the previous first sub-feature map and the current first sub-feature map, thereby generating a previous second sub-feature map and a current second sub-feature map, wherein the previous second sub-feature map has the same number of channels as the current feature map; it then applies the ReLU function to each of the current second sub-feature map and the previous second sub-feature map, thereby generating a current second sub-feature map and a previous second sub-feature map; and finally performs the 1×1 convolution operation on each of the current second sub-feature map and the previous second sub-feature map. The process involves several steps: first, generating a current third sub-feature map and a previous third sub-feature map with one channel; second, applying a soft sigmoid function to both the current and previous third sub-feature maps to generate a current soft prediction attention map; third, applying a ReLU function to both the previous and current second sub-feature maps and performing a 1×1 convolution operation to generate a previous and current third sub-feature map with one channel; fourth, applying a sigmoid function to the current third sub-feature map to generate a current prediction map; and finally, applying a soft sigmoid function to both the previous and current third sub-feature maps to generate the current soft prediction map and the updated previous soft prediction map. The soft sigmoid function is the activation function of the sigmoid function, which takes the input value and divides it by a preset hyperparameter.

5. The testing device on the equipment according to claim 4, wherein: In the process described in II), The processor causes the splicing layer to generate a current intermediate attention map by performing element-wise multiplication of the current feature map and the current attention map, and to generate a current attention feature map by performing element-wise addition of the current feature map and the current intermediate attention map.

6. The testing device on the equipment according to claim 4, wherein: The processor inputs the current candidate box into a binary converter, so that the binary converter binarizes the current candidate box and adjusts it to the size of the current feature map, thereby generating the current binary map.

Images