Object detection method, machine learning method and electronic device

Abstract

The invention relates to a continuous learning object detection technology, in particular to an object detection method, a machine learning method and an electronic device. An object detection method includes the following steps: detecting an environmental signal; determining a task mode according to the environment signal; an input image is extracted. Feature extraction is performed on the input image according to the task pattern through a sub-model of a neural network model, the sub-model of the neural network model comprises a task-specific layer corresponding to the task pattern, and a gateway switch of the task-specific layer determines the sub-model of the neural network model. And outputting an object detection result corresponding to the task mode. Therefore, the method can be used for a single-stage object detection model, and catastrophic forgetting is effectively avoided in classification and object positioning.

Description

Technical Field

[0001] The present invention relates to a continuous learning object detection technology, and more particularly to an object detection method, a machine learning method and an electronic device. Background Art

[0002] The goal of traditional object detection is to develop computational models and techniques that provide the information required by computer vision applications, including object location and category (such as people, animals, or cars). Over the past two decades, the development of object detection has gone through two major periods: the "traditional object detection period" (before 2014) and the "deep learning-based detection period" (after 2014). Early object detection algorithms were mainly based on complex features designed by hand, and various acceleration techniques were used to utilize limited computing resources. With the major breakthroughs in deep learning technology after 2014, object detection has become an unprecedented research hotspot. In recent years, due to the wide application of object detection in the fields of robots and self-driving cars, the field of continuous learning for single-stage and dual-stage object detection has received increasing attention.

[0003] Continuous learning is an important goal in the field of artificial intelligence research. Its main focus is to enable the model to learn knowledge of different tasks at different time points, while not forgetting previously learned knowledge and avoiding the problem of catastrophic forgetting. Continuous learning can not only improve the efficiency of learning new knowledge, but also reduce the hardware requirements of training equipment because it does not need to access data of old tasks. Task incremental learning is the most typical scenario in continuous learning. In this scenario, the data of the same task can all arrive at the same time point, so that the model can be trained under the assumption of independent and identical distribution.

[0004] In other words, the focus of continuous learning is how to classify features for task labels collected at different time points, and the feature spaces corresponding to different tasks do not intersect with each other. No matter how many tasks or categories need to be added, catastrophic forgetting will not occur.

[0005] Existing object detection technologies usually develop algorithms for specific network architectures, which may not be applicable to a variety of object detectors. They have poor generalization and flexibility for neural networks and training graphs, and may not reduce the burden on users. In addition, past research on task incremental learning has mainly focused on dual-stage object detectors, rather than single-stage object detectors, which are faster in actual application scenarios.

[0006] Therefore, how to share the same single-stage target detector neural network model for multiple tasks so that the overall network size and computational complexity can be significantly reduced, and how to avoid catastrophic forgetting without having to access and store old task data is a topic of concern to technical personnel in this field. Summary of the invention

[0007] In view of this, the present invention provides an object detection method, a machine learning method and an electronic device. Based on the network architecture of a single-stage object detector, an object detection method is provided that can perform incremental task learning without modifying the original network layer architecture. The method is not only widely applicable to various single-stage object detection models, but also can effectively avoid the problem of catastrophic forgetting.

[0008] An embodiment of the present invention provides an object detection method, comprising: detecting an environmental signal; determining a task mode according to the environmental signal; extracting an input image; performing feature extraction on the input image according to the task mode through a sub-model of a neural network model, wherein the sub-model of the neural network model includes a task-specific layer corresponding to the task mode, and a gate switch of the task-specific layer determines the sub-model of the neural network model; and outputting an object detection result corresponding to the task mode.

[0009] An embodiment of the present invention provides a machine learning method, comprising: receiving an environmental signal, wherein the environmental signal indicates a task mode; receiving training data associated with the task mode, wherein the training data includes a training image, a category label corresponding to the training image, and a bounding box label corresponding to the training image; setting a task-specific layer of a neural network model according to the task mode, wherein a gateway switch of the task-specific layer determines a sub-model of the neural network model; determining a loss function according to the category label, the bounding box label, and the gateway switch of the task-specific layer; determining a back-propagation gradient according to the loss function; and updating the neural network model and the gateway switch of the task-specific layer according to the back-propagation gradient.

[0010] An embodiment of the present invention provides an electronic device, comprising: a storage medium storing a plurality of modules; and a processor coupled to the storage medium, the processor being configured to execute the plurality of modules, wherein the plurality of modules include: an environmental perception module detecting an environmental signal; an image extraction module extracting an input image; an inference module determining a task mode according to the environmental signal, performing feature extraction on the input image according to the task mode through a sub-model of a neural network model, wherein the sub-model of the neural network model includes a task-specific layer corresponding to the task mode, a gate switch of the task-specific layer determines the sub-model of the neural network model, and outputs an object detection result corresponding to the task mode.

[0011] Based on the above, the embodiment of the present invention proposes a task incremental learning technology architecture based on object detection, which can determine different sub-models by corresponding important convolution neural parameters of a neural network model to different task modes through task-specific layers, and allow a neural network model to automatically determine whether to reuse important neural network parameters for other old tasks based on the loss function in combination with the gateway switch of the learned task, and use this parameter to help optimize the new task, so that the neural network model can achieve better reusability. Therefore, the single-level object detection model can effectively avoid catastrophic forgetting in both classification and object positioning. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 is a schematic diagram of an electronic device according to an embodiment of the present invention;

[0013] Figure 2 is a flow chart of an object detection method according to an embodiment of the present invention;

[0014] Figure 3 is a flow chart of a machine learning method according to an embodiment of the present invention;

[0015] Figure 4 is a schematic diagram of a neural network model of a YOLO object detection model architecture in an embodiment of the present invention;

[0016] Figure 5 is a schematic diagram of a task-specific layer of a neural network model according to an embodiment of the present invention;

[0017] Fig. 6A is a schematic diagram of training a neural network model in an embodiment of the present invention;

[0018] Figure 6B is a schematic diagram of back-propagating gradients to update a neural network model and a gate switch of a task-specific layer in an embodiment of the present invention;

[0019] Figure 7 Schematic diagram of switching task modes of object detection in an embodiment of the present invention.

[0020] [Description of Reference Numerals]

[0021] 10. Electronic devices;

[0022] 110. Processor;

[0023] 120. Storage medium;

[0024] 130. Image extraction device;

[0025] 140. Sensor;

[0026] 1201, image extraction module;

[0027] 1202, environmental perception module;

[0028] 1203, inference module;

[0029] 1205, training module;

[0030] IMG,IMG_H,IMG_L,image;

[0031] BOX, BOX_1, BOX_2, bounding box;

[0032] ES, environmental signal;

[0033] NN, neural network model;

[0034] FN, feature extraction network;

[0035] OUT, output layer;

[0036] BBN, backbone network;

[0037] NKN, Neck Network;

[0038] HDN, head network;

[0039] TSK_1,TSK_2,TSK_3, task specific layer;

[0040] FPN-1,…,FPN-N, Feature Pyramid Network;

[0041] S210, S220, S230, S240, S250, S310, S320, S330, S340, S350, S360, S60, S61, S62, S63, steps;

[0042] t,t+1,task;

[0043] Bn task-1,…,Bn task-t,610,630,650,610',630',650', batch normalization layer;

[0044] CNN,620,640,620',640',convolutional layer;

[0045] 701,702, object detection results;

[0046] C l , number of channels;

[0047] L, positive integer;

[0048] l,c, variables;

[0049] Feature map;

[0050]

[0051] Scaling factor;

[0052] The cumulative maximum value of the scaling factor;

[0053] average value;

[0054] Standard deviation;

[0055] Translation factor;

[0056] Loss function;

[0057] L CE , cross entropy loss;

[0058] R s (r), layer-by-layer sparsification regularization term;

[0059] λ1,k, parameter;

[0060] The average value of the scaling factors;

[0061] g,g ′ ,g l ,g l ′ ,gradient;

[0062] Activation parameters;

[0063] y1,y2,y3,z1,z2, neurons. DETAILED DESCRIPTION

[0064] Some embodiments of the present invention will be described in detail below in conjunction with the accompanying drawings. The referenced element symbols in the following description will be regarded as the same or similar elements when the same element symbols appear in different drawings. These embodiments are only part of the present invention and do not disclose all possible implementation methods of the present invention. More specifically, these embodiments are only examples of methods and devices within the scope of the patent application of the present invention. Wherever possible, elements / components / steps with the same reference numbers in the drawings and implementations represent the same or similar parts. Elements / components / steps with the same reference numbers or the same terms in different embodiments can refer to the relevant descriptions of each other.

[0065] Figure 11 is a schematic diagram of an electronic device according to an embodiment of the present invention. The electronic device 10 includes a processor 110 and a storage medium 120 .

[0066] The electronic device 10 may be an edge computing device implemented on an embedded platform. In an embodiment of the present invention, the electronic device 10 may be a drone. In some embodiments, the electronic device 10 may also be a device used in various application scenarios such as image recognition, access control management, identity verification, digital monitoring, financial industry, retail industry, unmanned stores, smart factories, mechanical surgery or medical diagnosis. The electronic device 10 may also be a desktop computer, a laptop computer, a server, a smart phone or a tablet computer.

[0067] The processor 110 is, for example, a central processing unit (CPU), or other programmable general-purpose or special-purpose micro control unit (MCU), microprocessor, digital signal processor (DSP), programmable controller, application specific integrated circuit (ASIC), graphics processing unit (GPU), tensor processing unit (TPU), image signal processor (ISP), image processing unit (IPU), arithmetic logic unit (ALU), complex programmable logic device (CPLD), field programmable gate array (FPGA), or other similar elements or combinations thereof. The processor 110 may be coupled to the storage medium 120, and access and execute multiple modules or various applications stored in the storage medium 120.

[0068] The storage medium 120 is, for example, any type of fixed or removable random access memory (RAM), read-only memory (ROM), flash memory, hard disk drive (HDD), solid state drive (SSD) or similar element or a combination of the above elements, and is used to store multiple modules, computer programs or various applications that can be executed by the processor 110. In this embodiment, the multiple modules stored in the storage medium 120 include an image extraction module 1201, an environment perception module 1202, an inference module 1203 and a training module 1205, and their functions will be described later.

[0069] In one embodiment, the electronic device 10 may further include an image capturing device 130. The processor 110 may be coupled to the image capturing device 130. The image capturing device 130 may be, for example, a digital camera, a video camera, or a camera lens having a lens and a photosensitive element. The photosensitive element is used to sense the intensity of light entering the lens, thereby generating an image.

[0070] In one embodiment, the electronic device 10 may further include a sensor 140. The processor 110 may be coupled to the sensor 140. The sensor 140 may detect the environment in which the electronic device 10 is located and generate an environmental signal. The processor 110 may receive the environmental signal to determine the state of the electronic device 10 relative to the environment based on the environmental signal. In one embodiment, the sensor 140 is a light-sensitive sensor that can detect the brightness of the environment. In one embodiment, the sensor 140 is a depth sensor that can calculate the depth of field of the environment of the electronic device 10. In one embodiment, the sensor 140 is an altitude sensor that includes a gyroscope, an accelerometer, a magnetometer and / or a barometer, and the altitude of the electronic device 10 in the environment can be determined by signals from the gyroscope, accelerometer, magnetometer and / or barometer.

[0071] Figure 2 is a flow chart of an object detection method according to an embodiment of the present invention. Figure 2 The object detection method shown is suitable for Figure 1 Please refer to the electronic device 10. Figure 1 and Figure 2 .

[0072] In step S210 , the environment sensing module 1202 detects an environment signal through the sensor 140 .

[0073] In step S220, the inference module 1203 determines a task mode according to the environmental signal. In one embodiment, the task mode is, for example, a day and night mode switched according to different ambient brightness. In one embodiment, the task mode is, for example, different task modes switched according to the depth of field setting of the image capture device 130. In one embodiment, the task mode is, for example, different task modes switched according to the height of the electronic device 10.

[0074] In step S230, the image extraction module 1201 extracts an input image. For example, the image extraction module 1201 may extract an input image through the image extraction device 130. The image extraction module 1201 may also extract an input image from a database. Alternatively, the image extraction module 1201 may receive an input image through a network.

[0075] In step S240, the inference module 1203 performs feature extraction on the input image according to the task mode through a sub-model of a neural network model, wherein the sub-model of the neural network model includes a task-specific layer corresponding to the task mode, and the gate switch (polarization mask) of the task-specific layer determines the sub-model of the neural network model.

[0076] In step S250 , the inference module 1203 outputs an object detection result corresponding to the task mode.

[0077] Figure 3 is a flow chart of a machine learning method according to an embodiment of the present invention. Figure 3 The machine learning approach shown is suitable for training a neural network model for single-stage object detection.

[0078] In step S310 , the environment sensing module 1202 receives an environment signal, wherein the environment signal indicates a task mode.

[0079] In step S320 , the training module 1205 receives a training data associated with the task mode, wherein the training data includes a training image, a category label corresponding to the training image, and a bounding box label corresponding to the training image.

[0080] In step S330, the training module 1205 sets a task-specific layer of a neural network model according to the task mode.

[0081] In step S340 , the training module 1205 determines a loss function according to the category label, the bounding box label, and the gate switch of the task-specific layer.

[0082] In step S350 , the training module 1205 determines a back-propagation gradient according to the loss function.

[0083] In step S360, the training module 1205 updates the neural network model and the gate switch of the task-specific layer according to the back-propagation gradient.

[0084] Figure 4 is a schematic diagram of a neural network model of the YOLO object detection model architecture in an embodiment of the present invention. In an embodiment of the present invention, the neural network model NN includes a feature extraction network FN and a head network. The head network includes an output layer OUT. The input image IMG obtains an input feature map through the feature extraction network FN. The output layer OUT of the head network can output an object detection result for the input feature map. Specifically, the input feature map obtains the predicted category of the input image IMG and the bounding box BOX of the input image IMG through the output layer OUT of the head network. In one embodiment, the input image IMG is extracted through the feature extraction network FN to obtain an input feature map. The input feature map is passed through multiple layers of convolutional layers CNN to obtain an output feature map. The output feature map then passes through the output layer OUT of the head network to obtain the predicted category of the input image IMG and the bounding box BOX of the input image IMG.

[0085] In one embodiment, the neural network model NN may be a convolutional neural network architecture. In one embodiment, the neural network model NN includes a convolutional layer and a batch normalization layer. In one embodiment, the neural network model NN is a YOLOR model.

[0086] In one embodiment of the present invention, the gateway switch of the task-specific layer includes multiple scaling factors of a batch normalization layer of the feature extraction network FN, and the multiple scaling factors are associated with multiple convolution kernels of a convolution layer of the feature extraction network FN. In one embodiment of the present invention, the inference module 1203 passes the input image IMG through the convolution layer and performs normalization operation through the multiple scaling factors of the batch normalization layer to obtain the input feature map.

[0087] In the network architecture of YOLOR, each convolution layer is connected to a batch normalization layer, and the number of output channels of each convolution layer is the same as the number of channels of the batch normalization layer. In the network architecture of YOLOR, the scaling factor in the batch normalization layer can be used as a gate switch for the convolution parameters. When the parameter is considered to be beneficial to reduce the loss in the training task, the scaling factor is activated so that the corresponding convolution parameters can be forward propagated. The scaling factor in the batch normalization layer is used to measure the importance of each convolution kernel in the convolution layer. The scaling factor has the function of scaling (i.e., adding multiple times) the feature map transmitted from the convolution layer. When the reverse gradient corrects the neuron parameters of the neural network model NN, the scaling factor corresponding to each convolution kernel will be activated. In this way, for different tasks, multiple scaling factors of the batch normalization layer are set as task-specific layers, so that the neural network model NN can retain the neuron parameters with high importance after training without changing the architecture of the object detector.

[0088] In one embodiment of the present invention, the feature extraction network FN includes a backbone network and a neck network. In one embodiment, the inference module 1203 passes the input image through the backbone network to obtain a first feature map. In one embodiment, the processor 110 can pass the first feature map through the neck network to obtain a second feature map that integrates information of different sizes. In one embodiment, the neck network may include multiple feature pyramid networks (FPNs), such as Figure 5 The feature pyramid network FPN-1, ..., FPN-N shown. Specifically, the neck network can fuse the different size information of the first feature map through the feature extraction of the feature pyramid network FPN-1, ..., FPN-N to obtain the second feature map.

[0089] In one embodiment, the inference module 1203 passes the second feature map through the neck network to obtain the input feature map. In one embodiment, the inference module 1203 extracts the output feature map through the head network, and the output feature map passes through multiple output layers OUT of the head network to obtain a predicted category of the input image IMG and a bounding box BOX of the input image IMG.

[0090] by Figure 4 Taking the input image IMG as an example, the input image IMG may be an image of a tree. After the inference module 1203 performs object detection on the input image IMG, an object detection result may be obtained, including a predicted category of the object "tree" and a confidence value corresponding to the object "tree", as well as a bounding box BOX for marking the object "tree".

[0091] In one embodiment of the present invention, the gateway switch of the task-specific layer includes a plurality of first scaling factors of a first batch normalization layer of the backbone network and a plurality of second scaling factors of a second batch normalization layer of the neck network, wherein the plurality of first scaling factors are associated with a plurality of first convolution kernels of a first convolution layer of the backbone network, wherein the plurality of second scaling factors are associated with a plurality of second convolution kernels of a second convolution layer of the neck network. In one embodiment, the inference module 1203 passes the input image through the first convolution layer and performs normalization operation through the plurality of first scaling factors of the first batch normalization layer to obtain the first feature map. In one embodiment, the inference module 1203 passes the first feature map through the second convolution layer and performs normalization operation through the plurality of second scaling factors of the second batch normalization layer to obtain the input feature map.

[0092] In one embodiment of the present invention, the head network includes a third batch normalization layer and multiple output layers, wherein the gateway switch of the task-specific layer includes multiple third scaling factors of the third batch normalization layer, wherein the multiple third scaling factors are associated with multiple third convolution kernels of a third convolution layer of the head network. In one embodiment, the inference module 1203 passes the input feature map through the third convolution layer and performs a normalization operation through the multiple third scaling factors of the third batch normalization layer to obtain an output feature map. In one embodiment, the inference module 1203 passes the output feature map through the multiple output layers to obtain the predicted category and the bounding box BOX.

[0093] It should be noted that the general single-stage object detection model, such as the YOLOR model, includes three complete parts: the backbone network, the neck network, and the head network. Therefore, the gate switches of the task-specific layer need to be set for the backbone network, the neck network, and the head network respectively.

[0094] Figure 5 Schematic diagram of task-specific layers of a neural network model according to an embodiment of the present invention. The neural network model NN includes a backbone network BBN, a neck network NKN, and a head network HDN. For a task t, the backbone network BBN, the neck network NKN, and the head network HDN respectively set task-specific layers TSK_1, TSK_2, and TSK_3.

[0095] In one embodiment of the present invention, the gateway switch of the task-specific layer TSK_1 includes multiple first scaling factors of a first batch standardization layer (Bn task-t) of the backbone network BBN. ​​The gateway switch of the task-specific layer TSK_2 includes multiple second scaling factors of a second batch standardization layer (Bn task-t) of the neck network NKN. The multiple first scaling factors are associated with multiple first convolution kernels of a first convolution layer of the backbone network BBN. ​​The multiple second scaling factors are associated with multiple second convolution kernels of a second convolution layer of the neck network NKN. In one embodiment, the inference module 1203 passes the input image IMG through the first convolution layer and performs standardization operation through the multiple first scaling factors of the first batch standardization layer to obtain the first feature map. In one embodiment, the inference module 1203 passes the first feature map through the second convolution layer and performs standardization operation through the multiple second scaling factors of the second batch standardization layer to obtain the input feature map. It should be noted that, if Figure 5 As shown, multiple tasks (eg, task 1 to task t) are respectively provided with corresponding batch normalization layers Bn task-1 to batch normalization layers Bn task-t.

[0096] In one embodiment of the present invention, the head network HDN includes multiple output layers, wherein the gateway switch of the task-specific layer TSK_3 includes multiple third scaling factors of the third batch normalization layer. The multiple third scaling factors are associated with multiple third convolution kernels of a third convolution layer of the head network HDN. In one embodiment, the inference module 1203 passes the input feature map through the third convolution layer and performs a normalization operation through the multiple third scaling factors of the third batch normalization layer to obtain an output feature map. In one embodiment, the task-specific layer TSK_3 further includes multiple output layers (for example, the implicit-add layer, implicit-mul layer and Conv 1x1 convolution layer of the YOLOR architecture). In one embodiment, the inference module 1203 passes the output feature map through multiple output layers to obtain the predicted category and the bounding box. For example, in Figure 5 In the example, the input image IMG is an image including a puppy and a pedestrian. Specifically, the inference module 1203 performs object detection on the input image IMG and can detect objects including "puppy" and "pedestrian" in the input image IMG. In other words, in this example, the inference module 1203 outputs the feature map through multiple output layers of the head network HDN to obtain object detection results including: the predicted category of the object "puppy" and the confidence value corresponding to the object "puppy", and the bounding box BOX_1 used to mark the "puppy" object; the predicted category of the object "pedestrian" and the confidence value corresponding to the object "pedestrian", and the bounding box BOX_2 used to mark the "pedestrian" object.

[0097] It should be noted that for task t (such as Figure 5 As shown in the figure, task-t), the task-specific layer TSK_3 sets multiple output layers corresponding to task t, namely, implicit-add task-t layer, implicit-mul task-t layer and Conv 1x1 task-t convolution layer. For the head network of YOLOR's network architecture, not all convolution layers are followed by corresponding batch normalization layers. Therefore, when setting task-specific layers for individual different tasks, corresponding task-specific layers need to be set in multiple output layers of the head network of the YOLOR architecture (namely implicit-add task-t layer, implicit-mul task-t layer and Conv 1x1 convolution layer). It is worth mentioning that since multiple output layers are not as large in parameters as 3x3 convolution kernels, they are all one-dimensional neural network layers. Therefore, each new task only needs to increase 4 scalar receptive field (scale) dimensions multiplied by these 3 task-specific layers, a total of 12 layers, accounting for less than 1% of the total model parameters, and the cost is very low, but the task-specific layers can be used to record the data distribution of each different task. Even if you are constantly learning new tasks, you can use the same neural network model NN model to infer the task labels of the input image IMG, so as to achieve the function of learning new things without forgetting old ones.

[0098] The following details the parameter updates of the neural network model NN and the gateway switches of the task-specific layer during the training process.

[0099] Fig. 6A Schematic diagram of training a neural network model in an embodiment of the present invention. Fig. 6A As shown, the neural network model NN includes batch normalization layers 610, 630, 650 and convolution layers 620, 640. For example, the neural network model NN has been trained for t tasks. For input data X, feature extraction is performed through the convolution layer 620 to obtain a feature map X * , and then the batch normalization layer 630 performs normalization operation to obtain the feature map

[0100] Taking the batch normalization layer 630 as an example, the normalization operation can be calculated by the following formula (1):

[0101]

[0102] in is the standardized feature map, X * is the feature map after the convolution layer, is the scaling factor, is an average value, is one standard deviation, is a shifting factor.

[0103] In formula (1), the variable l=1,…L-1 represents the convolutional layer, L is a positive integer, and the variable t represents the current task. is the average value of the input feature maps of different channels in the same batch, is the standard deviation of the input feature maps of different channels in the same batch, the scaling factor and translation factor is a parameter that can be trained. In one embodiment, The initial value is set to 0.5. The initial value is set to 0. Scaling factor It has the function of scaling the feature map passed by the convolutional layer.

[0104] In one embodiment of the present invention, the gate switches of the task-specific layer include multiple scaling factors of a batch normalization layer of the neural network model NN. And the multiple scaling factors A plurality of convolution kernels associated with a convolution layer of the neural network model NN. Specifically, the scaling factor in the batch normalization layer It can be used to measure the importance of the corresponding convolution kernel in the convolution layer. In the embodiment of the present invention, a task-specific layer is provided for each new task to be learned, and the batch normalization layer of the task-specific layer can remember the data distribution of the specific task t ( and ), and use the scaling factor corresponding to task t Record multiple convolution kernel positions that are important for the target task.

[0105] Specifically, the scaling factor in the batch normalization layer is It can be used to measure the importance of the corresponding convolution kernel in the convolution layer. In the embodiment of the present invention, a specific batch normalization layer (task-specific layer) is set for each learning task, and each task uses its corresponding batch normalization layer to remember the data distribution of the specific task ( and ), and use the scaling factor corresponding to task t Record multiple convolution kernel positions that are important for the target task.

[0106] In detail, each convolution layer has a corresponding scaling factor on the dimension of the convolution kernel channel To represent the importance of the convolution kernel in the convolution layer. In one embodiment, the scaling factor The number of corresponding feature maps X * The number of channels. Due to the scaling factor The feature map X * The characteristic of scaling up or down is to limit the operation of a part of the neural network model NN as a gate switch of a task-specific layer. In one embodiment, when the scaling factor If it is not 0, it means that the feature map corresponding to this channel must participate in the calculation, so the feature map passes through the batch normalization layer and then to the next layer. If it is equal to 0, it means that the feature map corresponding to this channel is not important and there is no need to pass the feature map corresponding to this channel to the next layer.

[0107] That is, the scaling factor It can be determined whether to activate the neurons in the neural network model NN for a task t. Fig. 6A For example, for the current task t, the corresponding scaling factors among the batch normalization layers 610, 630, and 650 are as well as The neurons are activated and the feature map corresponding to this channel is passed to the next layer.

[0108] It should be noted that after step S60, for the feature graph X given by the new task t+1, the training module 1205 replaces the new batch normalization layers 610', 630', 650'. However, the convolutional layers 620', 640' retain the same convolutional layers 620, 640 to perform training. Fig. 6A As shown, for the new task t+1, the corresponding scaling factors in the new batch normalization layers 610', 630', and 650' are as well as The neurons are activated and the feature map corresponding to this channel is passed to the next layer.

[0109] Next, the training module 1205 determines the loss function based on the category label, the bounding box label, and the gateway switch of the task-specific layer. Training module 1205 with loss function Perform back-propagation gradient update on the neural network model NN and the gate switches of the task-specific layers.

[0110] In one embodiment of the present invention, the loss function includes a cross entropy loss and a layer-wise polarization regularization term.

[0111] Loss Function It is expressed by the following formula (2):

[0112]

[0113] Where L CE is the cross entropy loss, R s (r) is the layer-by-layer sparsification regularization term and λ1 is the parameter.

[0114] Cross entropy loss L CE Determined by the category labels of the training data.

[0115] The layer-by-layer sparsification regularization term is calculated by the following formulas (3)-(4):

[0116]

[0117]

[0118] Where R s (r) is the layer-by-layer sparsification regularization term, are the multiple scaling factors, L is the number of batch normalization layers and convolutional layers, C l is the number of channels of the multiple convolution kernels, and k is a parameter.

[0119] In the layer-by-layer sparse regularization term R in formula (4) s (r) can be used to reduce the number of activated neurons, where Represents the scaling factor for each channel in each batch normalization layer Take the absolute value, this Each scaling factor can be After training, it approaches 0. It is the scaling factor of each layer in the neural network model NN The average value of This drives all scaling factors. Increase The difference, or scaling factor There will be a polarization effect, making the difference between important and unimportant significant. The parameters k and λ1 in formulas (2) and (3) can be adjusted according to different training data sets.

[0120] During the back-propagation gradient process of learning the new task t+1, the scaling factor will receive the gradient g from the activation layer l , training module 1205 according to the scaling factor Determine whether multiple corresponding convolution parameters are important and activated if more than one of the t tasks has been trained. If the convolution parameters at the same position are of high importance, Indicates that this convolution parameter is activated by at least one scaling factor of the task. The training module 1205 adjusts the back-propagation gradient g according to the activation parameter act(x). lThe training module 1205 consists of the adjusted back-propagation gradient g l ′ Update the neural network model NN and the gate switches of the task-specific layer corresponding to the new task t+1 (scaling factor ).

[0121] The adjusted back-propagation gradient is calculated by the following formula (5):

[0122]

[0123] where g l ′ is the adjusted back-propagation gradient, g l is the back-propagation gradient, where It is the cumulative maximum value of the scaling factors of different tasks with the same convolution parameters.

[0124] Figure 6B Schematic diagram of back propagation gradient updating of neural network model NN and a corresponding task-specific layer gate switch in one embodiment of the present invention. Figure 6B Taking the batch normalization layer 630' as an example, in step S61, the batch normalization layer 630' obtains the back-propagation gradient g transmitted back by the neuron z1. In step S62, the training module 1205 trains the training module 1205 according to the multiple scaling factors of the batch normalization layer 630'. Determine multiple activation parameters act(x) and adjust the back propagation gradient g ′ And according to the adjusted back-propagation gradient g ′ Multiple scaling factors for batch normalization layer 610' And the parameters of neurons y1 and y3 are updated.

[0125] It should be noted that when performing reverse gradient training for a new task t+1, the training module 1205 records the activated scaling factors for the t tasks that have been trained. Since the corresponding convolution layer parameters are very important to a certain old task, they do not want to be modified by the gradient of the new task. The reverse gradient process for the new task will protect the important convolution neurons from being modified by the new task, but it does not limit whether the gate switch of the task-specific layer of the new task activates the neurons that are important to the old task. Therefore, the convolution kernels with high parameter importance can be reused by multiple tasks. For example, in Figure 6B The neurons y2 and z2 shown therein are retained convolution neurons, so during the back propagation gradient process of step S63, the parameters of neurons y2 and z2 will not be modified.

[0126] Figure 71 is a schematic diagram of switching the task mode of object detection in an embodiment of the present invention. In an embodiment of the present invention, the electronic device 10 is, for example, a drone. Figure 7 As shown, according to the change of the environmental signal ES (e.g., height), the electronic device 10 can switch between the first task mode (corresponding to the image IMG_H) and the second task mode (corresponding to the image IMG_L). In one embodiment, the training module 1205 determines a second task mode according to the environmental signal ES and receives a second training data associated with the second task mode.

[0127] The training module 1205 sets a second task-specific layer of the neural network model according to the second task mode, wherein a second gate switch of the second task-specific layer determines a second sub-model of the neural network model. The training module 1205 updates the neural network model and the second gate switch of the second task-specific layer according to the second training data.

[0128] For example, in an application scenario of using a drone for mountain search and rescue, when the drone flies at an altitude of more than 50 meters, the pixel size of the pedestrian in the image IMG_H only occupies a very small part of the entire image due to the high altitude, making it difficult for the object detection model or the human eye to recognize it. Therefore, when the flight altitude is high, the more appropriate detection object should be changed to a larger area such as a trail, a footpath, or a stream where people are likely to stay as the object detection result 701.

[0129] Once the remote-controlled drone lowers its altitude to below 50 meters for intensive search, according to the change of environmental signal ES (e.g., altitude), the interrogation training module 1205 is notified that the altitude has been lowered and the mission mode can be switched. At this time, the size of the pedestrian object in the camera lens image IMG_L will also increase, and the object detection model is used to identify the suspected object to be rescued as the object detection result 702. It should be noted that regardless of high or low flight altitude, the drone uses the same neural network model NN for training, but uses different sub-models for inference due to different mission modes. In this search and rescue application scenario, if the drone detects a pedestrian, it can immediately provide a relative GPS position, image or sound warning, and use the remote-controlled drone to drop necessary life-sustaining supplies before the rescue personnel arrive at the scene, thereby extending the life survival time of the victim and achieving the best operating efficiency of the remote-controlled drone.

[0130] In one embodiment, the electronic device 10 may be an edge computing device. In addition, the drone may store the images collected at different altitudes during flight. After the flight mission is completed and a stable power source is connected, the edge computing device may train the images of each single mission to update the sub-model. Since the number of images of a single mission is not large, the amount of computation and the time required on the electronic device 10 are greatly reduced. Compared with the traditional training method, the embodiment of the present invention updates the neural network model NN more quickly. The use of the edge computing device allows the newly added data to update the neural network model NN in a short time and perform inferences at a real-time speed, greatly enhancing the advantages of the edge computing device.

[0131] In summary, the embodiments of the present invention propose an object detection method that allows multiple tasks to share the same single-stage neural network model, so that the overall network size and the amount of computation can be greatly reduced, and catastrophic forgetting can be avoided without the need to access and store old task data, so that the object detection model retains the knowledge of old tasks while also having the ability to continuously learn new tasks.

[0132] Although the present invention has been disclosed as above by way of embodiments, it is not intended to limit the present invention. Any person having ordinary knowledge in the technical field may make some changes and modifications without departing from the spirit and scope of the present invention. Therefore, the protection scope of the present invention shall be determined by the scope of the claims.

Claims

1. A method for detecting an object, It is characterized in that include: detecting an environmental signal; Determine a task mode according to the environmental signal; extracting an input image; Performing feature extraction on the input image according to the task mode through a sub-model of a neural network model, wherein the sub-model of the neural network model includes a task-specific layer corresponding to the task mode, and a gate switch of the task-specific layer determines the sub-model of the neural network model; and Output the object detection results corresponding to the task mode.

2. The object detection method according to claim 1, It is characterized in that The neural network model includes a feature extraction network, wherein the step of extracting features from the input image by the sub-model of the neural network model includes: The input image is passed through the feature extraction network to obtain an input feature map.

3. The object detection method according to claim 2, It is characterized in that The gate switch of the task-specific layer includes a plurality of scaling factors of a batch normalization layer of the feature extraction network, and the plurality of scaling factors are associated with a plurality of convolution kernels of a convolution layer of the feature extraction network, wherein the step of passing the input image through the feature extraction network to obtain the input feature map includes: The input image is passed through the convolution layer and normalized by the multiple scaling factors of the batch normalization layer to obtain the input feature map.

4. The object detection method according to claim 2, It is characterized in that The feature extraction network includes a backbone network and a neck network, wherein the step of passing the input image through the feature extraction network to obtain the input feature map includes: Passing the input image through the backbone network to obtain a first feature map; and The first feature map is passed through the neck network to obtain the input feature map.

5. The object detection method according to claim 4, It is characterized in that The gateway switch of the task-specific layer includes a plurality of first scaling factors of a first batch normalization layer of the backbone network and a plurality of second scaling factors of a second batch normalization layer of the neck network, wherein the plurality of first scaling factors are associated with a plurality of first convolution kernels of a first convolution layer of the backbone network, wherein the plurality of second scaling factors are associated with a plurality of second convolution kernels of a second convolution layer of the neck network, wherein the step of passing the input image through the feature extraction network to obtain the input feature map includes: The input image is passed through the first convolutional layer and normalized by the first scaling factors of the first batch normalization layer to obtain the first feature map; The first feature map is passed through the second convolutional layer and normalized by the multiple second scaling factors of the second batch normalization layer to obtain the input feature map.

6. The object detection method according to claim 2, It is characterized in that The neural network model further includes a head network, wherein the step of outputting the object detection result corresponding to the task mode includes: The input feature map is passed through the head network to obtain the predicted category of the input image and the bounding box of the input image.

7. The object detection method according to claim 6, It is characterized in that The head network includes a third batch normalization layer and a plurality of output layers, wherein the gate switch of the task-specific layer includes a plurality of third scaling factors of the third batch normalization layer, wherein the plurality of third scaling factors are associated with a plurality of third convolution kernels of a third convolution layer of the head network, wherein the step of passing the input feature map through the head network to obtain the predicted category of the input image and the bounding box of the input image comprises: Passing the input feature map through the third convolutional layer and performing normalization operation via the plurality of third scaling factors of the third batch normalization layer to obtain an output feature map; The output feature map is passed through the multiple output layers to obtain the predicted category and the bounding box.

8. The object detection method according to claim 1, It is characterized in that The neural network model is the YOLOR model.

9. The object detection method according to claim 1, It is characterized in that Also includes: Receiving a training data associated with the task mode, wherein the training data includes a training image, a category label corresponding to the training image, and a bounding box label corresponding to the training image; Setting the task-specific layer of the neural network model according to the task mode; Determining a loss function according to the category label, the bounding box label, and the gate switch of the task-specific layer; Determine a back-propagation gradient according to the loss function; and The neural network model and the gate switch of the task-specific layer are updated according to the back-propagated gradient.

10. The object detection method according to claim 9, It is characterized in that The loss function includes: a cross entropy loss; and A layer-by-layer sparsification regularization term.

11. The object detection method according to claim 10, It is characterized in that The gate switch of the task-specific layer includes multiple scaling factors of a batch normalization layer of the neural network model, and the multiple scaling factors are associated with multiple convolution kernels of a convolution layer of the neural network model, wherein the layer-by-layer sparsification regularization term is calculated by the following formula: Where R s (r) is the layer-by-layer sparsification regularization term, are the multiple scaling factors, L is the number of batch normalization layers and convolutional layers, C l is the number of channels of the multiple convolution kernels, and k is a parameter.

12. The object detection method according to claim 9, It is characterized in that The step of updating the neural network model and the gate switch of the task-specific layer according to the back-propagation gradient includes: determining a plurality of activation parameters according to the gate switch of the task-specific layer; Adjusting the back-propagation gradient according to the plurality of activation parameters; The neural network model and the gate switch of the task-specific layer are updated according to the back-propagated gradient.

13. The object detection method according to claim 9, It is characterized in that Also includes: Determine a second task mode according to the environmental signal, and receive a second training data associated with the second task mode; Setting a second task-specific layer of the neural network model according to the second task mode, wherein a second gate switch of the second task-specific layer determines a second sub-model of the neural network model; The neural network model and the second gate switch of the second task specific layer are updated according to the second training data.

14. A machine learning method, It is characterized in that include: receiving an environmental signal, wherein the environmental signal indicates a task mode; Receiving a training data associated with the task mode, wherein the training data includes a training image, a category label corresponding to the training image, and a bounding box label corresponding to the training image; Setting a task-specific layer of a neural network model according to the task mode, wherein a gate switch of the task-specific layer determines a sub-model of the neural network model; Determining a loss function according to the category label, the bounding box label, and the gate switch of the task-specific layer; Determine a back-propagation gradient according to the loss function; and The neural network model and the gate switch of the task-specific layer are updated according to the back-propagated gradient.

15. The machine learning method according to claim 14, It is characterized in that The loss function includes: a cross entropy loss; and A layer-by-layer sparsification regularization term.

16. The machine learning method according to claim 15, It is characterized in that The gate switch of the task-specific layer includes multiple scaling factors of a batch normalization layer of the neural network model, and the multiple scaling factors are associated with multiple convolution kernels of a convolution layer of the neural network model, wherein the layer-by-layer sparsification regularization term is calculated by the following formula: Where R s (r) is the layer-by-layer sparsification regularization term, are the multiple scaling factors, L is the number of batch normalization layers and convolutional layers, C l is the number of channels of the multiple convolution kernels, and k is a parameter.

17. The machine learning method according to claim 14, It is characterized in that The step of updating the neural network model and the gate switch of the task-specific layer according to the back-propagation gradient includes: determining a plurality of activation parameters according to the gate switch of the task-specific layer; Adjusting the back-propagation gradient according to the plurality of activation parameters; The neural network model and the gate switch of the task-specific layer are updated according to the back-propagated gradient.

18. The machine learning method according to claim 14, It is characterized in that Also includes: In response to the environmental signal indicating a second task mode, receiving a second training data associated with the second task mode; Setting a second task-specific layer of the neural network model according to the second task mode, wherein a second gate switch of the second task-specific layer determines a second sub-model of the neural network model; The neural network model and the second gate switch of the second task specific layer are updated according to the second training data.

19. The machine learning method according to claim 14, It is characterized in that The neural network model is the YOLOR model.

20. An electronic device, It is characterized in that include: Storage medium, storing multiple modules; as well as A processor is coupled to the storage medium, and the processor is configured to execute the plurality of modules, wherein the plurality of modules include: An environmental sensing module detects an environmental signal; An image extraction module extracts an input image; The inference module determines a task mode according to the environmental signal. A sub-model of a neural network model is used to extract features from the input image according to the task mode, wherein the sub-model of the neural network model includes a task-specific layer corresponding to the task mode, and a gate switch of the task-specific layer determines the sub-model of the neural network model, and Output the object detection results corresponding to the task mode.

Images