Object recognition method and apparatus

Abstract

The application relates to the field of artificial intelligence. Specifically, it relates to the field of computer vision, and discloses a perception network based on multiple Headers, which comprises a backbone network and multiple parallel Headers connected with the backbone network. The backbone network is used for receiving an input picture, performing convolution processing on the input picture, and outputting feature maps with different resolutions corresponding to the picture. Each Header in the multiple parallel Headers is used for detecting a task object in a task according to the feature maps output by the backbone network, and outputting a 2D frame of a region where the task object is located and a confidence corresponding to each 2D frame. Each parallel Header completes detection of different task objects. The task object is an object that needs to be detected in the task. The higher the confidence is, the greater the probability that a task object corresponding to the task exists in the 2D frame corresponding to the confidence.

Description

[0001] This application is a divisional application. The application number of the original application is 201910493331.6, and the original application date is June 6, 2019. The entire content of the original application is incorporated into this application by reference. Technical Field

[0002] The present application relates to the field of artificial intelligence, and in particular to an object recognition method and device. Background Art

[0003] Computer vision is an integral part of various intelligent / autonomous systems in various application areas, such as manufacturing, inspection, document analysis, medical diagnosis, and the military. It involves the use of cameras and computers to acquire the data and information we need about the objects being photographed. Figuratively speaking, it's like equipping a computer with eyes (cameras) and a brain (algorithms) to replace the human eye in identifying, tracking, and measuring objects, thus enabling the computer to perceive its environment. Because perception can be viewed as extracting information from sensory signals, computer vision can also be considered the study of how artificial systems can "perceive" from images or multidimensional data. In short, computer vision uses various imaging systems to replace the visual organs to acquire input information, and then allows computers to replace the brain to process and interpret this input information. The ultimate goal of computer vision research is to enable computers to observe and understand the world through vision like humans do, and to have the ability to autonomously adapt to their environments.

[0004] Currently, visual perception networks are capable of performing an increasing number of functions, including image classification, 2D detection, semantic segmentation (masking), key point detection, linear object detection (such as lane line or stop line detection in autonomous driving technology), and drivable area detection. Furthermore, visual perception systems are characterized by low cost, non-contact operation, small size, and large information content. As the accuracy of visual perception algorithms continues to improve, they have become a key technology in many current artificial intelligence systems and are being increasingly widely used. For example, in advanced driver assistance systems (ADAS) and autonomous driving systems (ADS), dynamic obstacles (people or vehicles) and static objects (traffic lights, traffic signs, or traffic cones) on the road are recognized, and in the terminal vision camera beauty function, a slimming effect is achieved by recognizing masks and key points on the human body.

[0005] Most current mainstream visual perception networks focus on a single detection task, such as 2D detection, 3D detection, semantic segmentation, and keypoint detection. Implementing multiple functions often requires different networks. Running multiple networks simultaneously significantly increases hardware computational complexity and power consumption, slowing down the model's execution speed and making real-time detection difficult. Summary of the Invention

[0006] In order to reduce the amount of hardware calculation and power consumption and improve the operation speed of the perception network model, an embodiment of the present invention provides a perception network based on multiple headers. The perception network includes a backbone network and multiple parallel headers, and the multiple parallel headers are connected to the backbone network.

[0007] The backbone network is used to receive an input image, perform convolution processing on the input image, and output feature maps with different resolutions corresponding to the image;

[0008] The parallel header is used to detect a task object in a task based on the feature map output by the backbone network, and output a 2D box of the area where the task object is located and the confidence level corresponding to each 2D box. Each parallel header detects a different task object. The task object is the object that needs to be detected in the task. The higher the confidence level, the greater the probability that the object corresponding to the task exists in the 2D box corresponding to the confidence level. The parallel header is any one of the multiple parallel headers mentioned above, and the functions of each parallel header are similar.

[0009] Optionally, each parallel head end includes a candidate region generation network (RPN) module, a region of interest extraction (ROI-ALIGN) module, and a regional convolutional neural network (RCNN) module. The RPN module of one parallel head end is independent of the RPN modules of other parallel head ends; the ROI-ALIGN module of one parallel head end is independent of the ROI-ALIGN modules of other parallel head ends; and the RCNN module of one parallel head end is independent of the RCNN modules of other parallel head ends. For each parallel head end:

[0010] The RPN module is used to predict the area where the task object is located on one or more feature maps provided by the backbone network, and output a candidate 2D box matching the area;

[0011] The ROI-ALIGN module is used to: extract features of the region where the candidate 2D frame is located from a feature map provided by the backbone network according to the region predicted by the RPN module;

[0012] The RCNN module is used to: perform convolution processing on the features of the area where the candidate 2D box is located through a neural network to obtain the confidence that the candidate 2D box belongs to each object category; the each object category is the object category in the task corresponding to the one parallel head end; adjust the coordinates of the 2D box of the candidate area through the neural network so that the adjusted 2D candidate box is more closely matched to the shape of the actual object than the candidate 2D box, and select the adjusted 2D candidate box with a confidence greater than a preset threshold as the 2D box of the area.

[0013] Optionally, the 2D frame is a rectangular frame.

[0014] Optionally, in another aspect of an embodiment of the present application, the RPN module is used to: based on the template frame (Anchor) of the object corresponding to the task, predict the area where the task object exists on one or more feature maps provided by the backbone network to obtain a candidate area, and output a candidate 2D frame matching the candidate area; wherein the template frame is obtained based on the statistical features of the task object to which it belongs, and the statistical features include the shape and size of the object.

[0015] Optionally, in another aspect of the embodiment of the present application, the sensing network further includes at least one or more serial headers; the serial headers are connected to the one parallel header;

[0016] The serial header is used to: use the 2D frame of the task object of the task to which it belongs provided by the parallel header to which it is connected, extract the features of the area where the 2D frame is located on one or more feature maps on the backbone network, and predict the 3D information, Mask information or Keypiont information of the task object of the task to which it belongs based on the features of the area where the 2D frame is located.

[0017] Optionally, the RPN module predicts regions where objects of different sizes are located on feature maps of different resolutions.

[0018] Optionally, the RPN module completes detection of the area where large objects are located on low-resolution features, and the RPN module completes detection of the area where small objects are located on high-resolution feature maps.

[0019] On the other hand, an embodiment of the present invention further provides an object detection method, the method comprising:

[0020] Receive input image;

[0021] Perform convolution on the input image and output feature maps with different resolutions corresponding to the image;

[0022] According to the feature map, the task objects in each task are independently detected for different tasks, and a 2D box of the area where each task object is located and the confidence corresponding to each 2D box are output; wherein the task object is an object that needs to be detected in the task; the higher the confidence, the greater the probability that the object corresponding to the task is in the 2D box corresponding to the confidence.

[0023] Optionally, the detecting, for different tasks, independently detecting the task objects in each task based on the feature map, and outputting a 2D box of the area where each task object is located and a confidence score corresponding to each 2D box includes:

[0024] Predict the area where the task object is located on one or more feature maps, and output a candidate 2D box matching the area;

[0025] According to the area where the task object is located, extracting the features of the area where the candidate 2D box is located from a feature map;

[0026] Performing convolution processing on the features of the region where the candidate 2D frame is located to obtain confidence that the candidate 2D frame belongs to each object category; each object category is an object category in the one task;

[0027] The coordinates of the 2D frame of the candidate area are adjusted through a neural network so that the adjusted 2D candidate frame better matches the shape of the actual object than the candidate 2D frame, and the adjusted 2D candidate frame with a confidence greater than a preset threshold is selected as the 2D frame of the area.

[0028] Optionally, the 2D frame is a rectangular frame.

[0029] Optionally, the region where the task object is located is predicted on one or more feature maps, and a candidate 2D box matching the region is output as:

[0030] Based on the template frame (Anchor) of the object corresponding to the task, the area where the task object exists is predicted on one or more feature maps provided by the backbone network to obtain a candidate area, and a candidate 2D frame matching the candidate area is output; wherein the template frame is obtained based on the statistical features of the task object to which it belongs, and the statistical features include the shape and size of the object.

[0031] Optionally, the method further includes:

[0032] Based on the 2D frame of the task object of the task, the features of the area where the 2D frame is located are extracted on one or more feature maps on the backbone network, and the 3D information, Mask information or Keypiont information of the task object of the task is predicted according to the features of the area where the 2D frame is located.

[0033] Optionally, the detection of the area where the large object is located is completed on the low-resolution feature map, and the RPN module completes the detection of the area where the small object is located on the high-resolution feature map.

[0034] On the other hand, an embodiment of the present application provides a method for training a multi-task perception network based on partially labeled data, wherein the perception network includes a backbone network and multiple parallel heads. The method includes:

[0035] Determine the task to which each image belongs based on the labeled data type of each image; wherein each image is labeled with one or more data types, the multiple data types are a subset of all data types, and each data type corresponds to one task;

[0036] According to the task to which each image belongs, determine the Header that needs to be trained for each image;

[0037] Calculate the loss value of the Header required for training for each picture;

[0038] For each image, the gradient is returned through the header to be trained, and the parameters of the header to be trained and the backbone network are adjusted based on the loss value.

[0039] Optionally, data balancing is performed on images belonging to different tasks.

[0040] An embodiment of the present invention further provides a device for training a multi-task perception network based on partially labeled data, wherein the perception network includes a backbone network and multiple parallel heads. The device includes:

[0041] A task determination module, configured to determine the task to which each image belongs based on the data type annotated on each image; wherein each image is annotated with one or more data types, the multiple data types being a subset of all data types, and each data type corresponds to one task;

[0042] The Header decision module is used to determine the Header required to be trained for each image based on the task to which the image belongs;

[0043] The loss value calculation module is used to calculate the loss value of the header determined by the header decision module for each image;

[0044] The adjustment module calculates the gradient of the header determined by the header determination module for each image, and adjusts the parameters of the header and backbone network to be trained based on the loss value obtained by the loss value calculation module.

[0045] Optionally, the device further includes: a data balancing module, configured to perform data balancing on images belonging to different tasks.

[0046] An embodiment of the present invention further provides a perception network application system, comprising at least one processor, at least one memory, at least one communication interface, and at least one display device. The processor, memory, display device, and communication interface are connected via a communication bus and communicate with each other.

[0047] Communication interface, used for communicating with other devices or communication networks;

[0048] The memory is used to store application code for executing the above solution, and the execution is controlled by the processor. The processor is used to execute the application code stored in the memory.

[0049] The code stored in the memory 2002 may execute the above-provided method for perceiving an object based on a Multi-Header, or may be the method for training the perception network provided in the above-mentioned embodiment.

[0050] The display device is used to display the image to be recognized, the 2D, 3D, mask, key points and other information of the object of interest in the image.

[0051] The perception network provided by the embodiments of this application uses the same backbone network for all perception tasks, significantly reducing computational effort and improving the speed of the perception network model. The network structure is easily scalable, and 2D detection types can be expanded by simply adding one or more headers. Each parallel header has independent RPN and RCNN modules and only needs to detect objects for its task. This avoids accidentally detecting unlabeled objects from other tasks during training.

[0052] These and other aspects of the present application will become more readily apparent from the description of the following embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the embodiments of the invention of this application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0054] Figure 1 A schematic diagram of the system architecture provided in the embodiment of the present application;

[0055] Figure 2A schematic diagram of a CNN feature extraction model provided in an embodiment of the present application;

[0056] Figure 3 This is a schematic diagram of a chip hardware structure provided by an embodiment of the present application;

[0057] Figure 4 A schematic diagram of a perception network application system framework based on multiple parallel headers provided in an embodiment of the present application;

[0058] Figure 5 A schematic diagram of a sensing network structure based on multiple parallel headers provided in an embodiment of the present application;

[0059] Figure 6 A schematic diagram of the structure of an ADAS / AD perception system based on multiple parallel Headers provided in an embodiment of the present application;

[0060] Figure 7 A schematic diagram of a basic feature generation process provided in an embodiment of the present application;

[0061] Figure 8 A schematic diagram of the structure of another RPN layer provided in an embodiment of the present application;

[0062] Figure 9 A schematic diagram of the corresponding anchors of another RPN layer object provided in an embodiment of the present application;

[0063] Figure 10 A schematic diagram of another ROI-ALIGN process provided in an embodiment of the present application;

[0064] Figure 11 Schematic diagram of another RCNN implementation and structure provided in the embodiment of the present application;

[0065] Figure 12 This is another implementation and structural diagram of a serial header provided in an embodiment of the present application;

[0066] Figure 13 This is another implementation and structural diagram of a serial header provided in an embodiment of the present application;

[0067] Figure 14 A schematic diagram of the implementation and structure of a serial header provided in an embodiment of the present application;

[0068] Figure 15 A schematic diagram of a training method for partially labeled data provided in an embodiment of the present application;

[0069] Figure 16 A schematic diagram of another training method for partially labeled data provided in an embodiment of the present application;

[0070] Figure 17 A schematic diagram of another training method for partially labeled data provided in an embodiment of the present application;

[0071] Figure 18 A schematic diagram of another training method for partially labeled data provided in an embodiment of the present application;

[0072] Figure 19 A schematic diagram of an application of a perception network based on multiple parallel headers provided in an embodiment of the present application;

[0073] Figure 20 A schematic diagram of an application of a perception network based on multiple parallel headers provided in an embodiment of the present application;

[0074] Figure 21 A flowchart of a sensing method provided in an embodiment of the present application;

[0075] Figure 22 A schematic diagram of a 2D detection process provided in an embodiment of the present application;

[0076] Figure 23 A schematic diagram of a 3D detection process of a terminal device provided in an embodiment of the present application;

[0077] Figure 24 A schematic diagram of a Mask prediction process provided in an embodiment of the present application;

[0078] Figure 25 A schematic diagram of a key point coordinate prediction process provided in an embodiment of the present application;

[0079] Figure 26 A schematic diagram of a training process of a perception network provided in an embodiment of the present application;

[0080] Figure 27 A schematic diagram of a perception network implementation structure based on multiple parallel headers provided in an embodiment of the present application;

[0081] Figure 28 A schematic diagram of a perception network implementation structure based on multiple parallel headers provided in an embodiment of the present application;

[0082] Figure 29 A diagram of a device for training a multi-task perception network based on partially labeled data provided in an embodiment of the present application;

[0083] Figure 30 A flowchart of an object detection method provided in an embodiment of the present application;

[0084] Figure 31This is a flowchart of an embodiment of the present application for training a multi-task perception network based on partially labeled data. DETAILED DESCRIPTION

[0085] First, the abbreviations used in the examples of this application are listed as follows:

[0086]

[0087] Table 1

[0088] It should be noted that some of the drawings in the embodiments of the present invention use English descriptions to better conform to the terminology used in the industry, and the corresponding Chinese definitions are also given in the embodiments. The embodiments of the present application are described below in conjunction with the drawings.

[0089] The embodiment of the present application is mainly used in the fields of driving assistance, automatic driving, mobile terminals, etc. that need to complete multiple perception tasks. The application system framework of the present invention is as follows Figure 4 As shown, a single image is obtained by extracting frames from the video. This image is fed into the Multi-Header perception network of the present invention to obtain 2D, 3D, Mask (mask), key point and other information of the object of interest in the image. These detection results are output to the post-processing module for processing. For example, in the autonomous driving system, they are sent to the planning control unit for decision-making, and in the mobile phone terminal, they are sent to the beauty algorithm for processing to obtain the beautified image. The following is a brief introduction to the two application scenarios of ADAS / ADS visual perception system and mobile phone beauty.

[0090] Application Scenario 1: ADAS / ADS Visual Perception System

[0091] like Figure 19As shown in the figure, ADAS and ADS require real-time detection of multiple types of 2D objects, including dynamic obstacles (pedestrians, cyclists, tricycles, cars, trucks, and buses), static obstacles (traffic cones, traffic sticks, fire hydrants, motorcycles, and bicycles), traffic signs, guide signs, billboards, red traffic lights (TrafficLight_Red), yellow traffic lights (TrafficLight_Yellow), green traffic lights (TrafficLight_Green), and black traffic lights (TrafficLight_Black), and road signs. Furthermore, to accurately determine the area occupied by dynamic obstacles in 3D space, 3D estimation of the dynamic obstacles is required, and 3D bounding boxes must be output. To fuse data with the LiDAR, it's necessary to obtain a mask for dynamic obstacles, thereby filtering out the laser point cloud that hits them. To accurately locate a parking space, it's necessary to simultaneously detect the four key points of the parking space. To perform composition positioning, it's necessary to detect the key points of static targets. Using the technical solutions provided in the embodiments of this application, all of the above functions can be accomplished within a single perception network.

[0092] Application scenario 2: mobile phone beauty function

[0093] like Figure 20 As shown, in a mobile phone, the mask and key points of the human body are detected through the perception network provided in the embodiment of the present application, and the corresponding parts of the human body can be enlarged or reduced, such as performing waist-tightening and buttock-beautifying operations, thereby outputting beauty pictures.

[0094] Application scenario 3: Image classification scenario

[0095] After acquiring an image to be classified, the object recognition device uses the object recognition method of this application to obtain the category of the object in the image to be classified. The image to be classified can then be classified based on the category of the object in the image to be classified. Photographers take many photos every day, including animals, people, and plants. Using the method of this application, photos can be quickly classified according to their content into photos containing animals, photos containing people, and photos containing plants.

[0096] When there are a large number of images, manual classification is inefficient, and people are easily fatigued when dealing with the same thing for a long time, and the classification results will have large errors. However, the method of the present application can quickly classify images without errors.

[0097] Application Scenario 4: Product Classification

[0098] After the object recognition device acquires an image of a product, it uses the object recognition method of this application to determine the category of the product in the image, and then classifies the product according to its category. For large shopping malls or supermarkets with a wide variety of products, the object recognition method of this application can quickly complete the classification of products, reducing time and labor costs.

[0099] The method and apparatus provided in the embodiments of the present application can also be used to expand the training database, such as Figure 1 The I / O interface 112 of the execution device 110 shown can send the image processed by the execution device (such as an image block or image containing an object) and the object category input by the user as a training data pair to the database 130, so that the training data maintained by the database 130 is richer, thereby providing richer training data for the training work of the training device 120.

[0100] The following describes the method provided by this application from the perspectives of model training and model application:

[0101] The method for training a CNN feature extraction model provided in an embodiment of the present application relates to computer vision processing, and can be specifically applied to data processing methods such as data training, machine learning, and deep learning. The training data (such as the image or image block of the object and the category of the object in this application) is subjected to symbolic and formalized intelligent information modeling, extraction, preprocessing, and training, and finally a trained CNN feature extraction model is obtained; and, in an embodiment of the present application, input data (such as the image of the object in this application) is input into the trained CNN feature extraction model to obtain output data (such as the 2D, 3D, Mask, key point, and other information of the object of interest in the image in this application).

[0102] Since the embodiments of the present application involve the application of a large number of neural networks, in order to facilitate understanding, the relevant terms and related concepts such as neural networks involved in the embodiments of the present application are first introduced below.

[0103] (1) Object recognition: using image processing, machine learning, computer graphics and other related methods to determine the category of the object in the image.

[0104] (2) Neural Network

[0105] A neural network can be composed of neural units. A neural unit can refer to an operation unit with xs and intercept 1 as input. The output of the operation unit can be:

[0106]

[0107] Where s = 1, 2, ... n, n is a natural number greater than 1, Ws is the weight of xs, and b is the bias of the neural unit. f is the activation function of the neural unit, which is used to introduce nonlinear characteristics into the neural network to convert the input signal in the neural unit into the output signal. The output signal of the activation function can be used as the input of the next convolutional layer. The activation function can be a sigmoid function. A neural network is a network formed by connecting many of the above-mentioned single neural units together, that is, the output of one neural unit can be the input of another neural unit. The input of each neural unit can be connected to the local receptive field of the previous layer to extract the features of the local receptive field. The local receptive field can be an area composed of several neural units.

[0108] (3) Deep Neural Networks

[0109] Deep Neural Network (DNN), also known as multi-layer neural network, can be understood as a neural network with many hidden layers. There is no special metric for "many" here. Based on the position of different layers in DNN, the neural network inside DNN can be divided into three categories: input layer, hidden layer, and output layer. Generally speaking, the first layer is the input layer, the last layer is the output layer, and the layers in between are all hidden layers. The layers are fully connected, that is, any neuron in the i-th layer must be connected to any neuron in the i+1-th layer. Although DNN looks complicated, the work of each layer is actually not complicated. Simply put, it is the following linear relationship expression: in, is the input vector, is the output vector, is the offset vector, W is the weight matrix (also called coefficient), and α() is the activation function. Each layer is just an input vector After such a simple operation, the output vector Since there are many DNN layers, the coefficient W and the offset vector The definition of these parameters in DNN is as follows: Take the coefficient W as an example: Assume that in a three-layer DNN, the linear coefficient from the 4th neuron in the second layer to the 2nd neuron in the third layer is defined as The superscript 3 represents the layer number of the coefficient W, while the subscript corresponds to the output of the third layer index 2 and the input of the second layer index 4. In summary, the coefficient from the kth neuron in the L-1th layer to the jth neuron in the Lth layer is defined as It's important to note that the input layer has no W parameter. In deep neural networks, more hidden layers allow the network to better capture complex real-world situations. Theoretically, a model with more parameters has higher complexity and greater "capacity," meaning it can handle more complex learning tasks. Training a deep neural network is essentially the process of learning the weight matrix, with the ultimate goal of obtaining the weight matrices for all layers of a trained deep neural network (a weight matrix formed by the vectors W across many layers).

[0110] (4) Convolutional Neural Networks

[0111] A convolutional neural network (CNN) is a deep neural network with a convolutional architecture. A CNN consists of a feature extractor consisting of convolutional layers and subsampling layers. This feature extractor can be viewed as a filter, and the convolution process can be thought of as convolving an input image or feature map with a trainable filter. A convolutional layer is the layer of neurons in a CNN that performs convolution on the input signal. Within a convolutional layer, a neuron can only connect to a subset of neurons in adjacent layers. A convolutional layer typically contains several feature planes, each composed of a rectangular arrangement of neurons. Neurons within the same feature plane share weights, referred to as the convolution kernel. Shared weights can be understood as ensuring that the method for extracting image information is independent of position. The underlying principle is that the statistical information of one part of the image is the same as that of another. This means that image information learned in one part can also be applied to other parts. Therefore, the same learned image information can be used at all positions in the image. In the same convolutional layer, multiple convolution kernels can be used to extract different image information. Generally speaking, the more convolution kernels there are, the richer the image information reflected by the convolution operation.

[0112] Convolution kernels can be initialized as matrices of random size, and during the training process of the convolutional neural network, the convolution kernels can be learned to obtain reasonable weights. In addition, the direct benefit of shared weights is that they reduce the number of connections between the layers of the convolutional neural network, while also reducing the risk of overfitting.

[0113] (5) Recurrent Neural Networks (RNN) are used to process sequence data. In traditional neural network models, the layers are fully connected from the input layer to the hidden layer and then to the output layer, while the nodes within each layer are disconnected. Although this ordinary neural network solves many difficult problems, it is still powerless to solve many problems. For example, if you want to predict the next word in a sentence, you generally need to use the previous word because the previous and next words in a sentence are not independent. The reason why RNN is called a recurrent neural network is that the current output of a sequence is also related to the previous output. The specific manifestation is that the network will remember the previous information and apply it to the calculation of the current output, that is, the nodes between the hidden layers are no longer disconnected but connected, and the input of the hidden layer includes not only the output of the input layer but also the output of the hidden layer at the previous moment. In theory, RNN can process sequence data of any length. The training of RNN is the same as the training of traditional CNN or DNN. This approach also uses the backpropagation algorithm, but with one key difference: if the RNN is expanded, its parameters, such as W, are shared; this is not the case with traditional neural networks, as in the example above. Furthermore, when using gradient descent, the output of each step depends not only on the state of the network at the current step but also on the state of the network at several previous steps. This learning algorithm is called Backpropagation Through Time (BPTT).

[0114] Since we already have convolutional neural networks, why do we still need recurrent neural networks? The reason is simple. Convolutional neural networks assume that elements are independent of each other, and that inputs and outputs are also independent, such as cats and dogs. However, in the real world, many elements are interconnected, such as the changes in stock prices over time. Or, for example, someone says, "I love traveling, and my favorite place is Yunnan. I must visit it someday." Humans should know to fill in the blank with "Yunnan." This is because humans make inferences based on context, but how can machines do the same? That's where RNNs come in. RNNs aim to give machines the ability to remember, like humans do. Therefore, the output of an RNN depends on both the current input and historical memory.

[0115] (6) Loss function

[0116] During the training of a deep neural network, because we want the output of the deep neural network to be as close as possible to the desired predicted value, we can compare the current network's predicted value with the desired target value and then update the weight vector of each layer of the neural network based on the difference between the two. (Of course, before the first update, there is usually an initialization process, which pre-configures the parameters for each layer in the deep neural network.) For example, if the network's predicted value is too high, the weight vector is adjusted to make it predict a lower value. This adjustment is continued until the deep neural network can predict the desired target value or a value very close to the desired target value. Therefore, it is necessary to predefine "how to compare the difference between the predicted value and the target value." This is the loss function (or objective function), which is an important equation used to measure the difference between the predicted value and the target value. For example, the loss function output value (loss) indicates a greater difference, so training a deep neural network becomes a process of minimizing this loss.

[0117] (7) Backpropagation algorithm

[0118] Convolutional neural networks can use the back propagation (BP) algorithm to correct the size of the parameters in the initial super-resolution model during training, reducing the reconstruction error loss of the super-resolution model. Specifically, the forward propagation of the input signal to the output generates an error loss. This error loss information is then backpropagated to update the parameters of the initial super-resolution model, thereby converging the error loss. The BP algorithm is a backward propagation movement dominated by the error loss, aiming to obtain the optimal super-resolution model parameters, such as the weight matrix.

[0119] The following describes the system architecture provided by the embodiments of the present application.

[0120] See also Figure 1, an embodiment of the present application provides a system architecture 100. As shown in the system architecture 100, the data acquisition device 160 is used to collect training data. In the embodiment of the present application, the training data includes: images or image blocks of objects and categories of objects; and the training data is stored in the database 130. The training device 120 obtains a CNN feature extraction model based on the training data maintained in the database 130 (Explanation: the CNN feature extraction model here is the model obtained through training in the training phase introduced above, which can be a perception network for feature extraction, etc.). The following will describe in more detail how the training device 130 obtains a CNN feature extraction model based on training data using Example 1. The CNN feature extraction model can be used to implement the perception network provided in the embodiment of the present application, that is, the image or image block to be identified is input into the CNN feature extraction model after relevant pre-processing, and the 2D, 3D, Mask, key points and other information of the object of interest in the image or image block to be identified can be obtained. The CNN feature extraction model in the embodiment of the present application can specifically be a CNN convolutional neural network. It should be noted that, in actual applications, the training data maintained in the database 130 may not all be collected by the data acquisition device 160, but may also be received from other devices. It should also be noted that the training device 120 may not train the CNN feature extraction model entirely based on the training data maintained by the database 130, but may also obtain training data from the cloud or other places for model training. The above description should not be used as a limitation on the embodiments of the present application.

[0121] The CNN feature extraction model trained by the training device 120 can be applied to different systems or devices, such as Figure 1 The execution device 110 shown in the figure can be a terminal, such as a mobile phone terminal, a tablet computer, a laptop computer, AR / VR, a vehicle terminal, etc., or a server or a cloud terminal. Figure 1 In the embodiment of the present application, the execution device 110 is configured with an I / O interface 112 for data interaction with an external device. The user can input data to the I / O interface 112 through the client device 140. The input data may include: an image to be identified, an image block, or a picture.

[0122] When the execution device 110 preprocesses the input data, or when the computing module 111 of the execution device 110 performs calculations and other related processing (such as implementing the functions of the perception network in this application), the execution device 110 can call the data, code, etc. in the data storage system 150 for corresponding processing, and can also store the data, instructions, etc. obtained from the corresponding processing in the data storage system 150.

[0123] Finally, the I / O interface 112 returns the processing results, such as the image or image block obtained above, or the 2D, 3D, mask, key point and other information of the object of interest in the picture, to the client device 140 for providing to the user.

[0124] Optionally, the client device 140 may be a planning control unit in an autonomous driving system or a beautification algorithm module in a mobile phone terminal.

[0125] It is worth noting that the training device 120 can generate corresponding target models / rules based on different training data for different goals or different tasks. The corresponding target models / rules can be used to achieve the above goals or complete the above tasks, thereby providing users with the desired results.

[0126] In the attached Figure 1 In the case shown in FIG, the user can manually input data, which can be operated through the interface provided by I / O interface 112. In another case, client device 140 can automatically send input data to I / O interface 112. If the automatic transmission of input data by client device 150 requires user authorization, the user can set the corresponding permissions in client device 140. The user can view the results output by execution device 110 on client device 140, which can be presented in a specific form such as display, sound, action, etc. Client device 140 can also serve as a data acquisition terminal, collecting input data input into I / O interface 112 and output results from I / O interface 112 as new sample data and storing them in database 130. Of course, it is also possible to bypass client device 150 for collection, and instead have I / O interface 112 directly store the input data input into I / O interface 112 and output results from I / O interface 112 as new sample data in database 130.

[0127] It is worth noting that the Figure 1 This is only a schematic diagram of a system architecture provided by an embodiment of the present invention. The positional relationship between the devices, components, modules, etc. shown in the figure does not constitute any limitation. For example, in the attached Figure 1 In the embodiment, the data storage system 150 is an external memory relative to the execution device 110. In other cases, the data storage system 150 can also be placed in the execution device 110.

[0128] like Figure 1 As shown, a CNN feature extraction model is obtained through training with the training device 120. The CNN feature extraction model can be a CNN convolutional neural network in the embodiment of the present application or a perception network based on multiple headers to be introduced in the following embodiment.

[0129] As mentioned in the previous basic concepts, a convolutional neural network is a deep neural network with a convolutional structure. It is a deep learning architecture, which uses machine learning algorithms to perform multiple levels of learning at different levels of abstraction. As a deep learning architecture, a CNN is a feed-forward artificial neural network in which each neuron responds to an image input.

[0130] like Figure 2 As shown, the convolutional neural network (CNN) 200 may include an input layer 210 , a convolutional layer / pooling layer 220 (wherein the pooling layer is optional), and a neural network layer 230 .

[0131] Convolutional layer / pooling layer 220:

[0132] Convolutional layer:

[0133] like Figure 2 The convolutional layer / pooling layer 220 shown may include layers 221-226. For example, in one implementation, layer 221 is a convolutional layer, layer 222 is a pooling layer, layer 223 is a convolutional layer, layer 224 is a pooling layer, layer 225 is a convolutional layer, and layer 226 is a pooling layer. In another implementation, layers 221 and 222 are convolutional layers, layer 223 is a pooling layer, layers 224 and 225 are convolutional layers, and layer 226 is a pooling layer. That is, the output of a convolutional layer can be used as the input of a subsequent pooling layer, or as the input of another convolutional layer to continue the convolution operation.

[0134] The following will take convolutional layer 221 as an example to introduce the internal working principle of a convolutional layer.

[0135] Convolution layer 221 can include multiple convolution operators, also known as kernels. In image processing, a convolution operator acts as a filter that extracts specific information from an input image matrix. A convolution operator is essentially a weight matrix, which is typically predefined. During the convolution operation, the weight matrix is ​​typically applied horizontally to the input image, pixel by pixel (or two pixels by two pixels, depending on the stride), thereby extracting specific features from the image. The size of the weight matrix should be related to the image size. It is important to note that the depth dimension of the weight matrix is ​​the same as the depth dimension of the input image. During the convolution operation, the weight matrix extends across the entire depth of the input image. Therefore, convolution with a single weight matrix produces a convolution output with a single depth dimension. However, in most cases, a single weight matrix is ​​not used. Instead, multiple weight matrices of the same size (rows × columns) are applied, i.e., multiple homogeneous matrices. The outputs of each weight matrix are stacked to form the depth dimension of the convolved image, where the dimension is determined by the "multiple" mentioned above. Different weight matrices can be used to extract different features from an image. For example, one weight matrix can be used to extract edge information, another weight matrix can be used to extract specific colors, and yet another weight matrix can be used to blur unwanted noise in the image. The weight matrices have the same size (rows × columns), and the feature maps extracted by these weight matrices are also the same size. The extracted feature maps of the same size are then merged to form the output of the convolution operation.

[0136] The weight values ​​in these weight matrices need to be obtained through a lot of training in practical applications. The weight matrices formed by the weight values ​​obtained through training can be used to extract information from the input image, so that the convolutional neural network 200 can make correct predictions.

[0137] When the convolutional neural network 200 has multiple convolutional layers, the initial convolutional layer (for example, 221) often extracts more general features, which can also be called low-level features. As the depth of the convolutional neural network 200 increases, the features extracted by the later convolutional layers (for example, 226) become more and more complex, such as high-level semantic features. Features with higher semantics are more suitable for the problem to be solved.

[0138] Pooling layer:

[0139] Since it is often necessary to reduce the number of training parameters, it is often necessary to periodically introduce a pooling layer after the convolution layer, such as Figure 2Each layer 221-226 illustrated in Figure 220 can be a convolutional layer followed by a pooling layer, or multiple convolutional layers followed by one or more pooling layers. In the image processing process, the sole purpose of the pooling layer is to reduce the spatial size of the image. The pooling layer can include an average pooling operator and / or a maximum pooling operator to sample the input image to obtain a smaller image. The average pooling operator can calculate the pixel values ​​in the image within a specific range to produce an average value as the result of average pooling. The maximum pooling operator can take the pixel with the largest value within a specific range as the result of maximum pooling. In addition, just as the size of the weight matrix used in the convolutional layer should be related to the image size, the operators in the pooling layer should also be related to the image size. The size of the image output after processing by the pooling layer can be smaller than the size of the image input to the pooling layer. Each pixel in the image output by the pooling layer represents the average value or maximum value of the corresponding sub-region of the image input to the pooling layer.

[0140] Neural Network Layer 230:

[0141] After being processed by the convolution layer / pooling layer 220, the convolution neural network 200 is not sufficient to output the required output information. As mentioned above, the convolution layer / pooling layer 220 only extracts features and reduces the parameters brought by the input image. However, in order to generate the final output information (the required class information or other related information), the convolution neural network 200 needs to use the neural network layer 230 to generate one or a group of outputs of the required number of classes. Therefore, the neural network layer 230 may include multiple hidden layers (such as Figure 2 231, 232 to 23n) and the output layer 240 shown, the parameters contained in the multiple hidden layers can be pre-trained based on relevant training data of specific task types, for example, the task types may include image recognition, image classification, image super-resolution reconstruction, etc.

[0142] After the multiple hidden layers in the neural network layer 230, that is, the last layer of the entire convolutional neural network 200 is the output layer 240, which has a loss function similar to the classification cross entropy, specifically used to calculate the prediction error. Once the forward propagation of the entire convolutional neural network 200 (such as Figure 2 The propagation from 220 to 240 is forward propagation) and the reverse propagation (such as Figure 2 The propagation from 240 to 220 is called back propagation) and the weight values ​​and biases of the aforementioned layers will begin to be updated to reduce the loss of the convolutional neural network 200 and the error between the result output by the convolutional neural network 200 through the output layer and the ideal result.

[0143] It should be noted that if Figure 2The convolutional neural network 200 shown is only an example of a convolutional neural network. In specific applications, the convolutional neural network may also exist in the form of other network models.

[0144] The following describes a chip hardware structure provided by an embodiment of the present application.

[0145] Figure 3 The embodiment of the present invention provides a chip hardware structure, which includes a neural network processor 30. The chip can be set as follows Figure 1 The execution device 110 shown in FIG. 1 is used to complete the calculation work of the calculation module 111. The chip can also be set in Figure 1 The training device 120 shown in FIG. 1 is used to complete the training work of the training device 120 and output the target model / rule. Figure 2 The algorithms of each layer in the convolutional neural network shown in the figure can be used in Figure 3 is implemented in the chip shown.

[0146] The neural network processor NPU 30 is mounted on the host CPU as a coprocessor and is assigned tasks by the host CPU. The core of the NPU is the arithmetic circuit 303. The controller 304 controls the arithmetic circuit 303 to extract data from the memory (weight memory or input memory) and perform calculations.

[0147] In some implementations, the arithmetic circuit 303 includes multiple processing units (PEs). In some implementations, the arithmetic circuit 303 is a two-dimensional systolic array. The arithmetic circuit 303 can also be a one-dimensional systolic array or other electronic circuitry capable of performing mathematical operations such as multiplication and addition. In some implementations, the arithmetic circuit 303 is a general-purpose matrix processor.

[0148] For example, assume there are input matrix A, weight matrix B, and output matrix C. The computation circuit retrieves the corresponding data of matrix B from weight memory 302 and caches it on each PE in the computation circuit. The computation circuit then retrieves the data of matrix A from input memory 301 and performs a matrix operation on it with matrix B. The partial or final matrix result is stored in accumulator 308.

[0149] The vector calculation unit 307 can further process the output of the operation circuit, such as vector multiplication, vector addition, exponential operation, logarithmic operation, size comparison, etc. For example, the vector calculation unit 307 can be used for network calculations of non-convolutional / non-FC layers in a neural network, such as pooling, batch normalization, local response normalization, etc.

[0150] In some implementations, the vector calculation unit 307 can store the processed output vector to the unified buffer 306. For example, the vector calculation unit 307 can apply a nonlinear function to the output of the operation circuit 303, such as a vector of accumulated values, to generate an activation value. In some implementations, the vector calculation unit 307 generates a normalized value, a merged value, or both. In some implementations, the processed output vector can be used as an activation input to the operation circuit 303, for example, for use in a subsequent layer in a neural network.

[0151] The operation of the perception network provided in the embodiment of the present application can be performed by 303 or 307.

[0152] The unified memory 306 is used to store input data and output data.

[0153] The weight data is directly transferred from the external memory to the input memory 301 and / or the unified memory 306 through the memory unit access controller 305 (Direct Memory Access Controller, DMAC), the weight data in the external memory is stored in the weight memory 302, and the data in the unified memory 306 is stored in the external memory.

[0154] The bus interface unit (BIU) 310 is used to implement interaction between the main CPU, DMAC and instruction fetch memory 309 through the bus.

[0155] An instruction fetch buffer 309 connected to the controller 304 for storing instructions used by the controller 304;

[0156] The controller 304 is used to call the instructions cached in the memory 309 to control the working process of the computing accelerator.

[0157] Optionally, the input data here in this application is a picture, and the output data is 2D, 3D, Mask, key point and other information of the object of interest in the picture.

[0158] Generally, the unified memory 306, the input memory 301, the weight memory 302 and the instruction fetch memory 309 are all on-chip memories, and the external memory is a memory outside the NPU, which can be a double data rate synchronous dynamic random access memory (DDR SDRAM), a high bandwidth memory (HBM) or other readable and writable memory.

[0159] Figure 1 and Figure 2 The program algorithm is completed by the main CPU and NPU together.

[0160] in, Figure 2 The operations of each layer in the convolutional neural network shown can be performed by the operation circuit 303 or the vector calculation unit 307.

[0161] See also Figure 5 , Figure 5 This is a schematic diagram of the structure of a multi-head sensing network provided in an embodiment of the present application. Figure 5 As shown, the perception network includes:

[0162] It mainly consists of two parts: the backbone network (Backbone) and multiple parallel Headers 0~N.

[0163] The backbone network is used to receive an input image, perform convolution processing on the input image, and output feature maps with different resolutions corresponding to the image; that is, to output feature maps of different sizes corresponding to the image;

[0164] In other words, Backbone completes the extraction of basic features and provides corresponding features for subsequent detection.

[0165] Any parallel head end is used to detect the task object in a task based on the feature map output by the backbone network, and output the 2D box of the area where the task object is located and the confidence corresponding to each 2D box; wherein each parallel Header completes the detection of different task objects; wherein the task object is the object that needs to be detected in the task; the higher the confidence, the greater the probability that the object corresponding to the task is in the 2D box corresponding to the confidence.

[0166] That is to say, parallel headers complete different 2D detection tasks. For example, parallel header 0 completes car detection and outputs the 2D box and confidence of car, truck, and bus; parallel header 1 completes person detection and outputs the 2D box and confidence of pedestrian, cyclist, and tricyle; parallel header 2 completes traffic light detection and outputs the 2D box and confidence of red_trafficlight, green_trafficlight, yellow_trafficlight, and black_trafficlight.

[0167] Alternatively, as Figure 5 As shown in FIG, the sensing network may further include multiple serial headers, and the sensing network may further include at least one or more serial head ends; the serial head ends are connected to a parallel head end; it should be emphasized here that although Figure 5 In order to better display, multiple serial headers are drawn, but in fact, serial headers are not necessary. For scenes that only need to detect 2D boxes, there is no need to include serial headers.

[0168] The serial header is used to: use the 2D frame of the task object of the task to which it belongs provided by the parallel header to which it is connected, extract the features of the area where the 2D frame is located on one or more feature maps on the backbone network, and predict the 3D information, Mask information or Keypiont information of the task object of the task to which it belongs based on the features of the area where the 2D frame is located.

[0169] The serial header is optionally connected in series behind the parallel header. Based on the detection of the 2D box of the task, the 3D / Mask / Keypoint detection of the object inside the 2D box is completed.

[0170] For example, the serial 3D_Header0 completes the estimation of the vehicle's orientation, center of mass, length, width and height, thereby outputting the vehicle's 3D frame; the serial Mask_Header0 predicts the vehicle's fine mask, thereby segmenting the vehicle; and the serial Keypont_Header0 completes the estimation of the vehicle's key points.

[0171] Serial headers are optional. Some tasks don't require 3D / Mask / Keypoint detection, so serial headers don't need to be connected. For example, traffic light detection only requires 2D bounding boxes, so serial headers are not required. Additionally, some tasks may require one or more serial headers depending on their specific needs. For example, parking lot detection requires not only 2D bounding boxes but also key points of parking spaces. Therefore, only one serial Keypoint_Header is needed for this task, and 3D and Mask headers are not required.

[0172] Each module is described in detail below.

[0173] Backbone: The backbone network performs a series of convolutions on the input image to generate feature maps at different scales. These feature maps provide the basis for the subsequent detection module. Backbone networks can take various forms, such as VGG (Visual Geometry Group), Resnet (Residual Neural Network), and Inception-net (the core structure of Google LeNet).

[0174] Parallel Header: Parallel Header mainly completes the 2D box detection of a task based on the basic features provided by Backbone, and outputs the 2D box of the object of this task and the corresponding confidence level.

[0175] Optionally, the parallel Header of each task includes three modules: RPN, ROI-ALIGN and RCNN.

[0176] RPN module: used to predict the area where the task object is located on one or more feature maps provided by the backbone network, and output candidate 2D boxes matching the area;

[0177] Or you can understand it this way, RPN stands for Region Proposal Network, which predicts the areas where the task object may exist on one or more feature maps of Backbone and gives the boxes of these areas, which are called proposal areas.

[0178] For example, when parallel Header0 is responsible for detecting cars, its RPN layer predicts candidate boxes where cars may exist; when parallel Header1 is responsible for detecting people, its RPN layer predicts candidate boxes where people may exist. Of course, these proposals are inaccurate. On the one hand, they may not contain the objects of the task, and on the other hand, these boxes are not compact.

[0179] ROI-ALIGN module: used to extract the features of the region where the candidate 2D box is located from a feature map provided by the backbone network based on the region predicted by the RPN module;

[0180] In other words, the ROI-ALIGN module extracts the features of the region where each proposal is located on a feature map in Backbone based on the proposal provided by the RPN module, and resizes it to a fixed size to obtain the features of each proposal. It is understood that the ROI-ALIGN module can use, but is not limited to, feature extraction methods such as ROI-POOLING (region of interest pooling), ROI-ALIGN (region of interest extraction), PS-ROIPOOLING (position-sensitive region of interest pooling), and PS-ROIALIGN (position-sensitive region of interest extraction).

[0181] RCNN module: used to perform convolution processing on the features of the area where the candidate 2D box is located through a neural network to obtain the confidence that the candidate 2D box belongs to each object category; the each object category is the object category in the task corresponding to the one parallel head end; adjust the coordinates of the 2D box of the candidate area through the neural network so that the adjusted 2D candidate box is more closely matched to the shape of the actual object than the candidate 2D box, and select the adjusted 2D candidate box with a confidence greater than a preset threshold as the 2D box of the area.

[0182] In other words, the RCNN module primarily refines the features of each proposal generated by the ROI-ALIGN module, deriving confidence scores for each proposal in each category (for example, for the car task, four scores are given: background, car, truck, and bus). It also adjusts the coordinates of the proposal's 2D bounding box to output a more compact 2D bounding box. These 2D bounding boxes are then merged using NMS (Non Maximum Suppression) and output as the final 2D bounding box.

[0183] As mentioned above, in some practical application scenarios, the perception network can also include a serial header. The serial header is mainly connected in series with the parallel header. Based on the 2D frame detection, it further performs 3D / Mask / Keypoint detection. Therefore, there are three types of serial headers:

[0184] Serial 3D Header: Based on the 2D boxes provided by the front-end parallel header (the 2D boxes at this time are accurate and compact 2D boxes), the serial 3D header extracts the features of the area where these 2D boxes are located on a feature map of Backbone through the ROI-ALIGN module, and then passes it through a small network ( Figure 5 The 3D_Header in the 2D box is used to regress the center of mass coordinates, orientation angle, length, width and height of the object inside the 2D box, thereby obtaining complete 3D information.

[0185] Serial Mask Header: The serial Mask Header extracts the features of the area where these 2D boxes are located on a feature map of Backbone through the ROI-ALIGN module based on the 2D boxes provided by the front-end parallel header (the 2D boxes at this time are accurate and compact 2D boxes), and then passes through a small network ( Figure 5 The Mask_Header in the 2D box is used to return the mask of the object inside the 2D box, thereby segmenting the object.

[0186] Serial Keypoint Header: The serial Keypoint Header extracts the features of the area where these 2D boxes are located on a feature map of Backbone through the ROI-ALIGN module based on the 2D box provided by the front-end parallel header (the 2D box at this time is an accurate and compact 2D box), and then passes it through a small network ( Figure 5 The key point coordinates of the object inside the 2D box are returned by the Keypoint_Header in the 2D box.

[0187] Alternatively, as Figure 29 As shown, an embodiment of the present invention further provides a device for training a multi-task perception network based on partially labeled data. The perception network includes a backbone network and multiple parallel heads. The structure of the perception network has been described in detail in the previous embodiment and will not be repeated here. The device includes:

[0188] A task determination module 2900 is configured to determine the task to which each image belongs based on the data type annotated on each image; wherein each image is annotated with one or more data types, the multiple data types being a subset of all data types, and each data type corresponds to one task;

[0189] Header determination module 2901, used to determine the header required to be trained for each image according to the task to which each image belongs determined by task determination module 2900;

[0190] The loss value calculation module 2902 is used to calculate the loss value of the header determined by the header determination module 2901 for each image;

[0191] The adjustment module 2903 performs gradient backpropagation on the header determined by the header determination module 2901 for each image, and adjusts the parameters of the header to be trained and the backbone network based on the loss value obtained by the loss value calculation module 2902.

[0192] Alternatively, in one embodiment, Figure 29 As shown in the dotted box, the device may further include:

[0193] The data balancing module 2904 is used to perform data balancing on images belonging to different tasks.

[0194] like Figure 6 As shown, the following takes the ADAS / AD visual perception system as an example to introduce the embodiment of the present invention in detail.

[0195] In the visual perception system of ADAS / AD, multiple types of 2D target detection need to be performed in real time, including: dynamic obstacles (Pedestrian, Cyclist, Tricycle, Car, Truck, Bus), static obstacles (TrafficCone, TrafficStick, FireHydrant, Motorcycle, Bicycle), and traffic signs (TrafficSign, GuideSign, Billboard). In addition, in order to accurately obtain the area occupied by the vehicle in 3D space, it is also necessary to perform 3D estimation of dynamic obstacles and output 3D frames. In order to fuse with the data of the lidar, it is necessary to obtain the mask of the dynamic obstacle, so as to filter out the laser point cloud that hits the dynamic obstacle; in order to accurately park, it is necessary to detect the four key points of the parking space at the same time. Using the technical solution provided by this embodiment, all the above functions can be completed in one network. This embodiment is described in detail below.

[0196] 1. Division of each Header task and overall network diagram

[0197] According to the similarity of the objects to be detected and the abundance and scarcity of training samples, in this embodiment, the 20 types of objects to be detected are divided into 8 major categories, as shown in Table 2.

[0198]

[0199] Table 2 Object categories and extended functions that each Header needs to detect

[0200] Based on business requirements, Header0 needs to complete not only 2D vehicle detection but also 3D and mask detection. Header1 needs to complete not only 2D person detection but also mask detection. Header2 needs to complete not only 2D parking space frame detection but also key point detection.

[0201] It should be noted that the task division in Table 2 is only an example in this embodiment. Different task divisions may be performed in other embodiments and are not limited to the task division in Table 2.

[0202] According to the task division in Table 2, the overall structure of the perception network in this embodiment is as follows: Figure 6 shown.

[0203] The perception network mainly consists of three parts: Backbone, Parallel Header and Serial Header. It should be noted that, as described in the previous embodiment, the serial header is not necessary. The reason has been described in the above embodiment and will not be repeated here. Among them, 8 parallel headers simultaneously complete the 8 major categories of 2D detection in Table 1. Several serial headers are connected in series after Header0~2 to further complete the 3D / Mask / Keypoint detection. Figure 6 It can be seen that the present invention can flexibly add and delete headers according to business requirements, thereby achieving different functional configurations.

[0204] 2. Basic feature generation

[0205] The basic feature generation process is Figure 6 The Backbone implementation performs convolution on the input image to generate several convolution feature maps of different scales. Each feature map is a H*W*C matrix, where H is the height of the feature map, W is the width of the feature map, and C is the number of channels of the feature map.

[0206] Backbone can use a variety of existing convolutional network frameworks, such as VGG16, Resnet50, Inception-Net, etc. The following uses Resnet18 as the backbone to illustrate the basic feature generation process. Figure 7 shown.

[0207] Assume that the resolution of the input image is H*W*3 (height H, width W, number of channels is 3, that is, three RBG channels). The input image is convolved by the first convolution module of Resnet18 (Res18-Conv1 in the figure, which consists of several convolutional layers, and the subsequent convolution modules are similar) to generate Featuremap (feature map) C1. This feature map is downsampled twice relative to the input image, and the number of channels is expanded to 64, so the resolution of C1 is H / 4*W / 4*64; C1 is convolved by the second convolution module (Res18-Conv2) of Resnet18 to obtain Featuremap C2, and the resolution of this feature map is consistent with C1; C2 continues to be processed by the third convolution module (Res18-Conv3) of Resnet18 to generate Featuremap C3. This feature map is further downsampled relative to C2, the number of channels is doubled, and its resolution is H / 8*W / 8*128; finally, C3 is processed by Res18-Conv4 to generate Featuremap C4, whose resolution is H / 16*W / 16*256.

[0208] from Figure 7 As can be seen, Resnet18 performs multiple layers of convolution on the input image, generating feature maps of different scales: C1 / C2 / C3 / C4. The bottom-level feature maps are larger in width and height and have fewer channels, primarily representing low-level image features (such as edges and textures). The higher-level feature maps are smaller in width and height and have more channels, primarily representing high-level image features (such as shape and object features). The subsequent 2D detection process will use these feature maps for further prediction.

[0209] 3. 2D candidate region prediction process

[0210] The 2D candidate region prediction process consists of Figure 6The implementation of the RPN module for each parallel header predicts the regions where the objects of the task may exist based on the feature maps (C1 / C2 / C3 / C4) provided by the Backbone, and gives the candidate boxes (also called candidate regions, Proposal) for these regions. In this embodiment, parallel header 0 is responsible for detecting vehicles, and its RPN layer predicts the candidate boxes where vehicles may exist; parallel header 1 is responsible for detecting people, and its RPN layer predicts the candidate boxes where people may exist, and so on, which will not be elaborated here.

[0211] The basic structure of the RPN layer is as Figure 8 shown. Through a 3*3 convolution on C4, the feature map RPNHidden is generated. The RPN layer of each subsequent parallel header will predict the Proposal from RPN Hidden. Specifically, the RPN layer of parallel header 0 predicts the coordinates and confidence of the Proposal at each position of RPN Hidden through two 1*1 convolutions respectively. The higher this confidence, the greater the probability that this Proposal exists the object of the task. For example, the larger the score of a certain Proposal in parallel header 0, the greater the probability that it exists a vehicle. The Proposal predicted by each RPN layer needs to pass through the Proposal merging module, and the redundant Proposal is removed according to the overlapping degree between the Proposals (this process can adopt but is not limited to the NMS algorithm), and the N (N<K) Proposals with the largest score are selected from the remaining K Proposals as the candidate regions where the objects may exist. From Figure 8 it can be seen that these Proposals are inaccurate. On the one hand, they may not contain the objects of the task, and on the other hand, these boxes are not compact. Therefore, the RPN module is only a rough detection process, and the subsequent RCNN module is needed for subdivision.

[0212] When the RPN module regresses the coordinates of the Proposal, it does not directly regress the absolute value of the coordinates, but regresses the coordinates relative to the Anchor. When these Anchors match the actual objects better, the probability that PRN can detect the objects is greater. In this invention, a framework with multiple headers is adopted, and the corresponding Anchors can be designed for the scale and aspect ratio of the objects of each RPN layer, so as to improve the recall rate of each PRN layer. As Figure 9 shown.

[0213] For parallel Header1, it is responsible for detecting people, and the main shape of people is slender, so the Anchor can be designed to be slender; for parallel Header4, it is responsible for detecting traffic signs, and the main shape of traffic signs is square, so the Anchor can be designed to be square.

[0214] 4. 2D candidate region feature extraction process

[0215] The 2D candidate region feature extraction process mainly consists of Figure 6 The ROI-ALIGN module in each parallel Header is implemented, which extracts the features of each Proposal on a feature map provided by Backbone based on the coordinates of the Proposal provided by the PRN layer. The ROI-ALIGN process is as follows: Figure 10 shown.

[0216] In this embodiment, features are extracted from Backbone's C4 feature map. Each Proposal has an area on C4 such as Figure 10 The dark area indicated by the arrow in the figure uses interpolation and sampling to extract fixed-resolution features. Assuming the number of proposals is N and the width and height of the features extracted by ROI-ALIGN are both 14, the size of the features output by ROI-ALIGN is N*14*14*256 (the number of channels of the features extracted by ROI-ALIGN is the same as that of C4, both 256 channels). These features are then sent to the subsequent RCNN module for segmentation.

[0217] 5. 2D candidate region subclassification

[0218] 2D candidate region subdivision is mainly composed of Figure 6 The RCNN module of each parallel Header is implemented, which further regresses the features of each Proposal extracted by the ROI-ALIGN module to obtain a more compact 2D box coordinate, and classifies the Proposal and outputs the confidence that it belongs to each category.

[0219] There are many possible implementations of RCNN, one of which is Figure 11 The following is an analysis of it.

[0220] The ROI-ALIGN module outputs features of size N*14*14*256. In the RCNN module, these features are first processed by the fifth convolutional module (Res18-Conv5) of Resnet18, resulting in an output feature size of N*7*7*512. This feature is then processed through a Global Avg Pool layer, averaging the 7*7 features within each channel of the input features to produce N*512 features, where each 1*512-dimensional feature vector represents the characteristics of each proposal. Next, two fully connected layers (FCs) are used to regress the precise coordinates of the bounding box (outputting N*4 vectors representing the x / y coordinates of the box's center point and the width and height of the box), as well as the confidence score of the box's category (in Header0, the score of whether the box is background, car, truck, or bus). Finally, a box merging operation selects the boxes with the highest scores, and a NMS operation is performed to remove duplicate boxes, resulting in a compact box output.

[0221] 6. 3D inspection process

[0222] The 3D inspection process consists of Figure 6 The serial 3D_Header0 in the 2D detection process is completed. It predicts the 3D information such as the center of mass point coordinates, orientation angle, length, width, and height of each object inside the 2D box based on the 2D box provided by the "2D detection" process and the feature map provided by Backbone. A possible implementation of the serial 3D_Header is as follows Figure 12 shown.

[0223] The ROI-ALIGN module extracts the features of the area where each 2D box is located on C4 based on the accurate 2D box provided by the parallel Header. Assuming that the number of 2D boxes is M, the feature size of the ROI-ALIGN module output is M*14*14*256. It is first processed by the 5th convolution module (Res18-Conv5) of Resnet18, and the output feature size is M*7*7*512. Then it is processed by a Global Avg Pool (average pooling layer) to average the 7*7 features of each channel in the input feature to obtain N*512 features, where each 1*512-dimensional feature vector represents the feature of each 2D box. Next, the orientation angle of the object in the box is regressed through three fully connected layers FC ( Figure 12 orientation, M*1 vector), centroid coordinates ( Figure 12 The centroid, M*2 vector, these two values ​​represent the x / y coordinates of the center of mass) and the length, width and height ( Figure 12 (dimention)

[0224] 7. Mask detection process

[0225] The Mask detection process consists of Figure 6 The Mask_Header0 sequence in the 2D detection process is completed, which predicts the fine mask of the object inside each 2D box based on the 2D box provided by the "2D detection" process and the feature map provided by Backbone. A possible implementation of the Mask_Header sequence is as follows: Figure 13 shown.

[0226] The ROI-ALIGN module extracts the features of the area where each 2D box is located on C4 based on the accurate 2D box provided by the parallel Header. Assuming that the number of 2D boxes is M, the feature size output by the ROI-ALIGN module is M*14*14*256. It is first processed by the 5th convolution module (Res18-Conv5) of Resnet18, and the output feature size is N*7*7*512. Then, it is further convolved through the deconvolution layer Deconv to obtain M*14*14*512 features. Finally, through a convolution, a Mask confidence output of M*14*14*1 is obtained. Each 14*14 matrix in this output represents the confidence of the mask of the object in each 2D box. Each 2D box is equally divided into 14*14 regions. This 14*14 matrix marks the possibility of the existence of an object in each region. By thresholding this confidence matrix (for example, outputting 1 if it is greater than a threshold of 0.5, and outputting 0 otherwise), the mask of the object can be obtained.

[0227] 8. Keypoint detection process

[0228] The Keypoint detection process consists of Figure 6 The serial Keypoint_Header2 in the "2D detection" process is completed, which predicts the key point coordinates of the object inside each 2D box based on the 2D box provided by the "2D detection" process and the feature map provided by Backbone. A possible implementation of the serial Keyponit_Header is as follows Figure 14 shown.

[0229] The ROI-ALIGN module extracts the features of the area where each 2D box is located on C4 based on the accurate 2D box provided by the parallel Header. Assuming that the number of 2D boxes is M, the feature size output by the ROI-ALIGN module is M*14*14*256. It is first processed by the fifth convolution module (Res18-Conv5) of Resnet18, and the output feature size is N*7*7*512. Then it is processed by a Global Avg Pool, averaging the 7*7 features of each channel in the input features to obtain M*512 features, where each 1*512-dimensional feature vector represents the feature of each 2D box. Next, a fully connected layer FC is used to regress the key point coordinates of the objects in the box (Keypoint in the figure, M*8 vector, these 8 values ​​represent the x / y coordinates of the four corner points of the parking space).

[0230] Based on the task division in Table 2, the embodiment of the present application also provides a detailed description of the training process of the perception network.

[0231] A. Preparation of training data

[0232] Based on the task division in Table 2, we need to provide labeled data for each task. For example, for Header0 training, we need to provide labeled data for cars, with the 2D bounding boxes and class labels for Car, Truck, and Bus in the dataset. For Header1 training, we need to provide labeled data for people, with the 2D bounding boxes and class labels for Pedestrian, Cyclist, and Tricycle in the dataset. For Header3, we need to provide labeled data for traffic lights, with the 2D bounding boxes and class labels for TrafficLight_Red, Yellow, Green, and Black in the dataset. And so on.

[0233] Each type of data only needs to be labeled with a specific type of object, allowing for targeted collection without having to label all objects of interest in every image, thus reducing data collection and labeling costs. Furthermore, this approach to data preparation offers flexible scalability. To add new types of objects to be detected, simply add one or more headers and provide the labeling data type for the newly added objects, eliminating the need to label the new objects in the existing data.

[0234] In addition, in order to train the 3D detection function in Header0, independent 3D annotation data is required to mark the 3D information of each car in the dataset (center of mass coordinates, orientation angle, length, width, and height); in order to train the Mask detection function in Header0, independent Mask annotation data is required to mark the mask of each car in the dataset; in particular, the Parking slot detection in Header2 requires detecting key points. This task requires the dataset to mark both the 2D box and the key points of the parking space (in fact, only the key points need to be marked, and the 2D box of the parking space can be automatically generated from the coordinates of the key points)

[0235] Generally, you only need to provide independent training data for each task, but you can also provide mixed annotated data. For example, you can annotate the 2D boxes and class labels of Car / Truck / Bus / Pedestrian / Cyclist / Tricycle on the dataset at the same time, so that you can use this data to train the parallel headers of Header0 and Header1 at the same time; you can also annotate the 2D / 3D / Mask data of Car / Truck / Bus on the dataset at the same time, so that this data can be used to train the parallel Header0, serial 3D_Header0 and serial Mask_Header0 at the same time.

[0236] We can assign a label to each image. This label determines which Headers in the network can be trained with this image. This will be described in detail in the subsequent training process.

[0237] In order to ensure that each Header gets an equal training opportunity, the data needs to be balanced. Specifically, the data with a small amount is expanded. The expansion method includes but is not limited to replication expansion. The balanced data is randomly shuffled and then sent to the network for training. Figure 15 shown.

[0238] B. Training a fully functional network based on partially labeled data

[0239] When training a fully functional network based on partially annotated data, the loss of the corresponding header is calculated based on the type of task for each input image. The gradient is then propagated back through this loss, and the gradients of the parameters on the corresponding header and backbone are calculated. The corresponding header and backbone are then adjusted based on the gradients. Headers that are not in the annotation task of the current input image are not adjusted.

[0240] If an image is only annotated with 2D data for a task, then when this image is fed into the network for training, only one corresponding parallel Header will be trained. Figure 16 shown.

[0241] The current image only has the 2D box of the traffic light marked. During training, only the parallel Header3 is used to obtain the predicted result of the traffic light for this input image, and the predicted result is compared with the true value to obtain the loss cost 2D_Loss3 of this Header. Since only one loss is generated, the overall loss value Final Loss = 2D_Loss3. In other words, the input image of the traffic light only flows through Backbone and parallel Header3, and the other headers do not participate in the training. Figure 16 As shown by the thick arrow without "X" in the figure. After getting the Final Loss, follow Figure 16 The gradients in Header3 and Backbone are calculated in parallel in the opposite direction of the thick arrow without "X". Then, the gradients are used to update the parameters of Header3 and Backbone to adjust the network and make it better predict traffic lights.

[0242] If a picture is labeled with 2D data of multiple tasks, then when this picture is sent to the network training, multiple corresponding parallel Headers will be trained. Figure 17 shown.

[0243] The current image is annotated with 2D frames of both people and cars. During training, the prediction results of people and cars in this input image will be obtained through parallel Header0 and parallel Header1, and compared with the true value to obtain the loss cost 2D_Loss0 / 2D_Loss1 of these two Headers. Since multiple cost losses are generated, the overall cost loss value is the average of each loss, that is, Final Loss = (2D_Loss0+2D_Loss1) / 2. In other words, the input image labeled with people and cars only flows through Backbone and parallel Header0 / 1, and the other Headers do not participate in the training. Figure 17 As shown by the thick arrow without "X" in the figure. After getting the Final Loss, follow Figure 17 The gradients in Header0 / 1 and Backbone are calculated in parallel in the opposite direction of the thick arrow without "X". Then, the gradients are used to update the parameters of Header0 / 1 and Backbone to adjust the network and make it better predict people and vehicles.

[0244] The training of serial header requires an independent data set for training, and the following is an example of 3D training of a car. Figure 18 shown.

[0245] The input image at this point is annotated with the true 2D and 3D values ​​of the car. During training, the data flow is shown by the thick arrows without "X" in the figure. Thick arrows with "X" indicate headers to which data does not flow. When this image is fed into the network, both the 2D and 3D loss functions are calculated simultaneously, resulting in the final Final Loss = (2D_Loss0 + 3D_Loss0) / 2. The gradients of the serial 3D Header0, parallel Header0, and Backbone are then calculated in the opposite direction of the thick arrows without "X". These gradients are then used to update their parameters, adjusting the network to better predict the 2D and 3D characteristics of the car.

[0246] When each image is fed into the network for training, only the corresponding header and backbone are adjusted to improve performance for that task. During this process, the performance of other tasks may deteriorate, but when the task's image is subsequently trained, the deteriorated header can be adjusted to compensate. Because the training data for all tasks is pre-balanced, each task receives equal training opportunities, preventing overtraining of any one task. This training approach enables the backbone to learn common features across tasks, while each header learns features specific to its task.

[0247] Currently, perception tasks require an increasing number of functions. Using multiple networks to implement a single function results in excessively high computational overhead. This embodiment of the present invention proposes a high-performance, scalable multi-header-based perception network. Each perception task uses the same backbone network, significantly reducing computational overhead and network parameter requirements. Table 3 shows the computational and parameter requirements for implementing a single function in a single-header network.

[0248] Single-Header-Model@720p GFlops Parameters(M) Vehicle (Car / Truck / Tram) 235.5 17.76 Vehicle+Mask+3D 235.6 32.49 Person(Pedestrian / Cyclist / Tricycle) 235.5 17.76 Person+Mask 235.6 23.0 Motorcycle / Bicycle 235.5 17.76 TrafficLight(Red / Green / Yellow / Black) 235.6 17.76 TrafficSign(Trafficsign / Guideside / Billboard) 235.5 17.75 TrafficCone / TrafficStick / FireHydrant 235.5 17.75 Parking slot (with keypoint) 235.6 18.98 Fully functional network (multiple single-header networks) 1648.9 145.49

[0249] Table 3 Statistics of computational complexity and parameter count of Single-Header network

[0250] As can be seen from the table, if eight networks are used to implement all the functions in this embodiment, the total computing power required is 1648.9 GFlops, and the total number of network parameters is 145.49M. This amount of computing power and network parameters is very huge and will put a lot of pressure on the hardware.

[0251] Table 4 shows the computational complexity and parameter requirements for implementing all functions of this embodiment using a Multi-Header network.

[0252] Multi-Header-Model@720p GFlops Parameters(M) Fully functional network (single Multi-Header network) 236.6 42.16

[0253] Table 4 Statistics of computational complexity and parameter count of Multi-Header network

[0254] As can be seen from the table, the computational complexity and parameter count of the Multi-Header network are only 1 / 7 and 1 / 3 of those of the Single-Header network, greatly reducing computational consumption.

[0255] In addition, the Multi-Header network can achieve the same detection performance as the Single-Header. Table 5 shows the performance comparison between Multi-Header and Single-Header in some categories.

[0256] category Single-Header Multi-Header Car 91.7 91.6 Tram 81.8 80.1 Pedestrian 73.6 75.2 Cyclist 81.8 83.3 TrafficLight 98.3 97.5 TrafficSign 95.1 94.5 Parkingslot(point precision / recall) 94.01 / 80.61 95.17 / 78.89 3D(mean_orien_err / mecentroid_dist_err) 2.95 / 6.78 2.88 / 6.34

[0257] Table 5 Detection performance comparison between Single-Header and Multi-Header networks

[0258] As can be seen from the table, the performance of the two is comparable. Therefore, the Multi-Header network does not cause performance degradation while saving computation and video memory.

[0259] The embodiment of the present invention proposes a high-performance, scalable perception network based on Multi-Header, which simultaneously implements different perception tasks (2D / 3D / key points / semantic segmentation, etc.) on the same network. Each perception task in the network shares the same backbone network, saving computational effort. The network structure is easy to expand, and only one function needs to be added by adding a Header. In addition, the embodiment of the present invention also proposes a method for training a multi-task perception network based on partially labeled data. Each task uses an independent data set, and there is no need to perform full task labeling on the same image. The training data of different tasks can be easily balanced, and the data of different tasks will not produce mutual inhibition.

[0260] like Figure 30 As shown, an embodiment of the present invention further provides an object detection method, the method comprising:

[0261] S3001, receiving an input image;

[0262] S3002, performing convolution processing on the input image, and outputting feature maps with different resolutions corresponding to the image;

[0263] S3003, based on the feature map, independently detect the task objects in each task for different tasks, and output the 2D box of the area where each task object is located and the confidence corresponding to each 2D box; wherein the task object is the object that needs to be detected in the task; the higher the confidence, the greater the probability that the object corresponding to the task is in the 2D box corresponding to the confidence.

[0264] Optionally, in one embodiment, S3002 may include the following four steps:

[0265] 1. Predict the area where the task object is located on one or more feature maps, and output a candidate 2D box matching the area;

[0266] Optionally, based on the template frame (Anchor) of the object corresponding to the task, the area where the task object exists can be predicted on one or more feature maps provided by the backbone network to obtain a candidate area, and a candidate 2D frame matching the candidate area can be output; wherein the template frame is obtained based on the statistical features of the task object to which it belongs, and the statistical features include the shape and size of the object.

[0267] 2. According to the area where the task object is located, extract the features of the area where the candidate 2D box is located from a feature map;

[0268] 3. Perform convolution processing on the features of the region where the candidate 2D box is located to obtain the confidence that the candidate 2D box belongs to each object category; each object category is an object category in the task;

[0269] 4. Adjust the coordinates of the 2D frame of the candidate area through a neural network so that the adjusted 2D candidate frame better matches the shape of the actual object than the candidate 2D frame, and select the adjusted 2D candidate frame with a confidence level greater than a preset threshold as the 2D frame of the area.

[0270] Optionally, the 2D frame may be a rectangular frame.

[0271] Optionally, the method further includes:

[0272] S3004, based on the 2D box of the task object of the task to which it belongs, extract the features of the area where the 2D box is located on one or more feature maps on the backbone network, and predict the 3D information, Mask information or Keypiont information of the task object of the task to which it belongs according to the features of the area where the 2D box is located.

[0273] Optionally, the detection of the area where the large object is located can be completed on the low-resolution feature map, and the RPN module can complete the detection of the area where the small object is located on the high-resolution feature map.

[0274] like Figure 31 As shown, an embodiment of the present invention further provides a method for training a multi-task perception network based on partially labeled data, the method comprising:

[0275] S3101: Determine the task to which each image belongs based on the labeled data type of each image; wherein each image is labeled with one or more data types, the multiple data types are a subset of all data types, and each data type corresponds to one task;

[0276] S3102, determining the header required to be trained for each image based on the task to which each image belongs;

[0277] S3103, calculating the loss value of the header required for training for each image;

[0278] S3104: For each image, gradient backpropagation is performed through the header to be trained, and parameters of the header to be trained and the backbone network are adjusted based on the loss value.

[0279] Alternatively, as Figure 31 As shown in the dotted box, before step S3102, the method further includes:

[0280] S31020, balance the data of images belonging to different tasks.

[0281] The present invention also provides a multi-header-based object perception method. The perception method provided by the present invention includes two parts: an inference process and a training process. These are described below.

[0282] 1. Perception process:

[0283] The process of the perception method provided by the embodiment of the present invention is as follows: Figure 21 shown.

[0284] In step S210, the image is input into the network;

[0285] In step S220, enter the "basic feature generation" process.

[0286] In this process, the image is Figure 5 Backbone performs basic feature extraction to obtain feature maps of different scales. After the basic features are generated, Figure 21 The dashed box in the figure represents the core process. Within the core process, each task has an independent "2D detection" process, as well as optional "3D detection," "Mask detection," and "Keypoint detection." The core process is described below.

[0287] 1. 2D inspection process

[0288] The "2D detection" process predicts the 2D box and confidence of each task based on the feature map generated by the "basic feature generation" process. Specifically, the "2D detection" process can be further divided into the "2D candidate region prediction" process, the "2D candidate region feature extraction" process, and the "2D candidate region sub-classification" process, such as Figure 22 shown.

[0289] The "2D candidate region prediction" process consists of Figure 5 The RPN module in the algorithm is implemented, which predicts the areas where the task object may exist on one or more feature maps provided by the "basic feature generation" process and gives the boxes of these areas (Proposal).

[0290] The "2D candidate region feature extraction" process consists of Figure 5 The ROI-ALIGN module in the algorithm is implemented. It extracts the features of the area where each Proposal is located on a feature map provided by the "Basic Feature Generation" process based on the Proposal provided by the "2D Candidate Region Prediction" process, and resizes it to a fixed size to obtain the features of each Proposal.

[0291] The "2D candidate region subdivision" process consists of Figure 5 The RCNN module in

[15] is implemented, which uses a neural network to further predict the features of each Proposal, outputs the confidence of each Proposal belonging to each category, and adjusts the coordinates of the Proposal's 2D box to output a more compact 2D box.

[0292] 2. 3D inspection process

[0293] The "3D Detection" process predicts the 3D information of the object inside each 2D frame, including the center of mass coordinates, orientation angle, length, width, and height, based on the 2D frame provided by the "2D Detection" process and the feature map generated by the "Basic Feature Generation" process. Specifically, the "3D Detection" process consists of two sub-processes, such as Figure 23 shown.

[0294] The sub-process analysis is as follows:

[0295] The "2D candidate region feature extraction" process consists of Figure 5The ROI-ALIGN module in the

[15] is implemented. It extracts the features of the area where each 2D box is located on a feature map provided by the "basic feature generation" process according to the coordinates of the 2D box, and resizes it to a fixed size to obtain the features of each 2D box.

[0296] The "3D centroid / orientation / length, width and height prediction" process consists of Figure 5 The 3D_Header implementation in , which mainly regresses the 3D information such as the center of mass coordinates, orientation angle, length, width and height of the object inside the 2D box based on the characteristics of each 2D box.

[0297] 3. Mask detection process

[0298] The "Mask Detection" process predicts the detailed mask of the object inside each 2D box based on the 2D box provided by the "2D Detection" process and the feature map generated by the "Basic Feature Generation" process. Specifically, the "Mask Detection" consists of two sub-processes, such as Figure 24 shown.

[0299] The sub-process analysis is as follows:

[0300] The "2D candidate region feature extraction" process consists of Figure 5 The ROI-ALIGN module in the

[15] is implemented. It extracts the features of the area where each 2D box is located on a feature map provided by the "basic feature generation" process according to the coordinates of the 2D box, and resizes it to a fixed size to obtain the features of each 2D box.

[0301] The Mask Prediction process consists of Figure 5 The Mask_Header implementation in , which mainly regresses the mask of the object inside the 2D box based on the features of each 2D box.

[0302] 4. Keypoint detection process

[0303] The "Keypoint Prediction" process predicts the mask of the object inside each 2D box based on the 2D box provided by the "2D Detection" process and the feature map generated by the "Basic Feature Generation" process. Specifically, the "Keypoint Prediction" consists of two sub-processes, such as Figure 25 shown.

[0304] The sub-process analysis is as follows:

[0305] The "2D candidate region feature extraction" process consists of Figure 5The ROI-ALIGN module in the

[15] is implemented. It extracts the features of the area where each 2D box is located on a feature map provided by the "basic feature generation" process according to the coordinates of the 2D box, and resizes it to a fixed size to obtain the features of each 2D box.

[0306] The "key point coordinate prediction" process consists of Figure 5 The Keypoint_Header implementation in , which mainly regresses the coordinates of the key points of the objects inside the 2D box based on the features of each 2D box.

[0307] 2. Training Process

[0308] The training process of the embodiment of the present invention is as follows Figure 26 shown.

[0309] The red frame represents the core training process, which is described below.

[0310] 1. Data balancing process between tasks

[0311] The amount of data for each task is extremely unbalanced. For example, the number of images containing people is much larger than that of traffic signs. To ensure that the headers for each task have equal training opportunities, the data must be balanced across tasks. Specifically, this involves expanding the smaller amount of data, including but not limited to replication.

[0312] 2. Calculate the Loss process based on the task to which the image belongs

[0313] Each image can be assigned to one or more tasks based on the type of data it is labeled with. For example, if a picture only has a traffic sign labeled on it, then this picture belongs only to the traffic sign task; if a picture is labeled with both a person and a car, then this picture belongs to both the person and car tasks. When calculating the loss, only the loss of the header corresponding to the task to which the current image belongs is calculated; the losses of other tasks are not calculated. For example, if the current input training image belongs to the person and car task, then only the loss of the person and car header is calculated; the losses of other tasks (such as traffic lights and traffic signs) are not calculated.

[0314] 3. Return the gradient process according to the task to which the image belongs

[0315] After calculating the loss, gradients need to be propagated back. At this point, gradients are propagated only through the header for the current task; headers not involved in the current task are excluded. This allows the current header to be adjusted for the current image, enabling it to better learn the current task. Because the data for each task has been balanced, each header receives equal training opportunities. Through this iterative adjustment process, each header learns task-specific features, while the backbone learns features shared across tasks.

[0316] This application addresses the shortcomings of existing methods and proposes a high-performance, scalable multi-header-based perception network. This network allows for simultaneous implementation of different perception tasks (2D / 3D / keypoint / semantic segmentation, etc.) on the same network. Each perception task in this network shares the same backbone network, significantly reducing computational overhead. Furthermore, the network structure is easily scalable, requiring only the addition of one or more headers for functionality expansion.

[0317] In addition, an embodiment of the present application also proposes a method for training a multi-task perception network based on partially labeled data. Each task uses an independent data set, and there is no need to perform full task labeling on the same image. The training data of different tasks can be easily balanced, and the data of different tasks will not produce mutual inhibition.

[0318] like Figure 5 The sensor network shown can be Figure 27 The structure in is implemented, Figure 27 Figure 2 is a schematic diagram of an application system for a perception network. As shown in the figure, the perception network 2000 includes at least one processor 2001, at least one memory 2002, at least one communication interface 2003, and at least one display device 2004. Processor 2001, memory 2002, display device 2004, and communication interface 2003 are connected via a communication bus and communicate with each other.

[0319] The communication interface 2003 is used to communicate with other devices or communication networks, such as Ethernet, radio access network (RAN), wireless local area network (WLAN), etc.

[0320] The memory 2002 may be a read-only memory (ROM) or other type of static storage device that can store static information and instructions, a random access memory (RAM) or other type of dynamic storage device that can store information and instructions, or an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, an optical disc storage (including a compact disc, laser disc, optical disc, digital versatile disc, Blu-ray disc, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory may exist independently and be connected to the processor via a bus. The memory may also be integrated with the processor.

[0321] The memory 2002 is used to store application code for executing the above solution, and the execution is controlled by the processor 2001. The processor 2001 is used to execute the application code stored in the memory 2002.

[0322] The code stored in the memory 2002 can execute the above-provided method for perceiving an object based on a Multi-Header.

[0323] The display device 2004 is used to display the image to be recognized, and information such as 2D, 3D, Mask, and key points of the object of interest in the image.

[0324] The processor 2001 may also be used as one or more integrated circuits to execute relevant programs to implement the Multi-Header-based object perception method or model training method of the embodiment of the present application.

[0325] Processor 2001 may also be an integrated circuit chip with signal processing capabilities. During implementation, each step of the recommended method of the present application may be performed by hardware integrated logic circuits or software instructions in processor 2001. During implementation, each step of the training method of the embodiment of the present application may be performed by hardware integrated logic circuits or software instructions in processor 2001. The aforementioned processor 2001 may also be a general-purpose processor, a digital signal processor (DSP), an ASIC, a field programmable gate array (FPGA), or other programmable logic device, a discrete gate or transistor logic device, or a discrete hardware component. It may implement or execute the various methods, steps, and module block diagrams disclosed in the embodiments of the present application. A general-purpose processor may be a microprocessor or any conventional processor. The steps of the method disclosed in conjunction with the embodiments of the present application may be directly implemented and executed by a hardware decoding processor, or by a combination of hardware and software modules in the decoding processor. The software module may be located in a storage medium well-known in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or the like. The storage medium is located in the memory 2002, and the processor 2001 reads the information in the memory 2002 and combines its hardware to complete the object perception method or model training method of the embodiment of the present application.

[0326] The communication interface 2003 uses a transceiver such as, but not limited to, a transceiver to implement communication between the recommendation device or training device and other devices or a communication network. For example, the communication interface 2003 can be used to obtain images to be recognized or training data.

[0327] The bus may include a path for transmitting information between various components of the device (e.g., memory 2002, processor 2001, communication interface 2003, and display device 2004). In one possible embodiment, processor 2001 specifically performs the following steps: receiving an input image; performing convolution processing on the input image, outputting feature maps of different resolutions corresponding to the image; independently detecting objects corresponding to different tasks based on the feature maps provided by the backbone network, and outputting 2D boxes of candidate regions of the objects corresponding to each task and the confidence level corresponding to each 2D box.

[0328] In a possible embodiment, when executing the step of independently detecting the object corresponding to each task for different tasks according to the feature map provided by the backbone network, and outputting the 2D box of the candidate area of ​​the object corresponding to each task and the confidence corresponding to each 2D box, the processor 2001 specifically performs the following steps: predicting the area where the task object exists on one or more feature maps to obtain a candidate area, and outputting a candidate 2D box matching the candidate area; deducting the features of the area where the candidate area is located from a feature map according to the candidate area obtained by the RPN module; refining the features of the candidate area to obtain the confidence of each object category corresponding to the candidate area; each object is an object in a corresponding task; adjusting the coordinates of the candidate area to obtain a second candidate 2D box, the second 2D candidate box is more matched with the actual object than the candidate 2D box, and selecting the 2D candidate box with a confidence greater than a preset threshold as the 2D box of the candidate area.

[0329] In a possible embodiment, when predicting the area where the task object exists on one or more feature maps to obtain a candidate area and outputting a candidate 2D box matching the candidate area, the processor 2001 specifically performs the following steps:

[0330] Based on the template frame (Anchor) of the object corresponding to the task, the area where the object of the task exists is predicted on one or more feature maps to obtain a candidate area, and a candidate 2D frame matching the candidate area is output; wherein: the template frame is obtained based on the statistical features of the task object to which it belongs, and the statistical features include the shape and size of the object.

[0331] In a possible embodiment, the processor 2001 further performs the following steps:

[0332] Based on the 2D box of the object corresponding to the task, the features of the object are extracted on one or more feature maps on the backbone network, and the 3D, Mask or Keypoint of the object is predicted.

[0333] In a possible embodiment, detection of candidate regions of large objects is performed on a low-resolution feature map, and detection of candidate regions of small objects is performed on a high-resolution feature map.

[0334] In a possible embodiment, the 2D frame is a rectangular frame.

[0335] Alternatively, as Figure 28 As shown, the structure of the perception network can be implemented by the server, which can Figure 28The server 2110 includes at least one processor 2101, at least one memory 2102, and at least one communication interface 2103. The processor 2101, the memory 2102, and the communication interface 2103 are connected via a communication bus and communicate with each other.

[0336] The communication interface 2103 is used to communicate with other devices or communication networks, such as Ethernet, RAN, WLAN, etc.

[0337] The memory 2102 may be a ROM or other type of static storage device capable of storing static information and instructions, a RAM or other type of dynamic storage device capable of storing information and instructions, or an EEPROM, a CD-ROM or other optical disk storage, an optical disc storage (including a compact disc, laser disc, optical disc, digital versatile disc, Blu-ray disc, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but is not limited thereto. The memory may be independent and connected to the processor via a bus. The memory may also be integrated with the processor.

[0338] The memory 2102 is used to store application code for executing the above solution, and the execution is controlled by the processor 2101. The processor 2101 is used to execute the application code stored in the memory 2102.

[0339] The codes stored in the memory 2102 can execute the above-provided method for perceiving objects based on a Multi-Header.

[0340] The processor 2101 may also be used as one or more integrated circuits to execute relevant programs to implement the multi-header-based object perception method or model training method of the embodiment of the present application.

[0341] Processor 2101 may also be an integrated circuit chip with signal processing capabilities. During implementation, each step of the recommended method of this application may be performed by hardware integrated logic circuits or software instructions within processor 2101. During implementation, each step of the training method of the embodiments of this application may be performed by hardware integrated logic circuits or software instructions within processor 2101. The aforementioned processor 2001 may also be a general-purpose processor, DSP, ASIC, FPGA, or other programmable logic device, discrete gate or transistor logic device, or discrete hardware component. It may implement or execute the various methods, steps, and module block diagrams disclosed in the embodiments of this application. A general-purpose processor may be a microprocessor or any conventional processor. The steps of the methods disclosed in conjunction with the embodiments of this application may be directly implemented and executed by a hardware decoding processor, or by a combination of hardware and software modules within the decoding processor. The software modules may be located in storage media well-known in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or the like. The storage medium is located in the memory 2102, and the processor 2101 reads the information in the memory 2102 and combines its hardware to complete the object perception method or model training method of the embodiment of the present application.

[0342] The communication interface 2103 uses a transceiver such as, but not limited to, a transceiver to enable communication between the recommendation device or training device and other devices or a communication network. For example, the communication interface 2103 can be used to obtain images to be recognized or training data.

[0343] The bus may include a path for transmitting information between various components of the device (e.g., memory 2102, processor 2101, and communication interface 2103). In one possible embodiment, the processor 2101 specifically performs the following steps: predicting the area where the task object exists on one or more feature maps to obtain a candidate area, and outputting a candidate 2D frame matching the candidate area; extracting features of the area where the candidate area is located from a feature map based on the candidate area obtained by the RPN module; refining the features of the candidate area to obtain confidence scores for each object category corresponding to the candidate area; each object is an object in a corresponding task; adjusting the coordinates of the candidate area to obtain a second candidate 2D frame, the second 2D candidate frame being more closely matched to the actual object than the candidate 2D frame, and selecting a 2D candidate frame with a confidence score greater than a preset threshold as the 2D frame of the candidate area.

[0344] The present application provides a computer-readable medium storing program codes for execution by a device. The program codes include a program for executing the following Figure 21 、 22, 23, 24 or 25, the relevant content of the object perception method of the embodiment shown.

[0345] The present application provides a computer-readable medium storing program codes for execution by a device. The program codes include a program for executing the following Figure 26 Relevant content of the training method of the illustrated embodiment.

[0346] The present application provides a computer program product comprising instructions, which, when executed on a computer, enables the computer to execute the above-mentioned Figure 21 、 22 , 23, 24 or 25, the relevant content of the perception method of the embodiment shown.

[0347] The present application provides a computer program product comprising instructions, which, when executed on a computer, enables the computer to execute the above-mentioned Figure 26 Relevant content of the training method of the illustrated embodiment.

[0348] The present application provides a chip, which includes a processor and a data interface. The processor reads instructions stored in a memory through the data interface and executes the following instructions: Figure 21 、 22 , 23, 24, 25 or 26 of the embodiments shown in the relevant content of the perception method.

[0349] The present application provides a chip, which includes a processor and a data interface. The processor reads instructions stored in a memory through the data interface and executes the following instructions: Figure 26 Relevant content of the training method of the illustrated embodiment.

[0350] Optionally, as an implementation, the chip may further include a memory, wherein instructions are stored in the memory, and the processor is configured to execute the instructions stored in the memory. When the instructions are executed, the processor is configured to execute the following steps: Figure 21 、 22 , 23, 24 or 25, or perform the following Figure 26 Relevant content of the training method of the illustrated embodiment.

[0351] It should be noted that for the aforementioned method embodiments, for simplicity of description, they are all expressed as a series of action combinations. However, those skilled in the art should be aware that the present invention is not limited by the order of the actions described, because according to the present invention, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in this specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present invention.

[0352] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0353] In summary, the beneficial effects of the embodiments of the present application are summarized as follows:

[0354] (1) Each perception task shares the same backbone network, saving computational effort exponentially. The network structure is easily scalable, and the 2D detection type can be expanded by simply adding one or more Headers. Each parallel Header has independent RPN and RCNN modules, and only needs to detect the objects of the task to which it belongs. This avoids accidental detection of unlabeled objects from other tasks during training. In addition, by using an independent RPN layer, a dedicated anchor can be customized for the scale and aspect ratio of the object for each task, thereby increasing the overlap ratio between the anchor and the object, and thus improving the recall rate of the RPN layer for objects.

[0355] (2) 3D, Mask, and Keypoint detection functions can be implemented in a flexible and convenient way. These functional extensions share the same backbone network with the 2D part, which does not significantly increase the amount of calculation. Using one network to implement multiple functions is easy to implement on the chip.

[0356] (3) Each task uses an independent dataset, so there is no need to label all tasks on the same image, saving labeling costs. Task expansion is flexible and simple. When adding a new task, only the data for the new task needs to be provided, and no new objects need to be labeled on the original data. The training data of different tasks can be easily balanced, so that each task has equal training opportunities and avoids large amounts of data overwhelming small amounts of data.

[0357] In the several embodiments provided in this application, it should be understood that the disclosed devices can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical or other forms.

[0358] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0359] In addition, the functional units in the various embodiments of the present invention may be integrated into a single processing unit, each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0360] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable memory. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a memory and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present invention. The aforementioned memory includes various media that can store program code, such as a USB flash drive, ROM, RAM, a mobile hard disk, a magnetic disk, or an optical disk.

[0361] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing the relevant hardware through a program, and the program can be stored in a computer-readable memory, which may include: a flash drive, ROM, RAM, a magnetic disk or an optical disk, etc.

[0362] The above is a detailed introduction to the embodiments of the present application. Specific examples are used herein to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea. At the same time, for those skilled in the art, according to the ideas of the present invention, there may be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting the present invention.

Claims

1. A device for a perception network based on multiple headends, characterized in that: The sensing network includes a backbone network and multiple parallel headers, and the multiple parallel headers are respectively connected to the backbone network; The backbone network is configured to receive an input image, perform convolution processing on the input image, and output one or more feature maps corresponding to the image; The first parallel header is used to detect the first category of task objects in a task based on the one or more feature maps, and output one or more 2D boxes in the area where the task object is located and the confidence corresponding to each 2D box. The first parallel header is any one of the multiple parallel headers; the confidence indicates the probability that the first category of task objects exists in the 2D box.

2. The sensing network device according to claim 1, characterized in that Different parallel headers complete the detection of task objects of different tasks.

3. The sensing network device according to claim 1, characterized in that It also includes a second parallel header, which is a parallel header different from the first parallel header among the multiple parallel headers. The second parallel header detects the task object in another task based on the one or more feature maps, and outputs a 2D box of the area where the task object in the other task is located.

4. The sensing network device according to claim 1, characterized in that The first category is the category of objects detected in the task corresponding to the first parallel Header. A higher confidence level indicates a greater probability that the task object corresponding to the task is present in the 2D box corresponding to the confidence level.

5. The sensing network device according to any one of claims 1 to 4, characterized in that: The perception network also includes one or more serial headers, a serial header is connected to a parallel header, the first serial header predicts the 3D information, mask information or keyword information of the task object of the task according to the characteristics of the area where the 2D box is located, and the first serial header is connected to the first parallel header.

6. The sensing network device according to any one of claims 1 to 4, characterized in that: The first parallel Header includes a candidate region generation network module, a region of interest extraction module and a regional convolutional neural network module; the candidate region generation network module is used to predict the region where the task object is located based on the one or more feature maps, and output a candidate 2D box matching the region; the region of interest extraction module is used to deduct the features of the region where the candidate 2D box is located from the one or more feature maps based on the region predicted by the candidate region generation network module; the regional convolutional neural network module adjusts the coordinates of the candidate 2D box according to the features of the region where the candidate 2D box is located to obtain the 2D box of the region where the task object is located.

7. The sensing network device according to claim 6, characterized in that: The regional convolutional neural network module is used to perform convolution processing on the features of the region where the candidate 2D frame is located through a neural network to obtain a confidence level that the candidate 2D frame belongs to the first category; adjust the coordinates of the candidate 2D frame through the neural network so that the adjusted 2D candidate frame better matches the shape of the actual object than the candidate 2D frame, and select the adjusted 2D candidate frame with a confidence level greater than a preset threshold as the 2D frame of the region.

8. The sensing network device according to claim 6, characterized in that: The candidate region generation network module is used to predict the area where the task object exists on one or more feature maps provided by the backbone network based on the template frame of the object corresponding to the task to obtain a candidate region, and output a candidate 2D frame matching the candidate region; the template frame is obtained based on the statistical features of the task object to which it belongs, and the statistical features include the shape and size of the object.

9. The sensing network device according to claim 1, characterized in that: The feature map of the image output by the backbone network includes multiple feature maps of different resolutions.

10. The sensing network device according to claim 6, characterized in that: The candidate region generation network module is used to predict regions where objects of different sizes are located on feature maps of different resolutions.

11. The sensing network device according to any one of claims 1 to 4, characterized in that: The 2D frame is a rectangular frame.

12. The sensing network device according to any one of claims 1 to 4, characterized in that: The multiple parallel headers are used to detect at least two of the following task objects: static objects, dynamic objects, traffic lights, and signs.

13. An object detection method, characterized in that: Applied to a perception network, the perception network includes a backbone network and multiple parallel headers, the multiple parallel headers are connected to the backbone network; the method includes: The backbone network receives an input image, performs convolution processing on the input image, and outputs one or more feature maps corresponding to the image; The first parallel header detects the first category of task objects in a task based on the one or more feature maps, and outputs one or more 2D boxes of the area where the task object is located and the confidence corresponding to each 2D box. The first parallel header is any one of the multiple parallel headers; the confidence indicates the probability that the first category of task objects exists in the 2D box.

14. The object detection method according to claim 13, wherein: Different parallel headers complete the detection of task objects of different tasks.

15. The object detection method according to claim 13, wherein: The perception network also includes a second parallel header, which is a parallel header different from the first parallel header among the multiple parallel headers. The second parallel header detects the task object in another task based on the one or more feature maps, and outputs a 2D box of the area where the task object in the other task is located.

16. The object detection method according to claim 13, wherein: The first category is the object category detected in the task corresponding to the first parallel Header; a higher confidence level indicates a greater probability that the task object corresponding to the task is present in the 2D box corresponding to the confidence level.

17. The object detection method according to any one of claims 13 to 16, characterized in that: The sensing network further includes one or more serial headers, one serial header is connected to one parallel header, and the method further includes: The first serial header predicts the 3D information, mask information or keyword information of the task object of the task according to the characteristics of the area where the 2D frame is located. The first serial header is connected to the first parallel header.

18. The object detection method according to any one of claims 13 to 16, characterized in that: The first parallel Header includes a candidate region generation network module, a region of interest extraction module and a regional convolutional neural network module; The first parallel header detects a task object in a task based on the one or more feature maps, and outputs a 2D frame of the area where the task object is located, including: The candidate region generation network module is used to predict the region where the task object is located based on the one or more feature maps, and output a candidate 2D box that matches the region; the region of interest extraction module is used to deduct the features of the region where the candidate 2D box is located from the one or more feature maps based on the region predicted by the candidate region generation network module; the regional convolutional neural network module adjusts the coordinates of the candidate 2D box based on the features of the region where the candidate 2D box is located to obtain a 2D box of the region where the task object is located.

19. The object detection method according to claim 18, wherein: The regional convolutional neural network module adjusts the coordinates of the candidate 2D frame according to the characteristics of the region where the candidate 2D frame is located to obtain the 2D frame of the region where the task object is located, including: The regional convolutional neural network module is used to perform convolution processing on the features of the region where the candidate 2D frame is located through a neural network to obtain a confidence level that the candidate 2D frame belongs to the first category; adjust the coordinates of the candidate 2D frame through the neural network so that the adjusted 2D candidate frame better matches the shape of the actual object than the candidate 2D frame, and select the adjusted 2D candidate frame with a confidence level greater than a preset threshold as the 2D frame of the region.

20. The object detection method according to claim 18, wherein: The candidate region generation network module is used to predict the region where the task object is located based on the one or more feature maps, and output a candidate 2D box matching the region, including: The candidate region generation network module is used to predict the area where the task object exists on the one or more feature maps based on the template frame of the object corresponding to the task to obtain a candidate region, and output a candidate 2D frame matching the candidate region; the template frame is obtained based on the statistical features of the task object to which it belongs, and the statistical features include the shape and size of the object.

21. The object detection method according to claim 18, wherein: The feature map output by the backbone network includes multiple feature maps of different resolutions.

22. The object detection method according to claim 18, wherein: The candidate region generation network module is used to predict regions where objects of different sizes are located on feature maps of different resolutions.

23. The object detection method according to any one of claims 13 to 16, characterized in that: The 2D frame is a rectangular frame.

24. The object detection method according to any one of claims 13 to 16, characterized in that: The multiple parallel headers are used to detect at least two of the following task objects: static objects, dynamic objects, traffic lights, and signs.

25. A perceptual network training method, characterized in that: The method comprises: Obtaining a training set, wherein the training set includes pictures of multiple categories; The first perception network is trained based on the training set, and pictures of different categories are used to train different parallel headers in the first perception network, thereby obtaining a perception network device as described in any one of claims 1-12.

26. The perceptual network training method according to claim 25, characterized in that: Different types of images are used to train different serial headers in the first perception network.

27. A perception network application system, characterized in that: The perception network application system includes a processor and a memory, the memory stores application code, and the processor is configured to obtain the application code and execute the method according to any one of claims 13 to 26.

28. A computer storage medium, characterized in that The computer storage medium stores one or more instructions, which, when executed by one or more computers, cause the one or more computers to implement the method according to any one of claims 13 to 26.

29. A computer program product comprising instructions, characterized in that When the computer program product is run on a computer, the computer is caused to perform the method according to any one of claims 13 to 26.

Images