Perception network, training method of perception network, object recognition method and device

Abstract

A perception network (800), a training method (1200) of the perception network, and an object recognition method (1500) and device, the perception network (800) comprising a backbone network (810), a candidate region generation network RPN (820), a region of interest extraction module (830), and a classification regression network (840), a plurality of perception tasks sharing the RPN (820), the RPN (820) predicting a region where an object to be detected in the plurality of tasks is located, and a target 2D box and a classification result being obtained by the classification regression network. The perception network (800) can reduce the parameter quantity and the calculation quantity in the multi-task perception network, reduce the power consumption of hardware, and improve the running speed of the model.

Description

Technical Field

[0001] This application relates to the field of computer vision, and more specifically, to a perceptual network, a training method for the perceptual network, an object recognition method, and an apparatus. Background Technology

[0002] Computer vision is an integral part of various intelligent / autonomous systems across diverse application fields, such as manufacturing, inspection, document analysis, and medical diagnosis. Figuratively speaking, computer vision is like equipping a computer with eyes (cameras / video cameras) and a brain (algorithms), enabling it to perceive its environment. Computer vision uses various imaging systems to acquire input information instead of visual organs, and then the computer processes and interprets this input information, replacing the brain.

[0003] With the development of visual perception technology and the increasing demand for artificial intelligence (AI) perception in real-world scenarios, more and more perception networks are being deployed across various fields. For example, perception networks deployed in advanced driving assistance systems (ADAS) and autonomous driving systems (ADS) can be used to identify obstacles on the road. Currently, most perception networks can only perform one detection task. To achieve multiple detection tasks, different networks are typically deployed to perform different tasks. However, running multiple perception networks simultaneously increases hardware power consumption and reduces model speed. Moreover, many fields use chips with low computing power, making it difficult to deploy large-scale perception networks, let alone multiple perception networks.

[0004] Therefore, how to reduce the hardware power consumption during the operation of multi-task awareness networks has become an urgent problem to be solved. Summary of the Invention

[0005] This application provides a perceptual network, a training method for the perceptual network, an object recognition method, and an apparatus, which can reduce the number of parameters and computational load in multi-task perceptual networks, reduce hardware power consumption, and improve the running speed of the model.

[0006] Firstly, a perceptual network is provided, comprising: a backbone network, a region proposal network (RPN), a region of interest extraction module, and a classification and regression network; the RPN is used to output the position information of candidate two-dimensional (2D) bounding boxes of target objects based on a second feature map, the target objects including objects to be detected in multiple tasks, each of the multiple tasks including at least one category, and the second feature map being determined based on a first feature map; the region of interest extraction module is used to extract first feature information on a third feature map based on the position information of the candidate 2D bounding boxes, the first feature information being the features of the region where the candidate 2D bounding boxes are located, and the third feature map being determined based on the first feature map; the classification and regression network is used to process the first feature information and output the target 2D bounding box of the target object and first indication information, the number of target 2D bounding boxes being less than or equal to the number of candidate 2D bounding boxes, and the first indication information being used to indicate the category to which the target object belongs.

[0007] According to the scheme of the embodiments of this application, a single perception network is used to complete multiple perception tasks, and multiple tasks share a single RPN. That is, a single RPN predicts the region where the object to be detected is located in multiple tasks. While ensuring the performance of the perception network, the number of parameters and the amount of computation of the perception network are reduced, the processing efficiency is improved, and it is beneficial to deploy in scenarios with high real-time requirements. At the same time, it reduces the pressure on the hardware and saves costs.

[0008] The "first feature map" refers to the feature map output by the backbone network. Any feature map output by the backbone network can be called the first feature map.

[0009] The first feature map of the input image can be one or more.

[0010] Multiple tasks can also be understood as multiple broad categories. A broad category includes at least one subcategory. Or, a broad category is a set of at least one subcategory. The criteria for dividing tasks can be set as needed. For example, objects to be detected can be divided into multiple tasks based on the similarity of the objects to be detected.

[0011] In this embodiment of the application, multiple tasks share the same RPN, which can also be called a single-header multitasking RPN.

[0012] The second feature map can be one or multiple.

[0013] For example, the second feature map may include one or more of the first feature maps.

[0014] For example, the third feature map can be one of the first feature maps.

[0015] In conjunction with the first aspect, in some implementations of the first aspect, the perceptual network also includes feature pyramid networks (FPNs), which are connected to the backbone network and are used to perform feature fusion on the first feature map and output the fused feature map.

[0016] In this case, the second feature map may include one or more of the fused feature maps.

[0017] For example, the third feature map can be one of the first feature maps or one of the fused feature maps output by the FPN.

[0018] According to the scheme of the embodiments of this application, the first feature map is fused using FPN, which can generate a more expressive feature map for subsequent modules, thereby improving the performance of the model.

[0019] In conjunction with the first aspect, in some implementations of the first aspect, the classification and regression network is specifically used for: processing the first feature information to obtain the confidence scores of the candidate 2D boxes belonging to each category in multiple tasks; adjusting the position information of the candidate 2D boxes to obtain the adjusted candidate 2D boxes; determining the target 2D box based on the adjusted candidate 2D boxes; and determining the first indication information based on the confidence scores of the target 2D boxes belonging to each category.

[0020] For example, the position information of the candidate 2D bounding box is adjusted so that the adjusted candidate 2D bounding box matches the shape of the actual object better than the candidate 2D bounding box, that is, the adjusted candidate 2D bounding box is a more compact candidate 2D bounding box.

[0021] Furthermore, a box merging operation is performed on the adjusted candidate 2D boxes to obtain the target 2D box. For example, non-maximum suppression (NMS) merging is performed on the adjusted 2D boxes to obtain the target 2D box.

[0022] In conjunction with the first aspect, in some implementations of the first aspect, the classification and regression network includes a first region convolutional neural network (RCNN). The first RCNN includes a hidden layer, multiple sub-classification fully connected layers, and multiple sub-regression fully connected layers. The hidden layer is connected to multiple sub-classification fully connected layers, and the hidden layer is connected to multiple sub-regression fully connected layers. The multiple sub-classification fully connected layers correspond one-to-one with multiple tasks, and the multiple sub-regression fully connected layers correspond one-to-one with multiple tasks. The hidden layer is used to process the first feature information to obtain the second feature information. The sub-classification fully connected layers are used to obtain the confidence of the candidate 2D box belonging to the object category in the task corresponding to the sub-classification fully connected layer based on the second feature information. The sub-regression fully connected layers are used to adjust the position information of the candidate 2D box based on the second feature information to obtain the adjusted candidate 2D box.

[0023] For example, the hidden layer may include at least one of the following: a convolutional layer or a fully connected layer. Since multiple tasks share the hidden layer, the convolutional layer in the hidden layer may also be called a shared convolutional layer (shared conv), and the fully connected layer in the hidden layer may also be called a shared fully connected layer (shared fc).

[0024] The first R-CNN includes hidden layers and multiple sub-classification fully connected layers and multiple sub-regression fully connected layers corresponding to multiple tasks. Each task can have an independent sub-classification fully connected layer and a sub-regression fully connected layer. The sub-classification fully connected layer and the sub-regression fully connected layer corresponding to each task can perform the detection of the object to be detected in that task. Specifically, the sub-classification fully connected layer can output the confidence of the candidate 2D box belonging to the object category in the task, and the sub-regression fully connected layer can output the adjusted candidate 2D box.

[0025] A first R-CNN includes multiple sub-classification fully connected layers and sub-regression fully connected layers; therefore, a single first R-CNN can perform object detection in multiple tasks. First R-CNN can also be called a single-head multi-task R-CNN.

[0026] According to the scheme of this application embodiment, multiple tasks share the hidden layers of the first RCNN, which further reduces the number of parameters and computational cost of the perceptual network and improves processing efficiency. Moreover, each task corresponds to an independent sub-classification fully connected layer (fc) and sub-regression fc, which improves the scalability of the perceptual network. The perceptual network can flexibly implement functional configuration by adding or removing sub-classification fc and sub-regression fc.

[0027] In conjunction with the first aspect, in some implementations of the first aspect, the classification and regression network includes a second RCNN. The second RCNN includes a hidden layer, a classification fully connected layer, and a regression fully connected layer. The hidden layer is connected to the classification fully connected layer and the regression fully connected layer. The hidden layer is used to process the first feature information to obtain the third feature information. The classification fully connected layer is used to obtain the confidence of the candidate 2D box belonging to each category based on the third feature information. The regression fully connected layer is used to adjust the position information of the candidate 2D box based on the third feature information to obtain the adjusted candidate 2D box.

[0028] A second R-CNN can perform object detection in multiple tasks. A second R-CNN can also be called a single-head multi-task R-CNN.

[0029] In the scheme of this application embodiment, a second R-CNN is used as the classification and regression network. Multiple tasks share the hidden layers of the second R-CNN, further reducing the number of parameters and computational load of the perceptual network and improving processing efficiency. Furthermore, the output of the hidden layers in the first R-CNN needs to be input into all sub-classification fully connected layers and sub-regression fully connected layers for multiple matrix operations, while the output of the hidden layers in the second R-CNN only needs to be input into the classification fully connected layer and the regression fully connected layer for matrix operations. This further reduces matrix operation operations, is more hardware-friendly, further reduces computation time, and improves processing efficiency.

[0030] In conjunction with the first aspect, in some implementations of the first aspect, the classification fully connected layer is obtained by merging multiple sub-classification fully connected layers in the first RCNN, and the regression fully connected layer is obtained by merging multiple sub-regression fully connected layers in the first RCNN. The first RCNN includes hidden layers, multiple sub-classification fully connected layers, and multiple sub-regression fully connected layers. The hidden layers are connected to multiple sub-classification fully connected layers, and the hidden layers are connected to multiple sub-regression fully connected layers. The multiple sub-classification fully connected layers correspond one-to-one with multiple tasks, and the multiple sub-regression fully connected layers correspond one-to-one with multiple tasks. The sub-classification fully connected layer is used to obtain the confidence score of the candidate 2D box belonging to the object category in the task corresponding to the sub-classification fully connected layer based on the third feature information. The sub-regression fully connected layer is used to adjust the position information of the candidate 2D box based on the third feature information to obtain the adjusted candidate 2D box.

[0031] In the scheme of this application embodiment, the multiple sub-classification fc and sub-regression fc in the first RCNN are merged, and the second RCNN is used as the classification and regression network. This can further reduce matrix operation operations, make it more hardware-friendly, further reduce computation time, and improve processing efficiency.

[0032] Secondly, a training method for a perceptual network is provided. The perceptual network includes a region candidate generation network (RPN), which is used to predict the position information of candidate 2D bounding boxes of target objects in sample images. The target objects include objects to be detected for multiple tasks, and each task includes at least one category. The target objects include a first task object and a second task object. The method includes: acquiring training data, which includes sample images, labeled data of the first task object on the sample images, and pseudo-bounding boxes of the second task object on the sample images. The labeled data includes the class label of the first task object and the labeled 2D bounding boxes of the first task object. The pseudo-bounding boxes of the second task object are obtained by reasoning about the sample images using other perceptual networks. The perceptual network is trained based on the training data.

[0033] When the labeled data is only partially labeled, i.e., only includes the labeled data of the objects in the first task, training the perceptual network solely based on this labeled data can lead to mutual inhibition between the training data of different tasks, since multiple tasks share a single Resource Planning (RPN). Specifically, because the labeled data is partial—for example, a sample image may only contain the labeled data of the objects to be detected for one task—the RPN parameters will be adjusted when training with this labeled data to more accurately predict the candidate 2D bounding boxes of those objects, but it may fail to accurately predict the candidate 2D bounding boxes of other objects in the same image. Similarly, when training with the labeled data of objects to be detected for another task, the RPN parameters will be adjusted, potentially causing the adjusted RPN to fail to accurately predict the candidate 2D bounding boxes of other objects. Thus, the training data of different tasks may inhibit each other, resulting in the RPN failing to predict all target objects in the image.

[0034] According to the scheme in this application embodiment, the perceptual network is trained based on pseudo-boundary boxes and labeled data. When the labeled data only includes the labeled data of the first task object (i.e., partial labeled data), pseudo-boundary boxes of the second task object are provided. This allows for a more comprehensive set of bounding boxes for the objects to be detected on the same sample image, serving as the target output of the RPN. This adjusts the RPN parameters so that the RPN output continuously approaches the target data, avoiding mutual inhibition between different tasks. This helps the RPN obtain more comprehensive and accurate candidate 2D bounding boxes, while improving recall. In this application embodiment, the labeled data of the sample images can be partial labeled data, allowing for targeted acquisition—that is, acquiring sample images for specific tasks without labeling all objects to be detected for every task in every sample image. This reduces data acquisition and labeling costs and helps balance the training data for different tasks. Furthermore, the partial labeled data scheme offers flexible scalability. When adding tasks, only the labeled data for the new tasks needs to be provided, without needing to label new objects to be detected on top of the existing training data.

[0035] The first task object can include the object to be detected in one or more tasks. These one or more tasks constitute the task to which the first task object belongs. The first task object in different sample images in the training set can be the same or different.

[0036] The second task object can include objects to be detected from one or more tasks. These one or more tasks refer to the tasks in which the second task object resides. The second task object and the first task object may contain the same objects to be detected. That is, the first and second task objects can have overlapping objects to be detected, or they can be completely different. The second task objects in different sample images in the training set can be the same or different.

[0037] Other perceptual networks refer to perceptual networks other than the one to be trained. For example, other perceptual networks can be multi-head, multi-task perceptual networks or multiple single-task perceptual networks, etc.

[0038] In conjunction with the second aspect, in some implementations of the second aspect, the perception network further includes a backbone network, a region of interest extraction module, and a classification and regression network. Training the perception network based on training data includes: calculating a first loss function value based on the difference between the labeled 2D bounding box of the first task object and the target 2D bounding box of the second task object and the candidate 2D bounding boxes of the target object in the sample image predicted by the RPN; calculating a second loss function value of the perception network based on the labeled data; backpropagating the first and second loss function values ​​to adjust the parameters of the parts of the perception network that need to be trained. The parts of the perception network that need to be trained include the parts of the classification and regression network that need to be trained, the region of interest extraction module, the RPN, and the backbone network. The parts of the classification and regression network that need to be trained are determined based on the first task object.

[0039] The labeled 2D bounding box of the first task object and the target 2D bounding box of the second task object are compared with the candidate 2D bounding boxes of the target object predicted by RPN to obtain the loss function value of the RPN stage, i.e., the first loss function value.

[0040] By comparing the labeled data of the sample images with the output of the classification and regression network, the loss function value of the first task object in the classification and regression network stage is obtained, which is the second loss function value.

[0041] The gradients of the parameters related to the first loss function value are calculated, and backpropagation is performed based on the first loss function value. That is, the parameters related to the first loss function value are adjusted based on the gradients of the parameters to adjust the perceptual network so that the RPN can predict candidate boxes more comprehensively.

[0042] The parameters related to the first loss function value are the parameters in the perceptual network used in calculating the first loss function value, such as the backbone parameters and the RPN parameters. Furthermore, if the perceptual network includes an FPN, the parameters related to the first loss function value also include the FPN.

[0043] Backpropagation based on the second loss function value is used to calculate the gradient of the parameters related to the second loss function value. Then, the parameters related to the second loss function value are adjusted based on the gradient of the parameters to adjust the perceptual network, so that the classification and regression network can better correct the output 2D box and improve the accuracy of category prediction.

[0044] The parameters related to the second loss function are the parameters in the perceptual network used in calculating the value of the second loss function. These include, for example, the parameters of the backbone, the RPN, the region of interest extraction module, and the parameters of the parts of the classification and regression network that need to be trained. Furthermore, if the perceptual network includes an FPN, the parameters related to the value of the second loss function also include the parameters of the FPN. The parameters related to the second loss function are essentially the parameters of the parts of the perceptual network that need to be trained.

[0045] According to the scheme in the embodiments of this application, the parts shared by different tasks in the perceptual network, such as the backbone network, RPN, and region of interest extraction module, all participate in the training process based on the labeled data of different tasks. This enables the shared parts of the perceptual network to learn the common features of each task. Different parts corresponding to different tasks in the perceptual network, such as the parts corresponding to each task in the classification and regression network, only participate in the training process based on the labeled data of their respective tasks. This enables the different parts corresponding to different tasks in the perceptual network to learn their task-specific features, improving the accuracy of the model. Simultaneously, during training, the parts of the classification and regression network that need to be trained are determined according to the task. The different parts of the classification and regression networks corresponding to different tasks do not affect each other during training, ensuring the independence of each task and making the model highly flexible.

[0046] In conjunction with the second aspect, in some implementations of the second aspect, a backbone network is used to perform convolution processing on the sample image and output a first feature map of the sample image; an RPN is used to output the position information of candidate 2D boxes of the target object based on the second feature map, the second feature map being determined based on the first feature map; a region of interest extraction module is used to extract first feature information on a third feature map based on the position information of the candidate 2D boxes, the first feature information being the features of the region where the candidate 2D boxes are located, the third feature map being determined based on the first feature map; a classification and regression network is used to process the first feature information and output the target 2D box of the target object and first indication information, the number of target 2D boxes being less than or equal to the number of candidate 2D boxes, the first indication information being used to indicate the category to which the target object belongs.

[0047] In conjunction with the second aspect, in some implementations of the second aspect, the classification and regression network includes a first region convolutional neural network (RCNN). The first RCNN includes a hidden layer, multiple sub-classification fully connected layers, and multiple sub-regression fully connected layers. The hidden layer is connected to multiple sub-classification fully connected layers, and the multiple sub-classification fully connected layers are connected to multiple sub-regression fully connected layers. The multiple sub-classification fully connected layers correspond one-to-one with multiple tasks, and the multiple sub-regression fully connected layers correspond one-to-one with multiple tasks. The hidden layer is used to process the first feature information to obtain the second feature information. The sub-classification fully connected layers are used to obtain the confidence level of the candidate 2D box belonging to the object category in the task corresponding to the sub-classification fully connected layer based on the second feature information. The sub-regression fully connected layers are used to adjust the position information of the candidate 2D box based on the second feature information to obtain the adjusted candidate 2D box. The training part of the classification and regression network includes the hidden layer and the sub-classification fully connected layer and sub-regression fully connected layer corresponding to the task where the object of the first task is located.

[0048] According to the method in this application embodiment, the shared parts of the perceptual network across different tasks—namely, the backbone network, RPN, region of interest extraction module, and hidden layers of the classification and regression network—are all trained during training based on labeled data from different tasks. This allows the shared parts of the perceptual network to learn the common features of each task. The different parts of the perceptual network corresponding to different tasks—namely, the sub-classification fully connected layers and sub-regression fully connected layers of the classification and regression network—are trained only during training based on labeled data from their respective tasks. This allows the different parts of the perceptual network corresponding to different tasks to learn their task-specific features, improving the model's accuracy.

[0049] Thirdly, an object recognition method is provided. The perceptual network includes a backbone network, a candidate region generation network (RPN), a region of interest extraction module, and a classification and regression network. The method includes: using the backbone network to perform convolution processing on the input image to obtain a first feature map of the input image; using the RPN to output the position information of candidate 2D bounding boxes of the target object based on the second feature map, where the target object includes objects to be detected in multiple tasks, and each task in the multiple tasks includes at least one category, and the second feature map is determined based on the first feature map; using the region of interest extraction module to extract first feature information on the third feature map based on the position information of the candidate 2D bounding boxes, where the first feature information is the feature of the region where the candidate 2D bounding box is located, and the third feature map is determined based on the first feature map; and using the classification and regression network to process the first feature information to obtain the target 2D bounding box of the target object and first indication information, where the number of target 2D bounding boxes is less than or equal to the number of candidate 2D bounding boxes, and the first indication information is used to indicate the category to which the target object belongs.

[0050] According to the scheme of the embodiments of this application, a single perception network is used to complete multiple perception tasks, and multiple tasks share a single RPN. That is, a single RPN predicts the region where the object to be detected is located in multiple tasks. While ensuring the performance of the perception network, the number of parameters and the amount of computation of the perception network are reduced, the processing efficiency is improved, and it is beneficial to deploy in scenarios with high real-time requirements. At the same time, it reduces the pressure on the hardware and saves costs.

[0051] In conjunction with the third aspect, in some implementations of the third aspect, a classification and regression network is used to process the first feature information to obtain the target 2D bounding box of the target object and the first indication information, including: processing the first feature information using a classification and regression network to obtain the confidence scores of the candidate 2D bounding boxes belonging to each category in multiple tasks; adjusting the position information of the candidate 2D bounding boxes using a classification and regression network to obtain the adjusted candidate 2D bounding boxes; determining the target 2D bounding box based on the adjusted candidate 2D bounding boxes; and determining the first indication information based on the confidence scores of the target 2D bounding box belonging to each category.

[0052] In conjunction with the third aspect, in some implementations of the third aspect, the classification and regression network includes a first region convolutional neural network (RCNN). The first RCNN includes a hidden layer, multiple sub-classification fully connected layers, and multiple sub-regression fully connected layers. The hidden layer is connected to multiple sub-classification fully connected layers, and the multiple sub-classification fully connected layers are connected to multiple sub-regression fully connected layers. The multiple sub-classification fully connected layers correspond one-to-one with multiple tasks, and the multiple sub-regression fully connected layers correspond one-to-one with multiple tasks. The classification and regression network processes the first feature information to output the target 2D bounding box of the target object and the first indication information, including: processing the first feature information using the hidden layer to obtain second feature information; using the sub-classification fully connected layers to obtain the confidence level of the candidate 2D bounding box belonging to the object category in the task corresponding to the sub-classification fully connected layer based on the second feature information; and using the sub-regression fully connected layers to adjust the position information of the candidate 2D bounding box based on the second feature information to obtain the adjusted candidate 2D bounding box.

[0053] In conjunction with the third aspect, in some implementations of the third aspect, the classification and regression network includes a second RCNN, which includes a hidden layer, a classification fully connected layer, and a regression fully connected layer. The hidden layer is connected to the classification fully connected layer, and the hidden layer is connected to the regression fully connected layer. The classification and regression network is used to process the first feature information and output the target 2D bounding box of the target object and the first indication information, including: using the hidden layer to process the first feature information to obtain the third feature information; using the classification fully connected layer to obtain the confidence of the candidate 2D bounding box belonging to each category based on the third feature information; and using the regression fully connected layer to adjust the position information of the candidate 2D bounding box based on the third feature information to obtain the adjusted candidate 2D bounding box.

[0054] In conjunction with the third aspect, in some implementations of the third aspect, the classification fully connected layer is obtained by merging multiple sub-classification fully connected layers in the first RCNN, and the regression fully connected layer is obtained by merging multiple sub-regression fully connected layers in the first RCNN. The first RCNN includes hidden layers, multiple sub-classification fully connected layers, and multiple sub-regression fully connected layers. The hidden layers are connected to multiple sub-classification fully connected layers, and the hidden layers are connected to multiple sub-regression fully connected layers. The multiple sub-classification fully connected layers correspond one-to-one with multiple tasks, and the multiple sub-regression fully connected layers correspond one-to-one with multiple tasks. The sub-classification fully connected layer is used to obtain the confidence score of the candidate 2D box belonging to the object category in the task corresponding to the sub-classification fully connected layer based on the third feature information. The sub-regression fully connected layer is used to adjust the position information of the candidate 2D box based on the third feature information to obtain the adjusted candidate 2D box.

[0055] The target perceptual network can be obtained by using the training method of the perceptual network in the second aspect. The target perceptual network can be a pre-trained image recognition model, and the pre-trained perceptual network can be used to process the image to be processed.

[0056] Fourthly, a training apparatus for a perceptual network is provided, the apparatus including a module or unit for performing the methods of the second aspect and any implementation thereof.

[0057] Fifthly, an object recognition device is provided, the device including a module or unit for performing the method in the third aspect and any implementation thereof.

[0058] It should be understood that the extensions, limitations, interpretations and descriptions of the relevant content in the first and second aspects above also apply to the same content in the third, fourth and fifth aspects.

[0059] In a sixth aspect, a training apparatus for a perceptual network is provided, the apparatus comprising: a processor and a transmission interface, the processor receiving or transmitting data through the transmission interface, the processor being configured to invoke program instructions stored in memory to execute the methods of the second aspect and any implementation thereof.

[0060] The processor mentioned in the sixth aspect above can be a central processing unit (CPU) or a combination of a CPU and a neural network processing processor. The neural network processing processor can include a graphics processing unit (GPU), a neural network processing unit (NPU), and a tensor processing unit (TPU), etc. The TPU is a dedicated integrated circuit developed by Google for a fully customized artificial intelligence accelerator for machine learning.

[0061] In a seventh aspect, an object recognition device is provided, comprising: a processor and a transmission interface, wherein the processor receives or transmits data through the transmission interface, and the processor is configured to invoke program instructions stored in a memory to execute the methods of the third aspect and any implementation thereof.

[0062] The processor mentioned in the seventh aspect above can be a central processing unit (CPU) or a combination of a CPU and a neural network processing processor. The neural network processing processor can include a graphics processing unit (GPU), a neural network processor, and a tensor processor, among others. The TPU is a Google application-specific integrated circuit (ASIC) for a fully custom-designed AI accelerator for machine learning.

[0063] Eighthly, a computer-readable storage medium is provided that stores program code for execution by a device, which, when run on a computer or processor, causes the computer or processor to perform the method in any of the implementations of the second or third aspect.

[0064] Ninth aspect, a computer program product containing instructions is provided, which, when run on a computer, causes the computer to perform the method in any one of the implementations of the second or third aspect described above.

[0065] In a tenth aspect, a chip is provided, the chip including a processor and a data interface, wherein the processor reads instructions stored in a memory through the data interface and executes the method in any one of the implementations of the second or third aspect described above.

[0066] Optionally, as one implementation, the chip may further include a memory storing instructions, and the processor is used to execute the instructions stored in the memory. When the instructions are executed, the processor is used to execute the method in either the first aspect or the second aspect.

[0067] The aforementioned chip can be a field-programmable gate array (FPGA) or an application-specific integrated circuit (ASIC).

[0068] Eleventhly, an electronic device is provided, which includes the means of any one of the fourth to seventh aspects described above. Attached Figure Description

[0069] Figure 1 A schematic diagram illustrating an application scenario provided in an embodiment of this application; Figure 2 A schematic diagram illustrating another application scenario provided by an embodiment of this application; Figure 3 A schematic diagram of a system architecture provided in an embodiment of this application; Figure 4 This is a schematic diagram of the structure of a convolutional neural network provided in an embodiment of this application; Figure 5 A schematic diagram of the hardware structure of a chip provided in an embodiment of this application; Figure 6 A schematic diagram of a system architecture provided for an embodiment of this application; Figure 7 A schematic block diagram of a multi-head, multi-task sensing network; Figure 8 A schematic structural diagram of a sensing network provided for an embodiment of this application; Figure 9 A schematic structural diagram of another sensing network provided in an embodiment of this application; Figure 10 A schematic structural diagram of another sensing network provided in an embodiment of this application; Figure 11 A schematic structural diagram of another sensing network provided in an embodiment of this application; Figure 12 A schematic flowchart illustrating the training method for a perceptual network provided in an embodiment of this application; Figure 13 A schematic diagram illustrating a training process for a perceptual network provided in an embodiment of this application; Figure 14 A schematic block diagram of the perceptual network during the training process provided in the embodiments of this application; Figure 15 A schematic flowchart illustrating an object recognition method provided in an embodiment of this application; Figure 16A schematic diagram of an object recognition process provided in an embodiment of this application; Figure 17 A schematic block diagram of the perceptual network in the inference process provided in the embodiments of this application; Figure 18 A schematic diagram illustrating the conversion process of the sensing network provided in an embodiment of this application; Figure 19 A schematic block diagram of an apparatus provided in an embodiment of this application; Figure 20 A schematic block diagram of another device provided in an embodiment of this application. Detailed Implementation

[0070] The technical solutions in this application will now be described with reference to the accompanying drawings.

[0071] This application can be applied to fields requiring multiple perception tasks, such as driver assistance, autonomous driving, mobile terminals, monitoring, and security. Images are input into the perception network of this application to obtain detection results of objects of interest in the images. These detection results can be processed in a post-processing module; for example, they can be sent to a planning and control unit for decision-making in an autonomous driving system, or to a security system for anomaly detection.

[0072] The following is a brief introduction to three application scenarios of Advanced Driving Assistant Systems (ADAS) / Autonomous Driving Systems (ADS): visual perception system, photo album classification, and monitoring.

[0073] ADAS / ADS visual perception system: like Figure 1As shown, ADAS and ADS require real-time detection of various types of targets, including dynamic obstacles, static obstacles, and traffic signs. Examples include pedestrians, cyclists, tricycles, cars, trucks, buses, wheels, car lights, traffic cones, traffic sticks, fire hydrants, motorcycles and bicycles, traffic signs, guide signs, billboards, road signs, poles, traffic lights, and road markings. Traffic lights include red, yellow, green, and black traffic lights. Road signs include Around, Straight, Left / Right, Straight and Left, Straight and Right, Straight and U-turn, Left and U-turn, Left and Right, Left bend, Right bend, and Pedestrian crossing signs.

[0074] Using the solution of the embodiments of this application, the detection tasks of the above-mentioned multiple objectives can be achieved in a single sensing network. That is, multiple objects to be detected can be detected in a single sensing network. After processing, the detection results can be sent to the planning and control unit for decision-making, such as obstacle avoidance, traffic light decision-making, or traffic sign decision-making.

[0075] Photo album categories: When users store a large number of pictures on terminal devices (such as mobile phones) or cloud storage, recognizing the images in the album can make it easier for users or the system to classify and manage the album, thus improving the user experience.

[0076] The solution implemented in this application provides a perceptual network suitable for classifying photos in an album. This perceptual network is then used to classify images, for example, into different categories such as photos containing animals or photos containing people, thereby tagging images of different categories for easy viewing and searching by users. Furthermore, these image classification tags can be provided to the album management system for categorization management, saving users' management time, improving album management efficiency, and enhancing the user experience.

[0077] monitor: The monitoring scenarios include: smart cities, field monitoring, indoor monitoring, outdoor monitoring, and vehicle monitoring.

[0078] like Figure 2 As shown, a smart city perception system needs to complete a variety of detection tasks, such as detecting vehicles, license plates, people, and faces. After processing, the detection results can be used to judge traffic violations and predict the degree of traffic congestion.

[0079] The solution adopted in this application embodiment can process the input road image in a single perception network to complete the detection tasks of the aforementioned multiple targets. Furthermore, the detection tasks of the perception network can be increased or decreased according to actual needs. For example, the current perception network's detection tasks include vehicle detection and pedestrian detection. If it is necessary to add traffic sign detection to the perception network's detection tasks, the structure of the perception network can be adjusted to add this detection task. Detailed descriptions can be found below, for example... Figure 14 .

[0080] Since the embodiments of this application involve a large number of neural network applications, for ease of understanding, the relevant terms and concepts of neural networks that may be involved in the embodiments of this application will be introduced below.

[0081] (1) Neural Network Neural networks can be composed of neural units, which can refer to units represented by x. s The arithmetic unit that takes an intercept of 1 as input can output the following: (1-1) Where s = 1, 2, ..., n, n is a natural number greater than 1, W s For x sThe weights are denoted by b, where b is the bias of the neural unit. f is the activation function of the neural unit, used to introduce non-linear characteristics into the neural network to convert the input signal into the output signal. The output signal of this activation function can be used as the input to the next convolutional layer; the activation function can be the sigmoid function. A neural network is a network formed by connecting multiple individual neural units, meaning the output of one neural unit can be the input of another. The input of each neural unit can be connected to the local receptive field of the previous layer to extract features from the local receptive field, which can be a region composed of several neural units.

[0082] (2) Deep Neural Networks A deep neural network (DNN), also known as a multilayer neural network, can be understood as a neural network with multiple hidden layers. Based on the position of the layers, the internal neural network of a DNN can be divided into three categories: input layer, hidden layer, and output layer. Generally, the first layer is the input layer, the last layer is the output layer, and the layers in between are hidden layers. The layers are fully connected, meaning that any neuron in the i-th layer is connected to any neuron in the (i+1)-th layer.

[0083] Although DNNs seem complex, the operation of each layer is actually not complicated. Simply put, it involves the following linear relationship expression: ,in, It is the input vector. It is the output vector. It is an offset vector. It is the weight matrix (also called coefficients). It's an activation function. Each layer simply applies the input vector... The output vector is obtained through such a simple operation. Because DNNs have many layers, the coefficients... and offset vector The number of parameters is also quite large. These parameters are defined in DNNs as follows: [as coefficients] For example: Suppose in a three-layer DNN, the linear coefficient from the fourth neuron in the second layer to the second neuron in the third layer is defined as... The superscript 3 represents the coefficient. The index corresponds to the third-level index 2 in the output and the second-level index 4 in the input.

[0084] In summary, the Lth The coefficients from the k-th neuron in layer 1 to the j-th neuron in layer L are defined as follows: .

[0085] It should be noted that the input layer does not have... In deep neural networks, more hidden layers allow the network to better represent complex real-world situations. Theoretically, the more parameters a model has, the higher its complexity and "capacity," meaning it can perform more complex learning tasks. Training a deep neural network is essentially the process of learning the weight matrix, and its ultimate goal is to obtain the weight matrix of all layers in the trained deep neural network (composed of vectors from many layers). The resulting weight matrix.

[0086] (3) Convolutional Neural Network A convolutional neural network (CNN) is a deep neural network with a convolutional structure. A CNN contains a feature extractor consisting of convolutional layers and subsampling layers. This feature extractor can be viewed as a filter, and the convolution process can be seen as using a trainable filter to convolve with an input image or a convolutional feature map. A convolutional layer is a layer of neurons in a CNN that performs convolution processing on the input signal. In a convolutional layer of a CNN, a neuron may only be connected to some of its neighboring neurons. A convolutional layer typically contains several feature planes, each composed of rectangularly arranged neural units. Neural units on the same feature plane share weights, which are the convolutional kernel. Shared weights can be understood as the way image information is extracted regardless of location. The underlying principle is that the statistical information of one part of the image is the same as that of other parts. This means that image information learned in one part can also be used in another part. Therefore, the same learned image information can be used for all locations in the image. In the same convolutional layer, multiple convolutional kernels can be used to extract different image information. Generally, the more convolutional kernels there are, the richer the image information reflected by the convolution operation.

[0087] Convolutional kernels can be initialized as matrices of random size, and during the training of a convolutional neural network, they can learn appropriate weights. Furthermore, sharing weights directly reduces the number of connections between layers in the convolutional neural network, while also lowering the risk of overfitting.

[0088] (4) Loss function In training deep neural networks, to ensure the output closely approximates the desired predicted value, we compare the network's prediction with the target value and update the weight vector of each layer based on the difference. (Of course, there's usually a pre-configuration process before the first update, where parameters are pre-configured for each layer.) For example, if the prediction is too high, the weight vector is adjusted to predict a lower value. This adjustment continues until the deep neural network can predict the target value or a value very close to it. Therefore, we need to predefine "how to compare the difference between the predicted and target values," which is the loss function or objective function. These are important equations used to measure the difference between the predicted and target values. Taking the loss function as an example, a higher output value (loss) indicates a greater difference, and training the deep neural network becomes a process of minimizing this loss. Generally, a smaller loss indicates higher training quality, while a larger loss indicates lower training quality. Similarly, smaller loss fluctuations result in more stable training, while larger loss fluctuations lead to less stable training.

[0089] (5) Backpropagation algorithm Neural networks can employ backpropagation (BP) algorithms to correct the parameters in the initial model during training, thereby reducing the reconstruction error loss. Specifically, forward propagation of the input signal to the output generates an error loss; this error loss information is then propagated back to update the parameters of the initial super-resolution model, leading to convergence of the error loss. The backpropagation algorithm is an error-loss-driven backpropagation process aimed at obtaining the optimal parameters of the super-resolution model, such as the weight matrix.

[0090] The method provided in this application is described below from the perspectives of model training and model application.

[0091] The perceptual network training method provided in this application involves computer vision processing. Specifically, it can apply data processing methods such as data training, machine learning, and deep learning to perform symbolic and formal intelligent information modeling, extraction, preprocessing, and training on training data, ultimately obtaining a trained perceptual network. Furthermore, the object recognition method provided in this application can utilize the aforementioned trained perceptual network, inputting input data (such as the image to be processed in this application) into the trained perceptual network to obtain output data (such as the first indication information and the target 2D bounding box of the target object in this application). It should be noted that the perceptual network training method and the object recognition method provided in this application are based on the same concept and can also be understood as two parts of a system or two stages of an overall process: such as the model training stage and the model application stage.

[0092] like Figure 3 As shown, this application embodiment provides a system architecture 100. In Figure 3 In this embodiment, the data acquisition device 160 is used to acquire training data. For the training method of the perceptual network in this application, the training data may include sample images, annotation data of the sample images, and pseudo-boundaries on the sample images.

[0093] After collecting the training data, the data acquisition device 160 stores the training data in the database 130, and the training device 120 trains the target model / rule 101 based on the training data maintained in the database 130.

[0094] The following section will describe in detail the specific method by which the training device 120 obtains the target model / rule 101 based on the training data. The target model / rule 101 can be used to implement the object recognition method of this application embodiment; that is, by inputting the image to be processed into the target model / rule 101, the detection result of the object of interest in the image to be processed can be obtained. It should be noted that in practical applications, the training data maintained in the database 130 may not all come from the data acquisition device 160; it may also be received from other devices. Furthermore, it should be noted that the training device 120 may not necessarily train the target model / rule 101 entirely based on the training data maintained in the database 130; it may also obtain training data from the cloud or other sources for model training. The above description should not be construed as limiting the embodiments of this application.

[0095] The target model / rule 101 trained using training device 120 can be applied to different systems or devices, such as... Figure 3The execution device 110 shown can be a terminal, such as a mobile phone, tablet, laptop, augmented reality (AR) / virtual reality (VR) device, in-vehicle terminal, etc., or it can be a server or cloud device. Figure 3 In this embodiment, the execution device 110 is configured with an input / output (I / O) interface 112 for data interaction with external devices. Users can input data to the I / O interface 112 through the client device 140. The input data may include an image to be processed.

[0096] During the preprocessing of input data by the execution device 110, or during the calculation module 111 of the execution device 110 performing calculations and other related processes, the execution device 110 can call data, code, etc. in the data storage system 150 for corresponding processing, or store the data, instructions, etc. obtained from the corresponding processing into the data storage system 150.

[0097] Finally, I / O interface 112 returns the processing results, such as the detection results obtained above, to client device 140, thereby providing them to the user.

[0098] For example, customer equipment 140 can be a planning control unit in an autonomous driving system.

[0099] It is worth noting that the training device 120 can generate corresponding target models / rules 101 based on different training data for different objectives or tasks. The corresponding target models / rules 101 can be used to achieve the above objectives or complete the above tasks, thereby providing the user with the required results.

[0100] exist Figure 3In the scenario shown, the user can manually provide input data, which can be done through the interface provided by I / O interface 112. Alternatively, the client device 140 can automatically send input data to I / O interface 112. If user authorization is required for the client device 140 to automatically send input data, the user can set the corresponding permissions in the client device 140. The user can view the output results of the execution device 110 on the client device 140, which can be presented in various forms such as display, sound, or animation. The client device 140 can also act as a data acquisition terminal, collecting the input data and output results of the input I / O interface 112 as new sample data and storing them in the database 130. Alternatively, data can be collected directly from the I / O interface 112 without going through the client device 140, using the input data and output results of the input I / O interface 112 as new sample data and storing them in the database 130.

[0101] It is worth noting that, Figure 3 This is merely a schematic diagram of a system architecture provided in an embodiment of this application. The positional relationships between the devices, components, modules, etc., shown in the diagram do not constitute any limitation. For example, in Figure 3 In this context, the data storage system 150 is an external memory relative to the execution device 110. In other cases, the data storage system 150 may also be placed within the execution device 110.

[0102] like Figure 3 As shown, a target model / rule 101 is obtained by training the training device 120. In this embodiment of the application, the target model / rule 101 may be a perceptual network.

[0103] Since CNN is a very common type of neural network, the following will combine... Figure 4 This section focuses on a detailed explanation of the structure of CNNs. As mentioned in the basic concept introduction above, a Convolutional Neural Network (CNN) is a deep neural network with a convolutional structure. It is a deep learning architecture, which refers to an algorithm that updates a neural network model by performing multi-level learning at different abstraction levels. As a deep learning architecture, CNN is a feed-forward artificial neural network, in which each neuron can respond to the input image.

[0104] The image recognition method in this application embodiment specifically employs a neural network structure as follows: Figure 4 As shown. In Figure 4In this example, the Convolutional Neural Network (CNN) 200 may include an input layer 210, convolutional / pooling layers 220 (where pooling layers are optional), and a fully connected layer 230. The input layer 210 acquires the image to be processed and passes it to the convolutional / pooling layers 220 and the subsequent fully connected layer 230 for processing to obtain the image processing result. The following section... Figure 4 This section provides a detailed introduction to the internal layer structure of CNN200.

[0105] Convolutional / pooling layer 220: Convolutional layers: like Figure 4 The convolutional / pooling layer 220 shown may include layers as in Examples 221-226. For instance, in one implementation, layer 221 is a convolutional layer, layer 222 is a pooling layer, layer 223 is a convolutional layer, layer 224 is a pooling layer, layer 225 is a convolutional layer, and layer 226 is a pooling layer; in another implementation, layers 221 and 222 are convolutional layers, layer 223 is a pooling layer, layers 224 and 225 are convolutional layers, and layer 226 is a pooling layer. That is, the output of the convolutional layer can be used as the input to a subsequent pooling layer, or as the input to another convolutional layer to continue the convolution operation.

[0106] The following section will use convolutional layer 221 as an example to introduce the internal working principle of a convolutional layer.

[0107] Convolutional layer 221 can include multiple convolution operators, also known as kernels. In image processing, a convolution operator acts as a filter to extract specific information from the input image matrix. Essentially, a convolution operator can be a weight matrix, which is usually predefined. During the convolution operation, the weight matrix typically processes the input image pixel by pixel (or two pixels by two pixels, depending on the stride) along the horizontal direction, thus extracting specific features from the image. The size of the weight matrix should be related to the image size. It's important to note that the depth dimension of the weight matrix is ​​the same as the depth dimension of the input image; during convolution, the weight matrix extends to the entire depth of the input image. Therefore, convolution with a single weight matrix produces a single-depth convolutional output. However, in most cases, a single weight matrix is ​​not used; instead, multiple weight matrices of the same size (rows × columns) are applied—multiple identical matrices. The outputs of each weight matrix are stacked to form the depth dimension of the convolutional image; this dimension can be understood as being determined by the "multiple" mentioned above. Different weight matrices can be used to extract different features from an image. For example, one weight matrix can be used to extract image edge information, another weight matrix can be used to extract specific colors of the image, and yet another weight matrix can be used to blur unwanted noise in the image. These multiple weight matrices have the same size (rows × columns), and the convolutional feature maps extracted by these multiple weight matrices of the same size also have the same size. Then, the multiple convolutional feature maps of the same size are merged to form the output of the convolution operation.

[0108] The weight values ​​in these weight matrices need to be obtained through extensive training in practical applications. The weight matrices formed by the weight values ​​obtained through training can be used to extract information from the input image, thereby enabling the convolutional neural network 200 to make correct predictions.

[0109] When a convolutional neural network 200 has multiple convolutional layers, shallow convolutional layers (such as 221) tend to extract more general features, which can also be called low-level features. As the depth of the convolutional neural network 200 increases, the features extracted by later convolutional layers (such as 226) become more and more complex, such as high-level semantic features. Features with higher semantic levels are more suitable for the problem to be solved.

[0110] Pooling layer: Because it is often necessary to reduce the number of training parameters, pooling layers are often introduced periodically after convolutional layers, such as... Figure 4Layers 221-226 in example 220 can be a convolutional layer followed by a pooling layer, or multiple convolutional layers followed by one or more pooling layers. In image processing, the sole purpose of pooling layers is to reduce the spatial size of the image. Pooling layers can include average pooling and / or max pooling operators to sample the input image to obtain a smaller image size. The average pooling operator calculates the average value of pixel values ​​within a specific range as the result of average pooling. The max pooling operator takes the pixel with the largest value within a specific range as the result of max pooling. Furthermore, just as the size of the weight matrix in a convolutional layer should be related to the image size, the operators in a pooling layer should also be related to the image size. The size of the output image after processing by the pooling layer can be smaller than the size of the input image of the pooling layer. Each pixel in the output image of the pooling layer represents the average or maximum value of the corresponding sub-region of the input image of the pooling layer.

[0111] Fully connected layer 230: After processing by the convolutional / pooling layers 220, the convolutional neural network 200 is still insufficient to output the required information. As mentioned earlier, the convolutional / pooling layers 220 only extract features and reduce the parameters introduced by the input image. However, to generate the final output information (the required class information or other relevant information), the convolutional neural network 200 needs to utilize fully connected layers 230 to generate one or a set of outputs representing the required number of classes. Therefore, the fully connected layers 230 can include multiple hidden layers (such as...). Figure 4 As shown in layers 231, 232 to 23n) and output layer 240, the parameters contained in these multi-layer hidden layers can be pre-trained based on relevant training data for specific task types, such as image recognition, image classification, image super-resolution reconstruction, etc.

[0112] After the multiple hidden layers in the fully connected layer 230, the final layer of the entire convolutional neural network 200 is the output layer 240. This output layer 240 has a loss function similar to the classification cross-entropy, specifically used to calculate the prediction error. Once the entire convolutional neural network 200 has propagated forward (e.g., ... Figure 4 Propagation from 210 to 240 degrees is considered forward propagation, while backward propagation (e.g.) is completed. Figure 4 The propagation from 240 to 210 (backpropagation) will begin to update the weight values ​​and biases of the layers mentioned above in order to reduce the loss of the convolutional neural network 200, that is, the error between the output of the convolutional neural network 200 through the output layer and the ideal result.

[0113] It should be noted that, Figure 4The convolutional neural network shown is only one possible example of a convolutional neural network. In specific applications, convolutional neural networks can also exist in the form of other network models.

[0114] Figure 5 The present application provides a hardware structure for a chip, which includes a neural network processor 50. This chip can be configured as follows: Figure 3 The execution device 110 shown is used to perform the calculations of the calculation module 111. This chip can also be located in, for example... Figure 3 The training device 120 shown is used to complete the training work of the training device 120 and output the target model / rule 101. The method in this embodiment can be used as follows: Figure 5 This is achieved in the chip shown.

[0115] The Neural Processing Unit (NPU) 50 is mounted as a coprocessor on the main central processing unit (CPU) (host CPU), and tasks are assigned by the host CPU. The core of the NPU is the arithmetic circuit 503, and the controller 504 controls the arithmetic circuit 503 to retrieve data from the memory (weight memory or input memory) and perform calculations.

[0116] In some implementations, the arithmetic circuit 503 internally includes multiple process engines (PEs). In some implementations, the arithmetic circuit 503 is a two-dimensional pulsating array. The arithmetic circuit 503 can also be a one-dimensional pulsating array or other electronic circuitry capable of performing mathematical operations such as multiplication and addition. In some implementations, the arithmetic circuit 503 is a general-purpose matrix processor.

[0117] For example, suppose we have an input matrix A, a weight matrix B, and an output matrix C. The arithmetic circuit retrieves the corresponding data of matrix B from the weight memory 502 and caches it in each PE of the arithmetic circuit. The arithmetic circuit retrieves the data of matrix A from the input memory 501 and performs matrix operations with matrix B. The partial result or the final result of the obtained matrix is ​​stored in the accumulator 508.

[0118] The vector computation unit 507 can further process the output of the arithmetic circuit, such as vector multiplication, vector addition, exponentiation, logarithmic operations, size comparisons, etc. For example, the vector computation unit 507 can be used for network computation in non-convolutional / non-FC layers of neural networks, such as pooling, batch normalization (BN), local response normalization, etc.

[0119] In some implementations, the vector computation unit 507 can store the processed output vector into a unified buffer 506. For example, the vector computation unit 507 can apply a nonlinear function to the output of the arithmetic circuit 503, such as a vector of accumulated values, to generate activation values. In some implementations, the vector computation unit 507 generates normalized values, merged values, or both. In some implementations, the processed output vector can be used as activation input to the arithmetic circuit 503, for example, for use in subsequent layers of a neural network.

[0120] The operations of the perception network provided in this application embodiment can be performed by the operation circuit 503 or the vector calculation unit 507.

[0121] The unified memory 506 is used to store input data and output data.

[0122] The weight data is directly transferred from the external memory to the input memory 501 and / or the unified memory 506 through the direct memory access controller 505 (DMAC), the weight data in the external memory is stored in the weight memory 502, and the data in the unified memory 506 is stored in the external memory.

[0123] The bus interface unit (BIU) 510 is used to enable interaction between the main CPU, DMAC and instruction fetch memory 509 via a bus.

[0124] The instruction fetch buffer 509, connected to the controller 504, is used to store the instructions used by the controller 504. The controller 504 is used to call the instructions cached in the instruction memory 509 to control the operation of the computing accelerator.

[0125] Generally, the unified memory 506, input memory 501, weight memory 502, and instruction fetch memory 509 are all on-chip memories, while the external memory is memory outside the NPU. This external memory can be double data rate synchronous dynamic random access memory (DDR SDRAM), high bandwidth memory (HBM), or other readable and writable memory.

[0126] The above-mentioned Figure 3 The execution device 110 or Figure 5The chip in the chip is capable of executing the various steps of the object recognition method described in the embodiments of this application. Figure 3 Training equipment 120 or Figure 5 The chip in the application is capable of executing the various steps of the training method for the perceptual network according to the embodiments of this application.

[0127] like Figure 6 As shown, this application embodiment provides a system architecture 300. The system architecture includes a local device 301, a local device 302, an execution device 310, and a data storage system 350, wherein the local devices 301 and 302 are connected to the execution device 310 through a communication network.

[0128] In one implementation, the execution device 310 can be implemented by one or more servers. Optionally, the execution device 310 can be used in conjunction with other computing devices, such as data storage devices, routers, load balancers, etc. The execution device 310 can be deployed on a single physical site or distributed across multiple physical sites. The execution device 310 can use data in the data storage system 350 or call program code in the data storage system 350 to implement the training method of the perceptual network of this application embodiment.

[0129] Specifically, in one implementation, the perceptual network includes: a candidate region generation network (RPN), which is used to predict the position information of candidate 2D bounding boxes of target objects in a sample image. The target objects include objects to be detected for multiple tasks, and each of the multiple tasks includes at least one category. The target objects include a first task object and a second task object.

[0130] The execution device 110 can perform the following processes: Acquire training data, which includes sample images, labeled data of the first task object on the sample images, and pseudo-boundary boxes of the second task object on the sample images. The labeled data includes the class label of the first task object and the labeled 2D bounding box of the first task object. The pseudo-boundary box of the second task object is the target 2D bounding box of the second task object obtained by reasoning about the sample images through other perceptual networks. Train the perceptual network based on the training data.

[0131] The device 110 can acquire a sensing network through the above process, which can be used for the detection of various tasks.

[0132] Users can interact with execution device 310 by operating their respective user devices (such as local device 301 and local device 302). Each local device can represent any computing device, such as a surveillance camera, personal computer, computer workstation, smartphone, tablet, smart camera, smart car or other type of cellular phone, media consumption device, wearable device, set-top box or game console, etc.

[0133] Each user's local device can interact with the execution device 310 through a communication network of any communication mechanism / standard. The communication network can be a wide area network, a local area network, a point-to-point connection, or any combination thereof.

[0134] In one implementation, local devices 301 and 302 obtain relevant parameters of the perception network from execution device 310, deploy the perception network on local devices 301 and 302, and use the perception network to perform target detection.

[0135] In another implementation, a sensing network can be directly deployed on the execution device 310. The execution device 310 acquires the image to be processed from local devices 301 and 302 and processes the image using the sensing network.

[0136] The aforementioned execution device 310 can also be a cloud device, in which case the execution device 310 can be deployed in the cloud; or, the aforementioned execution device 310 can also be a terminal device, in which case the execution device 310 can be deployed on the user terminal side. This application embodiment does not limit this.

[0137] For example, the perception network can be deployed on computing nodes of vehicle-mounted visual perception devices, urban perception devices, or security perception devices to process images and obtain detection results of objects of interest in the images. For example, the computing node can be... Figure 3 Execution device 110 in Figure 5 The execution device 310 or local device, etc.

[0138] Most current perceptrons can only perform one detection task. To achieve multiple detection tasks, different networks are usually needed to perform different tasks. However, running multiple perceptrons simultaneously increases hardware power consumption and reduces model execution speed. Existing perceptrons capable of performing multiple detection tasks also suffer from long execution times. For example, ... Figure 7The diagram illustrates a multi-header, multi-task perceptual network. This network comprises a backbone and multiple headers. Each header includes a region proposal network (RPN), a region of interest align (ROI-Align) module, and region convolutional neural networks (RCNNs). RPN's proposal generation is time-consuming, making it unsuitable for scenarios with high real-time requirements. Furthermore, the number of headers increases with the number of detection tasks, leading to a rapid increase in memory consumption, computational power, and processing time. Many fields utilize chips with limited computing power, making it difficult to deploy large-scale perceptual networks, let alone multiple perceptual networks.

[0139] This application provides a perception network that can reduce the number of parameters and computational load in the perception network, reduce hardware power consumption, and improve the running speed of the model.

[0140] Figure 8 A schematic diagram of a sensing network according to an embodiment of this application is shown. Figure 8 The sensing network 800 in the middle includes a backbone network 810 and a header network.

[0141] The sensing network in this application embodiment can be implemented by hardware, software, or a combination of both.

[0142] The backbone network 810 is used to perform convolution processing on the input image to obtain the first feature map of the input image.

[0143] The backbone network 810 can extract basic features through a series of convolutional processes, providing corresponding features for subsequent detection.

[0144] In this embodiment of the application, "first feature map" refers to the feature map output by the backbone network. Any feature map output by the backbone network can be called the first feature map.

[0145] The first feature map of the input image can be one or more.

[0146] For example, the backbone network 810 can output feature maps of the input image at different scales. The feature maps at different scales can all be understood as the first feature maps of the input image, and these feature maps can provide basic features for subsequent detection.

[0147] Feature maps at different scales can be understood as feature maps at different resolutions, or feature maps of different sizes.

[0148] For example, the backbone network 810 can adopt various forms of networks, such as visual geometry group (VGG), residual neural network (ResNet), or inception network (inception-net), which is the core structure of GoogleNet, etc.

[0149] The header is used to detect target objects based on the second feature map, outputting a 2D bounding box of the target object and a first indication. Target objects include objects to be detected in multiple tasks. The second feature map is determined based on the first feature map. The first indication indicates the category to which the target object belongs.

[0150] In other words, the header is used to perform target detection based on the second feature map, and outputs the target 2D bounding box of the target object and the first indication information.

[0151] For example, the first indication information may include the confidence level of the target object belonging to each category. That is, the category to which the target object belongs can be indicated by the confidence level of the target object belonging to each category. The higher the confidence level, the greater the probability that the target object belongs to the category corresponding to that confidence level. For example, the category corresponding to the highest confidence level is the category to which the target object belongs. Alternatively, the first indication information may be the category to which the target object belongs. Or, the first indication information may include the confidence level of the category to which the target object belongs. Each category includes object categories in multiple tasks. The embodiments of this application do not limit the specific form of the first indication information.

[0152] A header can perform the detection of objects in multiple tasks, that is, it is used to detect whether the objects to be detected in the input image exist in multiple tasks.

[0153] Multiple tasks can also be understood as multiple broad categories. A broad category includes at least one subcategory. Or, a broad category is a set of at least one subcategory. The criteria for dividing tasks can be set as needed. For example, objects to be detected can be divided into multiple tasks based on the similarity of the objects to be detected.

[0154] The objects to be detected are divided into multiple tasks. This can also be understood as dividing the objects into multiple broad categories. The object categories in each task can be the same or different.

[0155] For example, based on the similarity of the objects to be detected and the abundance and scarcity of the training samples, the 31 categories of objects to be detected are divided into 8 major categories, i.e. 8 tasks, as shown in Table 1.

[0156] Table 1

[0157] It should be noted that the division method in Table 1 is only an example. In other embodiments, different task division methods may be used than those in Table 1. This application embodiment does not limit this.

[0158] A header can be used to perform multiple object detection tasks. For example, a header can perform the eight tasks in Table 1 above, outputting the 2D bounding box of the target object and the confidence score of the target object belonging to the 31 object classes.

[0159] Optionally, the perception network 800 may also include other processing modules connected to the header. These other processing modules are used to obtain additional detection information of the target object based on the target 2D bounding box of the target object output by the header.

[0160] For example, other processing modules can extract features of the region where the target 2D bounding box is located from the feature map output by the backbone network based on the target 2D bounding box output by the header, and complete 3D detection of the target object or key point detection within the target 2D bounding box based on the extracted features.

[0161] It should be understood that the above is for illustrative purposes only, and other processing modules are optional and can be set according to actual needs. This application embodiment does not limit them.

[0162] The header will be described in detail below.

[0163] Specifically, the header includes RPN820, region of interest extraction module830, and classification and regression network840.

[0164] The RPN820 is used to predict the region where the target object is located on the second feature map and output the position information of the candidate 2D bounding boxes that match the region where the target object is located, i.e., the position information of the candidate 2D bounding boxes of the target object. The target object includes objects to be detected in multiple tasks, each of which includes at least one category, and the second feature map is determined based on the first feature map.

[0165] The region of interest extraction module 830 is used to extract first feature information on the third feature map based on the position information of the candidate 2D box. The first feature information is the feature of the region where the candidate 2D box is located. The third feature map is determined based on the first feature map.

[0166] The classification and regression network 840 is used to process the first feature information and output the target 2D bounding box of the target object and the first indication information. The number of target 2D bounding boxes is less than or equal to the number of candidate 2D bounding boxes. The first indication information is used to indicate the category to which the target object belongs.

[0167] For example, such as Figure 8 As shown, the classification and regression network can output the target 2D bounding box and class label of the object to be detected in multiple tasks. The class label of the target object can serve as the primary indication information. It should be understood that... Figure 8 The use of class labels as the first indication information is merely an example and does not constitute a limitation on the solutions of this application.

[0168] In this embodiment of the application, multiple tasks share the same RPN, which can also be called a single-header multitasking RPN.

[0169] The RPN820 can predict regions where a target object may exist on the second feature map and provide bounding boxes that match these regions. These regions are called candidate regions (proposals), and the bounding boxes that match the candidate regions are called candidate 2D bounding boxes. The bounding boxes that match the proposals can also be called the 2D bounding boxes of the proposals.

[0170] The target object includes objects to be detected in multiple tasks, such as the objects to be detected in the eight tasks in Table 1. The RPN820 is used to predict the regions where the objects to be detected in these eight tasks may exist.

[0171] For example, the target object can include the object to be detected in all tasks of the perception network. That is, the RPN can be used to predict the regions of the object to be detected in all tasks that may exist on the second feature map. In other words, all tasks of the perception network share the same RPN.

[0172] The second feature map can be one or multiple.

[0173] For example, the second feature map may include one or more of the first feature maps.

[0174] Optionally, the perception network 800 also includes feature pyramid networks (FPN).

[0175] The FPN is connected to the backbone 810 and is used to perform feature fusion on the feature map output by the backbone 810. Specifically, it fuses the first feature map of the input image and outputs a fused feature map. The fused feature map is then input into the RPN. In this case, the second feature map may include one or more features from the fused feature map.

[0176] Specifically, FPN takes feature maps of different scales output by backbone 810 as input, and generates more expressive feature maps by vertical feature fusion within FPN and horizontal feature fusion with backbone 810 in the same layer, which are then provided to subsequent modules to improve the performance of the model.

[0177] In other words, FPN can be used to achieve multi-scale feature fusion.

[0178] In the case where the perception network does not include FPN, such as Figure 8 As shown, the backbone 810 is connected to the RPN820.

[0179] The region of interest extraction module 830 is used to extract features of the region where the candidate 2D box is located on the third feature map based on the candidate 2D box output by RPN820.

[0180] For example, the third feature map is determined based on the first feature map and includes: In the case where the perceptual network includes FPN, the third feature map can be one of the feature maps output by the backbone (i.e., the first feature map) or one of the fused feature maps output by the FPN; In the case where the perceptron does not include FPN, the third feature map can be one of the feature maps output by the backbone (i.e., the first feature map).

[0181] For example, the region of interest extraction module 830 extracts the features of the region where each proposal is located on a certain feature map output by the backbone or FPN based on the proposal provided by RPN820, and resizes it to a fixed size to obtain the features of each proposal.

[0182] For example, the region of interest extraction module 830 may employ feature extraction methods such as region of interest pooling, region of interest alignment, position-sensitive region of interest pooling (PS-ROIPOOLING), or position-sensitive region of interest alignment (PS-ROIALIGN).

[0183] For example, the region of interest extraction module 830 uses the difference and sampling method in the region where the proposal is located to extract features at a fixed resolution, and inputs the extracted features into subsequent modules.

[0184] Optionally, the classification and regression network 840 is specifically used for: processing the first feature information to obtain the confidence scores of the candidate 2D boxes belonging to each category in multiple tasks; adjusting the position information of the candidate 2D boxes to obtain the adjusted candidate 2D boxes; determining the target 2D box based on the adjusted candidate 2D boxes; and determining the first indication information based on the confidence scores of the target 2D boxes belonging to each category.

[0185] For example, the position information of the candidate 2D bounding box is adjusted so that the adjusted candidate 2D bounding box matches the shape of the actual object better than the candidate 2D bounding box, that is, the adjusted candidate 2D bounding box is a more compact candidate 2D bounding box.

[0186] Furthermore, a box merging operation is performed on the adjusted candidate 2D boxes to obtain the target 2D box. For example, non-maximum suppression (NMS) merging is performed on the adjusted 2D boxes to obtain the target 2D box.

[0187] For example, for the eight tasks in Table 1, the classification and regression network 840 refines each proposal provided by the region of interest extraction module 830, obtaining the confidence score of each proposal belonging to one of the 31 categories in the eight tasks. Simultaneously, the coordinates of the 2D bounding boxes for each proposal are adjusted to obtain adjusted candidate 2D bounding boxes. Further, the adjusted candidate 2D bounding boxes are merged using NMS to obtain the target 2D bounding boxes and the first indication information. The number of candidate 2D bounding boxes is greater than or equal to the number of target 2D bounding boxes.

[0188] In one possible implementation, the classification and regression network 840 includes multiple third R-CNNs, each corresponding to a different task. That is, each third R-CNN performs the detection of objects to be detected in a different task.

[0189] Figure 9 A schematic block diagram of a sensing network provided in an embodiment of this application is shown. For example, such as Figure 9 As shown, the perceptual network includes a backbone, FPN, RPN, ROI-Align module, and n third RCNNs.

[0190] Specifically, the third RCNN is used to: process the features of the region where the candidate 2D box is located to obtain the confidence that the candidate 2D box belongs to the object category in the task corresponding to the third RCNN; and adjust the position information of the candidate 2D box to make the adjusted candidate 2D box.

[0191] That is, any one of the multiple third R-CNNs can predict the confidence score of the candidate 2D box belonging to the object category in the task corresponding to that third R-CNN, and obtain the adjusted candidate 2D box. The multiple third R-CNNs can then obtain the confidence score of the candidate 2D box belonging to each category, as well as the adjusted candidate 2D box obtained by each third R-CNN.

[0192] Furthermore, after the adjusted candidate 2D boxes are merged by NMS, the target 2D box and the first indication information are obtained.

[0193] For example, the task corresponding to the third RCNN1# is the car detection task in Table 1. The third RCNN1# outputs the confidence score for each proposal belonging to the three categories of sedan, truck, and bus, along with adjusted candidate 2D bounding boxes. The task corresponding to the third RCNN2# is the wheel and headlight detection task in Table 1. The third RCNN2# outputs the confidence score for each proposal belonging to the two categories of wheels and headlights, along with adjusted candidate 2D bounding boxes. Thus, for any given proposal, after processing by the third RCNN1# and third RCNN2#, a total of five categories of confidence scores and adjusted candidate 2D bounding boxes can be obtained.

[0194] Figure 9 The perceptron in the image is used to implement n tasks, for example, the n tasks include Figure 9 The algorithm consists of tasks 0, 1, ..., n-1, where n is an integer greater than 1. There are n third-party R-CNNs, each corresponding to one of the n tasks. Taking task 0 as an example, the third-party R-CNN for task 0 outputs the confidence score of each proposal belonging to each object category in task 0, along with the adjusted candidate 2D bounding boxes. The n third-party R-CNNs corresponding to the n tasks each obtain the confidence score of each proposal belonging to each object category in its respective task. The classification and regression network can then obtain the confidence score of each proposal belonging to each category.

[0195] It should be noted that, Figure 9 FPN is an optional module. Figure 9 The ROI-Align module used in this example for region of interest extraction is just that—an example. Other methods can also be used to extract the corresponding features. For details, please refer to the previous text.

[0196] In another possible implementation, the classification and regression network includes a first RCNN, which includes hidden layers, multiple subclassification fully connected layers (cls fc), and multiple subregression fully connected layers (reg fc). The hidden layers are connected to the multiple subclassification fully connected layers, and the multiple subclassification fully connected layers are connected to the multiple subregression fully connected layers. The multiple subclassification fully connected layers correspond one-to-one with multiple tasks, and the multiple subregression fully connected layers correspond one-to-one with multiple tasks.

[0197] Alternatively, the first R-CNN includes hidden layers and multiple subclass classification functions (FCs) and subregulation functions (RCs) corresponding to multiple tasks. Each task can have an independent subclassification function (FC) and subregression function (FC).

[0198] Figure 10 A schematic block diagram of another sensing network provided in an embodiment of this application is shown. For example, such as Figure 10 As shown, the perceptual network includes a backbone, FPN, RPN, ROI-Align module, and a first RCNN.

[0199] The hidden layer is used to process the first feature information to obtain the second feature information.

[0200] In other words, the hidden layer is used to process the features of the region where the candidate 2D box is located, and the processed results are input into multiple sub-classification fully connected layers and multiple sub-regression fully connected layers respectively.

[0201] For example, the hidden layer may include at least one of the following: a convolutional layer or a fully connected layer. Since multiple tasks share the hidden layer, the convolutional layer in the hidden layer may also be called a shared convolutional layer (shared conv), and the fully connected layer in the hidden layer may also be called a shared fully connected layer (shared fc).

[0202] The subclassification fully connected layer is used to obtain the confidence level of the candidate 2D box in the task corresponding to the subclassification fully connected layer based on the second feature information.

[0203] The sub-regression fully connected layer is used to adjust the position information of the candidate 2D boxes based on the second feature information, resulting in adjusted candidate 2D boxes. Furthermore, the sub-regression fully connected layer can perform box merging operations, such as removing duplicate boxes using the NMS operation, to output more compact candidate 2D boxes.

[0204] Each task's corresponding sub-classification fully connected layer and sub-regression fully connected layer can detect the objects to be detected in that task. Specifically, the sub-classification fully connected layer can output the confidence score of the candidate 2D bounding box belonging to the object category in that task, and the sub-regression fully connected layer can output the adjusted candidate 2D bounding box. In other words, a first R-CNN can complete the detection of objects to be detected in multiple tasks. The first R-CNN can also be called a single-head multi-task R-CNN.

[0205] The first RCNN can predict the confidence level of the candidate 2D box belonging to the object category in multiple tasks corresponding to the first RCNN, and obtain the adjusted candidate box.

[0206] For example, if the first RCNN corresponds to multiple tasks including the 8 tasks in Table 1, then the first RCNN includes 8 sub-cls fcs and 8 regis fcs, which correspond to the 8 tasks respectively. Each sub-cls fc outputs the confidence score of each proposal belonging to the object category in the task corresponding to the sub-cls fc, and each regis fc outputs the adjusted candidate 2D bounding box. In this way, the first RCNN can obtain the confidence score of each proposal belonging to the 31 object categories in the 8 tasks and the adjusted candidate 2D bounding box.

[0207] Figure 10 The perceptron in the image is used to implement n tasks, for example, the n tasks include Figure 10 The task is defined as task 0, task 1, ..., task n-1, where n is an integer greater than 1. The first R-CNN includes hidden layers and n sub-cls fcs and n sub-reg fcs corresponding to the n tasks. The hidden layers may include shared fcs and / or shared conv.

[0208] Taking Task 0 as an example, the sub-cls fc corresponding to Task 0 in the first R-CNN outputs the confidence score of each proposal belonging to each object category in Task 0, and the sub-reg fc corresponding to Task 0 outputs the adjusted candidate 2D bounding boxes. In this way, the n sub-cls fcs corresponding to n tasks obtain the confidence score of each proposal belonging to each object category in its respective task, and the first R-CNN can then obtain the confidence score of each proposal belonging to each category.

[0209] It should be noted that, Figure 10 FPN is an optional module. Figure 10 The ROI-Align module used in this example for region of interest extraction is just that—an example. Other methods can also be used to extract the corresponding features. For details, please refer to the previous text.

[0210] In another possible implementation, the classification and regression network includes a second RCNN, which includes hidden layers, a classification fully connected layer, and a regression fully connected layer. The hidden layers are connected to the classification fully connected layer and the regression fully connected layer.

[0211] Figure 11 A schematic block diagram of yet another sensing network provided in an embodiment of this application is shown. For example, such as Figure 11 As shown, the perception network includes a backbone, FPN, RPN, ROI-Align module, and a second RCNN.

[0212] The hidden layer is used to process the first feature information to obtain the third feature information.

[0213] In other words, the hidden layer is used to process the features of the region where the candidate 2D box is located, and the processed results are input into the classification fully connected layer and the regression fully connected layer respectively.

[0214] For example, the hidden layer may include at least one of the following: a convolutional layer or a fully connected layer. A detailed description can be found in the first R-CNN, and will not be repeated here.

[0215] A fully connected classification layer is used to obtain the confidence level of candidate 2D boxes belonging to each category based on the third feature information.

[0216] A regressive fully connected layer is used to adjust the position information of the candidate 2D boxes based on the third feature information, resulting in adjusted candidate 2D boxes. Further, a box merging operation is performed on the adjusted candidate 2D boxes to obtain the target 2D box.

[0217] In other words, a second R-CNN performs the detection of objects in multiple tasks. The second R-CNN can also be called a single-head multi-task R-CNN.

[0218] Specifically, the classification fully connected layer is obtained by merging multiple sub-classification fully connected layers in the first R-CNN. The regression fully connected layer is obtained by merging multiple sub-regression fully connected layers in the first R-CNN. In this case, the first feature information and the third feature information are the same.

[0219] Merging multiple sub-classification fully connected layers can be understood as concatenating the weight matrices of the multiple sub-classification fully connected layers. Similarly, merging multiple sub-regression fully connected layers can be understood as concatenating the weight matrices of the multiple sub-regression fully connected layers.

[0220] The first RCNN can use the sigmoid function to normalize the label logits obtained from the subclassing fc, which is equivalent to performing a binary classification process for each category. The confidence that the target object belongs to one category is independent of other categories. Merging the subclassing fc of multiple tasks in the model into a single classification fc will not affect the inference results of the model. That is, the output of multiple subclassing fcs and the output of the classification fc obtained by merging multiple subclassing fcs are the same.

[0221] In other words, the second R-CNN and the first R-CNN can perform the same tasks and output the same results. However, for accelerators such as NPUs, only one matrix operation is performed at a time. In the first R-CNN, the output of the hidden layer needs to be input into the subclassification flow table and subregression flow table corresponding to each task for multiple matrix operations. The number of matrix multiplications in the first R-CNN increases with the number of tasks, while the number of matrix multiplications in the second R-CNN is not affected by the number of tasks. That is, with the same number of parameters in the first and second R-CNNs, the time required to execute the second R-CNN is less than the time required to execute the first R-CNN.

[0222] Therefore, by merging the sub-classification fully connected layers corresponding to multiple tasks in the first RCNN to obtain the classification fully connected layer of the second RCNN, and merging the sub-regression fully connected layers corresponding to multiple tasks in the first RCNN to obtain the regression fully connected layer of the second RCNN, the number of matrix multiplication operations in the neural network accelerator can be reduced, making it more hardware-friendly and further reducing the time consumption.

[0223] Figure 11 The perceptron network is used to implement n tasks, which include Figure 10 The task is defined as task 0, task 1, ..., task n-1, where n is an integer greater than 1. The second R-CNN includes hidden layers, a cls fc, and a reg fc. The hidden layers can include a shared fc and / or a shared conv. The cls fc can be... Figure 10 The reg fc is obtained by merging the n subcls fcs in the formula, and can be obtained by... Figure 10 It is obtained by merging the n subregs fc in the middle.

[0224] In this way, cls fc can output the confidence score of each proposal belonging to each category, and reg fc can output the adjusted candidate 2D bounding boxes.

[0225] It should be noted that, Figure 11 FPN is an optional module. Figure 11The ROI-Align module used in this example for region of interest extraction is just that—an example. Other methods can also be used to extract the corresponding features. For details, please refer to the previous text.

[0226] For example, during the training of the perceptual network, the classification and regression network adopts the first RCNN. After training, the second RCNN is obtained based on the first RCNN. That is, in the perceptual network used for inference, the classification and regression network can adopt the second RCNN.

[0227] For example, Figure 10 The perceptual network in the training process can be applied to the training side. The first R-CNN in the trained perceptual network is then merged to obtain... Figure 11 The sensor network shown is, Figure 11 The model parameters in the data are based on Figure 10 The parameters are obtained from the model. Figure 11 The perceptual network in the model can be applied to the inference side to reduce time consumption.

[0228] According to the scheme of the embodiments of this application, a single perception network is used to complete multiple perception tasks. Multiple tasks share a single RPN, and the RPN predicts the region where the object to be detected is located in multiple tasks. While ensuring the performance of the perception network, the number of parameters and computation of the perception network is reduced, the processing efficiency is improved, and it is beneficial to deploy in scenarios with high real-time requirements. At the same time, it reduces the pressure on the hardware and saves costs.

[0229] Furthermore, in the scheme of this application embodiment, a first RCNN or a second RCNN is used as the classification and regression network. Multiple tasks share the hidden layers of the RCNN, further reducing the number of parameters and computational load of the perceptual network and improving processing efficiency. Moreover, when training with the first RCNN, each task corresponds to an independent sub-classification fc and sub-regression fc, improving the scalability of the perceptual network. The perceptual network can flexibly implement functional configuration by adding or removing sub-classification fc and sub-regression fc, that is, flexibly adding or removing detection tasks by adding or removing sub-classification fc and sub-regression fc.

[0230] Furthermore, in the scheme of this application embodiment, the multiple sub-classification fc and sub-regression fc in the first RCNN are merged, and the second RCNN is used as the classification and regression network, which can further reduce matrix operation operations, is more hardware-friendly, further reduce computation time, and improve processing efficiency.

[0231] The perception network in this embodiment can be trained using existing training methods.

[0232] However, when using existing training methods with fully labeled sample data, it is necessary to label all the objects to be detected for each task in the sample images of the dataset, resulting in a high labeling cost. Moreover, if the perceptual network needs to be expanded, i.e., a new task is added, the sample images in the entire dataset need to be relabeled to add the objects to be detected for the new task, further increasing the labeling cost and reducing the scalability of the perceptual network.

[0233] If partially labeled sample images are used for training, it's unnecessary to label all objects to be detected for all tasks on a single sample image, reducing labeling costs. However, since multiple tasks share a single Resource Planning Network (RPN), training data from different tasks may inhibit each other during RPN training. This can cause the RPN to fail to predict candidate regions for all objects to be detected across all tasks, thus affecting the accuracy of the perceptual network. Specifically, because the labeled data is partial—for example, a sample image may only contain annotations for objects to be detected for one task—when training with this data, the RPN parameters are adjusted to more accurately predict candidate 2D bounding boxes for that task, but not for other tasks. Similarly, when training with annotations from another task, the RPN parameters are adjusted, potentially leading to inaccurate predictions of candidate 2D bounding boxes for other tasks. Thus, training data from different tasks may inhibit each other, causing the RPN to fail to predict all target objects in the image.

[0234] This application provides a training method for a perceptual network. By using sample images from other perceptual network inference training sets, pseudo bounding boxes (Pseudo Bboxes) are provided for unlabeled objects to be detected in the sample images. Then, the RPN is trained together based on the pseudo bounding boxes and labeled data, which is beneficial for obtaining candidate 2D boxes of objects to be detected in multiple tasks.

[0235] Figure 12 This application illustrates a training method 1200 for a perceptual network according to an embodiment of the present application. Method 1200 can be executed by a neural network model training device, which can be a cloud service device or a terminal device, such as a computer, server, or other device with sufficient computing power to execute the neural network model training method. It can also be a system composed of a cloud service device and a terminal device. Exemplarily, method 1200 can be executed by… Figure 3 Training equipment 120 Figure 5 The neural network processor 50 or Figure 6The execution device 310 performs the operation. The perceptual network includes: RPN, which is used to predict the location information of candidate 2D bounding boxes of target objects in sample images. The target objects include objects to be detected in multiple tasks, each of which includes at least one category.

[0236] Optionally, the perception network can be Figure 8 The perceptual network shown is illustrated. To avoid unnecessary repetition, relevant descriptions of the training method are omitted. During training, the input image is simply replaced with the sample image.

[0237] Method 1200 includes steps S1210 to S1220, which are described below.

[0238] S1210, Obtain training data.

[0239] The target objects include a first task object and a second task object. The training data includes sample images, labeled data of the first task object on the sample images, and pseudo-boundary boxes of the second task object on the sample images. The labeled data includes the class label and labeled 2D bounding box of the first task object. The pseudo-boundary box of the second task object is the target 2D bounding box of the second task object obtained by inference from the sample images by other perceptual networks.

[0240] Annotated data can also be understood as ground truth. The class labels in the annotations indicate the true category to which the task object belongs. The annotation data for the first task object can also be understood as the annotation data for the sample image. Full annotation data for the sample image includes the class labels and 2D bounding boxes for all objects to be detected in all tasks on the sample image. Full annotation data includes the annotation information for all objects of interest. Partial annotation data includes the class labels and 2D bounding boxes for some objects to be detected in a portion of the sample image. Partial annotation data only includes the annotation information for some objects of interest.

[0241] The first task object may include the object to be detected in one or more tasks. These one or more tasks refer to the task in which the first task object is located. The first task object in different sample images in the training set may be the same or different. In the embodiments of this application, the term "first" in "first task object" is only used to limit the object to be detected that has a ground truth value in the sample image and has no other limiting effect.

[0242] For example, the annotation data for sample image 1# is the annotation data for vehicles, meaning that the first task object in sample image 1# includes the objects to be detected in the vehicle detection task, such as trucks, cars, buses, etc.; the annotation data for sample image 2# is the annotation data for wheels and headlights, meaning that the first task object in sample image 2# includes the objects to be detected in the wheel and headlight detection task, such as wheels and headlights; the annotation data for sample image 3# includes the annotation data for vehicles and the annotation data for wheels and headlights, meaning that the first task object in sample image 3# includes the objects in the vehicle detection task and the objects to be detected in the wheel and headlight detection task.

[0243] In other words, the labeled data for the sample images in this embodiment can be partially labeled data. This allows for targeted data collection, meaning that sample images are collected for specific tasks without having to label all objects to be detected for every task in every sample image, thus reducing data collection and labeling costs. Furthermore, the partially labeled data approach offers flexible scalability. When adding tasks, only the labeled data for the new tasks needs to be provided, without needing to label new objects to be detected on top of the existing training data.

[0244] The Pseudo bounding boxes on the sample images are the target 2D bounding boxes of the second task object obtained through inference from the sample images by other perceptual networks. The Pseudo bounding boxes on the sample images can also be understood as the Pseudo bounding boxes of the second task object.

[0245] Other perceptual networks refer to perceptual networks other than the one to be trained. For example, other perceptual networks can be multi-head, multi-task perceptual networks.

[0246] For example, using Figure 7 The perceptual network shown infers from sample images in the training set to obtain inference results for the sample images, including the target 2D bounding boxes of the target objects on the sample images.

[0247] For example, other sensing networks may also include multiple single-task sensing networks.

[0248] For example, multiple single-task perception networks are used to infer the sample images in the training set, and the inference results of the sample images are obtained respectively. The inference result of each single-task perception network includes the target 2D bounding box of the object to be detected in the sample image in that task. Based on the inference results of the multiple single-task perception networks, the target 2D bounding boxes of the objects to be detected in the sample images in multiple tasks can be obtained.

[0249] The second task object may include objects to be detected in one or more tasks. These one or more tasks refer to the tasks in which the second task object resides. The second task object and the first task object may contain the same objects to be detected. The second task objects in different sample images in the training set may be the same or different. In the embodiments of this application, the term "second" in "second task object" is only used to define objects to be detected with pseudo-boundaries in the sample images and has no other limiting effect.

[0250] For example, when the same object to be detected exists in both the first and second task objects, the bounding boxes in the labeled data are used as the target output of the RPN. The labeled data is usually manually labeled, and its accuracy is generally higher than that of pseudo-boundaries obtained by other perceptual networks. Using the labeled boxes as the target output can improve the accuracy of the trained model.

[0251] For example, the perceptual network needs to complete multiple tasks, including the eight tasks in Table 1. The annotation data for sample image 1# is the annotation data for cars. The first task object in sample image 1# includes the objects to be detected in the car detection task, such as trucks, cars, and buses; that is, the annotation data for sample image 1# is partially annotated. By inferring from sample image 1# through other perceptual networks, the target 2D bounding box, or pseudo-bounding box, for the second task object is obtained. For example, by inferring from sample image 1# through seven single-task perceptual networks used to complete the seven tasks in Table 1 other than the car detection task, the target 2D bounding box for the second task object can be obtained. In this case, the second task object can include the objects in the seven tasks in Table 1 other than the car detection task. For example, ... Figure 7 The multi-head, multi-task perceptual network shown can be used to complete the eight tasks in Table 1. Using this perceptual network to infer from sample image 1#, the target 2D bounding box of the second task object can be obtained. In this case, the second task object can include the objects to be detected in the eight tasks in Table 1. Thus, by merging the pseudo-boundary and the labeled bounding box, the region containing the objects to be detected in the eight tasks of sample image 1# can be obtained.

[0252] By using pseudo-boundaries to supplement unlabeled objects to be detected in sample images, we can avoid mutual inhibition between partially labeled data of different tasks when RPN is trained based on partially labeled data, which would affect the training of RPN and improve the recall rate of RPN. This is beneficial for RPN to predict the location of the objects to be detected in all tasks that need to be detected.

[0253] Furthermore, other perceptual networks infer from the sample images to obtain the target 2D bounding box of the second task object on the sample images and the confidence score of the category to which the second task object belongs. If the confidence score is greater than or equal to a first threshold, the target 2D bounding box of the second task object on the sample images inferred by other perceptual networks is used as a pseudo-bounding box on the sample images. In other words, if the confidence score is greater than or equal to the first threshold, the inference results of other perceptual networks are used for training.

[0254] For example, a low threshold can be used for filtering. For instance, a first threshold of 0.05 means that 2D bounding boxes with a confidence level greater than or equal to 0.05 can be used as pseudo-bounding boxes on the sample image and participate in the training of the perceptual network along with the labeled data. It should be understood that the first threshold can be set as needed, and this embodiment does not limit its application.

[0255] S1220 trains the perception network based on training data.

[0256] Specifically, step S1220 may include steps S1221 to S1223.

[0257] S1221, calculate the first loss function value based on the difference between the labeled 2D bounding box of the first task object and the target 2D bounding box of the second task object and the candidate 2D bounding box of the target object in the sample image predicted by RPN.

[0258] In other words, the labeled 2D bounding box of the first task object and the target 2D bounding box of the second task object are compared with the candidate 2D bounding boxes of the target object predicted by RPN to obtain the loss function value of the RPN stage, i.e., the first loss function value.

[0259] The forward propagation of the perceptual network is performed based on the sample images. The RPN predicts candidate 2D bounding boxes of target objects on the sample images. For details of the forward propagation process, please refer to [link to relevant documentation]. Figure 8 This will not be elaborated upon here.

[0260] S1222, calculate the second loss function value of the perceptual network based on the labeled data of the sample images.

[0261] The second loss function value of the perceptual network is the second loss function value of the part of the perceptual network that needs to be trained. The part of the perceptual network that needs to be trained includes the part of the classification and regression network that needs to be trained, the region of interest extraction module, the RPN, and the backbone network. The part of the classification and regression network that needs to be trained is determined based on the first task object.

[0262] The part of a perceptual network that needs to be trained refers to the part of the perceptual network that needs to be trained, determined by the sample images.

[0263] Classification and regression networks can predict the confidence level of candidate 2D boxes belonging to each category and the target 2D box of the target object.

[0264] Specifically, after the RPN predicts candidate 2D bounding boxes for the target object, the region of interest extraction module extracts the features of the candidate 2D bounding boxes from the feature map. These features are then input into the training portion of the classification and regression network to obtain the confidence score that the candidate 2D bounding box belongs to the object category in the task corresponding to the first task object. The training portion of the classification and regression network is determined based on the first task object. In other words, the training portion of the classification and regression network is determined based on the task in which the first task object belongs.

[0265] Optionally, the classification and regression network includes multiple third RCNNs, and the part of the classification and regression network that needs to be trained includes the third RCNN corresponding to the task in which the first task object is located.

[0266] For example, a perception network can be as follows Figure 9 As shown. In sample image 1# (an example of a sample image), the task of the first task object includes the car detection task, and the first task object includes the object to be detected in the car detection task. The features of the candidate 2D box are input into the third RCNN corresponding to the car detection task, thereby obtaining the confidence that the candidate 2D box belongs to the three categories of sedan, truck, and bus, as well as the target 2D box. For sample image 1#, the part of the classification and regression network that needs to be trained is the third RCNN corresponding to the car detection task.

[0267] Optionally, the classification and regression network includes a first RCNN, and the parts of the classification and regression network that need to be trained include the hidden layers in the first RCNN and the subclassification fc and subregression fc corresponding to the task to which the first task object is located.

[0268] For example, a perception network can be as follows Figure 10 As shown. The task of the first object in sample image 1# includes the car detection task, and the first object includes the object to be detected in the car detection task. The features of the candidate 2D bounding box are passed through the hidden layer of the first RCNN and then input into the sub-classification fc and sub-regression fc corresponding to the car detection task, thereby obtaining the confidence that the candidate 2D bounding box belongs to the three categories of sedan, truck and bus, as well as the target 2D bounding box. For sample image 1#, the parts that need to be trained in the classification and regression network are the hidden layer of the first RCNN and the sub-classification fc and sub-regression fc corresponding to the car detection task.

[0269] The labeled data of the sample images is compared with the output of the classification and regression network to obtain the loss function value of the task in which the first task object belongs in the classification and regression network stage, i.e., the second loss function value. In other words, the loss of other tasks not involved in the labeled data of the sample images is not calculated.

[0270] S1223, backpropagation is performed based on the first loss function value and the second loss function value to adjust the parameters of the part of the perceptual network that needs to be trained.

[0271] Backpropagation based on the first loss function value is used to calculate the gradient of the parameters related to the first loss function value. Then, the parameters related to the first loss function value are adjusted based on the gradient of the parameters to adjust the perceptual network so that the RPN can predict candidate boxes more comprehensively.

[0272] The parameters related to the first loss function value are the parameters in the perceptual network used in calculating the first loss function value, such as the backbone parameters and the RPN parameters. Furthermore, if the perceptual network includes an FPN, the parameters related to the first loss function value also include the FPN.

[0273] Backpropagation based on the second loss function value is used to calculate the gradient of the parameters related to the second loss function value. Then, the parameters related to the second loss function value are adjusted based on the gradient of the parameters to adjust the perceptual network, so that the classification and regression network can better correct the output 2D box and improve the accuracy of category prediction.

[0274] The parameters related to the second loss function are the parameters in the perceptual network used in calculating the value of the second loss function. These include, for example, the parameters of the backbone, the RPN, the region of interest extraction module, and the parameters of the parts of the classification and regression network that need to be trained. Furthermore, if the perceptual network includes an FPN, the parameters related to the value of the second loss function also include the parameters of the FPN. The parameters related to the second loss function are essentially the parameters of the parts of the perceptual network that need to be trained.

[0275] If the training termination condition is met, the training is terminated, and the trained perceptual network is obtained.

[0276] For example, if the perceptron converges, terminate the training and output the weights of the trained perceptron.

[0277] It should be understood that steps S1221 to S1223 are only one implementation of step S1220, and step S1220 can also be implemented in other ways.

[0278] For example, step S1220 includes steps S1 to S3.

[0279] S1, calculate the first loss function value based on the difference between the labeled 2D bounding box of the first task object and the target 2D bounding box of the second task object and the candidate 2D bounding box of the target object on the sample image predicted by RPN.

[0280] In other words, the labeled 2D bounding box of the first task object and the target 2D bounding box of the second task object are compared with the candidate 2D bounding boxes of the target object predicted by RPN to obtain the loss function value of the RPN stage, i.e., the first loss function value.

[0281] The forward propagation of the perceptual network is performed based on the sample images. The RPN predicts candidate 2D bounding boxes of target objects on the sample images. For details of the forward propagation process, please refer to [link to relevant documentation]. Figure 8 This will not be elaborated upon here.

[0282] S2, calculate the second loss function value of the part of the perceptual network that needs to be trained based on the labeled data of the sample image, the pseudo-boundary on the sample image, and the pseudo-label of the second task object on the sample image. The part of the perceptual network that needs to be trained includes the part of the classification and regression network that needs to be trained, the region of interest extraction module, the RPN, and the backbone network. The part of the classification and regression network that needs to be trained is determined based on the first task object and the second task object. The pseudo-label on the sample image is the class label of the second task object on the sample image obtained by inference from the sample image by other perceptual networks.

[0283] Classification and regression networks can predict the confidence level of candidate 2D boxes belonging to each category and the target 2D box of the target object.

[0284] Specifically, after the RPN predicts candidate 2D bounding boxes for the target object, the region of interest extraction module extracts the features of the candidate 2D bounding boxes from the feature map. These features are then input into the training portion of the classification and regression network to obtain the confidence scores for the candidate 2D bounding boxes belonging to the object category in the task of the first task object and the object category in the task of the second task object. The training portion of the classification and regression network is determined based on the first and second task objects. In other words, the training portion of the classification and regression network is determined based on the tasks of the first and second task objects.

[0285] For example, the classification and regression network includes multiple third RCNNs, such as a perceptual network. Figure 9As shown. The task of the first object in sample image 1# includes car detection, and the first object includes the object to be detected in the car detection task. The features of the candidate 2D bounding box are input into the third R-CNN corresponding to the car detection task, thus obtaining the confidence score of the candidate 2D bounding box belonging to the three categories of sedan, truck, and bus, as well as the target 2D bounding box. The task of the second object in sample image 1# includes wheel and headlight detection, and the second object includes the object in the wheel and headlight detection task. The features of the candidate 2D bounding box are input into the third R-CNN corresponding to the wheel and headlight detection task, thus obtaining the confidence score of the candidate 2D bounding box belonging to the two categories of wheel and headlight, as well as the target 2D bounding box.

[0286] For sample image 1#, the parts that need to be trained in the classification and regression network are the third RCNN corresponding to the car detection task and the third RCNN corresponding to the wheel and headlight detection tasks.

[0287] For example, the classification and regression network includes a first RCNN, such as a perceptual network, which can be as follows: Figure 10 As shown. The first task object in sample image 1# belongs to the car detection task, and the first task object includes the object to be detected in the car detection task. The features of the candidate 2D bounding box are passed through the hidden layer of the first R-CNN and then input into the sub-classification fc and sub-regression fc corresponding to the car detection task, thus obtaining the confidence score of the candidate 2D bounding box belonging to the three categories of sedan, truck, and bus, as well as the target 2D bounding box. The second task object in sample image 1# belongs to the wheel and headlight detection task, and the second task object includes the object in the wheel and headlight detection task. The features of the candidate 2D bounding box are passed through the hidden layer of the first R-CNN and then input into the sub-classification fc and sub-regression fc corresponding to the wheel and headlight detection task, thus obtaining the confidence score of the candidate 2D bounding box belonging to the two categories of wheel and headlight, as well as the target 2D bounding box.

[0288] For sample image 1#, the parts that need to be trained in the classification and regression network include the hidden layers in the first RCNN, the subclassification fc and subregression fc corresponding to the car detection task, and the subclassification fc and subregression fc corresponding to the wheel and headlight detection tasks.

[0289] The labeled data of the sample images is compared with the output of the classification and regression network to obtain the loss function value for the first task object and the loss function value for the second task object in the classification and regression network stage, i.e., the second loss function value. In other words, the loss of the labeled data of the sample images and other tasks not involved in the pseudo-labels are not calculated.

[0290] S3, based on the first loss function value and the second loss function value, performs backpropagation to adjust the parameters of the part of the perceptual network that needs to be trained.

[0291] Backpropagation based on the first loss function value is used to calculate the gradient of the parameters related to the first loss function value. Then, the parameters related to the first loss function value are adjusted based on the gradient of the parameters to adjust the perceptual network so that the RPN can predict candidate boxes more comprehensively.

[0292] The parameters related to the first loss function value are the parameters in the perceptual network used in calculating the first loss function value, such as the backbone parameters and the RPN parameters. Furthermore, if the perceptual network includes an FPN, the parameters related to the first loss function value also include the FPN.

[0293] Backpropagation based on the second loss function value is used to calculate the gradient of the parameters related to the second loss function value. Then, the parameters related to the second loss function value are adjusted based on the gradient of the parameters to adjust the perceptual network, so that the classification and regression network can better correct the output 2D box and improve the accuracy of category prediction.

[0294] The parameters related to the second loss function are the parameters in the perceptual network used in calculating the value of the second loss function. These include, for example, the parameters of the backbone, the RPN, the region of interest extraction module, and the parameters of the parts of the classification and regression network that need to be trained. Furthermore, if the perceptual network includes an FPN, the parameters related to the value of the second loss function also include the parameters of the FPN. The parameters related to the second loss function are essentially the parameters of the parts of the perceptual network that need to be trained.

[0295] If the training termination condition is met, the training is terminated, and the trained perceptual network is obtained.

[0296] For example, if the perceptron converges, terminate the training and output the weights of the trained perceptron.

[0297] According to the scheme in this application embodiment, the perceptual network is trained based on pseudo-boundary boxes and labeled data. When the labeled data only includes the labeled data of the first task object (i.e., partial labeled data), pseudo-boundary boxes of the second task object are provided. This allows for a more comprehensive set of bounding boxes for the objects to be detected on the same sample image, serving as the target output of the RPN. This adjusts the RPN parameters so that the RPN output continuously approaches the target data, avoiding mutual inhibition between different tasks. This helps the RPN obtain more comprehensive and accurate candidate 2D bounding boxes, while improving recall. In this application embodiment, the labeled data of the sample images can be partial labeled data, allowing for targeted acquisition—that is, acquiring sample images for specific tasks without labeling all objects to be detected for every task in every sample image. This reduces data acquisition and labeling costs and helps balance the training data for different tasks. Furthermore, the partial labeled data scheme offers flexible scalability. When adding tasks, only the labeled data for the new tasks needs to be provided, without needing to label new objects to be detected on top of the existing training data.

[0298] Furthermore, according to the scheme in the embodiments of this application, the parts shared by different tasks in the perceptual network, such as the backbone network, RPN, and region of interest extraction module, all participate in the training process based on the labeled data of different tasks. This enables the shared parts of the perceptual network to learn the common features of each task. Different parts corresponding to different tasks in the perceptual network, such as the parts corresponding to each task in the classification and regression network, only participate in the training process based on the labeled data of their respective tasks. This enables the different parts corresponding to different tasks in the perceptual network to learn their task-specific features, improving the accuracy of the model. Simultaneously, during training, the parts of the classification and regression network that need to be trained are determined according to the task. The different parts of the classification and regression networks corresponding to different tasks do not affect each other during training, ensuring the independence of each task and making the model highly flexible.

[0299] Figure 13 This application illustrates a training method for a perceptual network according to an embodiment of the present application. Figure 13 The method shown can be regarded as Figure 12 The above is a specific implementation of the method shown. For a related description, please refer to the description in method 1200. To avoid unnecessary repetition, appropriate omissions are made when describing method 1300.

[0300] The following section uses the visual perception system of ADAS / ADS as an example to provide a detailed description of the solution in this application. The visual perception system of ADAS / ADS needs to perform target detection for various tasks, such as: dynamic obstacles, static obstacles, traffic signs, traffic lights, road markings (e.g., left turn signs or straight-ahead signs), and zebra crossings.

[0301] The solution in this application embodiment can complete the target detection of the above-mentioned multiple tasks in a single perception network. The solution in this application embodiment will be described in detail below.

[0302] The training method of the perception network in this application embodiment will be described in detail below, taking the task division in Table 1 as an example.

[0303] Before training begins, training data is prepared. The target objects include the first task object and the second task object. The training data includes sample images, the annotation data of the first task object on the sample images, and the pseudo-boundary of the second task object on the sample images. The annotation data includes the class label of the first task object and the annotation 2D bounding box of the first task object.

[0304] Based on the task division in Table 1, labeled data is provided for each task. For example, for the training process of task 0, labeled data for cars is provided, with 2D bounding boxes for Car / Truck / Bus and class labels annotated on one or more sample images in the dataset; for the training of task 1, labeled data for people is provided, with 2D bounding boxes for Pedestrian / Cyclist / Tricycle and class labels annotated on one or more sample images in the dataset; for task 2, labeled data for wheels and lights is provided, with 2D bounding boxes for Wheel / Car_light and class labels annotated on one or more sample images in the dataset; for task 3, labeled data for traffic lights is provided, with 2D bounding boxes for TrafficLight_Red / Yellow / Green / Black and class labels annotated on one or more sample images in the dataset, and so on. In this way, each sample image has labeled data for at least one task.

[0305] In one possible implementation, the sample images include annotation information for all objects of interest. That is, all objects of interest are labeled in each sample image. For example, the objects of interest are the objects to be detected in the eight major categories listed in Table 1.

[0306] In another possible implementation, each type of annotation data only needs to annotate a specific type of object. That is, the annotation data for each sample image can be partial annotation data.

[0307] For example, each sample image is labeled with only the class label and 2D bounding box of the object to be detected in the task.

[0308] Alternatively, each sample image can be labeled with class labels and 2D bounding boxes for objects to be detected from multiple tasks, providing hybrid labeled data. For example, 2D bounding boxes and class labels for Car / Truck / Bus / Pedestrian / Cyclist / Tricycle can be simultaneously labeled on sample images. In this way, the required training parts of the perceptual networks for two tasks can be trained simultaneously using this training data.

[0309] For example, a task label can be assigned to each sample image, which can be used to indicate the part of the sample image that is used to train the desired part of the perceptual network.

[0310] The annotation data for the sample images can be obtained using the method described above. For example, the annotation data can be stored in an annotation file, which is essentially a ground truth file.

[0311] Inference is performed on the sample images using other perceptual networks to obtain inference results. These inference results include Pseudo Bboxes on the sample images. Pseudo Bboxes can be used to complete the annotation data of the sample images for objects belonging to other tasks that are not labeled. For example, the inference results can be stored in an inference result file, which is essentially a Pseudo Bboxes file.

[0312] Each sample image can correspond to a labeling file and an inference result file. In one possible implementation, the labeled 2D bounding boxes and Pseudo Bboxes in the labeled data of the sample image can be merged to obtain the 2D bounding boxes of the objects to be detected in all tasks on the sample image.

[0313] For example, a multi-head, multi-task perceptual network can be used to infer from sample images to obtain inference results.

[0314] For example, multiple single-task perceptual networks can be used to perform reasoning on sample images to obtain reasoning results for multiple tasks, and then the reasoning results of multiple tasks can be merged together.

[0315] Furthermore, the inference results also include the confidence level of the category to which the second task object on the sample image belongs. A low threshold is used to filter the inference results, specifically filtering out inference results with a confidence level less than a first threshold. This ensures that the confidence levels of all Pseudo Bboxes used for training are greater than or equal to the first threshold. For example, the first threshold is 0.05.

[0316] The perceptual network is trained based on partially labeled data and Pseudo Bboxes. Specifically, method 1300 includes steps S1310 to S1350.

[0317] S1310, Obtain training data.

[0318] The training data is input into the perceptual network. The training data includes sample images, the labeled data of the first task object on the sample images, and the pseudo-boundary of the second task object on the sample images.

[0319] For example, input the sample image, the corresponding annotation file of the sample image, and the Pseudo Bboxes file into the perceptual network.

[0320] Step S1310 corresponds to step S1210 in method 1200, and a detailed description is given in step S1210.

[0321] Perform forward propagation of the perceptron based on the training data.

[0322] For example, based on the task division method in Table 1, the structure of the perceptual network used during training is as follows: Figure 14 As shown. Figure 14 As shown, the perception network includes: backbone, RPN, region of interest extraction module and first RCNN. Figure 14 The sensor network shown can be regarded as Figure 10 This is one specific implementation of the sensor network shown. Figure 14 The perception network in the system can simultaneously perform target detection for the eight major categories listed in Table 1. In other words, Figure 14 The perception network in the system can simultaneously perform object detection for the eight tasks listed in Table 1. Specifically, Figure 14 The first R-CNN in the dataset uses eight sub-classification and sub-regression function tables to simultaneously perform 2D object detection for the eight major categories listed in Table 1. Figure 14 As can be seen from this, the perceptual network of this application can flexibly add or remove classification fc and regression fc in the first RCNN according to the needs of the business, so as to train a perceptual network that can achieve target detection of different numbers of tasks.

[0323] S1320 calculates the loss for the RPN stage using labeled data and Pseudo Bboxes.

[0324] The annotation data for the sample images includes the annotated 2D bounding boxes and class labels for the first task objects. The Pseudo bounding boxes on the sample images include the Pseudo bounding boxes for the second task objects.

[0325] Step S1320 includes: calculating the loss of the RPN stage, i.e., the first loss function value, using the labeled 2D bounding box of the first task object and the Pseudo Bboxes of the second task object.

[0326] For example, Pseudo Bboxes with a confidence level greater than or equal to 0.05 in the Pseudo Bboxes file are merged with the labeled 2D boxes in the labeled data to obtain the 2D boxes of all target objects on the sample image. The 2D boxes of all target objects are compared with the candidate 2D boxes predicted by RPN to obtain the loss function value of the RPN stage, i.e., the first loss function value.

[0327] Step S1320 corresponds to step S1221 in method 1200. For a detailed description, please refer to step S1221.

[0328] S1330 uses labeled data to calculate the loss in the classification and regression network stage.

[0329] Sample images can be categorized into one or more tasks based on the data type they are labeled with, or in other words, they can be categorized into one or more tasks based on the task corresponding to their primary object. For example, if a sample image only labels traffic signs, then it belongs only to the "traffic signs" task. If a sample image labels both people and vehicles, then it belongs to both the "people" and "vehicles" tasks. When calculating the loss in the classification and regression network stage, only the loss corresponding to the task to which the current sample image belongs is calculated; the losses for other tasks are not calculated. For example, if the current input sample image belongs to the "people and vehicles" task, then only the loss corresponding to "people and vehicles" is calculated, and the losses for other tasks (such as traffic lights and traffic signs) are not calculated.

[0330] For example, such as Figure 14 As shown, the region of interest extraction module extracts features from a feature map based on the candidate 2D bounding boxes predicted by the RPN. After passing through shared function class (FC) and shared regression class (conv), these features are fed into the sub-classification FC and sub-regression FC corresponding to the task to which the sample image belongs, obtaining the prediction results. These results represent the confidence score of the candidate 2D bounding box belonging to the object category in that task, as well as the target 2D bounding box. The labeled data is then compared with the prediction results to obtain the loss, which is the loss of the classification and regression network stage corresponding to that task.

[0331] If the annotation data of the current sample image only includes annotation data for one task, then when the sample image is input into the network for training, for the multiple sub-classification fc and sub-regression fc in the first RCNN, only the sub-classification fc and sub-regression fc corresponding to that task in the first RCNN will be trained, without affecting the sub-classification fc and sub-regression fc corresponding to other tasks in the first RCNN.

[0332] For example, such as Figure 14 As shown in Table 1, if the current sample image only has 2D bounding boxes for traffic lights labeled, then the task for traffic lights is Task 3. During training, the prediction results for the traffic lights in the sample image are obtained only through the sub-classification fc and sub-regression fc corresponding to Task 3, and compared with the ground truth to obtain the loss value. In other words, the sample image of traffic lights only goes through the backbone, RPN, region of interest extraction module, and the sub-classification fc and sub-regression fc corresponding to traffic lights in the first RCNN. The sub-classification fc and sub-regression fc corresponding to other tasks do not participate in the calculation of the loss value.

[0333] If the annotation data of the current sample image includes annotation data for multiple tasks, then when the sample image is input into the network for training, for the multiple sub-classification fc and sub-regression fc in the first RCNN, only the sub-classification fc and sub-regression fc corresponding to the multiple tasks in the first RCNN are trained, without affecting the sub-classification fc and sub-regression fc corresponding to other tasks in the first RCNN.

[0334] For example, such as Figure 14 As shown in Table 1, if the current sample image is labeled with 2D bounding boxes for traffic lights and people, the task for traffic lights is Task 3, and the task for people is Task 1. During training, the prediction results for traffic lights in the sample image are obtained through the sub-classification fc and sub-regression fc corresponding to Task 3, and the prediction results for people in the sample image are obtained through the sub-classification fc and sub-regression fc corresponding to Task 1. These results are then compared with the ground truth to obtain the loss values ​​for the two tasks. In other words, the sample image only passes through the backbone, RPN, region of interest extraction module, the sub-classification fc and sub-regression fc corresponding to Task 3 in the first RCNN, and the sub-classification fc and sub-regression fc corresponding to Task 1 in the first RCNN. The sub-classification fc and sub-regression fc corresponding to other tasks do not participate in the calculation of the loss value. In this way, the loss for the classification and regression stages corresponding to the two tasks will be obtained, and the overall loss value for the classification and regression stages can be the average of these multiple losses.

[0335] S1340, gradient backhaul.

[0336] After calculating the loss, gradient backpropagation is required.

[0337] Based on the backpropagation of the loss (first loss function value) of the RPN stage and the loss (second loss function value) of the classification and regression network, the gradients of the relevant parameters are calculated and gradient backpropagation is performed.

[0338] Gradient backpropagation is performed on the parts of the perceptual network that need to be trained. The parts of the perceptual network that need to be trained are determined based on the task to which the sample image belongs. Parts that do not correspond to the task to which the sample image belongs do not participate in gradient backpropagation.

[0339] For example, such as Figure 14 As shown, the gradient is backpropagated along the subclassification fc and subregression fc corresponding to the task to which the sample image belongs, without affecting the subclassification fc and subregression fc corresponding to other tasks. The shared fc or conv of the first RCNN, as well as RPN and backbone, all participate in gradient backpropagation.

[0340] S1350, adjust the parameters of the sensing network.

[0341] The weight parameters of the part of the perceptron to be trained are updated using the backpropagated gradient.

[0342] In this way, the part of the perceptual network corresponding to the task to which the sample image belongs can be adjusted in a targeted manner, so that the part corresponding to the task to which the sample image belongs can learn the task to which the sample image belongs better.

[0343] S1360, determine whether the perceptron network has converged.

[0344] If the perceptron converges, the weight parameters of the perceptron are output.

[0345] If the perceptual network fails to converge, proceed to step S1310 to continue the training process.

[0346] In this embodiment, the labeled data for the sample images can be partially labeled, allowing for targeted data collection. This means collecting sample images specific to a particular task, rather than labeling all objects of interest in every image, thus reducing data collection and labeling costs. Furthermore, the method of preparing training data using partially labeled data offers flexible scalability. When adding a detection task, only the corresponding part of the detection task needs to be added to the classification and regression network. For example, adding a sub-classification flow table and a sub-regression flow table for that detection task, along with sample images containing labeled data of the newly added objects, is required. It is not necessary to label the newly added objects to be detected based on the existing training data.

[0347] Furthermore, by using pseudo-boundaries to supplement unlabeled objects to be detected in sample images, the mutual inhibition between partially labeled data of different tasks when the RPN is trained based on partially labeled data is avoided, which affects the training of the RPN. This is beneficial for the RPN to predict the location of the objects to be detected in all tasks that need to be detected.

[0348] Furthermore, the part corresponding to each task in the perceptual network only detects the objects to be detected in that task, which avoids false positives on objects from other tasks that are not labeled during training. Additionally, shared parts of the perceptual network, such as the backbone, RPN, and region of interest extraction module, learn common features across tasks, while the parts corresponding to each task in the classification and regression network learn task-specific features. For example, in the first RCNN, the sub-classification and sub-regression functions for each task learn their respective task-specific features.

[0349] This application embodiment also provides an object recognition method 1500, which can be executed by an object recognition device. The object recognition device can be a cloud service device or a terminal device, such as a vehicle, drone, robot, computer, server, or mobile phone, etc., with sufficient computing power to execute the object recognition method. It can also be a system composed of a cloud service device and a terminal device. Exemplarily, method 1500 can be executed by... Figure 3 Execution device 110 in Figure 5 The neural network processor 50 or Figure 6 The execution device 310 or the local device executes the execution.

[0350] For example, object recognition methods can be specifically derived from, for example... Figure 3 The execution device 110 shown performs the execution.

[0351] Optionally, the object recognition method can be processed by a GPU, or by a combination of a CPU and a GPU, or it can use other processors suitable for neural network computing instead of a GPU. This application does not impose any restrictions.

[0352] In method 1500, the perceptual network in the embodiments of this application is used to process the image. To avoid unnecessary repetition, repeated descriptions will be omitted when introducing method 1500 below.

[0353] Method 1500 includes steps S1510 to S1540, which are described below.

[0354] The perceptual network includes a backbone network, RPN, region of interest extraction module, and classification and regression network.

[0355] S1510 uses the backbone network to perform convolution processing on the input image and outputs the first feature map of the input image.

[0356] For example, the input image may be an image captured by a camera of a terminal device (or a computer, server, or other device or equipment), or the input image may be an image obtained from within the terminal device (or a computer, server, or other device or equipment) (e.g., an image stored in the photo album of the terminal device, or an image obtained by the terminal device from the cloud). This application embodiment does not limit this.

[0357] S1520, using RPN to output the position information of candidate 2D bounding boxes of the target object based on the second feature map, the target object includes objects to be detected in multiple tasks, each of the multiple tasks includes at least one category, and the second feature map is determined based on the first feature map.

[0358] S1530, the region of interest extraction module extracts the first feature information on the third feature map based on the position information of the candidate 2D box. The first feature information is the feature of the region where the candidate 2D box is located. The third feature map is determined based on the first feature map.

[0359] S1540, the first feature information is processed using a classification and regression network to obtain the target 2D bounding box of the target object and the first indication information. The number of target 2D bounding boxes is less than or equal to the number of candidate 2D bounding boxes. The first indication information is used to indicate the category to which the target object belongs.

[0360] Optionally, the first feature information is processed using a classification and regression network to obtain the target 2D bounding box of the target object and the first indication information, including: processing the first feature information using a classification and regression network to obtain the confidence scores of the candidate 2D bounding boxes belonging to each category in multiple tasks; adjusting the position information of the candidate 2D bounding boxes using a classification and regression network to obtain the adjusted candidate 2D bounding boxes; determining the target 2D bounding box based on the adjusted candidate 2D bounding boxes; and determining the first indication information based on the confidence scores of the target 2D bounding boxes belonging to each category.

[0361] Optionally, the classification and regression network includes a first region convolutional neural network (RCNN). The first RCNN includes a hidden layer, multiple sub-classification fully connected layers, and multiple sub-regression fully connected layers. The hidden layer is connected to multiple sub-classification fully connected layers, and the multiple sub-classification fully connected layers are connected to multiple sub-regression fully connected layers. The multiple sub-classification fully connected layers correspond one-to-one with multiple tasks, and the multiple sub-regression fully connected layers correspond one-to-one with multiple tasks. The classification and regression network processes the first feature information to output the target 2D bounding box of the target object and the first indication information, including: processing the first feature information using the hidden layer to obtain second feature information; using the sub-classification fully connected layers to obtain the confidence score of the candidate 2D bounding box belonging to the object category in the task corresponding to the sub-classification fully connected layer based on the second feature information; and using the sub-regression fully connected layers to adjust the position information of the candidate 2D bounding box based on the second feature information to obtain the adjusted candidate 2D bounding box.

[0362] Optionally, the classification and regression network includes a second RCNN, which includes hidden layers, a classification fully connected layer, and a regression fully connected layer. The hidden layers are connected to the classification fully connected layer and the regression fully connected layer. The classification and regression network processes the first feature information to output the target 2D bounding box of the target object and the first indication information, including: processing the first feature information using the hidden layer to obtain third feature information; obtaining the confidence of the candidate 2D bounding box belonging to each category based on the third feature information using the classification fully connected layer; and adjusting the position information of the candidate 2D bounding box based on the third feature information using the regression fully connected layer to obtain the adjusted candidate 2D bounding box.

[0363] Optionally, the classification fully connected layer is obtained by merging multiple sub-classification fully connected layers in the first RCNN, and the regression fully connected layer is obtained by merging multiple sub-regression fully connected layers in the first RCNN. The first RCNN includes hidden layers, multiple sub-classification fully connected layers, and multiple sub-regression fully connected layers. The hidden layers are connected to multiple sub-classification fully connected layers, and the hidden layers are connected to multiple sub-regression fully connected layers. The multiple sub-classification fully connected layers correspond one-to-one with multiple tasks, and the multiple sub-regression fully connected layers correspond one-to-one with multiple tasks. The sub-classification fully connected layer is used to obtain the confidence score of the candidate 2D box belonging to the object category in the task corresponding to the sub-classification fully connected layer based on the third feature information. The sub-regression fully connected layer is used to adjust the position information of the candidate 2D box based on the third feature information to obtain the adjusted candidate 2D box.

[0364] Figure 16 The diagram illustrates the processing flow of the object recognition method provided in the embodiments of this application. Figure 16 The processing flow in can be regarded as Figure 15 One specific implementation of the method shown is... Figure 16 The method in the middle can be adopted Figure 8 The perceptual network shown is executed. For related descriptions, please refer to the description in Perceptual Network 800. To avoid unnecessary repetition, repeated descriptions are appropriately omitted when describing Method 1600.

[0365] The following section uses the visual perception system of ADAS / ADS as an example to provide a detailed description of the solution in the embodiments of this application.

[0366] Based on the task division method in Table 1, the structure of the perception network used in this embodiment is as follows: Figure 17 As shown. Figure 17 As shown, the perceptual network includes: backbone, RPN, region of interest extraction module and second RCNN. Figure 17 The sensor network shown can be regarded as Figure 11 This is one specific implementation of the sensor network shown. Figure 17The perception network in the system can simultaneously perform target detection for the eight major categories listed in Table 1. In other words, Figure 17 The perception network in the system can simultaneously perform target detection for the eight tasks listed in Table 1. Figure 17 The sensor network shown can be based on Figure 14 The sensory network shown is determined. For example, as... Figure 18 As shown, the classification fc in the second RCNN is obtained by merging multiple sub-classification fcs in the first RCNN, and the regression fc in the second RCNN is obtained by merging the sub-regression fcs in the first RCNN. Figure 18 As can be seen from this, the perceptual network of this application can flexibly add or remove subclassification fc and subregression fc in the first RCNN according to the needs of the business, thereby achieving target detection for different numbers of tasks.

[0367] Specifically, method 1600 includes steps S1610 to S1650.

[0368] S1610, Input the image to be processed.

[0369] S1620, Generate basic features.

[0370] For example, step S1620 can be performed by Figure 17 The backbone is executed.

[0371] Specifically, the backbone performs convolution processing on the input image to generate several feature maps of different scales, namely the first feature map.

[0372] For example, the backbone can be a variety of convolutional networks, such as VGG16, ResNet50, or Inception-Net.

[0373] Furthermore, if the perceptual network also includes an FPN, step S1620 may also include: performing feature fusion based on the first feature map and outputting the fused feature map.

[0374] The feature maps output by the backbone network or FPN can be used as basic features for subsequent modules.

[0375] S1630, predict candidate 2D boxes.

[0376] For example, step S1630 can be performed by Figure 17 RPN execution in the process.

[0377] RPN predicts the region containing the target object on the second feature map and outputs candidate 2D bounding boxes that match the region containing the target object. The target object includes objects to be detected in multiple tasks. The second feature map can include feature maps output by the backbone network or FPN.

[0378] Specifically, the RPN predicts regions where target objects may exist based on feature maps provided by the backbone or FPN, and outputs candidate bounding boxes for these regions, or the coordinates of the candidate regions (proposals). In this embodiment, the RPN can predict candidate bounding boxes for the eight categories of objects to be detected that may exist, as shown in Table 1.

[0379] S1640, extract features of candidate 2D bounding boxes.

[0380] For example, step S1640 can be performed by Figure 17 The region of interest extraction module is executed.

[0381] The region of interest extraction module extracts features from the region containing the candidate 2D bounding box on the third feature map. The third feature map can be a feature map provided by the backbone or FPN.

[0382] For example, the region of interest extraction module extracts the features of the region where each proposal is located on a feature map provided by the backbone or the FPN based on the coordinates of the proposal provided by the RPN, and resizes it to a fixed size to obtain the features of each proposal.

[0383] S1650, refines and classifies candidate 2D bounding boxes.

[0384] For example, step S1650 can be performed by Figure 17 The second RCNN is executed.

[0385] Specifically, in the second R-CNN, the hidden layers, such as shared fc / conv, further extract features from each proposal extracted by the region of interest extraction module and feed them into cls fc and reg fc. The cls fc classifies the proposals, obtaining the confidence score of each proposal belonging to each category. The reg fc adjusts the coordinates of the 2D bounding boxes of the proposals to obtain more compact 2D bounding box coordinates. Then, a box merge operation, such as NMS, is performed to merge the adjusted 2D bounding boxes, outputting the target 2D bounding box and the classification result. The classification result can serve as the first indication information.

[0386] Figure 17 The weights of classification fc in the second R-CNN are obtained by... Figure 16The weights of multiple subclass fc in the first RCNN are combined. Figure 17 The weights of the regression fc in the second R-CNN are obtained by... Figure 16 The weights of multiple sub-regression full-c in the first R-CNN are merged. During training, Figure 16 The first R-CNN in the model uses the sigmoid function to normalize the label logits obtained from the sub-classification full-catch (FC) to obtain the confidence score for each class. This is equivalent to performing a binary classification process for each class. The confidence score of the current class is independent of other classes. Therefore, during inference, the sub-classification FCs of all tasks can be merged into a single classification FC. Sub-regression FCs can also be merged into a single regression FC.

[0387] For example, if the candidate 2D bounding box is a rectangle, and its position information is represented by four values, the length of the hidden layer output feature is 1024, and the number of classes in each task is n, then the weight of the sub-regression fully connected (FC) in each task is 1024. A 4n tensor, where the weights of subclass fc in each task are 1024. A tensor of size n. Table 1 shows a total of 8 tasks and 31 types of task objects. The weights of the combined fully connected regression are 1024. A tensor of 124 has a weight of 1024 for the class fc. The second R-CNN has a tensor of 31. That is, the merged second R-CNN includes only one classification fc and one regression fc. Its input and output are consistent with the tensor shape of the merged weights. That is, the input of the classification fc and the regression fc is 1024, the output of the classification fc is 31, and the output of the regression fc is 124.

[0388] Table 2 shows the single-headed multi-tasking network in this embodiment and the existing multi-headed multi-tasking network when the input image size is 720. The parameters and computational cost of implementing 8 tasks in the case of 1280 (@720p) are statistically analyzed. Table 2 shows the parameters and computational cost of the single-headed task network and the multi-headed multi-task network with 8 tasks.

[0389] Table 2

[0390] As shown in Table 2, if a multi-head, multi-task network is used to implement the target detection of the eight tasks in this embodiment, the total computational load is 413.96 GFlops, and the number of network parameters is 142.76 M. This enormous computational load and number of network parameters would put significant pressure on the hardware. However, using the single-head, multi-task network provided in this embodiment reduces the computational load by 60% and the number of parameters by 71%, significantly reducing both computational load and parameter count, thus reducing computational consumption and hardware pressure.

[0391] Table 3 shows a comparison of inference time between the single-head multitasking network in the embodiments of this application and the existing multi-head multitasking network.

[0392] Table 3

[0393] As shown in Table 3, compared with the multi-head multi-task network, the single-head multi-task network of this application embodiment reduces the latency of 720p and 1080p images by 17% and 22% respectively, which significantly improves the processing efficiency and is beneficial for deployment in scenarios with high real-time requirements.

[0394] Furthermore, the single-head multi-task network in this embodiment can achieve the same detection performance as the multi-head multi-task network. Table 4 shows a performance comparison of the single-head multi-task network and the multi-head multi-task network in some categories.

[0395] Table 4

[0396] As shown in Table 4, the average precision (AP) of the single-head multi-task network in this embodiment is not significantly different from that of existing multi-head multi-task networks, meaning their performance is comparable. Therefore, the single-head multi-task network in this embodiment can save computational resources and GPU memory while maintaining model performance.

[0397] The following is combined with Figures 19 to 20 The apparatus of the embodiments of this application will be described below. It should be understood that the apparatus described below is capable of performing the methods of the foregoing embodiments of this application. To avoid unnecessary repetition, repeated descriptions will be appropriately omitted when describing the apparatus of the embodiments of this application below.

[0398] Figure 19 This is a schematic block diagram of an apparatus according to an embodiment of this application. Figure 19 The apparatus 4000 shown includes an acquisition unit 4010 and a processing unit 4020.

[0399] In one implementation, the device 4000 can serve as a training device for a perceptual network, and the acquisition unit 4010 and the processing unit 4020 can be used to execute the training method of the perceptual network of the present application embodiment. For example, they can be used to execute method 1200 or method 1300.

[0400] Specifically, the perceptual network includes a candidate region generation network (RPN), which is used to predict the position information of candidate 2D bounding boxes of target objects in sample images. The target objects include objects to be detected for multiple tasks, each of which includes at least one category. The target objects include objects from a first task and objects from a second task.

[0401] The acquisition unit 4010 is used to acquire training data, which includes sample images, annotation data of the first task object on the sample images, and pseudo-boundary boxes of the second task object on the sample images. The annotation data includes the class label of the first task object and the labeled 2D bounding box of the first task object. The pseudo-boundary box of the second task object is the target 2D bounding box of the second task object obtained by reasoning about the sample images through other perceptual networks.

[0402] The processing unit 4020 is used to train the perceptual network based on the training data.

[0403] Optionally, as an embodiment, the perception network further includes a backbone network, a region of interest extraction module, and a classification and regression network. The processing unit 4020 is specifically used to: calculate a first loss function value based on the difference between the labeled 2D bounding box of the first task object and the target 2D bounding box of the second task object and the candidate 2D bounding box of the target object in the sample image predicted by the RPN; calculate a second loss function value of the perception network based on the labeled data; backpropagate the first loss function value and the second loss function value to adjust the parameters of the part of the perception network that needs to be trained. The part of the perception network that needs to be trained includes the part of the classification and regression network that needs to be trained, the region of interest extraction module, the RPN, and the backbone network. The part of the classification and regression network that needs to be trained is determined based on the first task object.

[0404] Optionally, as an embodiment, the backbone network is used to perform convolution processing on the sample image and output a first feature map of the sample image; the RPN is used to output the position information of the candidate 2D boxes of the target object based on the second feature map, the second feature map being determined based on the first feature map; the region of interest extraction module is used to extract first feature information on the third feature map based on the position information of the candidate 2D boxes, the first feature information being the features of the region where the candidate 2D boxes are located, the third feature map being determined based on the first feature map; the classification and regression network is used to process the first feature information and output the target 2D box of the target object and first indication information, the number of target 2D boxes of the target object being less than or equal to the number of candidate 2D boxes of the target object, and the first indication information being used to indicate the category to which the target object belongs.

[0405] Optionally, as an embodiment, the classification and regression network includes a first region convolutional neural network (RCNN). The first RCNN includes a hidden layer, multiple sub-classification fully connected layers, and multiple sub-regression fully connected layers. The hidden layer is connected to the multiple sub-classification fully connected layers, and the multiple sub-classification fully connected layers are connected to multiple tasks one-to-one. The multiple sub-regression fully connected layers are also connected to multiple tasks one-to-one. The hidden layer is used to process the first feature information to obtain the second feature information. The sub-classification fully connected layers are used to obtain the confidence of the candidate 2D box belonging to the object category in the task corresponding to the sub-classification fully connected layer based on the second feature information. The sub-regression fully connected layers are used to adjust the position information of the candidate 2D box based on the result processed by the hidden layer to obtain the adjusted candidate 2D box. The training part of the classification and regression network includes the hidden layer and the sub-classification fully connected layer and the sub-regression fully connected layer corresponding to the task where the object of the first task is located.

[0406] In another implementation, device 4000 can serve as an object recognition device. The object recognition device includes an acquisition unit 4010 and a processing unit 4020. The perception network includes a backbone network, a candidate region generation network, a region of interest extraction module, and a classification and regression network.

[0407] The acquisition unit 4010 and the processing unit 4020 can be used to execute the object recognition method of the embodiments of this application, for example, they can be used to execute method 1500 or method 1600.

[0408] The acquisition unit 4010 is used to acquire the input image.

[0409] The processing unit 4020 is used to perform convolution processing on the input image using a backbone network to obtain a first feature map of the input image; it uses the RPN to output the position information of candidate 2D bounding boxes of the target object based on the second feature map, the target object includes objects to be detected in multiple tasks, each of the multiple tasks includes at least one category, and the second feature map is determined based on the first feature map; it uses the region of interest extraction module to extract first feature information on the third feature map based on the position information of the candidate 2D bounding boxes, the first feature information is the feature of the region where the candidate 2D bounding box is located, and the third feature map is determined based on the first feature map; it uses a classification regression network to process the first feature information to obtain the target 2D bounding box of the target object and first indication information, the number of target 2D bounding boxes of the target object is less than or equal to the number of candidate 2D bounding boxes of the target object, and the first indication information is used to indicate the category to which the target object belongs.

[0410] Optionally, as an embodiment, the processing unit 4020 is specifically used to: process the first feature information using a classification and regression network to obtain the confidence level of the candidate 2D box belonging to each category in multiple tasks; adjust the position information of the candidate 2D box using a classification and regression network to obtain the adjusted candidate 2D box; determine the target 2D box based on the adjusted candidate 2D box; and determine the first indication information based on the confidence level of the target 2D box belonging to each category.

[0411] Optionally, as an embodiment, the classification and regression network includes a first region convolutional neural network (RCNN). The first RCNN includes a hidden layer, multiple sub-classification fully connected layers, and multiple sub-regression fully connected layers. The hidden layer is connected to the multiple sub-classification fully connected layers, and the multiple sub-regression fully connected layers are connected to multiple tasks one-to-one. The processing unit is specifically used to: process the first feature information using the hidden layer to obtain the second feature information; use the sub-classification fully connected layers to obtain the confidence level of the candidate 2D box belonging to the object category in the task corresponding to the sub-classification fully connected layer based on the second feature information; and use the sub-regression fully connected layers to adjust the position information of the candidate 2D box based on the second feature information to obtain the adjusted candidate 2D box.

[0412] Optionally, as an embodiment, the classification regression network includes a second RCNN, which includes a hidden layer, a classification fully connected layer, and a regression fully connected layer. The hidden layer is connected to the classification fully connected layer and the regression fully connected layer. The processing unit 4020 is specifically used to: process the first feature information using the hidden layer to obtain the third feature information; use the classification fully connected layer to obtain the confidence of the candidate 2D box belonging to each category based on the obtained third feature information; and use the regression fully connected layer to adjust the position information of the candidate 2D box based on the obtained third feature information to obtain the adjusted candidate 2D box.

[0413] Optionally, as an embodiment, the classification fully connected layer is obtained by merging multiple sub-classification fully connected layers in the first RCNN, and the regression fully connected layer is obtained by merging multiple sub-regression fully connected layers in the first RCNN. The first RCNN includes a hidden layer, multiple sub-classification fully connected layers, and multiple sub-regression fully connected layers. The hidden layer is connected to multiple sub-classification fully connected layers, and the hidden layer is connected to multiple sub-regression fully connected layers. The multiple sub-classification fully connected layers correspond one-to-one with multiple tasks, and the multiple sub-regression fully connected layers correspond one-to-one with multiple tasks. The sub-classification fully connected layer is used to obtain the confidence score of the candidate 2D box belonging to the object category in the task corresponding to the sub-classification fully connected layer based on the obtained third feature information. The sub-regression fully connected layer is used to adjust the position information of the candidate 2D box based on the obtained third feature information to obtain the adjusted candidate 2D box.

[0414] It should be noted that the aforementioned device 4000 is embodied in the form of a functional unit. The term "unit" here can be implemented in software and / or hardware, without specific limitations.

[0415] For example, a "unit" can be a software program, a hardware circuit, or a combination of both that implements the above functions. The hardware circuit may include an application-specific integrated circuit (ASIC), electronic circuitry, a processor (e.g., a shared processor, a proprietary processor, or a group processor) and memory for executing one or more software or firmware programs, integrated logic circuitry, and / or other suitable components that support the described functions.

[0416] Therefore, the units of the various examples described in the embodiments of this application can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0417] Figure 20 This is a schematic diagram of the hardware structure of a device provided in an embodiment of this application. Figure 20 The device 6000 shown (which may specifically be a computer device) includes a memory 6001, a processor 6002, a communication interface 6003, and a bus 6004. The memory 6001, the processor 6002, and the communication interface 6003 are interconnected via the bus 6004.

[0418] In one implementation, device 6000 can be used as a training device for a perceptual network.

[0419] The memory 6001 can be a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 6001 can store programs, and when the program stored in the memory 6001 is executed by the processor 6002, the processor 6002 performs the various steps of the training method for the perceptual network according to the embodiments of this application. Specifically, the processor 6002 can perform the steps described above... Figure 12 Step S1220 in the method shown.

[0420] The processor 6002 may be a general-purpose central processing unit (CPU), microprocessor, application-specific integrated circuit (ASIC), graphics processing unit (GPU), or one or more integrated circuits, used to execute related programs to implement the training method of the perceptual network in the embodiments of this application.

[0421] The processor 6002 can also be an integrated circuit chip with signal processing capabilities; for example, it could be... Figure 5 The chip shown. In the implementation process, each step of the training method for the perceptual network of this application can be completed by the integrated logic circuit of the hardware in the processor 6002 or by instructions in the form of software.

[0422] The processor 6002 described above can also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software modules can be located in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. This storage medium is located in memory 6001. The processor 6002 reads the information in memory 6001 and, in conjunction with its hardware, completes the functions required by the units included in the training device in the embodiments of this application, or executes the methods of the embodiments of this application. Figure 12 The training method for the perceptual network is shown.

[0423] The communication interface 6003 uses transceiver devices, such as, but not limited to, transceivers, to enable communication between the device 6000 and other devices or communication networks. For example, training data can be acquired through the communication interface 6003.

[0424] Bus 6004 may include a pathway for transmitting information between various components of device 6000 (e.g., memory 6001, processor 6002, communication interface 6003).

[0425] In another implementation, device 6000 can be used as an object recognition device.

[0426] The memory 6001 can be a ROM, static storage device, or RAM. The memory 6001 can store programs, and when the program stored in the memory 6001 is executed by the processor 6002, the processor 6002 and the communication interface 6003 are used to execute the various steps of the object recognition method of this application embodiment. Specifically, the processor 6002 can execute the steps described above... Figure 15 Steps S1520 to S1540 in the method shown.

[0427] The processor 6002 may be a general-purpose CPU, microprocessor, ASIC, GPU, or one or more integrated circuits, used to execute relevant programs to achieve the functions required by the units in the object recognition device of this application embodiment, or to execute the object recognition method of this application method embodiment.

[0428] The processor 6002 can also be an integrated circuit chip with signal processing capabilities; for example, it could be... Figure 6 The chip shown. In the implementation process, each step of the object recognition method of this application embodiment can be completed by the integrated logic circuit of the hardware in the processor 6002 or by instructions in the form of software.

[0429] The processor 6002 described above can also be a general-purpose processor, DSP, ASIC, FPGA, or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software modules can be located in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. This storage medium is located in memory 6001. The processor 6002 reads the information in memory 6001 and, in conjunction with its hardware, completes the functions required by the units included in the object recognition device of this application embodiment, or executes the object recognition method of the method embodiment of this application.

[0430] The communication interface 6003 uses transceiver devices, such as, but not limited to, transceivers, to enable communication between the device 6000 and other devices or communication networks. For example, data to be processed can be obtained through the communication interface 6003.

[0431] Bus 6004 may include a pathway for transmitting information between various components of device 6000 (e.g., memory 6001, processor 6002, communication interface 6003).

[0432] It should be noted that although the above-described device 6000 only shows a memory, processor, and communication interface, those skilled in the art should understand that in specific implementations, device 6000 may also include other devices necessary for normal operation. Furthermore, depending on specific needs, those skilled in the art should understand that device 6000 may also include hardware devices for implementing other additional functions. Moreover, those skilled in the art should understand that device 6000 may only include the devices necessary for implementing the embodiments of this application, and may not necessarily include... Figure 20 All the devices shown.

[0433] This application provides a computer-readable medium storing program code that can be executed by a device. The program code includes functions for performing actions such as... Figure 15 or Figure 16 The content related to the object recognition method shown.

[0434] This application provides a computer-readable medium storing program code that can be executed by a device. The program code includes functions for performing actions such as... Figure 12 or Figure 13 The training method shown is described below.

[0435] This application provides a computer program product that, when run on a computer, causes the computer to perform actions such as... Figure 15 or Figure 16 The content related to the object recognition method shown.

[0436] This application provides a computer program product that, when run on a computer, causes the computer to perform actions such as... Figure 12 or Figure 13 The training method shown is described below.

[0437] This application provides a chip, which includes a processor and a data interface. The processor reads instructions from a memory through the data interface and executes them as follows: Figure 15 or Figure 16 Object recognition methods.

[0438] This application provides a chip, which includes a processor and a data interface. The processor reads instructions from a memory through the data interface and executes them as follows: Figure 12 or Figure 13 Training methods.

[0439] Optionally, as one implementation, the chip may further include a memory storing instructions, and the processor is configured to execute the instructions stored in the memory. When the instructions are executed, the processor is configured to perform... Figure 15 or Figure 16 Object recognition methods or Figure 12 or Figure 13 Training methods.

[0440] It should be understood that the processor in the embodiments of this application can be a central processing unit (CPU), or it can be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor, etc.

[0441] It should also be understood that the memory in the embodiments of this application can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate synchronous DRAM (DDR SDRAM), enhanced synchronous DRAM (ESDRAM), synchronous linked DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0442] The above embodiments can be implemented, in whole or in part, by software, hardware, firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more sets of available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium. A semiconductor medium can be a solid-state drive.

[0443] It should be understood that the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. A and B can be singular or plural. Additionally, the character " / " in this article generally indicates an "or" relationship between the preceding and following related objects, but it can also represent an "and / or" relationship. Please refer to the context for a more accurate understanding.

[0444] In this application, "at least one" means one or more, and "more than one" means two or more. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or multiple items. For example, at least one of a, b, or c can mean: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple.

[0445] It should be understood that in the various embodiments of this application, the order of the above-mentioned processes does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0446] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0447] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0448] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0449] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0450] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0451] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0452] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A sensing network, characterized in that, include: The network consists of a backbone network, a candidate region generation network (RPN), a region of interest extraction module, and a classification and regression network. The backbone network is used to perform convolution processing on the input image and output the first feature map of the input image; The RPN is used to output the position information of candidate 2D bounding boxes of target objects based on a second feature map. The target objects include objects to be detected in multiple tasks, each of which includes at least one category. The second feature map is determined based on the first feature map. The region of interest extraction module is used to extract first feature information on a third feature map based on the position information of the candidate 2D box. The first feature information is the feature of the region where the candidate 2D box is located. The third feature map is determined based on the first feature map. The classification and regression network is used to process the first feature information and output the target 2D bounding box of the target object and the first indication information. The number of target 2D bounding boxes is less than or equal to the number of candidate 2D bounding boxes. The first indication information is used to indicate the category to which the target object belongs.

2. The sensing network according to claim 1, characterized in that, The classification and regression network is specifically used for: The first feature information is processed to obtain the confidence level of the candidate 2D box belonging to each category in the multiple tasks; The position information of the candidate 2D bounding box is adjusted to obtain the adjusted candidate 2D bounding box; The target 2D bounding box is determined based on the adjusted candidate 2D bounding boxes; The first indication information is determined based on the confidence level of the target 2D bounding box belonging to each category.

3. The sensing network according to claim 2, characterized in that, The classification and regression network includes a first region convolutional neural network RCNN. The first RCNN includes a hidden layer, multiple sub-classification fully connected layers and multiple sub-regression fully connected layers. The hidden layer is connected to the multiple sub-classification fully connected layers and the multiple sub-regression fully connected layers. The multiple sub-classification fully connected layers correspond one-to-one with the multiple tasks and the multiple sub-regression fully connected layers correspond one-to-one with the multiple tasks. The hidden layer is used to process the first feature information to obtain the second feature information; The sub-classification fully connected layer is used to obtain the confidence level of the candidate 2D box belonging to the object category in the task corresponding to the sub-classification fully connected layer based on the second feature information; The sub-regression fully connected layer is used to adjust the position information of the candidate 2D box according to the second feature information to obtain the adjusted candidate 2D box.

4. The sensing network according to claim 2, characterized in that, The classification and regression network includes a second RCNN, which includes a hidden layer, a classification fully connected layer, and a regression fully connected layer. The hidden layer is connected to the classification fully connected layer, and the hidden layer is connected to the regression fully connected layer. The hidden layer is used to process the first feature information to obtain the third feature information; The classification fully connected layer is used to obtain the confidence level of the candidate 2D box belonging to each category based on the third feature information; The regression fully connected layer is used to adjust the position information of the candidate 2D box according to the third feature information to obtain the adjusted candidate 2D box.

5. The sensing network according to claim 4, characterized in that, The classification fully connected layer is obtained by merging multiple sub-classification fully connected layers in the first RCNN, and the regression fully connected layer is obtained by merging multiple sub-regression fully connected layers in the first RCNN. The first RCNN includes the hidden layer, multiple sub-classification fully connected layers and multiple sub-regression fully connected layers. The hidden layer is connected to the multiple sub-classification fully connected layers and the multiple sub-regression fully connected layers. The multiple sub-classification fully connected layers correspond one-to-one with the multiple tasks and the multiple sub-regression fully connected layers correspond one-to-one with the multiple tasks. The sub-classification fully connected layer is used to obtain the confidence level of the candidate 2D box belonging to the object category in the task corresponding to the sub-classification fully connected layer based on the third feature information; The sub-regression fully connected layer is used to adjust the position information of the candidate 2D box according to the third feature information to obtain the adjusted candidate 2D box.

6. A training method for a perceptual network, characterized in that, The perception network includes: a candidate region generation network (RPN), which is used to predict the position information of candidate 2D bounding boxes of target objects in a sample image. The target objects include objects to be detected for multiple tasks, and each of the multiple tasks includes at least one category. The target objects include a first task object and a second task object. The method includes: Acquire training data, which includes the sample image, the annotation data of the first task object on the sample image, and the pseudo-boundary of the second task object on the sample image. The annotation data includes the class label of the first task object and the annotation 2D bounding box of the first task object. The pseudo-boundary of the second task object is the target 2D bounding box of the second task object obtained by reasoning about the sample image through other perceptual networks. The perception network is trained based on the training data.

7. The training method according to claim 6, characterized in that, The perception network also includes a backbone network, a region of interest extraction module, and a classification and regression network. Training the perceptual network based on the training data includes: The first loss function value is calculated based on the difference between the labeled 2D bounding box of the first task object, the target 2D bounding box of the second task object, and the candidate 2D bounding box of the target object in the sample image predicted by the RPN. Calculate the second loss function value of the perception network based on the labeled data; The first loss function value and the second loss function value are backpropagated to adjust the parameters of the part of the perception network that needs to be trained. The part of the perception network that needs to be trained includes the part of the classification and regression network that needs to be trained, the region of interest extraction module, the RPN and the backbone network. The part of the classification and regression network that needs to be trained is determined based on the first task object.

8. The training method according to claim 7, characterized in that, The backbone network is used to perform convolution processing on the sample image and output the first feature map of the sample image. The RPN is used to output the position information of the candidate 2D bounding box of the target object based on the second feature map, which is determined based on the first feature map. The region of interest extraction module is used to extract first feature information on a third feature map based on the position information of the candidate 2D box. The first feature information is the feature of the region where the candidate 2D box is located. The third feature map is determined based on the first feature map. The classification and regression network is used to process the first feature information and output the target 2D bounding box of the target object and the first indication information. The number of target 2D bounding boxes is less than or equal to the number of candidate 2D bounding boxes. The first indication information is used to indicate the category to which the target object belongs.

9. The training method according to claim 8, characterized in that, The classification and regression network includes a first region convolutional neural network RCNN. The first RCNN includes a hidden layer, multiple sub-classification fully connected layers and multiple sub-regression fully connected layers. The hidden layer is connected to the multiple sub-classification fully connected layers and the multiple sub-regression fully connected layers. The multiple sub-classification fully connected layers correspond one-to-one with the multiple tasks and the multiple sub-regression fully connected layers correspond one-to-one with the multiple tasks. The hidden layer is used to process the first feature information to obtain the second feature information; The sub-classification fully connected layer is used to obtain the confidence level of the candidate 2D box belonging to the object category in the task corresponding to the sub-classification fully connected layer based on the second feature information; The sub-regression fully connected layer is used to adjust the position information of the candidate 2D box according to the second feature information to obtain the adjusted candidate 2D box; as well as The training components of the classification and regression network include the hidden layer and the sub-classification fully connected layer and sub-regression fully connected layer corresponding to the task to which the first task object belongs.

10. An object recognition method, characterized in that, The perceptual network includes: a backbone network, a region candidate generation network (RPN), a region of interest extraction module, and a classification and regression network. The method includes: The input image is convolved using the backbone network to obtain the first feature map of the input image; The RPN outputs the position information of candidate 2D bounding boxes of target objects based on the second feature map. The target objects include objects to be detected in multiple tasks, and each of the multiple tasks includes at least one category. The second feature map is determined based on the first feature map. The region of interest extraction module extracts first feature information on the third feature map based on the position information of the candidate 2D box. The first feature information is the feature of the region where the candidate 2D box is located. The third feature map is determined based on the first feature map. The first feature information is processed using the classification regression network to obtain the target 2D bounding box of the target object and the first indication information. The number of target 2D bounding boxes is less than or equal to the number of candidate 2D bounding boxes. The first indication information is used to indicate the category to which the target object belongs.

11. The method according to claim 10, characterized in that, The step of processing the first feature information using the classification and regression network to obtain the target 2D bounding box of the target object and the first indication information includes: The classification and regression network is used to process the first feature information to obtain the confidence level of the candidate 2D box belonging to each category in the multiple tasks; The position information of the candidate 2D bounding boxes is adjusted using the classification and regression network to obtain the adjusted candidate 2D bounding boxes; The target 2D bounding box is determined based on the adjusted candidate 2D bounding boxes; The first indication information is determined based on the confidence level of the target 2D bounding box belonging to each category.

12. The method according to claim 11, characterized in that, The classification and regression network includes a first region convolutional neural network RCNN. The first RCNN includes a hidden layer, multiple sub-classification fully connected layers and multiple sub-regression fully connected layers. The hidden layer is connected to the multiple sub-classification fully connected layers and the multiple sub-regression fully connected layers. The multiple sub-classification fully connected layers correspond one-to-one with the multiple tasks and the multiple sub-regression fully connected layers correspond one-to-one with the multiple tasks. as well as The step of processing the first feature information using the classification and regression network to output the target 2D bounding box of the target object and the first indication information includes: The first feature information is processed using the hidden layer to obtain the second feature information; The confidence level of the candidate 2D box belonging to the object category in the task corresponding to the sub-classification fully connected layer is obtained by using the second feature information based on the sub-classification fully connected layer. The position information of the candidate 2D box is adjusted using the sub-regression fully connected layer based on the second feature information to obtain the adjusted candidate 2D box.

13. The method according to claim 11, characterized in that, The classification and regression network includes a second RCNN, which includes a hidden layer, a classification fully connected layer, and a regression fully connected layer. The hidden layer is connected to the classification fully connected layer, and the hidden layer is connected to the regression fully connected layer. as well as The step of processing the first feature information using the classification and regression network to output the target 2D bounding box of the target object and the first indication information includes: The first feature information is processed using the hidden layer to obtain the third feature information; The confidence level of the candidate 2D box belonging to each category is obtained using the classification fully connected layer based on the third feature information; The position information of the candidate 2D box is adjusted by the regression fully connected layer based on the third feature information to obtain the adjusted candidate 2D box.

14. The method according to claim 13, characterized in that, The classification fully connected layer is obtained by merging multiple sub-classification fully connected layers in the first RCNN, and the regression fully connected layer is obtained by merging multiple sub-regression fully connected layers in the first RCNN. The first RCNN includes the hidden layer, the plurality of sub-classification fully connected layers and the plurality of sub-regression fully connected layers. The hidden layer is connected to the plurality of sub-classification fully connected layers and the plurality of sub-regression fully connected layers. The plurality of sub-classification fully connected layers correspond one-to-one with the plurality of tasks and the plurality of sub-regression fully connected layers correspond one-to-one with the plurality of tasks. The sub-classification fully connected layer is used to obtain the confidence level of the candidate 2D box belonging to the object category in the task corresponding to the sub-classification fully connected layer based on the third feature information; The sub-regression fully connected layer is used to adjust the position information of the candidate 2D box according to the third feature information to obtain the adjusted candidate 2D box.

15. A training device for a perceptual network, characterized in that, The perception network includes a candidate region generation network (RPN), which is used to predict the position information of candidate 2D bounding boxes of target objects in sample images. The target objects include objects to be detected for multiple tasks, each of the multiple tasks includes at least one category, and the target objects include first task objects and second task objects. The training device includes: The acquisition unit is used to acquire training data, which includes the sample image, the annotation data of the first task object on the sample image, and the pseudo-boundary of the second task object on the sample image. The annotation data includes the class label of the first task object and the labeled 2D bounding box of the first task object. The pseudo-boundary of the second task object is the target 2D bounding box of the second task object obtained by reasoning about the sample image through other perceptual networks. A processing unit is used to train the perception network based on the training data.

16. The training device according to claim 15, characterized in that, The perception network further includes a backbone network, a region of interest extraction module, and a classification and regression network, and the processing unit is specifically used for: The first loss function value is calculated based on the difference between the labeled 2D bounding box of the first task object, the target 2D bounding box of the second task object, and the candidate 2D bounding box of the target object in the sample image predicted by the RPN. Calculate the second loss function value of the perception network based on the labeled data; The first loss function value and the second loss function value are backpropagated to adjust the parameters of the part of the perception network that needs to be trained. The part of the perception network that needs to be trained includes the part of the classification and regression network that needs to be trained, the region of interest extraction module, the RPN and the backbone network. The part of the classification and regression network that needs to be trained is determined based on the first task object.

17. The training device according to claim 16, characterized in that, The backbone network is used to perform convolution processing on the sample image and output the first feature map of the sample image. The RPN is used to output the position information of candidate 2D bounding boxes of the target object based on the second feature map, which is determined based on the first feature map. The region of interest extraction module is used to extract first feature information on a third feature map based on the position information of the candidate 2D box. The first feature information is the feature of the region where the candidate 2D box is located. The third feature map is determined based on the first feature map. The classification and regression network is used to process the first feature information and output the target 2D bounding box of the target object and the first indication information. The number of target 2D bounding boxes of the target object is less than or equal to the number of candidate 2D bounding boxes of the target object. The first indication information is used to indicate the category to which the target object belongs.

18. The training device according to claim 17, characterized in that, The classification and regression network includes a first region convolutional neural network RCNN. The first RCNN includes a hidden layer, multiple sub-classification fully connected layers and multiple sub-regression fully connected layers. The hidden layer is connected to the multiple sub-classification fully connected layers and the multiple sub-regression fully connected layers. The multiple sub-classification fully connected layers correspond one-to-one with the multiple tasks and the multiple sub-regression fully connected layers correspond one-to-one with the multiple tasks. The hidden layer is used to process the first feature information to obtain the second feature information; The sub-classification fully connected layer is used to obtain the confidence level of the candidate 2D box belonging to the object category in the task corresponding to the sub-classification fully connected layer based on the second feature information; The sub-regression fully connected layer is used to adjust the position information of the candidate 2D box based on the result of the hidden layer processing, so as to obtain the adjusted candidate 2D box. as well as The training components of the classification and regression network include the hidden layer and the sub-classification fully connected layer and sub-regression fully connected layer corresponding to the task to which the first task object belongs.

19. An object recognition device, characterized in that, The perception network includes: a backbone network, a region candidate generation network (RPN), a region of interest extraction module, and a classification and regression network. The perception network is deployed on the device, which includes: The acquisition unit is used to acquire the input image; Processing unit, used for: The input image is convolved using the backbone network to obtain the first feature map of the input image; The RPN outputs the position information of candidate 2D bounding boxes of the target object based on the second feature map. The target object includes objects to be detected in multiple tasks. Each of the multiple tasks includes at least one category. The second feature map is determined based on the first feature map. The region of interest extraction module extracts first feature information on the third feature map based on the position information of the candidate 2D box. The first feature information is the feature of the region where the candidate 2D box is located. The third feature map is determined based on the first feature map. The first feature information is processed using the classification regression network to obtain the target 2D bounding box of the target object and the first indication information. The number of target 2D bounding boxes of the target object is less than or equal to the number of candidate 2D bounding boxes of the target object. The first indication information is used to indicate the category to which the target object belongs.

20. The apparatus according to claim 19, characterized in that, The processing unit is specifically used for: The classification and regression network is used to process the first feature information to obtain the confidence level of the candidate 2D box belonging to each category in the multiple tasks; The position information of the candidate 2D bounding boxes is adjusted using the classification and regression network to obtain the adjusted candidate 2D bounding boxes; The target 2D bounding box is determined based on the adjusted candidate 2D bounding boxes; The first indication information is determined based on the confidence level of the target 2D bounding box belonging to each category.

21. The apparatus according to claim 20, characterized in that, The classification and regression network includes a first region convolutional neural network (RCNN), which includes a hidden layer, multiple sub-classification fully connected layers, and multiple sub-regression fully connected layers. The hidden layer is connected to both the multiple sub-classification and sub-regression fully connected layers. Each sub-classification and sub-regression fully connected layer corresponds one-to-one with a specific task. The processing unit is specifically used for: The first feature information is processed using the hidden layer to obtain the second feature information; the confidence level of the candidate 2D box belonging to the object category in the task corresponding to the sub-classification fully connected layer is obtained using the second feature information. The position information of the candidate 2D box is adjusted using the sub-regression fully connected layer based on the second feature information to obtain the adjusted candidate 2D box.

22. The apparatus according to claim 20, characterized in that, The classification and regression network includes a second RCNN, which includes a hidden layer, a fully connected classification layer, and a fully connected regression layer. The hidden layer is connected to the fully connected classification layer and the fully connected regression layer. The processing unit is specifically used for: The first feature information is processed using the hidden layer to obtain the third feature information; The confidence level of the candidate 2D box belonging to each category is obtained by using the classification fully connected layer based on the obtained third feature information; The position information of the candidate 2D box is adjusted by the regression fully connected layer based on the obtained third feature information to obtain the adjusted candidate 2D box.

23. The apparatus according to claim 22, characterized in that, The classification fully connected layer is obtained by merging multiple sub-classification fully connected layers in the first RCNN, and the regression fully connected layer is obtained by merging multiple sub-regression fully connected layers in the first RCNN. The first RCNN includes the hidden layer, the plurality of sub-classification fully connected layers and the plurality of sub-regression fully connected layers. The hidden layer is connected to the plurality of sub-classification fully connected layers and the plurality of sub-regression fully connected layers. The plurality of sub-classification fully connected layers correspond one-to-one with the plurality of tasks and the plurality of sub-regression fully connected layers correspond one-to-one with the plurality of tasks. The sub-classification fully connected layer is used to obtain the confidence level of the candidate 2D box belonging to the object category in the task corresponding to the sub-classification fully connected layer based on the obtained third feature information; The sub-regression fully connected layer is used to adjust the position information of the candidate 2D box according to the obtained third feature information, so as to obtain the adjusted candidate 2D box.

24. A training device for a perceptual network, characterized in that, The device includes a processor and a transmission interface, the processor receiving or sending data through the transmission interface, and the processor being configured to invoke program instructions stored in memory to perform the method of any one of claims 6 to 9.

25. An object recognition device, characterized in that, The device includes a processor and a transmission interface, the processor receiving or sending data through the transmission interface, and the processor being configured to invoke program instructions stored in memory to perform the method of any one of claims 10 to 14.

26. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores program code for execution by the device, which, when run on a computer or processor, causes the computer or processor to perform the method as described in any one of claims 6 to 9 or 10 to 14.

27. A computer program product containing instructions, characterized in that, When the computer program product is run on a computer or processor, it causes the computer or processor to perform the method as described in any one of claims 6 to 9 or 10 to 14.

Images