Object detection method and device

Abstract

The present application discloses an object detection method, which relates to the field of artificial intelligence, including: obtaining a target image; detecting an object in the target image through a machine learning model, and outputting a first depth value of the object and target information, wherein the target information includes at least one of the following information: geometric features of the object, key point information of the object, and position features of the object; predicting at least one second depth value of the object according to the target information; fusing the first depth value and at least one second depth value to obtain the target depth of the object. The present application estimates the depth of the center point of the target based on different principles using the relevant attribute information (target information) of the object output by the machine learning model, thereby obtaining at least one differentiated and diversified depth prediction, and then determining a more accurate depth value of the object based on multiple depth values ​​of the object obtained through multiple different methods.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence, and in particular to an object detection method and device thereof. Background Art

[0002] Computer vision is an integral part of various intelligent / autonomous systems in various application fields, such as manufacturing, inspection, document analysis, medical diagnosis, and military. It is a discipline about how to use cameras / camcorders and computers to obtain the data and information of the objects we need. Figuratively speaking, it is to install eyes (cameras or camcorders) and brains (algorithms) on computers to replace human eyes to identify, track, and measure targets, so that computers can perceive the environment. Because perception can be seen as extracting information from sensory signals, computer vision can also be seen as a science that studies how to make artificial systems "perceive" from images or multidimensional data. In general, computer vision uses various imaging systems to replace visual organs to obtain input information, and then computers replace the brain to complete the processing and interpretation of these input information. The ultimate research goal of computer vision is to enable computers to observe and understand the world through vision like humans, and have the ability to adapt to the environment autonomously.

[0003] At present, the visual perception network can complete more and more functions, including image classification, 2D detection, semantic segmentation (Mask), key point detection, linear object detection (such as lane line or stop line detection in autonomous driving technology), drivable area detection, etc. In addition, the visual perception system has the characteristics of low cost, non-contact, small size, and large amount of information. With the continuous improvement of the accuracy of visual perception algorithms, it has become a key technology in many artificial intelligence systems today and has been widely used, such as: recognition of dynamic obstacles (people or cars) and static objects (traffic lights, traffic signs or traffic cones) on the road in advanced driving assistant systems (ADAS) and autonomous driving systems (ADS), and achieving slimming effects by recognizing the mask and key points of the human body in the terminal vision camera beauty function.

[0004] Whether in autonomous driving (ADS) or advanced driver assistance systems (ADAS), vehicles need to perceive other traffic participants in the road environment in real time to ensure driving safety and support their next driving decision. Among all the sensors in smart cars, cameras can provide information with the highest resolution and the richest details. In addition, the visual perception system has the characteristics of low cost and small size, so perceiving and modeling other traffic participants near the current vehicle through visual signals is a key technology in the perception system of smart cars. Among them, target detection of surrounding vehicles and pedestrians is the most important perception function, which has been widely used, such as estimating the position of surrounding vehicles in ADAS and ADS to avoid collisions and maintain a safe distance; predicting the direction and size of surrounding vehicles and predicting their future motion trajectory on this basis, so that autonomous driving vehicles can plan safe and comfortable routes, etc.

[0005] Detecting traffic participants in images and obtaining relevant information is the most challenging problem in camera-based autonomous driving perception systems. At the same time, there are many types of traffic participants, and various environmental factors such as their distance from the camera, degree of occlusion, and lighting will cause further interference. How to improve the accuracy of target detection is the key to enabling smart cars to accurately perceive the positions of surrounding traffic participants and make the next driving decision.

[0006] With the development of deep learning technology, the use of convolutional neural networks (CNN) for target detection has become a trend in this field. In existing research, according to the different ways of obtaining target depth information, target detection methods can generally be divided into two methods: direct estimation based on neural networks and geometric reasoning based on estimating the key points of the target and other information of the target. The direct estimation method requires the neural network to learn the depth of each target relative to the camera based on its appearance in the image, and directly obtain the estimation of the depth, orientation, and size of the target from the network during the reasoning stage. This method has a simple process but the reasoning accuracy is not accurate enough. Once the camera that takes the image changes, the accuracy often drops significantly. Summary of the invention

[0007] The present application provides an object detection method, which can improve the accuracy of depth values ​​obtained when detecting objects in an image.

[0008] In a first aspect, the present application provides an object detection method, comprising: acquiring a target image; detecting an object in the target image through a machine learning model, and outputting a first depth value and target information of the object, the target information comprising at least one of the following information: geometric features of the object, key point information of the object, and position features of the object; predicting at least one second depth value of the object based on the target information; and fusing the first depth value and the at least one second depth value to obtain a target depth of the object.

[0009] In a possible implementation, the target image may include at least one object, and the object in the embodiment of the present application may be any object in the target image.

[0010] In one possible implementation, the reasoning accuracy of the machine learning model is not accurate enough. Once the camera that captures the image changes or the shooting scene changes, the accuracy tends to drop significantly. In the embodiment of the present application, the relevant attribute information of the object output by the machine learning model (that is, the target information in the embodiment of the present application) is used to estimate the depth of the center point of the target based on different principles, so that at least one different and diversified depth prediction can be obtained, and then a more accurate depth value of the object can be determined based on multiple depth values ​​of the object obtained by multiple different methods.

[0011] In one possible implementation, after detecting the object in the target image through the machine learning model, the method also includes: outputting a first confidence of the first depth value and predicting a second confidence of each second depth value; and fusing the first confidence and the at least one second confidence to obtain a target confidence of the target depth.

[0012] This application uses different theoretical assumptions and target attribute combinations to simultaneously generate multiple groups of depth estimates and the uncertainty of each depth estimate, and in the subsequent process selects and fuses highly reliable depth predictions and discards unreliable erroneous predictions based on the size of the uncertainty, thereby ultimately generating robust depth estimation results, greatly improving the system's detection robustness and accuracy in different environments. In addition, the uncertainty also helps the system generate a more reliable evaluation score for the prediction results, further improving the detection accuracy and system interpretability.

[0013] In a possible implementation, at least one second depth value of the object may be predicted by an N-point perspective algorithm PnP.

[0014] Specifically, the key point information in the target information may include the 2D positions of multiple key points of the object on the image, the geometric features of the object include the physical size of the object, and the position features of the object include the orientation angle of the object. Then, at least one second depth value of the object may be predicted based on the target information by using an N-point perspective algorithm PnP.

[0015] In the embodiment of the present application, the depth of the target center point is of interest, so the 3D center point of the target can also be defined as a key point. When its 2D position (u c ,v c ), the center position of the object can be expressed in a form that is only related to z:

[0016]

[0017] Based on this, each key point can be expressed as two linear equations about the z value, so two sets of estimates of the z value of the target center point can be calculated.

[0018] In a possible implementation, depth prediction can also be performed based on the length of the object in a preset direction. Specifically, the geometric features of the object include multiple pixel sizes of the 3D envelope of the object in the preset direction and the physical size of the object in the preset direction; further, at least one second depth value of the object can be predicted based on the geometric relationship between the multiple pixel sizes and the physical size.

[0019] Taking the preset direction as the vertical direction as an example, the length of the preset direction is the height. The network can predict the physical height H of the target and the projection points of the corner points and center points of the target 3D bounding box on the upper and lower surfaces of the bounding box (i.e., the pre-defined key points). Since the physical height of the target is related to the distance between these key points, the depth estimation of the key points can be obtained by the following formula:

[0020]

[0021] Where f is the intrinsic parameter of the camera, which can be obtained in advance, H is the physical height of the target estimated by the network, and h is the vertical distance between a pair of key points on the upper and lower surfaces of the target 3D bounding box.

[0022] In a possible implementation, the fusing of the first confidence and the at least one second confidence includes: obtaining a first weight of the first confidence and each of the second confidences; and performing a weighted summation of the first confidence and the at least one second confidence according to the first weight.

[0023] The first confidence and the first weight of each second confidence may be understood as the first weight corresponding to the first confidence and the first weight corresponding to each second confidence.

[0024] In a possible implementation, the fusion method can be weighted summation. Specifically, the second weight of the first depth value and each of the second depth values ​​can be obtained; and according to the second weight, the first depth value and the at least one second depth value are weighted summed.

[0025] The second weight of the first depth value and each of the second depth values ​​may be understood as the second weight corresponding to the first depth value and the second weight corresponding to each of the second depth values.

[0026] In one possible implementation, the second weight of the first depth value and each of the second depth values ​​can be determined based on the confidence (or uncertainty) of the first depth value and each of the second depth values, wherein the confidence (or uncertainty) can describe the reliability of the depth value, and the higher the confidence of the depth value, the greater the corresponding weight, that is, the value of the second weight is positively correlated with the confidence of the corresponding depth value.

[0027] The confidence of the first depth value may be the output of the machine learning model, and the confidence of the second depth value obtained by different methods may be updated as a trainable parameter during model training, and when the model converges, the confidence of each updated second depth value may be obtained. In a possible implementation, the second weight of each second depth value may also be pre-set, which is not limited here.

[0028] In a possible implementation, the method further includes: obtaining the confidence of the 3D envelope box of the object output by the machine learning model; and fusing the confidence of the 3D envelope box of the object and the target confidence to obtain the geometric confidence of the object.

[0029] In a possible implementation, the fusing the confidence of the 3D envelope box of the object and the target confidence includes: obtaining a third weight of the confidence of the 3D envelope box of the object and the target confidence; and performing a weighted summation of the confidence of the 3D envelope box of the object and the target confidence according to the third weight.

[0030] In a possible implementation, the method further includes: obtaining the confidence of the 2D envelope box of the object output by the machine learning model; and obtaining the detection confidence of the object based on the confidence of the 2D envelope box of the object and the geometric confidence.

[0031] In a second aspect, the present application provides an object detection device, comprising:

[0032] An acquisition module, used for acquiring a target image;

[0033] An object detection module is used to detect an object in the target image through a machine learning model, and output a first depth value and target information of the object, wherein the target information includes at least one of the following information: geometric features of the object, key point information of the object, and position features of the object;

[0034] a depth value prediction module, configured to predict at least one second depth value of the object according to the target information;

[0035] A fusion module is used to fuse the first depth value and the at least one second depth value to obtain a target depth of the object.

[0036] In a possible implementation, the device further includes:

[0037] A confidence prediction module, configured to output a first confidence of the first depth value and predict a second confidence of each of the second depth values ​​after detecting the object in the target image by the machine learning model;

[0038] The fusion module is further used to fuse the first confidence and the at least one second confidence to obtain the target confidence of the target depth.

[0039] In a possible implementation, the key point information includes 2D positions of multiple key points of the object on the image;

[0040] The depth value prediction module is specifically used for:

[0041] At least one second depth value of the object is predicted according to the target information through an N-point perspective algorithm PnP.

[0042] In a possible implementation, the geometric feature of the object includes the physical size of the object, and the position feature of the object includes the orientation angle of the object.

[0043] In a possible implementation, the geometric features of the object include a plurality of pixel sizes of a 3D envelope box of the object in a preset direction, and a physical size of the object in the preset direction;

[0044] The depth value prediction module is specifically used for:

[0045] At least one second depth value of the object is predicted according to the geometric relationship between the multiple pixel sizes and the physical sizes.

[0046] In a possible implementation, the fusion module is specifically used to:

[0047] Obtaining the first confidence and a first weight of each of the second confidences;

[0048] A weighted sum is performed on the first confidence level and the at least one second confidence level according to the first weight.

[0049] In a possible implementation, the fusion module is specifically used to:

[0050] Obtaining the first depth value and a second weight of each of the second depth values;

[0051] A weighted sum is performed on the first depth value and the at least one second depth value according to the second weight.

[0052] In a possible implementation, the acquisition module is further used to:

[0053] Obtaining the confidence of the 3D envelope box of the object output by the machine learning model;

[0054] The fusion module is further used for:

[0055] The confidence of the 3D envelope box of the object and the target confidence are fused to obtain the geometric confidence of the object.

[0056] In a possible implementation, the fusion module is specifically used to:

[0057] Acquire the confidence of the 3D envelope box of the object and a third weight of the target confidence;

[0058] According to the third weight, a weighted sum is performed on the confidence of the 3D envelope box of the object and the target confidence.

[0059] In a possible implementation, the acquisition module is further used to:

[0060] Obtaining the confidence of the 2D envelope box of the object output by the machine learning model;

[0061] The confidence determination module is used to obtain the detection confidence of the object according to the confidence of the 2D envelope box of the object and the geometric confidence.

[0062] In a third aspect, an embodiment of the present application provides an object detection device, which may include a memory, a processor, and a bus system, wherein the memory is used to store programs, and the processor is used to execute the programs in the memory to perform the first aspect and any optional method thereof.

[0063] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, in which a computer program is stored. When the computer-readable storage medium is run on a computer, the computer executes the above-mentioned first aspect and any optional method thereof.

[0064] In a fifth aspect, an embodiment of the present application provides a computer program, which, when executed on a computer, enables the computer to execute the above-mentioned first aspect and any optional method thereof.

[0065] In a sixth aspect, the present application provides a chip system, which includes a processor for supporting an execution device or a training device to implement the functions involved in the above aspects, such as sending or processing the data involved in the above methods; or, information. In one possible design, the chip system also includes a memory, which is used to store program instructions and data necessary for the execution device or the training device. The chip system can be composed of chips, or it can include chips and other discrete devices. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] Figure 1 A structural diagram of the main framework of artificial intelligence;

[0067] Figure 2a and Figure 2b It is a schematic diagram of the application system framework of the present invention;

[0068] Figure 3 This is an illustration of an application scenario of this application;

[0069] Figure 4 This is an illustration of an application scenario of this application;

[0070] Figure 5 A system architecture diagram of the present application is shown below;

[0071] Figure 6 A schematic diagram of the structure of a neural network in an embodiment of the present application;

[0072] Figure 7 A schematic diagram of the structure of a neural network in an embodiment of the present application;

[0073] Figure 8 A hardware structure of a chip provided in an embodiment of the present application;

[0074] Fig. 9 A flowchart of an object detection method provided in an embodiment of the present application;

[0075] Fig.10a and Fig.10b It is the backbone network backbone of the embodiment of the present application;

[0076] Fig.11 This is a schematic diagram of the structure of an FPN;

[0077] Fig.12a It is a hint of a header;

[0078] Figure 12b This is a schematic diagram of the RPN layer of a header;

[0079] Fig.13 A schematic diagram of a network structure in this embodiment;

[0080] Fig.14 A heatmap is provided in accordance with an embodiment of the present invention;

[0081] Fig.15 An angle diagram provided for an embodiment of the present application;

[0082] Fig.16 A highly selected schematic diagram provided for an embodiment of the present application;

[0083] Fig.17a A schematic diagram of a system flow provided for an embodiment of the present application;

[0084] Fig.17b A schematic diagram of the structure of an object detection device provided in an embodiment of the present application;

[0085] Fig.18 A schematic diagram of the structure of an execution device provided in an embodiment of the present application;

[0086] Fig.19 A schematic diagram of the structure of a training device provided in an embodiment of the present application;

[0087] Fig. 20 A schematic diagram of the structure of a chip provided in an embodiment of the present application. DETAILED DESCRIPTION

[0088] The embodiments of the present invention are described below in conjunction with the accompanying drawings in the embodiments of the present invention. The terms used in the implementation mode of the present invention are only used to explain the specific embodiments of the present invention, and are not intended to limit the present invention.

[0089] The embodiments of the present application are described below in conjunction with the accompanying drawings. Those skilled in the art will appreciate that, with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0090] The terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and need not be used to describe a specific order or sequential order. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, which is only to describe the distinction mode adopted by the objects of the same attributes when describing in the embodiments of the present application. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, so that the process, method, system, product or equipment comprising a series of units need not be limited to those units, but may include other units that are not clearly listed or inherent to these processes, methods, products or equipment.

[0091] First, the overall workflow of the artificial intelligence system is described. Figure 1 , Figure 1 The figure shows a structural diagram of the main framework of artificial intelligence. The following is an explanation of the above artificial intelligence theme framework from the two dimensions of "intelligent information chain" (horizontal axis) and "IT value chain" (vertical axis). Among them, the "intelligent information chain" reflects a series of processes from data acquisition to processing. For example, it can be a general process of intelligent information perception, intelligent information representation and formation, intelligent reasoning, intelligent decision-making, intelligent execution and output. In this process, the data has undergone a condensation process of "data-information-knowledge-wisdom". The "IT value chain" reflects the value that artificial intelligence brings to the information technology industry from the underlying infrastructure of human intelligence, information (providing and processing technology implementation) to the industrial ecology process of the system.

[0092] (1) Infrastructure

[0093] The infrastructure provides computing power support for the AI ​​system, enables communication with the outside world, and supports it through the basic platform. It communicates with the outside world through sensors; computing power is provided by smart chips (CPU, NPU, GPU, ASIC, FPGA and other hardware acceleration chips); the basic platform includes distributed computing frameworks and networks and other related platform guarantees and support, which can include cloud storage and computing, interconnected networks, etc. For example, sensors communicate with the outside world to obtain data, and these data are provided to the smart chips in the distributed computing system provided by the basic platform for calculation.

[0094] (2) Data

[0095] The data on the upper layer of the infrastructure is used to represent the data sources in the field of artificial intelligence. The data involves graphics, images, voice, text, and IoT data of traditional devices, including business data of existing systems and perception data such as force, displacement, liquid level, temperature, and humidity.

[0096] (3) Data processing

[0097] Data processing usually includes data training, machine learning, deep learning, search, reasoning, decision-making and other methods.

[0098] Among them, machine learning and deep learning can symbolize and formalize data for intelligent information modeling, extraction, preprocessing, and training.

[0099] Reasoning refers to the process of simulating human intelligent reasoning in computers or intelligent systems, using formalized information to perform machine thinking and solve problems based on reasoning control strategies. Typical functions are search and matching.

[0100] Decision-making refers to the process of making decisions after intelligent information is reasoned, usually providing functions such as classification, sorting, and prediction.

[0101] (4) General ability

[0102] After the data has undergone the data processing mentioned above, some general capabilities can be further formed based on the results of the data processing, such as an algorithm or a general system, for example, translation, text analysis, computer vision processing, speech recognition, image recognition, etc.

[0103] (5) Smart products and industry applications

[0104] Smart products and industry applications refer to the products and applications of artificial intelligence systems in various fields. They are the encapsulation of the overall artificial intelligence solution, which productizes intelligent information decision-making and realizes practical applications. Its application areas mainly include: smart terminals, smart transportation, smart medical care, autonomous driving, smart cities, etc.

[0105] The embodiments of the present application are mainly used in fields such as driving assistance, automatic driving, and mobile terminals that require the completion of multiple sensing tasks. Figure 2a and Figure 2b As shown, the video is extracted to obtain a single picture, which is sent to the present invention. Figure 2a or Figure 2b The perception network shown in the figure obtains the 2D, 3D, Mask, key points and other information of the object of interest in the image. These detection results are output to the post-processing module for processing. For example, they are sent to the planning control unit for decision-making in the autonomous driving system and sent to the beautification algorithm in the mobile terminal for processing to obtain the beautified image. The following is a brief introduction to the two application scenarios of ADAS / ADS visual perception system and mobile phone beautification.

[0106] Application scenario 1: ADAS / ADS visual perception system

[0107] like Figure 3As shown in the figure, in ADAS and ADS, multiple types of 2D target detection need to be performed in real time, including: dynamic obstacles (pedestrians, cyclists, tricycles, cars, trucks, and buses), static obstacles (traffic cones, traffic sticks, fire hydrants, motorcycles, and bicycles), traffic signs (Traffic Sign, Guide Sign, Billboard, red traffic light (Traffic Light_Red) / yellow traffic light (Traffic Light_Yellow) / green traffic light (Traffic Light_Green) / black traffic light (Traffic Light_Black), and road signs). In addition, in order to accurately obtain the area occupied by dynamic obstacles in 3D space, it is also necessary to perform 3D estimation on the dynamic obstacles and output a 3D frame. In order to fuse with the laser radar data, it is necessary to obtain the mask of the dynamic obstacle, so as to filter out the laser point cloud that hits the dynamic obstacle; in order to accurately park, it is necessary to detect the four key points of the parking space at the same time; in order to perform composition positioning, it is necessary to detect the key points of the static target. Using the technical solution provided in the embodiment of this application, all or part of the above functions can be completed.

[0108] For example, the technical solution provided in the embodiments of the present application can be applied to adaptive cruise control in assisted driving and advanced assisted driving.

[0109] The adaptive cruise function in ADAS needs to adaptively adjust the speed of the vehicle according to the position and speed of the vehicle in front of the lane, so as to achieve automatic cruising without collision. When there are no other traffic participants (targets) in front of the lane of the vehicle, the vehicle moves forward according to the pre-set speed or road speed limit. For example, when the perception system of the vehicle detects that other traffic participants have entered the front of the lane, the vehicle will automatically reduce the speed of the vehicle according to the position and speed of the white car to avoid the front car from slowing down and colliding.

[0110] For example, the technical solution provided in the embodiments of the present application can be applied to target trajectory prediction in automatic assisted driving and monitoring.

[0111] Trajectory prediction uses cameras to perceive road scenes, and uses target detection algorithms to obtain information such as the position, direction, and size of important traffic participants in the environment. By accumulating multiple frames of detection results, the speed and direction of each target can be obtained, thereby predicting the target's future motion trajectory and serving as the basis for subsequent autonomous driving vehicle decision-making and control. For example, it can predict the future motion direction of surrounding vehicles for autonomous driving vehicles. For example, in a monitoring scene, by detecting the direction and position of pedestrians, their future motion can be predicted, thereby identifying possible crowd shapes in advance.

[0112] Application scenario 2: Mobile phone beauty function

[0113] like Figure 4 As shown, in a mobile phone, the object detection method provided in the embodiment of the present application can detect the mask and key points of the human body, and can enlarge and reduce the corresponding parts of the human body, such as performing waist-tightening and buttocks-beautifying operations, thereby outputting beauty pictures.

[0114] Application scenario 3: Image classification scenario:

[0115] After acquiring the image to be classified, the object recognition device uses the object recognition method of the present application to obtain the category of the object in the image to be classified, and then the image to be classified can be classified according to the category of the object in the image to be classified. For photographers, many photos are taken every day, including animals, people, and plants. The method of the present application can quickly classify the photos according to the content in the photos, which can be divided into photos containing animals, photos containing people, and photos containing plants.

[0116] When there are a large number of images, manual classification is inefficient, and people are easily fatigued when dealing with the same thing for a long time, and the classification results will have large errors. The method of the present application can be used to classify images.

[0117] Application scenario 4 Product classification:

[0118] After the object recognition device obtains the image of the commodity, the object recognition method of the present application is used to obtain the category of the commodity in the image of the commodity, and then the commodity is classified according to the category of the commodity. For a wide variety of commodities in large shopping malls or supermarkets, the object detection method of the present application can be used to complete the classification of commodities, reducing time expenditure and labor costs.

[0119] Since the embodiments of the present application involve the application of a large number of neural networks, in order to facilitate understanding, the relevant terms and related concepts such as neural networks involved in the embodiments of the present application are first introduced below.

[0120] (1) Object recognition: using image processing, machine learning, computer graphics and other related methods to determine the category of the object in the image.

[0121] (2) Neural Network

[0122] A neural network may be composed of neural units, and a neural unit may refer to an operation unit with xs and intercept 1 as input, and the output of the operation unit may be:

[0123]

[0124] Where s=1, 2, ...n, n is a natural number greater than 1, Ws is the weight of xs, and b is the bias of the neural unit. f is the activation function of the neural unit, which is used to introduce nonlinear characteristics into the neural network to convert the input signal in the neural unit into the output signal. The output signal of the activation function can be used as the input of the next convolution layer. The activation function can be a sigmoid function. A neural network is a network formed by connecting many of the above-mentioned single neural units together, that is, the output of one neural unit can be the input of another neural unit. The input of each neural unit can be connected to the local receptive field of the previous layer to extract the characteristics of the local receptive field. The local receptive field can be an area composed of several neural units.

[0125] (3) Deep Neural Networks

[0126] A deep neural network (DNN) can be understood as a neural network with many hidden layers. There is no special metric for "many" here. The multi-layer neural networks and deep neural networks we often talk about are essentially the same thing. According to the position of different layers of DNN, the neural network inside DNN can be divided into three categories: input layer, hidden layer, and output layer. Generally speaking, the first layer is the input layer, the last layer is the output layer, and the layers in between are all hidden layers. The layers are fully connected, that is, any neuron in the i-th layer must be connected to any neuron in the i+1-th layer. Although DNN looks complicated, the work of each layer is actually not complicated. Simply put, it is the following linear relationship expression: in, is the input vector, is the output vector, is the offset vector, W is the weight matrix (also called coefficient), and α() is the activation function. Each layer is just an input vector After such a simple operation, the output vector Since DNN has many layers, the coefficient W and the offset vector So how are the specific parameters defined in DNN? First, let's look at the definition of coefficient W. Take a three-layer DNN as an example, for example: the linear coefficient from the 4th neuron in the second layer to the 2nd neuron in the third layer is defined as The superscript 3 represents the layer number of the coefficient W, while the subscripts correspond to the third layer index 2 of the output and the second layer index 4 of the input. In summary, the coefficients from the kth neuron in the L-1th layer to the jth neuron in the Lth layer are defined as Note that the input layer does not have a W parameter. In a deep neural network, more hidden layers allow the network to better capture complex situations in the real world. In theory, a model with more parameters has higher complexity and greater "capacity", which means it can complete more complex learning tasks.

[0127] (4) Convolutional neural network (CNN) is a deep neural network with a convolutional structure. Convolutional neural network contains a feature extractor consisting of a convolution layer and a subsampling layer. The feature extractor can be regarded as a filter, and the convolution process can be regarded as using a trainable filter to convolve with an input image or convolution feature plane (feature map). The convolution layer refers to the neuron layer in the convolutional neural network that performs convolution processing on the input signal. In the convolution layer of the convolutional neural network, a neuron can only be connected to some neurons in the adjacent layer. A convolutional layer usually contains several feature planes, and each feature plane can be composed of some rectangularly arranged neural units. The neural units in the same feature plane share weights, and the shared weights here are the convolution kernels. Shared weights can be understood as the way to extract image information is independent of position. The implicit principle is that the statistical information of a part of the image is the same as that of other parts. This means that the image information learned in one part can also be used in another part. So for all positions on the image, we can use the same learned image information. In the same convolution layer, multiple convolution kernels can be used to extract different image information. Generally speaking, the more convolution kernels there are, the richer the image information reflected by the convolution operation.

[0128] The convolution kernel can be initialized in the form of a matrix of random size, and the convolution kernel can obtain reasonable weights through learning during the training process of the convolutional neural network. In addition, the direct benefit of shared weights is to reduce the connections between the layers of the convolutional neural network, while reducing the risk of overfitting.

[0129] (5) Back propagation algorithm

[0130] Convolutional neural networks can use the error back propagation (BP) algorithm to correct the size of the parameters in the initial super-resolution model during the training process, so that the reconstruction error loss of the super-resolution model becomes smaller and smaller. Specifically, the forward transmission of the input signal to the output will generate error loss, and the error loss information is back-propagated to update the parameters in the initial super-resolution model, so that the error loss converges. The back propagation algorithm is a back propagation movement dominated by error loss, aiming to obtain the optimal parameters of the super-resolution model, such as the weight matrix.

[0131] (6) Recurrent neural networks (RNN) are used to process sequence data.

[0132] In the traditional neural network model, from the input layer to the hidden layer and then to the output layer, the layers are fully connected, and the nodes within each layer are disconnected. Although this ordinary neural network has solved many difficult problems, it is still powerless to solve many problems. For example, if you want to predict the next word in a sentence, you generally need to use the previous word, because the previous and next words in a sentence are not independent. The reason why RNN is called a recurrent neural network is that the current output of a sequence is also related to the previous output. The specific manifestation is that the network will remember the previous information and apply it to the calculation of the current output, that is, the nodes between the hidden layers are no longer disconnected but connected, and the input of the hidden layer includes not only the output of the input layer but also the output of the hidden layer at the previous moment. In theory, RNN can process sequence data of any length. The training of RNN is the same as the training of traditional CNN or DNN.

[0133] Since we already have convolutional neural networks, why do we need recurrent neural networks? The reason is simple. In convolutional neural networks, there is a premise that the elements are independent of each other, and the input and output are also independent, such as cats and dogs. But in the real world, many elements are interconnected, such as the change of stocks over time, or someone said: I like traveling, and my favorite place is Yunnan. I must go there if I have the chance in the future. Fill in the blank here, humans should all know to fill in "Yunnan". Because humans will make inferences based on the content of the context, but how can machines do this? RNN came into being. RNN is designed to give machines the ability to remember like humans. Therefore, the output of RNN needs to rely on the current input information and historical memory information.

[0134] (7) Loss function

[0135] In the process of training a deep neural network, because we hope that the output of the deep neural network is as close as possible to the value we really want to predict, we can compare the predicted value of the current network with the target value we really want, and then update the weight vector of each layer of the neural network according to the difference between the two (of course, there is usually an initialization process before the first update, that is, pre-configuring parameters for each layer in the deep neural network). For example, if the predicted value of the network is high, adjust the weight vector to make it predict a lower value, and keep adjusting until the deep neural network can predict the target value we really want or a value very close to the target value we really want. Therefore, it is necessary to pre-define "how to compare the difference between the predicted value and the target value", which is the loss function or objective function, which are important equations used to measure the difference between the predicted value and the target value. Among them, taking the loss function as an example, the higher the output value (loss) of the loss function, the greater the difference, so the training of the deep neural network becomes a process of minimizing this loss as much as possible.

[0136] (8) Back propagation algorithm

[0137] Neural networks can use the error back propagation (BP) algorithm to correct the size of the parameters in the initial neural network model during the training process, so that the reconstruction error loss of the neural network model becomes smaller and smaller. Specifically, the forward transmission of the input signal to the output will generate error loss, and the parameters in the initial neural network model are updated by back propagating the error loss information, so that the error loss converges. The back propagation algorithm is a back propagation movement dominated by error loss, which aims to obtain the optimal parameters of the neural network model, such as the weight matrix.

[0138] The following describes the system architecture provided by the embodiments of the present application.

[0139] See also Figure 5, the embodiment of the present application provides a system architecture 100. As shown in the system architecture 100, the data acquisition device 160 is used to collect training data. In the embodiment of the present application, the training data includes: an image or image block of an object and the category of the object; and the training data is stored in the database 130. The training device 120 obtains a machine learning model based on the training data maintained in the database 130. The machine learning model may include a CNN feature extraction model (Explanation: the feature extraction model here is the model obtained by training in the training phase introduced above, which may be a machine learning model for feature extraction, etc.) and a header. The CNN feature extraction model can be used to implement the machine learning model provided in the embodiment of the present application, that is, the image or image block to be identified is input into the CNN feature extraction model after relevant preprocessing, and the 2D, 3D, Mask, key points and other information of the object of interest of the image or image block to be identified can be obtained. The CNN feature extraction model in the embodiment of the present application can be specifically a CNN convolutional neural network. It should be noted that in actual applications, the training data maintained in the database 130 may not all come from the collection of the data acquisition device 160, but may also be received from other devices. It should also be noted that the training device 120 does not necessarily train the CNN feature extraction model entirely based on the training data maintained by the database 130, and it is also possible to obtain training data from the cloud or other places for model training. The above description should not be regarded as a limitation on the embodiments of the present application.

[0140] The target model / rule obtained by training the training device 120 can be applied to different systems or devices, such as Figure 5 The execution device 110 shown in the figure may be a terminal, such as a mobile phone terminal, a tablet computer, a laptop computer, an augmented reality (AR) / virtual reality (VR), a vehicle terminal, etc., or a server or a cloud. Figure 5 In the embodiment, the execution device 110 is configured with an input / output (I / O) interface 112 for data interaction with an external device. A user may input data to the I / O interface 112 through a client device 140. The input data may include: an image to be identified, an image block, or a picture.

[0141] When the execution device 120 preprocesses the input data, or when the computing module 111 of the execution device 120 performs calculations and other related processing (such as implementing the functions of the machine learning model in this application), the execution device 120 can call the data, code, etc. in the data storage system 150 for corresponding processing, and can also store the data, instructions, etc. obtained from the corresponding processing in the data storage system 150.

[0142] Finally, the I / O interface 112 returns the processing results, such as the image or image block obtained above, or the 2D, 3D, Mask, key points, confidence and other information of the object of interest in the picture to the client device 140, so as to provide it to the user.

[0143] Optionally, the client device 140 may be a planning control unit in an autonomous driving system or a beautification algorithm module in a mobile phone terminal.

[0144] It is worth noting that the training device 120 can generate corresponding target models / rules based on different training data for different goals or different tasks, and the corresponding target models / rules can be used to achieve the above goals or complete the above tasks, thereby providing users with the desired results.

[0145] exist Figure 5 In the case shown in the figure, the user can manually give input data, and the manual giving can be operated through the interface provided by the I / O interface 112. In another case, the client device 140 can automatically send input data to the I / O interface 112. If the client device 140 is required to automatically send input data and needs to obtain the user's authorization, the user can set the corresponding authority in the client device 140. The user can view the results output by the execution device 110 on the client device 140, and the specific presentation form can be a specific method such as display, sound, action, etc. The client device 140 can also be used as a data acquisition terminal to collect the input data of the input I / O interface 112 and the output results of the output I / O interface 112 as shown in the figure as new sample data, and store them in the database 130. Of course, it is also possible not to collect through the client device 140, but the I / O interface 112 directly stores the input data of the input I / O interface 112 and the output results of the output I / O interface 112 as new sample data in the database 130.

[0146] It is worth noting that Figure 5 This is only a schematic diagram of a system architecture provided by an embodiment of the present application. The positional relationship between the devices, components, modules, etc. shown in the figure does not constitute any limitation. For example, in Figure 5 In the embodiment, the data storage system 150 is an external memory relative to the execution device 110. In other cases, the data storage system 150 may also be placed in the execution device 110.

[0147] like Figure 5 As shown, a CNN feature extraction model is obtained by training with the training device 120. The CNN feature extraction model in the embodiment of the present application can be a CNN convolutional neural network or a machine learning model to be introduced in the following embodiment.

[0148] Since CNN is a very common neural network, Figure 5 The focus is on a detailed introduction to the structure of CNN. As mentioned in the basic concept introduction above, a convolutional neural network is a deep neural network with a convolutional structure and a deep learning architecture. A deep learning architecture refers to multiple levels of learning at different levels of abstraction through machine learning algorithms. As a deep learning architecture, CNN is a feed-forward artificial neural network in which each neuron can respond to the image input into it.

[0149] The structure of the neural network specifically used in the image processing method of the embodiment of the present application can be as follows: Figure 6 As shown. Figure 6 In the example, the convolutional neural network (CNN) 100 may include an input layer 210, a convolutional layer / pooling layer 220 (where the pooling layer is optional), and a neural network layer 230. The input layer 210 may obtain an image to be processed, and the obtained image to be processed is passed to the convolutional layer / pooling layer 220 and the subsequent neural network layer 230 for processing, and the image processing result may be obtained. Figure 6 The internal layer structure of CNN 100 is introduced in detail.

[0150] Convolutional layer / pooling layer 220:

[0151] Convolutional Layer:

[0152] like Figure 6 The convolution layer / pooling layer 220 shown may include layers 221-226, for example: in one implementation, layer 221 is a convolution layer, layer 222 is a pooling layer, layer 223 is a convolution layer, layer 224 is a pooling layer, layer 225 is a convolution layer, and layer 226 is a pooling layer; in another implementation, layers 221 and 222 are convolution layers, layer 223 is a pooling layer, layer 224 and 225 are convolution layers, and layer 226 is a pooling layer. That is, the output of a convolution layer can be used as the input of a subsequent pooling layer, or as the input of another convolution layer to continue the convolution operation.

[0153] The following will take the convolution layer 221 as an example to introduce the internal working principle of a convolution layer.

[0154] The convolution layer 221 may include a plurality of convolution operators, which are also called kernels. The convolution operator is equivalent to a filter that extracts specific information from the input image matrix in image processing. The convolution operator can be essentially a weight matrix, which is usually predefined. In the process of performing convolution operations on the image, the weight matrix is ​​usually processed one pixel after another (or two pixels after two pixels... depending on the value of the step length stride) in the horizontal direction on the input image, thereby completing the work of extracting specific features from the image. The size of the weight matrix should be related to the size of the image. It should be noted that the depth dimension of the weight matrix is ​​the same as the depth dimension of the input image. In the process of performing convolution operations, the weight matrix will extend to the entire depth of the input image. Therefore, convolution with a single weight matrix will produce a convolution output with a single depth dimension, but in most cases, a single weight matrix is ​​not used, but multiple weight matrices of the same size (row × column), that is, multiple isotype matrices, are applied. The output of each weight matrix is ​​stacked to form the depth dimension of the convolution image, and the dimension here can be understood as being determined by the "multiple" mentioned above. Different weight matrices can be used to extract different features in the image, for example, one weight matrix is ​​used to extract image edge information, another weight matrix is ​​used to extract specific colors of the image, and another weight matrix is ​​used to blur unnecessary noise points in the image, etc. The multiple weight matrices have the same size (rows × columns), and the convolution feature maps extracted by the multiple weight matrices of the same size are also the same size. The extracted multiple convolution feature maps of the same size are then merged to form the output of the convolution operation.

[0155] The weight values ​​in these weight matrices need to be obtained through a lot of training in practical applications. The weight matrices formed by the weight values ​​obtained through training can be used to extract information from the input image, so that the convolutional neural network 200 can make correct predictions.

[0156] When the convolutional neural network 200 has multiple convolutional layers, the initial convolutional layer (for example, 221) often extracts more general features, which can also be called low-level features. As the depth of the convolutional neural network 200 increases, the features extracted by the later convolutional layers (for example, 226) become more and more complex, such as high-level semantic features. Features with higher semantics are more suitable for the problem to be solved.

[0157] Pooling layer:

[0158] Since it is often necessary to reduce the number of training parameters, it is often necessary to periodically introduce a pooling layer after the convolution layer. Figure 6Each layer 221-226 illustrated in 220 may be a convolution layer followed by a pooling layer, or may be multiple convolution layers followed by one or more pooling layers. In the image processing process, the only purpose of the pooling layer is to reduce the spatial size of the image. The pooling layer may include an average pooling operator and / or a maximum pooling operator for sampling the input image to obtain an image of smaller size. The average pooling operator may calculate the pixel values ​​in the image within a specific range to generate an average value as the result of average pooling. The maximum pooling operator may take the pixel with the largest value in the range within a specific range as the result of maximum pooling. In addition, just as the size of the weight matrix used in the convolution layer should be related to the image size, the operator in the pooling layer should also be related to the image size. The size of the image output after processing by the pooling layer may be smaller than the size of the image input to the pooling layer, and each pixel in the image output by the pooling layer represents the average value or maximum value of the corresponding sub-region of the image input to the pooling layer.

[0159] Neural Network Layer 230:

[0160] After being processed by the convolution layer / pooling layer 220, the convolution neural network 200 is not sufficient to output the required output information. As mentioned above, the convolution layer / pooling layer 220 only extracts features and reduces the parameters brought by the input image. However, in order to generate the final output information (the required class information or other related information), the convolution neural network 200 needs to use the neural network layer 230 to generate one or a group of outputs of the required number of classes. Therefore, the neural network layer 230 may include multiple hidden layers (such as Figure 6 231, 232 to 23n) and the output layer 240 shown, the parameters contained in the multi-layer hidden layer can be pre-trained according to relevant training data of a specific task type. For example, the task type may include image recognition, image classification, image super-resolution reconstruction, etc.

[0161] After the multiple hidden layers in the neural network layer 230, that is, the last layer of the entire convolutional neural network 200 is the output layer 240, which has a loss function similar to the classification cross entropy, specifically used to calculate the prediction error. Once the forward propagation of the entire convolutional neural network 200 (such as Figure 6 The propagation from 210 to 240 is forward propagation), and the reverse propagation (such as Figure 6 The propagation from 240 to 210 is called back propagation) and then starts to update the weight values ​​and biases of the aforementioned layers to reduce the loss of the convolutional neural network 200 and the error between the result output by the convolutional neural network 200 through the output layer and the ideal result.

[0162] It should be noted that the convolutional neural network 210 shown in FIG. 2 is only an example of a convolutional neural network. In specific applications, the convolutional neural network may also exist in the form of other network models.

[0163] The structure of the neural network specifically used in the image processing method of the embodiment of the present application can be as follows: Figure 7 As shown. Figure 7 In FIG. 1 , a convolutional neural network (CNN) 100 may include an input layer 110, a convolutional layer / pooling layer 120 (where the pooling layer is optional), and a neural network layer 130. Figure 6 compared to, Figure 7 The multiple convolutional layers / pooling layers in the convolutional layer / pooling layer 120 are operated in parallel, and the features extracted respectively are input to the full neural network layer 130 for processing.

[0164] It should be noted that Figure 6 and Figure 7 The convolutional neural network shown is only an example of two possible convolutional neural networks for the image processing method of an embodiment of the present application. In specific applications, the convolutional neural network adopted by the image processing method of the embodiment of the present application may also exist in the form of other network models.

[0165] In addition, the structure of the convolutional neural network obtained by the search method for the neural network structure of the embodiment of the present application can be as follows: Figure 6 and Figure 7 The convolutional neural network structure is shown in .

[0166] Figure 8 The hardware structure of a chip provided in the embodiment of the present application includes a neural network processor NPU50. The chip can be set as follows Figure 5 The execution device 110 shown in FIG. 1 is used to complete the calculation work of the calculation module 111. The chip can also be set in Figure 5 The training device 120 shown in FIG. 1 is used to complete the training work of the training device 120 and output the target model / rule. Figure 6 and Figure 7 The algorithms of each layer in the convolutional neural network shown in the figure can be used in Figure 8 is implemented in the chip shown.

[0167] The neural network processor NPU 50 is mounted on the host central processing unit (CPU) as a coprocessor, and the host CPU assigns tasks. The core part of the NPU is the operation circuit 503, and the controller 504 controls the operation circuit 503 to extract data from the memory (weight memory or input memory) and perform operations.

[0168] In some implementations, the operation circuit 503 includes multiple processing units (process engines, PEs) inside. In some implementations, the operation circuit 503 is a two-dimensional systolic array. The operation circuit 503 can also be a one-dimensional systolic array or other electronic circuits capable of performing mathematical operations such as multiplication and addition. In some implementations, the operation circuit 503 is a general-purpose matrix processor.

[0169] For example, assume there is an input matrix A, a weight matrix B, and an output matrix C. The operation circuit takes the corresponding data of matrix B from the weight memory 502 and caches it on each PE in the operation circuit. The operation circuit takes the matrix A data from the input memory 501 and performs matrix operation with matrix B, and the partial result or final result of the matrix is ​​stored in the accumulator 508.

[0170] The vector calculation unit 507 can further process the output of the operation circuit, such as vector multiplication, vector addition, exponential operation, logarithmic operation, size comparison, etc. For example, the vector calculation unit 507 can be used for network calculations of non-convolutional / non-FC layers in a neural network, such as pooling, batch normalization, local response normalization, etc.

[0171] In some implementations, the vector calculation unit 507 can store the processed output vector to the unified buffer 506. For example, the vector calculation unit 507 can apply a nonlinear function to the output of the operation circuit 503, such as a vector of accumulated values, to generate an activation value. In some implementations, the vector calculation unit 507 generates a normalized value, a merged value, or both. In some implementations, the processed output vector can be used as an activation input to the operation circuit 503, such as for use in a subsequent layer in a neural network.

[0172] The unified memory 506 is used to store input data and output data.

[0173] The weight data is directly transferred from the external memory to the input memory 501 and / or the unified memory 506 through the direct memory access controller 505 (DMAC), the weight data in the external memory is stored in the weight memory 502, and the data in the unified memory 506 is stored in the external memory.

[0174] The bus interface unit (BIU) 510 is used to implement the interaction between the main CPU, DMAC and instruction fetch memory 509 through the bus.

[0175] An instruction fetch buffer 509 connected to the controller 504 and used to store instructions used by the controller 504;

[0176] The controller 504 is used to call the instructions cached in the memory 509 to control the working process of the computing accelerator.

[0177] Optionally, the input data here in the present application is a picture, and the output data is 2D, 3D, Mask, key point and other information of the object of interest in the picture.

[0178] Generally, the unified memory 506, the input memory 501, the weight memory 502 and the instruction fetch memory 509 are all on-chip memories, and the external memory is a memory outside the NPU, which can be a double data rate synchronous dynamic random access memory (DDR SDRAM), a high bandwidth memory (HBM) or other readable and writable memory.

[0179] Introduced above Figure 5 The execution device 110 in the embodiment of the present application can execute each step of the object detection method. Figure 6 and Figure 7 The CNN model shown and Figure 8 The chip shown can also be used to execute each step of the object detection method of the embodiment of the present application. The object detection method of the embodiment of the present application is described in detail below with reference to the accompanying drawings.

[0180] Reference Fig. 9 , Fig. 9 The process diagram of an object detection method provided in an embodiment of the present application is as follows: Fig. 9 As shown, an object detection method provided in an embodiment of the present application may include steps 901 to 903, and these steps are described in detail below.

[0181] 901. Acquire a target image, and detect an object in the target image through a machine learning model, and output a first depth value of the object and target information, where the target information includes at least one of the following information: geometric features of the object, key point information of the object, and position features of the object.

[0182] In a possible implementation, the target image may include at least one object, and the object in the embodiment of the present application may be any object in the target image.

[0183] In the embodiment of the present application, the architecture of the machine learning model can be Figure 2a The architecture shown in , which can be composed of a feature extraction network and a head-end header, wherein the feature extraction network may include a backbone network backbone and a feature pyramid network (FPN), wherein FPN is optional.

[0184] In an embodiment of the present application, the backbone network backbone is used to receive an input image, perform convolution processing on the input image, and output feature maps with different resolutions corresponding to the image; that is, feature maps of different sizes corresponding to the image are output, that is, Backbone completes the extraction of basic features and provides corresponding features for subsequent detection.

[0185] Specifically, the backbone network can perform a series of convolutions on the input image to obtain feature maps at different scales. These feature maps will provide basic features for subsequent detection modules. The backbone network can take many forms, such as visual geometry group (VGG), residual neural network (resnet), GoogLeNet's core structure (Inception-net), etc.

[0186] The backbone network can perform convolution processing on the input image to generate several convolution feature maps of different scales. Each feature map is a H*W*C matrix, where H is the height of the feature map, W is the width of the feature map, and C is the number of channels of the feature map.

[0187] Backbone can use a variety of existing convolutional network frameworks, such as VGG16, Resnet50, Inception-Net, etc. The following uses Resnet18 as Backbone as an example. Fig.10a shown.

[0188] Assume that the resolution of the input image is H*W*3 (height H, width W, number of channels is 3, that is, RBG three channels). The input image is convolved through the first convolution module Res18-Conv1 of Resnet18 (convolution module 1 in the figure) to generate Featuremap C1. This feature map is downsampled twice relative to the input image, and the number of channels is expanded to 64. Therefore, the resolution of C1 is H / 4*W / 4*64. The convolution module 1 consists of several convolution layers, and the subsequent convolution modules are similar. Refer to Fig.10b , Fig.10b This is a schematic diagram of the structure of the convolution module, as shown in Fig.10b As shown in the figure, the convolution module 1 may include multiple convolution layers (convolution layer 1 to convolution layer N); C1 is convolved by the second convolution module Res18-Conv2 of Resnet18 (convolution module 2 in the figure) to obtain Featuremap C2, and the resolution of this feature map is consistent with that of C1; C2 is further processed by the third convolution module Res18-Conv3 of Resnet18 (convolution module 3 in the figure) to generate Featuremap C3, which is further downsampled relative to C2, and the number of channels is doubled, and its resolution is H / 8*W / 8*128; finally, C3 is processed by Res18-Conv4 (convolution module 4 in the figure) to generate Featuremap C4, whose resolution is H / 16*W / 16*256.

[0189] from Fig.10a It can be seen that Resnet18 performs multiple levels of convolution processing on the input image to obtain feature maps of different scales: C1 / C2 / C3 / C4. The width and height of the bottom feature map are relatively large, and the number of channels is relatively small. It is mainly the low-level features of the image (such as image edges and texture features). The width and height of the high-level feature map are relatively small, and the number of channels is relatively large. It is mainly the high-level features of the image (such as shape and object features). The subsequent 2D detection process will make further predictions based on these feature maps.

[0190] In an embodiment of the present application, the backbone network backbone includes multiple convolution modules, each convolution module includes multiple convolution layers, each convolution module can perform convolution processing on the input feature map to obtain feature maps of different resolutions. In an embodiment of the present application, the first convolution layer included in the backbone network backbone is one of the multiple convolution layers included in the backbone network backbone.

[0191] It should be noted that the backbone network in the embodiment of the present application may also be referred to as a backbone network, which is not limited here.

[0192] It should be noted that Fig.10a and Fig.10b The backbone network backbone shown in the figure is only one implementation method and does not constitute a limitation to the present application.

[0193] In an embodiment of the present application, FPN is connected to the backbone network backbone, and FPN can perform convolution processing on multiple feature maps of different resolutions generated by the backbone network backbone to construct a feature pyramid.

[0194] Reference Fig.11 , Fig.11 The figure is a schematic diagram of the structure of an FPN, in which the top-level feature map C4 is processed using a convolution module 1, and the convolution module 1 may include at least one convolution layer. Exemplarily, the convolution module 1 may use a hole convolution and a 1×1 convolution to reduce the number of channels of the top-level feature map C4 to 256, as the top-level feature map P4 of the feature pyramid; the output result of the feature map C3 of the next layer of the top layer is horizontally linked and the number of channels is reduced to 256 using a 1×1 convolution (convolution module 2), and then the feature map p4 is added pixel by pixel to obtain the feature map p3; and so on, from top to bottom, a feature pyramid Φp={feature map p4, feature map p3, feature map p2, feature map p1} is constructed.

[0195] In an embodiment of the present application, FPN includes multiple convolution modules, each convolution module includes multiple convolution layers, each convolution module can perform convolution processing on the input feature map, and the second convolution layer included in the FPN in the embodiment of the present application is one of the multiple convolution layers included in the FPN.

[0196] It should be noted that Fig.11 The FPN shown in the figure is only one implementation method and does not constitute a limitation to the present application.

[0197] In the embodiment of the present application, the header is connected to the FPN. The header can complete the detection of the 2D box of a task according to the feature map provided by the FPN, and output the 2D box, 3D box and corresponding confidence of the object of the task. The following describes a schematic diagram of the structure of the header. Fig.12a , Fig.12a This is a header representation, such as Fig.12a As shown in the figure, the Header includes three modules: Region Proposal Network (RPN), ROI-ALIGN and RCNN.

[0198] Among them, the RPN module can be used to predict the region where the task object is located on one or more feature maps provided by the FPN, and output candidate 2D boxes that match the region; or it can be understood in this way that the RPN predicts the regions where the task object may exist on one or more horizontal maps output by the FPN, and gives the boxes of these regions, and these regions are called candidate regions (Proposals). For example, when the Header is responsible for detecting vehicles, its RPN layer predicts candidate boxes where vehicles may exist; when the Header is responsible for detecting people, its RPN layer predicts candidate boxes where people may exist. Of course, these Proposals are inaccurate. On the one hand, they may not contain the object of this task, and on the other hand, these boxes are not tight.

[0199] The 2D candidate region prediction process can be implemented by the RPN module of the Header. Based on the feature maps provided by the FPN, it predicts the regions where the task object may exist and gives the candidate boxes (also called candidate regions, Proposals) of these regions. In this embodiment, if the Header is responsible for detecting vehicles, its RPN layer predicts candidate boxes where vehicles may exist.

[0200] The basic structure of the RPN layer can be as Figure 12b shown. Through the convolution module 1 (for example, a 3*3 convolution) on the feature maps provided by the FPN, the feature map RPNHidden is generated. The subsequent RPN layer of the Header will predict Proposals from RPN Hidden. Specifically, the RPN layer of the Header predicts the coordinates and confidence of the Proposals at each position of RPN Hidden through the convolution module 2 and the convolution module 3 (for example, 1*1 convolutions respectively). The higher this confidence, the greater the probability that this Proposal contains the object of this task. For example, the larger the score of a certain Proposal in the Header, the greater the probability that it contains a vehicle. Each Proposal predicted by each RPN layer needs to go through the Proposal merging module to remove redundant Proposals according to the overlapping degree between Proposals (this process can adopt but is not limited to the NMS algorithm), and select the N (N<K) Proposals with the largest scores from the remaining K Proposals as the candidate regions where the object may exist. From Figure 12bIt can be seen that these proposals are inaccurate. On the one hand, they do not necessarily contain the objects of the task, and on the other hand, these boxes are not compact. Therefore, the RPN module is only a rough detection process, which requires the subsequent RCNN module to subdivide. When the RPN module regresses the coordinates of the proposal, it does not directly regress the absolute value of the coordinates, but regresses the coordinates relative to the anchor. The higher the match between these anchors and the actual objects, the greater the probability that the RPN can detect the object.

[0201] The ROI-ALIGN module is used to extract the features of the region where the candidate 2D box is located from a feature map provided by the FPN according to the region predicted by the RPN module; that is, the ROI-ALIGN module mainly extracts the features of the region where each Proposal is located on a certain feature map based on the Proposal provided by the RPN module, and resizes it to a fixed size to obtain the features of each Proposal. It is understandable that the ROI-ALIGN module can use but is not limited to feature extraction methods such as ROI-POOLING (region of interest pooling) / ROI-ALIGN (region of interest extraction) / PS-ROIPOOLING (position-sensitive region of interest pooling) / PS-ROIALIGN (position-sensitive region of interest extraction).

[0202] The RCNN module is used to perform convolution processing on the features of the region where the candidate 2D frame is located through a neural network to obtain the confidence that the candidate 2D frame belongs to each object category; the coordinates of the 2D frame of the candidate region are adjusted through a neural network so that the adjusted 2D candidate frame is more closely matched to the shape of the actual object than the candidate 2D frame, and the adjusted 2D candidate frame with a confidence greater than a preset threshold is selected as the 2D frame of the region. In other words, the RCNN module mainly refines the features of each Proposal proposed by the ROI-ALIGN module to obtain the confidence that each Proposal belongs to each category (for example, for the task of the car, four scores of Background / Car / Truck / Bus will be given), and at the same time adjusts the coordinates of the Proposal's 2D frame to output a more compact 2D frame. These 2D frames are merged through nonmaximum suppression (NMS) and output as the final 2D frame.

[0203] 2D candidate region subdivision is mainly composed of Fig.12aThe RCNN module of the Header in the implementation further regresses the more compact 2D box coordinates based on the features of each Proposal extracted by the ROI-ALIGN module, and classifies the Proposal and outputs the confidence that it belongs to each category. There are many possible implementation forms of RCNN, one of which is Figure 12b As shown. The feature size output by the ROI-ALIGN module can be N*14*14*256 (Feature of proposals). In the RCNN module, it is first processed by the convolution module 4 (Res18-Conv5) of Resnet18, and the output feature size is N*7*7*512. Then it is processed by a Global Avg Pool (average pooling layer), averaging the 7*7 features in each channel of the input features to obtain N*512 features, where each 1*512-dimensional feature vector represents the feature of each Proposal. Next, two fully connected layers FC are used to regress the precise coordinates of the box (output N*4 vectors, these 4 numerical values ​​represent the x / y coordinates of the center point of the box, the width and height of the box), and the confidence of the box category (in Header0, the score of this box being Background / Car / Truck / Bus needs to be given). Finally, through the box merging operation, several boxes with the largest scores are selected, and duplicate boxes are removed through the NMS operation to obtain a compact box output.

[0204] In some practical application scenarios, the perception network may also include other Headers, which can further perform 3D / Mask / Keypoint detection based on the detection of 2D boxes. For example, taking 3D as an example, the ROI-ALIGN module extracts the features of the area where each 2D box is located on the feature map output by FPN based on the accurate 2D box provided by the Header. Assuming that the number of 2D boxes is M, the feature size output by the ROI-ALIGN module is M*14*14*256, which is first processed by the convolution module 5 of Resnet18 (for example, Res18-Conv5), and the output feature size is N*7*7*512, and then processed by a GlobalAvg Pool (average pooling layer), averaging the 7*7 features of each channel in the input features to obtain M*512 features, where each 1*512-dimensional feature vector represents the feature of each 2D box. Next, three fully connected layers FC are used to regress the orientation angle (orientation, M*1 vector), centroid coordinates (centroid, M*2 vector, these two values ​​represent the x / y coordinates of the centroid) and length, width and height (dimention) of the object in the frame.

[0205] In an embodiment of the present application, the header includes at least one convolution module, each convolution module includes at least one convolution layer, each convolution module can perform convolution processing on the input feature map, and the third convolution layer included in the header in the embodiment of the present application is one of the multiple convolution layers included in the header.

[0206] It should be noted that Fig.12a and Figure 12b The header shown in is only one implementation method and does not constitute a limitation of the present application.

[0207] Exemplarily, the feature extraction network can be implemented by a CNN module, which includes a feature extractor (encoder) and multiple prediction decoders (head). The encoder performs convolution processing on the input image (RGB image) to extract high-level semantic features of the image. The architecture can adopt any standard image classification model and gradually reduce the feature resolution through Pooling or stride convolution to finally obtain the high-dimensional semantic features of the image. Fig.13 As shown, this embodiment uses CenterNet as the encoder to downsample the image spatial resolution to 1 / 16 of the original size, and the output feature dimension is 64. The model parameters are initialized with the values ​​after training in the ImageNet image classification dataset.

[0208] In this embodiment, for each attribute to be predicted, an independent head network is constructed, which inputs the features obtained by the encoder, undergoes two convolutions and upsampling, and supplements the detailed information in the features with the features from the low level of the encoder to obtain high-resolution predictions. A BN layer and a nonlinear RELU layer are inserted between the two convolutions. Fig.13 As shown in head1 and head2, the intermediate structure of each head is exactly the same. Only in the final stage, different dimensions are output according to the predicted attributes. The spatial resolution of the output stage is 1 / 4 of the original image size.

[0209] It should be noted that the encoder and head structures here, as well as the hyperparameters involved, including the number of convolutional layers, activation layer types, intermediate feature dimensions, resolution, etc., are not limited, and any common architecture for dense prediction tasks can be used.

[0210] In an embodiment of the present application, a machine learning model can detect objects in an image and then predict the target attributes of the object, wherein the main purpose of target attribute prediction is to analyze the visual information within each pixel of the image and its neighborhood, and determine whether there is a target of interest around the pixel, as well as target-related information, wherein the prediction content may include but is not limited to heatmap, 2D Box offset, orientation, physical size, key point offset, and uncertainty of certain attributes.

[0211] Next, we will explain them separately:

[0212] In a possible implementation, the output dimension of the heatmap is C, each dimension corresponds to a target category, and expresses the probability that each pixel in the image feature map belongs to a target category, with a value range of 0 to 1. When the heatmap response value of a pixel is high in the cth dimension, it means that there is a participant (target object) near the pixel with a high probability.

[0213] In one possible implementation, each pixel position in the feature map predicts a 2D bounding box of the target. This bounding box gives the rectangular area occupied by the target in the image. This 2D box actually gives the 2D detection result of the target.

[0214] In one possible implementation, for each image processed by CNN, the predicted heatmap is first decoded to determine whether there is a target in the image. In this embodiment, the center point of the target's 3D bounding box can be used to express each target. The closer a pixel is to the target's centroid, the larger the heatmap value is, reaching a maximum value of 1 at the centroid position. In this embodiment, there are multiple categories of targets (such as pedestrians, vehicles, etc.). For the Heatmap of each pixel k in the input image I, C(k) can be calculated using the following formula:

[0215]

[0216] where f k (W,I) is the output of the CNN model at the kth pixel when the parameter is W and the input is I. The formula normalizes any model output into a probability value ranging from 0 to 1. Fig.14 As shown in the Heatmap diagram in the figure, the lighter the color in the figure, the larger the Heat value.

[0217] In one possible implementation, each pixel position in the feature map predicts the 3D size of an object, which corresponds to the length, width and height of the object in the real physical world.

[0218] In a possible implementation, each pixel position of the feature map predicts the orientation angle of a target relative to the camera's line of sight. Since this angle is related to the appearance of the target in the image, it can be directly predicted by the CNN model based on its appearance, which can be called the local angle. After obtaining this angle, combined with the target's line of sight angle relative to the camera, the orientation angle of the target in the camera coordinate system can be restored, which can be called the global angle. The relationship between the global angle and the local angle can be obtained by Fig.15 illustrate.

[0219] In a possible implementation, this embodiment obtains the network's orientation, size, and depth estimation offset of the target from each centroid position, and then decodes the global orientation angle, physical length, width, and height, and centroid depth of each target in the camera coordinate system according to the following formula: For a certain category c, this embodiment first counts the average length, width, and height of the target in the data Then get the size offset (δ h ,δ w ,δ l ), the predicted physical size of each target can be obtained by the following formula:

[0220]

[0221] In this embodiment, CNN is used to predict the center point depth offset z of the target. o , which is restored to the absolute depth z of the target center of mass by the following formula r :

[0222]

[0223] In one possible implementation, similar to a 2D bounding box, each pixel position in the feature map predicts the location of the target keypoint in the image.

[0224] For example, this example defines 11 key points {k i , i=1,…11}, which are the 8 corner points of each target’s 3D bounding box, the center point (i.e., centroid point) of each target’s 3D bounding box, and the projection points of the center point of the 3D bounding box on the upper and lower surfaces of the bounding box. The above key points are as follows: Fig.13 As shown in the yellow, red and blue dots in the middle. Similar to the 2D box decoding process, this embodiment obtains the position of the key point of each target in the image by obtaining the network's estimate of the corresponding key point offset at the centroid position.

[0225]

[0226] In one possible implementation, each pixel position in the feature map predicts the depth of the target center of mass, which is a physical quantity in 3D space and refers to the z value in the center (x, y, z) of the target.

[0227] In addition, in a possible implementation, in addition to predicting the above-mentioned 3D attribute values, each pixel position of the feature map is also capable of giving the uncertainty of these attribute predictions. The prediction of uncertainty uses the principle based on the Bayesian network, which will be obtained during the network training process. When the prediction of the above-mentioned attributes is modeled to conform to the Gaussian distribution, this uncertainty can be represented by the variance of the Gaussian distribution.

[0228] As described above, through the machine learning model, the pixel at the target's centroid can be used to directly predict the depth of the target. At this time, the first depth value of the object directly estimated based on the network can be obtained.

[0229] 902. Predict at least one second depth value of the object according to the target information.

[0230] In one possible implementation, the reasoning accuracy of the machine learning model is not accurate enough. Once the camera that captures the image changes or the shooting scene changes, the accuracy tends to drop significantly. In the embodiment of the present application, the relevant attribute information of the object output by the machine learning model (that is, the target information in the embodiment of the present application) is used to estimate the depth of the center point of the target based on different principles, so that at least one different and diversified depth prediction can be obtained, and then a more accurate depth value of the object can be determined based on multiple depth values ​​of the object obtained by multiple different methods.

[0231] Next, we will introduce how to predict at least one second depth value of the object based on the target information:

[0232] In a possible implementation, at least one second depth value of the object may be predicted by an N-point perspective algorithm PnP.

[0233] Specifically, the key point information in the target information may include the 2D positions of multiple key points of the object on the image, the geometric features of the object include the physical size of the object, and the position features of the object include the orientation angle of the object. Then, at least one second depth value of the object may be predicted based on the target information by using an N-point perspective algorithm PnP.

[0234] In one possible implementation, for each key point, a 3D point P in the camera coordinate system can be constructed. c With the 2D point P in image space 2d The correspondence between the two points can be expressed by the camera's intrinsic parameter matrix K:

[0235] λP 2d =KP c

[0236] Where P c and the position P of the key point in the local coordinate system of the target o The relationship can be obtained through the external parameters of the camera:

[0237] P c =RP o +T

[0238] Therefore, each key point P detected on the image 2d It can form a position T = [x, y, z] about the target center point in the camera coordinate system T The equation is:

[0239]

[0240]

[0241] A=[x o sinθ-z o cosθ]

[0242]

[0243] The above formula defines two linear equations about T. When the number of key points on the target exceeds 2, T can be solved. This method is the so-called N-point perspective algorithm PnP.

[0244] In the embodiment of the present application, the depth of the target center point is of interest, so the 3D center point of the target can also be defined as a key point. When its 2D position (u c ,v c ), the center position of the object can be expressed in a form that is only related to z:

[0245]

[0246] Based on this, each key point can be expressed as two linear equations about the z value, so two sets of estimates of the z value of the target center point can be calculated.

[0247] The target 3D information of this embodiment includes the physical length, width, and height of the target. The global heading angle r of the target y , and the positions of the center and corner points of the target's 3D bounding box on the image (9 key points). For each corner key point, the coordinates P in the target's local coordinate systemo It can be obtained by its length, width and height (for example ), and its coordinates in the image are obtained by the key point decoding module (u,v). At the same time, the coordinates of the image centroid key point are (u c ,v c ). According to the PnP-based depth solution principle, two sets of estimates of z values ​​can be obtained:

[0248]

[0249] For example, the 8 corner points of the 3D box in this embodiment can thus form 16 sets of depth estimates.

[0250] In a possible implementation, depth prediction can also be performed based on the length of the object in a preset direction. Specifically, the geometric features of the object include multiple pixel sizes of the 3D envelope of the object in the preset direction and the physical size of the object in the preset direction; further, at least one second depth value of the object can be predicted based on the geometric relationship between the multiple pixel sizes and the physical size.

[0251] Taking the preset direction as the vertical direction as an example, the length of the preset direction is the height. The network can predict the physical height H of the target and the projection points of the corner points and center points of the target 3D bounding box on the upper and lower surfaces of the bounding box (i.e., the pre-defined key points). Since the physical height of the target is related to the distance between these key points, the depth estimation of the key points can be obtained by the following formula:

[0252]

[0253] Where f is the intrinsic parameter of the camera, which can be obtained in advance, H is the physical height of the target estimated by the network, and h is the vertical distance between a pair of key points on the upper and lower surfaces of the target 3D bounding box.

[0254] Exemplarily, the 10 key points in this embodiment can be calculated to obtain 3 sets of pixel heights of the target, wherein the vertical distance difference of the target centroid projection point can form a set of pixel heights, such as Fig.16 As shown in the Zc line of (a); the vertical distances of the eight key points at the corner positions of the target 3D bounding box can form four groups of pixel heights, such as Fig.16 (b) as shown in Z1 and Z3, Fig.16 (c) is shown by the Z2 and Z4 lines. In this embodiment, because the centroid depth of the target is to be calculated, the module further divides the 4 groups of heights into two groups, where the pixel heights at the corner positions are averaged to obtain the pixel height at the centroid position. For each group of pixel heights, a target depth estimate can be solved, and this method can form 3 groups of depth estimates.

[0255] It should be understood that the above method of calculating at least one second depth value is only for illustration, and the depth value of the object can also be predicted by performing calculations on the target information, which is not limited in the present application.

[0256] 903. Fuse the first depth value and the at least one second depth value to obtain a target depth of the object.

[0257] In a possible implementation, after obtaining the first depth value and the at least one second depth value, the first depth value and the at least one second depth value may be fused to obtain the target depth of the object.

[0258] In a possible implementation, the fusion method can be weighted summation. Specifically, the second weight of the first depth value and each of the second depth values ​​can be obtained; and according to the second weight, the first depth value and the at least one second depth value are weighted summed.

[0259] In one possible implementation, the second weight of the first depth value and each of the second depth values ​​can be determined based on the confidence (or uncertainty) of the first depth value and each of the second depth values, wherein the confidence (or uncertainty) can describe the reliability of the depth value, and the higher the confidence of the depth value, the greater the corresponding weight, that is, the value of the second weight is positively correlated with the confidence of the corresponding depth value.

[0260] The confidence of the first depth value may be the output of the machine learning model, and the confidence of the second depth value obtained by different methods may be updated as a trainable parameter during model training, and when the model converges, the confidence of each updated second depth value may be obtained. In a possible implementation, the second weight of each second depth value may also be pre-set, which is not limited here.

[0261] Exemplarily, multiple groups of depth estimation values ​​and their uncertainties obtained by the depth solution system describe the reliability of each estimation value, which is used to filter abnormal estimation and fuse the remaining estimation values ​​in the embodiment of the present application. The fusion of the remaining estimation values ​​can be performed in a weighted sum manner. When there are N groups of estimations and variances in the estimation set, the weighted parameter of each group of estimations is:

[0262]

[0263] The essence of the above formula is the weighted summation of multiple random Gaussian variables. According to probability theory, the result of the summation is still a Gaussian random variable, whose mean and variance satisfy:

[0264]

[0265] In a possible implementation, a first confidence of the first depth value and a second confidence of each second depth value may be predicted and the first confidence and the at least one second confidence may be fused to obtain a target confidence of the target depth.

[0266] In a possible implementation, the fusion method can be weighted summation. Specifically, the first confidence and the first weight of each second confidence can be obtained; according to the first weight, the first confidence and the at least one second confidence are weighted summed. The method for determining the first weight can refer to the method for determining the second weight in the above embodiment, which will not be repeated here.

[0267] In one possible implementation, for each predicted object, the confidence of the prediction, that is, the reliability of the prediction, is calculated at the same time. The calculation formula for the overall confidence is as follows:

[0268] P m =P 3d|2d ·P 2d ;

[0269] This formula expresses two things at the same time: one is the probability of whether the target exists in the image, and the other is whether the shape and position of the predicted target in 3D space are accurate, that is, the geometric confidence. Since the uncertainty about the target position (target confidence) has been obtained, the uncertainty can be used to calculate the target geometric confidence, that is, when the uncertainty of the target estimate is higher, the confidence is lower.

[0270] In a possible implementation, the confidence of the 3D envelope box of the object output by the machine learning model can be obtained; the confidence of the 3D envelope box of the object and the target confidence are fused to obtain the geometric confidence of the object.

[0271] In a possible implementation, the confidence of the 3D envelope box of the object and a third weight of the target confidence may be obtained; and the confidence of the 3D envelope box of the object and the target confidence may be weighted summed according to the third weight.

[0272] Specifically, for each predicted target, the system will also calculate the confidence of the prediction, that is, the reliability of the prediction. The calculation formula for the overall confidence is as follows:

[0273] P m =P 3d|2d ·P 2d ;

[0274] This formula expresses two things at the same time: one is the probability of whether the target exists in the image, and the other is whether the shape and position of the predicted target in 3D space are accurate, that is, the geometric confidence. Since the uncertainty about the target position has been obtained, the uncertainty can be used to calculate the target geometric confidence, that is, when the uncertainty of the target estimate is higher, the confidence is lower:

[0275] d=1-min{σ 2 ,1};

[0276] where the overall geometric confidence is represented by the centroid depth confidence d c and the confidence d of the 3D bounding box shape b It consists of two parts:

[0277] P 3d|2d =ω c d c +ω b d b ;

[0278] Weight ω c With ω d The determination of can refer to the above description about the second weight, and the similarities are not repeated here.

[0279] In a possible implementation, the confidence of the 2D envelope box of the object output by the machine learning model can be obtained; and the detection confidence of the object can be obtained according to the confidence of the 2D envelope box of the object and the geometric confidence.

[0280] For example, refer to Fig.17a , Fig.17a It is a schematic diagram of an overall process of an embodiment of the present application;

[0281] Next, a training process diagram of the machine learning model in the embodiment of the present application is introduced:

[0282] The neural network used in this embodiment needs to be trained. During the training process, this embodiment uses the position, orientation, and size information of the target of interest in the camera coordinate system that is manually annotated in advance to supervise the network's 2D bounding box, center point heatmap, and 3D attribute output. The form of its objective function can use the relevant methods of the existing technology. The unique part of this embodiment is that while training 20 groups of depth estimates, its uncertainty is also trained. The supervision cost function for each group of depth and uncertainty is defined by the following formula:

[0283]

[0284] In addition, the target fusion depth and the target physical size Direction information Combined, the bounding box of the target in 3D space is generated, and the corner points of the predicted bounding box and the corner points of the true bounding box are used to calculate the cost function, as shown in the following formula: b :

[0285]

[0286] In order to demonstrate the technical effect of this embodiment, the experimental results of this embodiment on the industry's public authoritative data set Kitti are summarized as follows. As shown in Table 1, the estimation results generated by using network-based direct estimation, height estimation, PnP-based estimation, or a combination of two methods alone have a large accuracy gap with the diversified depth estimation system of the present invention. This result shows that the diversified depth estimation solution module of Example 1 can maximize the exploration of depth clues from multiple tasks and produce more accurate depth estimates.

[0287] Table 1 Verification experimental results of the diversified depth solution system of the embodiment on the Kitti verification set

[0288]

[0289] As shown in Table 2, for multiple groups of depth estimation, the accuracy produced by hard selection, average fusion, and direct weighted fusion cannot reach the accuracy of the depth selection fusion module in this embodiment. The experimental results show that the uncertainty-based abnormal estimation filtering and fusion strategy can better fuse the results of multiple groups of estimations and achieve performance improvement.

[0290] Table 2 Verification experimental results of the deep selection fusion module of the embodiment on the Kitti verification set

[0291]

[0292] As shown in Table 3, different 3D geometric confidence modeling methods have an impact on the final accuracy. When the 3D geometric confidence calculation module in this embodiment is not used, but only one of the uncertainty modeling confidences of the 20 sets of depth estimates is used instead, or only σ is used d and σ b The experimental results show that our uncertainty-based geometric confidence estimation can effectively model the position accuracy of the target in 3D space, improve the 3D detection accuracy, and is significantly better than 3D IOU estimation.

[0293] Table 3 Validation experiment results of the confidence calculation module of the embodiment on the Kitti validation set

[0294]

[0295] Table 4 is a comparison of this embodiment with the current industry best method on the Kitti test set. This embodiment is significantly better than the current industry best level in Kitti's main evaluation category car (17.14 vs 14.17 accuracy improvement of ~20%). This embodiment also ranks first in the cyclist category and second in the pedestrain category.

[0296] Table 4 Performance comparison of the embodiment with other solutions in the car category of the Kitti official test set

[0297]

[0298] Table 5 Performance comparison of the embodiment with other solutions in the cyclist and pedestrain categories of the Kitti official test set

[0299]

[0300] The embodiment of the present application provides an object detection method, including: acquiring a target image; detecting an object in the target image through a machine learning model, outputting a first depth value of the object and target information, wherein the target information includes at least one of the following information: geometric features of the object, key point information of the object, and position features of the object; predicting at least one second depth value of the object according to the target information; fusing the first depth value and the at least one second depth value to obtain the target depth of the object. The present application simultaneously generates multiple sets of depth estimates and the uncertainty of each depth estimate by using different theoretical assumptions and target attribute combinations, and selects and fuses highly reliable depth predictions and discards unreliable erroneous predictions according to the size of the uncertainty in the subsequent process, thereby ultimately generating a robust depth estimation result, which greatly improves the detection robustness and accuracy of the system for different environments. In addition, the uncertainty also helps the system to generate a more reliable evaluation score for the prediction results, further improving the detection accuracy and system interpretability.

[0301] Reference Fig.17b , Fig.17b FIG. 17 is a schematic diagram of the structure of an object detection device provided in an embodiment of the present application. As shown in FIG. 17 , an object detection device 1700 provided in an embodiment of the present application includes:

[0302] An acquisition module 1701 is used to acquire a target image;

[0303] The object detection module 1702 is used to detect the object in the target image through a machine learning model, and output a first depth value and target information of the object, where the target information includes at least one of the following information: geometric features of the object, key point information of the object, and position features of the object;

[0304] The description of the acquisition module 1701 and the object detection module 1702 may refer to the description of step 901 in the above embodiment, and will not be repeated here.

[0305] A depth value prediction module 1703, configured to predict at least one second depth value of the object according to the target information;

[0306] The description of the depth value prediction module 1703 may refer to the description of step 902 in the above embodiment, which will not be repeated here.

[0307] The fusion module 1704 is configured to fuse the first depth value and the at least one second depth value to obtain a target depth of the object.

[0308] The description of the fusion module 1704 may refer to the description of step 903 in the above embodiment, which will not be repeated here.

[0309] In a possible implementation, the device further includes:

[0310] A confidence prediction module, configured to output a first confidence of the first depth value and predict a second confidence of each of the second depth values ​​after detecting the object in the target image by the machine learning model;

[0311] The fusion module is further used to fuse the first confidence and the at least one second confidence to obtain the target confidence of the target depth.

[0312] In a possible implementation, the key point information includes 2D positions of multiple key points of the object on the image;

[0313] The depth value prediction module is specifically used for:

[0314] At least one second depth value of the object is predicted according to the target information through an N-point perspective algorithm PnP.

[0315] In a possible implementation, the geometric feature of the object includes the physical size of the object, and the position feature of the object includes the orientation angle of the object.

[0316] In a possible implementation, the geometric features of the object include a plurality of pixel sizes of a 3D envelope box of the object in a preset direction, and a physical size of the object in the preset direction;

[0317] The depth value prediction module is specifically used for:

[0318] At least one second depth value of the object is predicted according to the geometric relationship between the multiple pixel sizes and the physical sizes.

[0319] In a possible implementation, the fusion module is specifically used to:

[0320] Obtaining the first confidence and a first weight of each of the second confidences;

[0321] A weighted sum is performed on the first confidence level and the at least one second confidence level according to the first weight.

[0322] In a possible implementation, the fusion module is specifically used to:

[0323] Obtaining the first depth value and a second weight of each of the second depth values;

[0324] A weighted sum is performed on the first depth value and the at least one second depth value according to the second weight.

[0325] In a possible implementation, the acquisition module is further used to:

[0326] Obtaining the confidence of the 3D envelope box of the object output by the machine learning model;

[0327] The fusion module is further used for:

[0328] The confidence of the 3D envelope box of the object and the target confidence are fused to obtain the geometric confidence of the object.

[0329] In a possible implementation, the fusion module is specifically used to:

[0330] Acquire the confidence of the 3D envelope box of the object and a third weight of the target confidence;

[0331] According to the third weight, a weighted sum is performed on the confidence of the 3D envelope box of the object and the target confidence.

[0332] In a possible implementation, the acquisition module is further used to:

[0333] Obtaining the confidence of the 2D envelope box of the object output by the machine learning model;

[0334] The confidence determination module is used to obtain the detection confidence of the object according to the confidence of the 2D envelope box of the object and the geometric confidence.

[0335] Next, an execution device provided by an embodiment of the present application is introduced. Fig.18 , Fig.18 This is a schematic diagram of the structure of the execution device provided in the embodiment of the present application. The execution device 1800 can be specifically manifested as a virtual reality VR device, a mobile phone, a tablet, a laptop, a smart wearable device, a monitoring data processing device or a server, etc., which is not limited here. Specifically, the execution device 1800 includes: a receiver 1801, a transmitter 1802, a processor 1803 and a memory 1804 (wherein the number of processors 1803 in the execution device 1800 can be one or more, Fig.18 In the example of FIG. 1803 , the processor 1803 may include an application processor 18031 and a communication processor 18032. In some embodiments of the present application, the receiver 1801, the transmitter 1802, the processor 1803 and the memory 1804 may be connected via a bus or other means.

[0336] The memory 1804 may include a read-only memory and a random access memory, and provides instructions and data to the processor 1803. A portion of the memory 1804 may also include a non-volatile random access memory (NVRAM). The memory 1804 stores processor and operation instructions, executable modules or data structures, or subsets thereof, or extended sets thereof, wherein the operation instructions may include various operation instructions for implementing various operations.

[0337] The processor 1803 controls the operation of the execution device. In a specific application, the various components of the execution device are coupled together through a bus system, wherein the bus system includes not only a data bus but also a power bus, a control bus, and a status signal bus, etc. However, for the sake of clarity, various buses are referred to as bus systems in the figure.

[0338] The method disclosed in the above embodiment of the present application can be applied to the processor 1803, or implemented by the processor 1803. The processor 1803 can be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit or software instructions in the processor 1803. The above processor 1803 can be a general processor, a digital signal processor (digital signal processing, DSP), a microprocessor or a microcontroller, and can further include an application specific integrated circuit (application specific integrated circuit, ASIC), a field programmable gate array (field-programmable gate array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The processor 1803 can implement or execute the various methods, steps and logic block diagrams disclosed in the embodiments of the present application. The general processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the embodiment of the present application can be directly embodied as a hardware decoding processor to execute, or the hardware and software modules in the decoding processor can be combined and executed. The software module may be located in a storage medium mature in the art, such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 1804, and the processor 1803 reads the information in the memory 1804 and completes the steps of the above method in combination with its hardware.

[0339] The receiver 1801 can be used to receive input digital or character information and generate signal input related to the relevant settings and function control of the execution device. The transmitter 1802 can be used to output digital or character information through the first interface; the transmitter 1802 can also be used to send instructions to the disk group through the first interface to modify the data in the disk group; the transmitter 1802 can also include a display device such as a display screen.

[0340] The present application also provides a training device. Fig.19 , Fig.19It is a structural diagram of a training device provided in an embodiment of the present application. Specifically, the training device 1900 is implemented by one or more servers. The training device 1900 may have relatively large differences due to different configurations or performances, and may include one or more central processing units (CPU) 1919 (for example, one or more processors) and memory 1932, and one or more storage media 1930 (for example, one or more mass storage devices) storing application programs 1942 or data 1944. Among them, the memory 1932 and the storage medium 1930 can be short-term storage or permanent storage. The program stored in the storage medium 1930 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations in the training device. Furthermore, the central processing unit 1919 can be configured to communicate with the storage medium 1930 to execute a series of instruction operations in the storage medium 1930 on the training device 1900.

[0341] The training device 1900 may also include one or more power supplies 1926, one or more wired or wireless network interfaces 1950, one or more input and output interfaces 1958; or, one or more operating systems 1941, such as Windows ServerTM, Mac OS XTM, UnixTM, LinuxTM, FreeBSDTM, etc.

[0342] In the embodiment of the present application, the central processing unit 1919 is used to execute actions related to model training in the above embodiment.

[0343] Also provided in an embodiment of the present application is a computer program product which, when executed on a computer, enables the computer to execute the steps executed by the aforementioned execution device, or enables the computer to execute the steps executed by the aforementioned training device.

[0344] A computer-readable storage medium is also provided in an embodiment of the present application, which stores a program for signal processing. When the computer-readable storage medium is run on a computer, it enables the computer to execute the steps executed by the aforementioned execution device, or enables the computer to execute the steps executed by the aforementioned training device.

[0345] The execution device, training device or terminal device provided in the embodiments of the present application may specifically be a chip, and the chip includes: a processing unit and a communication unit, wherein the processing unit may be, for example, a processor, and the communication unit may be, for example, an input / output interface, a pin or a circuit, etc. The processing unit may execute the computer execution instructions stored in the storage unit so that the chip in the execution device executes the data processing method described in the above embodiment, or so that the chip in the training device executes the data processing method described in the above embodiment. Optionally, the storage unit is a storage unit in the chip, such as a register, a cache, etc. The storage unit may also be a storage unit located outside the chip in the wireless access device, such as a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM), etc.

[0346] For details, please refer to Fig. 20 , Fig. 20 A schematic diagram of the structure of a chip provided in an embodiment of the present application, the chip can be expressed as a neural network processor NPU 2000, NPU 2000 is mounted on the host CPU (Host CPU) as a coprocessor, and the host CPU assigns tasks. The core part of the NPU is the operation circuit 2003, which is controlled by the controller 2004 to extract matrix data from the memory and perform multiplication operations.

[0347] In some implementations, the operation circuit 2003 includes multiple processing units (Process Engine, PE) inside. In some implementations, the operation circuit 2003 is a two-dimensional systolic array. The operation circuit 2003 can also be a one-dimensional systolic array or other electronic circuits capable of performing mathematical operations such as multiplication and addition. In some implementations, the operation circuit 2003 is a general-purpose matrix processor.

[0348] For example, suppose there is an input matrix A, a weight matrix B, and an output matrix C. The operation circuit takes the corresponding data of matrix B from the weight memory 2002 and caches it on each PE in the operation circuit. The operation circuit takes the matrix A data from the input memory 2001 and performs matrix operation with matrix B. The partial result or final result of the matrix is ​​stored in the accumulator 2008.

[0349] The unified memory 2006 is used to store input data and output data. The weight data is directly transferred to the weight memory 2002 through the direct memory access controller (DMAC) 2005. The input data is also transferred to the unified memory 2006 through the DMAC.

[0350] BIU stands for Bus Interface Unit 2010 , which is used for the interaction between the AXI bus and the DMAC and the instruction fetch buffer (IFB) 2009 .

[0351] The bus interface unit 2010 (Bus Interface Unit, BIU for short) is used for the instruction fetch memory 2009 to obtain instructions from the external memory, and is also used for the storage unit access controller 2005 to obtain the original data of the input matrix A or the weight matrix B from the external memory.

[0352] DMAC is mainly used to transfer input data in the external memory DDR to the unified memory 2006 or to transfer weight data to the weight memory 2002 or to transfer input data to the input memory 2001.

[0353] The vector calculation unit 2007 includes multiple operation processing units, and further processes the output of the operation circuit 2003 when necessary, such as vector multiplication, vector addition, exponential operation, logarithmic operation, size comparison, etc. It is mainly used for non-convolutional / fully connected layer network calculations in neural networks, such as Batch Normalization, pixel-level summation, upsampling of feature planes, etc.

[0354] In some implementations, the vector calculation unit 2007 can store the processed output vector to the unified memory 2006. For example, the vector calculation unit 2007 can apply a linear function; or a nonlinear function to the output of the operation circuit 2003, such as linear interpolation of the feature plane extracted by the convolution layer, and then for example, a vector of accumulated values ​​to generate an activation value. In some implementations, the vector calculation unit 2007 generates a normalized value, a pixel-level summed value, or both. In some implementations, the processed output vector can be used as an activation input to the operation circuit 2003, for example, for use in a subsequent layer in a neural network.

[0355] An instruction fetch buffer 2009 connected to the controller 2004, for storing instructions used by the controller 2004;

[0356] The unified memory 2006, the input memory 2001, the weight memory 2002 and the instruction fetch memory 2009 are all on-chip memories. The external memory is private to the NPU hardware architecture.

[0357] The processor mentioned in any of the above places may be a general-purpose central processing unit, a microprocessor, an ASIC, or one or more integrated circuits for controlling the execution of the above program.

[0358] It should also be noted that the device embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed over multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. In addition, in the drawings of the device embodiments provided by the present application, the connection relationship between the modules indicates that there is a communication connection between them, which may be specifically implemented as one or more communication buses or signal lines.

[0359] Through the description of the above implementation mode, the technicians in the field can clearly understand that the present application can be implemented by means of software plus necessary general hardware, and of course, it can also be implemented by special hardware including special integrated circuits, special CPUs, special memories, special components, etc. In general, all functions completed by computer programs can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be various, such as analog circuits, digital circuits or special circuits. However, for the present application, software program implementation is a better implementation mode in more cases. Based on such an understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a readable storage medium, such as a computer floppy disk, a U disk, a mobile hard disk, a ROM, a RAM, a disk or an optical disk, etc., including a number of instructions to enable a computer device (which can be a personal computer, a training device, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0360] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product.

[0361] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website site, a computer, a training device, or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, training device, or data center. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device, a data center, etc. that includes one or more available media integrations. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)), etc.

Claims

1. An object detection method, characterized in that: include: Acquire a target image, and detect an object in the target image through a machine learning model, and output a first depth value and target information of the object, wherein the target information includes at least one of the following information: geometric features of the object, key point information of the object, and position features of the object, wherein the machine learning model includes a convolutional neural network; Predicting at least one second depth value of the object based on the target information; fusing the first depth value and the at least one second depth value to obtain a target depth of the object; The fusing the first depth value and the at least one second depth value comprises: Obtaining the first depth value and a second weight of each of the second depth values, wherein a value of the second weight is positively correlated with a confidence level of the corresponding depth value; A weighted sum is performed on the first depth value and the at least one second depth value according to the second weight.

2. The method according to claim 1, characterized in that After detecting the object in the target image by the machine learning model, the method further includes: Outputting a first confidence of the first depth value, and predicting a second confidence of each of the second depth values; The first confidence and the at least one second confidence are fused to obtain a target confidence of the target depth.

3. The method according to claim 1 or 2, characterized in that: The key point information includes 2D positions of multiple key points of the object on the image; The predicting at least one second depth value of the object according to the target information includes: At least one second depth value of the object is predicted according to the target information through an N-point perspective algorithm PnP.

4. The method according to claim 3, characterized in that The geometric features of the object include the physical size of the object, and the positional features of the object include the orientation angle of the object.

5. The method according to claim 1 or 2, characterized in that: The geometric features of the object include a plurality of pixel sizes of a 3D envelope box of the object in a preset direction, and a physical size of the object in the preset direction; The predicting at least one second depth value of the object according to the target information includes: At least one second depth value of the object is predicted according to the geometric relationship between the multiple pixel sizes and the physical sizes.

6. The method according to claim 2, characterized in that The fusing the first confidence level and the at least one second confidence level comprises: Obtaining the first confidence and a first weight of each of the second confidences; A weighted sum is performed on the first confidence level and the at least one second confidence level according to the first weight.

7. The method according to claim 2, characterized in that The method further comprises: Obtaining the confidence of the 3D envelope box of the object output by the machine learning model; The confidence of the 3D envelope box of the object and the target confidence are fused to obtain the geometric confidence of the object.

8. The method according to claim 7, characterized in that The fusing the confidence of the 3D envelope box of the object and the target confidence includes: Acquire the confidence of the 3D envelope box of the object and a third weight of the target confidence; According to the third weight, a weighted sum is performed on the confidence of the 3D envelope box of the object and the target confidence.

9. The method according to claim 7 or 8, characterized in that: The method further comprises: Obtaining the confidence of the 2D envelope box of the object output by the machine learning model; The detection confidence of the object is obtained according to the confidence of the 2D envelope box of the object and the geometric confidence.

10. An object detection device, characterized in that: include: An acquisition module, used for acquiring a target image; An object detection module is used to detect an object in the target image through a machine learning model, and output a first depth value and target information of the object, wherein the target information includes at least one of the following information: geometric features of the object, key point information of the object, and position features of the object, and the machine learning model includes a convolutional neural network; a depth value prediction module, configured to predict at least one second depth value of the object according to the target information; a fusion module, configured to fuse the first depth value and the at least one second depth value to obtain a target depth of the object; Wherein, the fusion module is specifically used for: Obtaining the first depth value and a second weight of each of the second depth values, wherein a value of the second weight is positively correlated with a confidence level of the corresponding depth value; A weighted sum is performed on the first depth value and the at least one second depth value according to the second weight.

11. The device according to claim 10, characterized in that The device also includes: A confidence prediction module, configured to output a first confidence of the first depth value and predict a second confidence of each of the second depth values ​​after detecting the object in the target image by the machine learning model; The fusion module is further used to fuse the first confidence and the at least one second confidence to obtain the target confidence of the target depth.

12. The device according to claim 10 or 11, characterized in that The key point information includes 2D positions of multiple key points of the object on the image; The depth value prediction module is specifically used for: At least one second depth value of the object is predicted according to the target information through an N-point perspective algorithm PnP.

13. The device according to claim 12, characterized in that The geometric features of the object include the physical size of the object, and the positional features of the object include the orientation angle of the object.

14. The device according to claim 10 or 11, characterized in that The geometric features of the object include a plurality of pixel sizes of a 3D envelope box of the object in a preset direction, and a physical size of the object in the preset direction; The depth value prediction module is specifically used for: At least one second depth value of the object is predicted according to the geometric relationship between the multiple pixel sizes and the physical sizes.

15. The device according to claim 11, characterized in that The fusion module is specifically used for: Obtaining the first confidence and a first weight of each of the second confidences; A weighted sum is performed on the first confidence level and the at least one second confidence level according to the first weight.

16. The device according to claim 11, characterized in that The acquisition module is further used for: Obtaining the confidence of the 3D envelope box of the object output by the machine learning model; The fusion module is further used for: The confidence of the 3D envelope box of the object and the target confidence are fused to obtain the geometric confidence of the object.

17. The device according to claim 16, characterized in that The fusion module is specifically used for: Acquire the confidence of the 3D envelope box of the object and a third weight of the target confidence; According to the third weight, a weighted sum is performed on the confidence of the 3D envelope box of the object and the target confidence.

18. The device according to claim 16 or 17, characterized in that The acquisition module is further used for: Obtaining the confidence of the 2D envelope box of the object output by the machine learning model; The confidence determination module is used to obtain the detection confidence of the object according to the confidence of the 2D envelope box of the object and the geometric confidence.

19. A computer storage medium, characterized in that: The computer storage medium stores one or more instructions which, when executed by one or more computers, cause the one or more computers to perform the operations of the method of any one of claims 1 to 9.

20. A computer program product, characterized in that The method comprises computer-readable instructions, and when the computer-readable instructions are executed on a computer device, the computer device is caused to execute the method according to any one of claims 1 to 9.

21. An electronic device, comprising at least one processor and at least one memory; the processor and the memory are connected via a communication bus and communicate with each other; The at least one memory is used to store code; The at least one processor is configured to execute the code to perform the method according to any one of claims 1 to 9.

Images