A perception network and data processing method

Abstract

This application discloses a perceptual network applicable to the field of artificial intelligence, comprising: a feature extraction network, wherein a first block in the feature extraction network performs convolution processing on the input data to obtain M target feature maps; at least one second block in the feature extraction network performs convolution processing on M1 target feature maps out of the M target feature maps to obtain M1 first feature maps; a target operation in the feature extraction network processes M2 target feature maps out of the M target feature maps to obtain M2 second feature maps; and a concatenation operation in the feature extraction network concatenates the M1 first feature maps and the M2 second feature maps to obtain a concatenated feature map. This application utilizes cross-layer target operations within the same stage to enable the perceptual network to generate features highly similar to key features, reducing the number of model parameters and thereby improving the running speed of the model on GPU devices, TPU devices, and NPU devices.

Description

[0001] This application is a divisional application of Chinese Patent Application No. 202110221934.8, filed with the Chinese Patent Office on February 27, 2021, entitled "A Sensing Network and Data Processing Method", the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of artificial intelligence, and in particular to a perceptual network and a data processing method. Background Technology

[0003] Artificial intelligence (AI) is the theory, methods, technology, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a branch of computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to possess the functions of perception, reasoning, and decision-making.

[0004] Computer vision is an integral part of various intelligent / autonomous systems across diverse application fields, such as manufacturing, inspection, document analysis, medical diagnosis, and the military. It's the science of how to use cameras / video cameras and computers to acquire the data and information we need about the objects being photographed. Figuratively speaking, it's about equipping computers with eyes (cameras / video cameras) and a brain (algorithms) to replace human eyes in identifying, tracking, and measuring targets, thus enabling computers to perceive their environment. Because perception can be seen as extracting information from sensory signals, computer vision can also be viewed as the science of how to enable artificial systems to "perceive" from images or multidimensional data. In short, computer vision uses various imaging systems to replace visual organs in acquiring input information, and then the computer replaces the brain in processing and interpreting this input information. The ultimate research goal of computer vision is to enable computers to observe and understand the world through vision like humans, possessing the ability to autonomously adapt to their environment.

[0005] Convolutional neural network (CNN)-based inference models are widely used in various computer vision-based terminal tasks such as image recognition, object detection, and image segmentation. Traditional basic neural networks often suffer from large parameter counts and computational demands, preventing them from operating in real-time for these tasks. Existing lightweight inference networks (such as MobileNet, EfficientNet, and ShuffleNet) are designed for mobile devices like CPUs and ARM (Advanced RISC) machines. Their performance on high-throughput processing units like GPUs, TPUs, and NPUs is unsatisfactory, with inference speeds sometimes even slower than traditional CNNs. Summary of the Invention

[0006] In a first aspect, this application provides a perceptual network, the perceptual network comprising: a feature extraction network, the feature extraction network comprising a first block, at least one second block connected in series, a target operation and a splicing operation, wherein the first block and the M second blocks are blocks within the same stage in the feature extraction network, and the number of parameters of the target operation is less than the number of parameters of the M second blocks;

[0007] The target operation, also known as the cheap operation, refers to a series of operations with fewer parameters, which is used to distinguish them from traditional convolution operations. The number of parameters describes the number of parameters contained in a neural network and is used to evaluate the size of the model.

[0008] The concat operation refers to merging feature maps without altering their data. For example, feature maps... Figure 1 The result of concatenating the feature map with feature map 2 is (feature map 2) Figure 1 Feature diagram 2), where features Figure 1 The order between feature map 1 and feature map 2 is not restricted. More specifically, concatenating a feature map with three semantic channels with a feature map with five semantic channels results in a feature map with eight semantic channels.

[0009] The first block is used to perform convolution processing on the input data to obtain M target feature maps, each target feature map corresponding to one channel;

[0010] The at least one second block is used to perform convolution processing on M1 target feature maps out of the M target feature maps to obtain M1 first feature maps, where M1 is less than M;

[0011] The target operation is used to process M2 target feature maps out of the M target feature maps to obtain M2 second feature maps, where M2 is less than M;

[0012] The splicing operation is used to splice the M1 first feature maps and the M2 second feature maps to obtain the spliced ​​feature map.

[0013] Within the same stage of a feature extraction network, the output features of different blocks exhibit high similarity, while the output features of blocks in different stages show low similarity. Therefore, this embodiment utilizes cross-layer target operations within the same stage to enable the perceptual network to generate features highly similar to key features, reducing the number of model parameters and thereby improving the model's running speed on GPU, TPU, and NPU devices.

[0014] This application provides a perceptual network comprising: a feature extraction network, the feature extraction network including a first block, at least one second block connected in series, a target operation, and a concatenation operation; the first block and the M second blocks being blocks within the same stage of the feature extraction network, and the number of parameters of the target operation being less than the number of parameters of the M second blocks; the first block being used to perform convolution processing on input data to obtain M target feature maps, each target feature map corresponding to one channel; the at least one second block being used to perform convolution processing on M1 target feature maps among the M target feature maps to obtain M1 first feature maps, where M1 is less than M; the target operation being used to process M2 target feature maps among the M target feature maps to obtain M2 second feature maps, where M2 is less than M; and the concatenation operation being used to concatenate the M1 first feature maps and the M2 second feature maps to obtain a concatenated feature map. By utilizing the above method, cross-layer target operations between the same stages are used to enable the perceptual network to generate features that are highly similar to key features, thereby reducing the number of model parameters and improving the running speed of the model on GPU devices, TPU devices, and NPU devices.

[0015] In one possible implementation, the intersection of the M1 target feature maps and the M2 target feature maps is empty, and the sum of the M1 and M2 is the M, and the number of channels of the concatenated feature map is the M.

[0016] Essentially, the output feature map and the number of channels of at least one second block are a portion of the number of channels of the target feature map output from the first block, with the remaining channels of the target feature map being processed by the target operation. Furthermore, since the number of parameters in the target operation is less than the number of parameters in at least one second block, the overall number of parameters in the perceptual network is reduced, thereby improving the running speed of the perceptual network on GPU, TPU, and NPU devices.

[0017] In one possible implementation, the target operation is a convolution operation with fewer parameters than the at least one second block; or,

[0018] The target operation is a residual connection operation between the output of the first block and the output of the splicing operation.

[0019] For example, if the target operation is a 1*1 convolution, or if the target operation is a residual connection operation between the output of the first block and the output of the concatenation operation, then M2 second feature maps are equivalent to M1 first feature maps, that is, M1 first feature maps are directly used as M2 second feature maps.

[0020] It should be understood that the output feature map of the first block can also be split into multiple sets of feature maps, and multiple target operations can process a set of feature maps. As long as the number of channels of the feature map output by each target operation and the sum of the feature maps output by at least one second block are the same as the number of channels of the feature map output by the first block, the number of channels of the feature maps output by different target operations can be different.

[0021] In one possible implementation, the at least one second block is used to perform convolution processing on M1 target feature maps among the M target feature maps to obtain feature maps output by each second block, wherein the output of the second block that is farthest from the first block among the at least one second block is the M1 first feature maps;

[0022] The feature extraction network also includes:

[0023] A fusion operation is used to fuse the feature maps output by each second block to obtain a fused feature map, the size of which is the same as the size of the M2 second feature maps;

[0024] An addition operation is performed on the fused feature map and the M2 second feature maps to obtain the processed M2 second feature maps;

[0025] The splicing operation is used to splice the M1 first feature maps and the processed M2 second feature maps to obtain a spliced ​​feature map.

[0026] In this process, the feature maps output from each second block can be concatenated to obtain a concatenated feature map (the number of channels is the sum of the feature maps output from each second block). Since the number of channels in the concatenated feature map is greater than the number of channels in the feature map output from the target operation (M2), in order to perform addition operations between the concatenated feature map and the output of the target operation (M2 second feature maps), it is necessary to perform a dimensionality reduction operation on the concatenated feature map so that the number of channels in the concatenated feature map is equal to M2. Then, matrix addition operations can be performed between the concatenated feature map and the output of the target operation.

[0027] In one possible implementation, the fusion operation is used to concatenate and reduce the dimensions of the output of each second block to obtain the fused feature map with the same size as the M2 second feature maps.

[0028] In one possible implementation, the first block and the M second blocks are blocks within the target stage of the feature extraction network, and the concatenated feature map is used as the output feature map of the target stage in the feature extraction network; or...

[0029] The target stage further includes at least one third block, which is used to perform a convolution operation on the concatenated feature map to obtain the output feature map of the target stage.

[0030] In this embodiment of the application, the stitched feature map can be used as the output feature map of the target stage, or the stitched feature map can be processed by other blocks (third blocks) included in the target stage, and the stitched feature map can be processed by at least one third block to obtain the output feature map of the target stage.

[0031] In one possible implementation, the first block can be the first block in the feature extraction network or a block in an intermediate layer. At least one third block can be connected before the first block, in which case the first block is used to perform convolution processing on the data output by at least one third block.

[0032] In one possible implementation, the feature extraction network is used to acquire an input image, extract features from the input image, and output a feature map of the input image;

[0033] The sensing network also includes:

[0034] The task network is used to process the corresponding task based on the feature map of the input image to obtain the processing result.

[0035] In one possible implementation, the task includes object detection, image segmentation, or image classification.

[0036] Secondly, this application provides a data processing method, the method comprising:

[0037] A feature extraction network is obtained, which includes a first block, at least one second block connected in series, a target operation, and a concatenation operation. The first block and the M second blocks are blocks within the same stage in the feature extraction network, and the number of parameters of the target operation is less than the number of parameters of the M second blocks.

[0038] The input data is convolved through the first block to obtain M target feature maps, each of which corresponds to one channel;

[0039] Through the at least one second block, convolution processing is performed on M1 target feature maps out of the M target feature maps to obtain M1 first feature maps, where M1 is less than M;

[0040] Through the target operation, M2 target feature maps out of the M target feature maps are processed to obtain M2 second feature maps, where M2 is less than M;

[0041] The splicing operation is used to splice the M1 first feature maps and the M2 second feature maps to obtain the spliced ​​feature map.

[0042] In one possible implementation, the intersection of the M1 target feature maps and the M2 target feature maps is empty, and the sum of the M1 and M2 is the M, and the number of channels of the concatenated feature map is the M.

[0043] In one possible implementation, the target operation is a convolution operation with fewer parameters than the at least one second block; or,

[0044] The target operation is a residual connection operation between the output of the first block and the output of the splicing operation.

[0045] In one possible implementation, the at least one second block is used to perform convolution processing on M1 target feature maps among the M target feature maps to obtain feature maps output by each second block, wherein the output of the second block that is farthest from the first block among the at least one second block is the M1 first feature maps;

[0046] The method further includes:

[0047] The feature maps output from each second block are fused through a fusion operation to obtain a fused feature map. The size of the fused feature map is the same as the size of the M2 second feature maps. An addition operation is performed on the fused feature map and the M2 second feature maps to obtain processed M2 second feature maps.

[0048] The process of stitching together the M1 first feature maps and the M2 second feature maps through the stitching operation includes:

[0049] The M1 first feature maps and the processed M2 second feature maps are concatenated to obtain the concatenated feature map.

[0050] In one possible implementation, the fusion operation, which fuses the feature maps output by each second block, includes:

[0051] By performing a fusion operation, the outputs of each second block are concatenated and dimensionality reduced to obtain a fused feature map with the same size as the M2 second feature maps.

[0052] In one possible implementation, the first block and the M second blocks are blocks within the target stage of the feature extraction network, and the concatenated feature map is used as the output feature map of the target stage in the feature extraction network; or...

[0053] The target stage further includes at least one third block, which is used to perform a convolution operation on the concatenated feature map to obtain the output feature map of the target stage.

[0054] In one possible implementation, the feature extraction network is used to acquire an input image, extract features from the input image, and output a feature map of the input image;

[0055] The method further includes:

[0056] The task network is used to process the corresponding task based on the feature map of the input image to obtain the processing result.

[0057] In one possible implementation, the task includes object detection, image segmentation, or image classification.

[0058] Thirdly, this application provides a data processing apparatus, the apparatus comprising:

[0059] An acquisition module is used to acquire a feature extraction network, wherein the feature extraction network includes a first block, at least one second block connected in series, a target operation, and a concatenation operation. The first block and the M second blocks are blocks within the same stage in the feature extraction network, and the number of parameters of the target operation is less than the number of parameters of the M second blocks.

[0060] The convolution processing module performs convolution processing on the input data through the first block to obtain M target feature maps, each target feature map corresponding to one channel;

[0061] Through the at least one second block, convolution processing is performed on M1 target feature maps out of the M target feature maps to obtain M1 first feature maps, where M1 is less than M;

[0062] Through the target operation, M2 target feature maps out of the M target feature maps are processed to obtain M2 second feature maps, where M2 is less than M;

[0063] The splicing module is used to splice the M1 first feature maps and the M2 second feature maps through the splicing operation to obtain the spliced ​​feature map.

[0064] In one possible implementation, the intersection of the M1 target feature maps and the M2 target feature maps is empty, and the sum of the M1 and M2 is the M, and the number of channels of the concatenated feature map is the M.

[0065] In one possible implementation, the target operation is a convolution operation with fewer parameters than the at least one second block; or,

[0066] The target operation is a residual connection operation between the output of the first block and the output of the splicing operation.

[0067] In one possible implementation, the at least one second block is used to perform convolution processing on M1 target feature maps among the M target feature maps to obtain feature maps output by each second block, wherein the output of the second block that is farthest from the first block among the at least one second block is the M1 first feature maps;

[0068] The device further includes:

[0069] The fusion module is used to fuse the feature maps output by each second block through a fusion operation to obtain a fused feature map, the size of which is the same as the size of the M2 second feature maps; and to perform an addition operation on the fused feature map and the M2 second feature maps to obtain processed M2 second feature maps.

[0070] The splicing module is used to splice the M1 first feature maps and the processed M2 second feature maps to obtain a spliced ​​feature map.

[0071] In one possible implementation, the fusion module is used to perform a fusion operation on the output of each second block, concatenating and reducing the dimensionality to obtain a fused feature map with the same size as the M2 second feature maps.

[0072] In one possible implementation, the first block and the M second blocks are blocks within the target stage of the feature extraction network, and the concatenated feature map is used as the output feature map of the target stage in the feature extraction network; or...

[0073] The target stage further includes at least one third block, which is used to perform a convolution operation on the concatenated feature map to obtain the output feature map of the target stage.

[0074] In one possible implementation, the feature extraction network is used to acquire an input image, extract features from the input image, and output a feature map of the input image;

[0075] The device further includes:

[0076] The task processing module is used to process the corresponding task based on the feature map of the input image through a task network to obtain the processing result.

[0077] In one possible implementation, the task includes object detection, image segmentation, or image classification.

[0078] Fourthly, embodiments of this application provide a data processing apparatus, which may include a memory, a processor, and a bus system, wherein the memory is used to store a program, and the processor is used to execute the program in the memory to run the first aspect described above and any optional sensing network thereof.

[0079] Fifthly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when run on a computer, causes the computer to perform the first aspect described above and any of its optional sensing networks.

[0080] In a sixth aspect, embodiments of this application provide a computer program, including code, which, when executed, is used to run the first aspect described above and any optional perceptual network thereof.

[0081] In a seventh aspect, embodiments of this application provide a data processing apparatus, which may include a memory, a processor, and a bus system, wherein the memory is used to store a program, and the processor is used to execute the program in the memory to perform the methods described in the first aspect and any of its optional methods.

[0082] Eighthly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when run on a computer, causes the computer to perform the methods described in the first aspect and any of its alternatives.

[0083] Ninthly, embodiments of this application provide a computer program, including code, which, when executed, is used to implement the first aspect described above and any of its optional methods.

[0084] In a tenth aspect, this application provides a chip system including a processor for supporting an execution device or training device in implementing the functions involved in the foregoing aspects, such as transmitting or processing data involved in the foregoing methods; or, information. In one possible design, the chip system further includes a memory for storing program instructions and data necessary for the execution device or training device. This chip system may be composed of chips or may include chips and other discrete devices.

[0085] This application provides a perceptual network comprising: a feature extraction network, the feature extraction network including a first block, at least one second block connected in series, a target operation, and a concatenation operation; the first block and the M second blocks being blocks within the same stage of the feature extraction network, and the number of parameters of the target operation being less than the number of parameters of the M second blocks; the first block being used to perform convolution processing on input data to obtain M target feature maps, each target feature map corresponding to one channel; the at least one second block being used to perform convolution processing on M1 target feature maps among the M target feature maps to obtain M1 first feature maps, where M1 is less than M; the target operation being used to process M2 target feature maps among the M target feature maps to obtain M2 second feature maps, where M2 is less than M; and the concatenation operation being used to concatenate the M1 first feature maps and the M2 second feature maps to obtain a concatenated feature map. By utilizing the above method, cross-layer target operations between the same stages are used to enable the perceptual network to generate features that are highly similar to key features, thereby reducing the number of model parameters and improving the running speed of the model on GPU devices, TPU devices, and NPU devices. Attached Figure Description

[0086] Figure 1 A structural diagram illustrating the main framework of artificial intelligence;

[0087] Figure 2a A schematic diagram illustrating an application scenario provided in an embodiment of this application;

[0088] Figure 2b A schematic diagram illustrating an application scenario provided in an embodiment of this application;

[0089] Figure 2c A schematic diagram of a convolutional neural network provided in an embodiment of this application;

[0090] Figure 2d A schematic diagram of a convolutional neural network provided in an embodiment of this application;

[0091] Figure 3 A schematic diagram of a system architecture provided for an embodiment of this application;

[0092] Figure 4 This is an illustration of an embodiment of a data processing method provided in this application.

[0093] Figure 5a A schematic diagram of a sensing network provided in an embodiment of this application;

[0094] Figure 5bA schematic diagram of a sensing network provided in an embodiment of this application;

[0095] Figure 5c A schematic diagram of a sensing network provided in an embodiment of this application;

[0096] Figure 5d A schematic diagram of a sensing network provided in an embodiment of this application;

[0097] Figures 6 to 14 A schematic diagram of a sensing network provided in an embodiment of this application;

[0098] Figure 15 A schematic diagram of a data processing apparatus provided in an embodiment of this application;

[0099] Figure 16 A schematic diagram of the structure of the execution device provided in the embodiments of this application;

[0100] Figure 17 A schematic diagram of the structure of the training device provided in the embodiments of this application;

[0101] Figure 18 This is a schematic diagram of a chip structure provided in an embodiment of this application. Detailed Implementation

[0102] The embodiments of the present invention will now be described with reference to the accompanying drawings. The terminology used in the embodiments section is for illustrative purposes only and is not intended to limit the scope of the invention.

[0103] The embodiments of this application will now be described with reference to the accompanying drawings. Those skilled in the art will recognize that, with technological advancements and the emergence of new scenarios, the technical solutions provided in the embodiments of this application are equally applicable to similar technical problems.

[0104] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such terms are interchangeable where appropriate; this is merely a way of distinguishing objects with the same attributes in the embodiments of this application. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion, so that a process, method, system, product, or apparatus that comprises a series of elements is not necessarily limited to those elements, but may include other elements not explicitly listed or inherent to those processes, methods, products, or apparatuses.

[0105] First, the overall workflow of the artificial intelligence system is described; please refer to [link / reference]. Figure 1 , Figure 1The diagram illustrates a structural framework for artificial intelligence (AI). The framework is further elaborated below along two dimensions: the "Intelligent Information Chain" (horizontal axis) and the "IT Value Chain" (vertical axis). The "Intelligent Information Chain" reflects a series of processes from data acquisition to processing. For example, it could be the general process of intelligent information perception, intelligent information representation and formation, intelligent reasoning, intelligent decision-making, and intelligent execution and output. In this process, data undergoes a condensation process of "data—information—knowledge—wisdom." The "IT Value Chain" reflects the value that AI brings to the information technology industry, from the underlying infrastructure of human intelligence and information (provided and processed through technological means) to the industrial ecosystem of the system.

[0106] (1) Infrastructure

[0107] Infrastructure provides computing power to support artificial intelligence systems, enabling communication with the external world and providing support through a basic platform. This communication occurs through sensors; computing power is provided by intelligent chips (hardware acceleration chips such as CPUs, NPUs, GPUs, ASICs, and FPGAs); and the basic platform includes distributed computing frameworks and related platform guarantees and support, which may include cloud storage and computing, interconnected networks, etc. For example, sensors communicate with the outside world to acquire data, and this data is provided to intelligent chips in the distributed computing system provided by the basic platform for computation.

[0108] (2) Data

[0109] The data at the next layer of infrastructure is used to represent the data sources in the field of artificial intelligence. The data involves graphics, images, voice, text, and IoT data from traditional devices, including business data from existing systems and sensor data such as force, displacement, liquid level, temperature, and humidity.

[0110] (3) Data processing

[0111] Data processing typically includes methods such as data training, machine learning, deep learning, search, reasoning, and decision-making.

[0112] Among them, machine learning and deep learning can perform intelligent information modeling, extraction, preprocessing, and training on data, including symbolization and formalization.

[0113] Reasoning refers to the process in which, in a computer or intelligent system, the machine thinks and solves problems by simulating human intelligent reasoning, based on reasoning control strategies and using formalized information. Typical functions include search and matching.

[0114] Decision-making refers to the process of making decisions based on intelligent information after reasoning, and it typically provides functions such as classification, sorting, and prediction.

[0115] (4) General ability

[0116] After the data processing mentioned above, the results of the data processing can be used to form some general capabilities, such as algorithms or a general system, for example, translation, text analysis, computer vision processing, speech recognition, image recognition, etc.

[0117] (5) Smart Products and Industry Applications

[0118] Intelligent products and industry applications refer to products and applications of artificial intelligence systems in various fields. They are the encapsulation of overall artificial intelligence solutions, productizing intelligent information decision-making and realizing practical applications. Their application areas mainly include: intelligent terminals, intelligent transportation, intelligent healthcare, autonomous driving, smart cities, etc.

[0119] This application primarily focuses on fields requiring multiple perception tasks, such as driver assistance, autonomous driving, and mobile terminals. Video is frame-by-frame extracted to obtain a single image, which is then fed into the perception network of this invention to obtain information such as the 2D and 3D dimensions of the object of interest, its mask, and key points. These detection results are output to a post-processing module for further processing. For example, in an autonomous driving system, they are fed into the planning and control unit for decision-making; in a mobile terminal, they are fed into a beautification algorithm to obtain a beautified image. The following sections provide a brief introduction to two application scenarios: ADAS / ADS visual perception systems and mobile beautification.

[0120] Application Scenario 1: ADAS / ADS Visual Perception System

[0121] like Figure 2aAs shown, ADAS and ADS require real-time detection of multiple types of 2D objects, including: dynamic obstacles (pedestrians, cyclists, tricycles, cars, trucks, buses), static obstacles (traffic cones, traffic sticks, fire hydrants, motorcycles, bicycles), and traffic signs (traffic signs, guide signs, billboards, red / yellow / green / black traffic lights, road signs). Furthermore, to accurately determine the area occupied by dynamic obstacles in 3D space, 3D estimation of the dynamic obstacles is also required, outputting 3D bounding boxes. To fuse data with lidar, it is necessary to acquire the mask of dynamic obstacles, thereby filtering out the laser point clouds hitting the dynamic obstacles; for precise parking, it is necessary to simultaneously detect four key points of the parking space; for mapping and localization, it is necessary to detect the key points of static targets. Using the technical solution provided in the embodiments of this application, all or part of the above functions can be completed in the perception network.

[0122] Application Scenario 2: Mobile Phone Beauty Function

[0123] like Figure 2b As shown, in a mobile phone, the human body's mask and key points are detected through the perception network provided in this application embodiment. The corresponding parts of the human body can be enlarged or reduced, such as waist slimming and buttock shaping operations, thereby outputting a beautified image.

[0124] Application Scenario 3: Image Classification Scenario:

[0125] After acquiring an image to be classified, the object recognition device uses the object recognition method of this application to obtain the category of the objects in the image, and then classifies the image according to the category of the objects in the image. For photographers, many photos are taken every day, including animals, people, and plants. The method of this application can quickly classify photos according to their content, such as photos containing animals, photos containing people, and photos containing plants.

[0126] When dealing with a large number of images, manual classification is inefficient, and people are prone to fatigue when processing the same task for a long time, which can lead to significant errors in the classification results. However, the method described in this application can quickly classify images without errors.

[0127] Application Scenario 4: Product Classification

[0128] After acquiring an image of a product, the object recognition device uses the object recognition method of this application to obtain the product category in the image, and then classifies the product according to the product category.

[0129] Since the embodiments of this application involve a large number of neural network applications, for ease of understanding, the relevant terms and concepts such as neural networks involved in the embodiments of this application will be introduced below.

[0130] (1) Object recognition: using image processing and machine learning, computer graphics and other related methods to determine the category of objects in an image.

[0131] (2) Neural Network

[0132] A neural network can be composed of neural units, which can be operational units that take xs and an intercept of 1 as inputs, and whose output can be:

[0133]

[0134] Where s = 1, 2, ..., n, where n is a natural number greater than 1, Ws is the weight of xs, and b is the bias of the neural unit. f is the activation function of the neural unit, used to introduce nonlinear characteristics into the neural network to convert the input signal in the neural unit into the output signal. The output signal of this activation function can be used as the input of the next convolutional layer. The activation function can be the sigmoid function. A neural network is a network formed by connecting many of the above-mentioned individual neural units together, that is, the output of one neural unit can be the input of another neural unit. The input of each neural unit can be connected to the local receptive field of the previous layer to extract the features of the local receptive field, which can be a region composed of several neural units.

[0135] (3) Deep Neural Networks

[0136] A Deep Neural Network (DNN) can be understood as a neural network with many hidden layers. There's no specific metric for "many" layers; the terms "multi-layer neural network" and "deep neural network" are essentially the same thing. DNNs can be categorized into three layers based on their position: input layer, hidden layer, and output layer. Generally, the first layer is the input layer, the last layer is the output layer, and the layers in between are hidden layers. All layers are fully connected, meaning that any neuron in the i-th layer is connected to any neuron in the (i+1)-th layer. Although DNNs seem complex, the operation of each layer is actually quite simple, resembling a linear relationship as follows: in, It is the input vector. It is the output vector. α is the offset vector, W is the weight matrix (also called coefficients), and α() is the activation function. Each layer is simply an adjustment of the input vector. The output vector is obtained through such a simple operation. Because DNNs have many layers, the coefficients W and the offset vector... The number is quite large. So, how are the specific parameters defined in a DNN? First, let's look at the definition of the coefficient W. Taking a three-layer DNN as an example, the linear coefficient from the 4th neuron in the second layer to the 2nd neuron in the third layer is defined as... The superscript 3 represents the layer number where coefficient W resides, while the subscript corresponds to the output third layer index 2 and the input second layer index 4. In summary, the coefficients from the k-th neuron in layer L-1 to the j-th neuron in layer L are defined as... Note that the input layer does not have a W parameter. In deep neural networks, more hidden layers allow the network to better represent complex real-world situations. Theoretically, the more parameters a model has, the higher its complexity and "capacity," meaning it can accomplish more complex learning tasks.

[0137] (4) A Convolutional Neural Network (CNN) is a deep neural network with a convolutional structure. A CNN contains a feature extractor consisting of convolutional layers and subsampling layers. This feature extractor can be viewed as a filter, and the convolution process can be seen as using a trainable filter to convolve with an input image or a convolutional feature map. A convolutional layer refers to the neuron layer in a CNN that performs convolution processing on the input signal. In a convolutional layer of a CNN, a neuron can be connected to only some of the neurons in its neighboring layers. A convolutional layer typically contains several feature planes, each of which can be composed of rectangularly arranged neural units. Neural units on the same feature plane share weights, which are the convolutional kernels. Shared weights can be understood as the way image information is extracted regardless of location. The underlying principle is that the statistical information of one part of the image is the same as that of other parts. This means that image information learned in one part can also be used in another part. Therefore, for all locations on the image, we can use the same learned image information. In the same convolutional layer, multiple convolutional kernels can be used to extract different image information. Generally, the more convolutional kernels there are, the richer the image information reflected by the convolution operation.

[0138] Convolutional kernels can be initialized as matrices of random size, and during the training of a convolutional neural network, they can learn appropriate weights. Furthermore, sharing weights directly reduces the number of connections between layers in the convolutional neural network, while also lowering the risk of overfitting.

[0139] Since CNN is a very common type of neural network, the following will combine... Figure 2c This section focuses on a detailed explanation of the structure of CNNs. As mentioned in the basic concept introduction above, a Convolutional Neural Network (CNN) is a deep neural network with a convolutional structure. It is a deep learning architecture, which refers to learning at multiple levels of abstraction through machine learning algorithms. As a deep learning architecture, CNN is a feed-forward artificial neural network, where each neuron responds to the input image.

[0140] exist Figure 2c In this example, the Convolutional Neural Network (CNN) 100 may include an input layer 120, convolutional / pooling layers 120 (where pooling layers are optional), and a neural network layer 130. The input layer 120 acquires the image to be processed and then passes the acquired image to the convolutional / pooling layers 120 and the subsequent neural network layer 130 for processing to obtain the image processing result. The following section... Figure 2c This section provides a detailed introduction to the internal layer structure of CNN 100.

[0141] Convolutional / pooling layers 120:

[0142] Convolutional layers:

[0143] like Figure 2c The convolutional / pooling layer 120 shown may include layers as in Examples 121-126. For instance, in one implementation, layer 121 is a convolutional layer, layer 122 is a pooling layer, layer 123 is a convolutional layer, layer 124 is a pooling layer, layer 125 is a convolutional layer, and layer 126 is a pooling layer; in another implementation, layers 121 and 122 are convolutional layers, layer 123 is a pooling layer, layers 124 and 125 are convolutional layers, and layer 126 is a pooling layer. That is, the output of the convolutional layer can be used as the input to a subsequent pooling layer, or as the input to another convolutional layer to continue the convolution operation.

[0144] The following section will use convolutional layer 121 as an example to introduce the internal working principle of a convolutional layer.

[0145] Convolutional layer 121 can include multiple convolution operators, also known as kernels. In image processing, a convolution operator acts as a filter, extracting specific information from the input image matrix. Essentially, a convolution operator can be a weight matrix, which is usually predefined. During convolution, the weight matrix processes the input image pixel by pixel (or two pixels by two pixels, depending on the stride) along the horizontal direction, extracting specific features. The size of the weight matrix should be related to the image size. It's important to note that the depth dimension of the weight matrix is ​​the same as the depth dimension of the input image; during convolution, the weight matrix extends to the entire depth of the input image. Therefore, convolution with a single weight matrix produces a single-depth convolutional output. However, in most cases, a single weight matrix is ​​not used; instead, multiple weight matrices of the same size (rows × columns) are applied—multiple identical matrices. The outputs of each weight matrix are stacked to form the depth dimension of the convolutional image; this dimension can be understood as being determined by the "multiple" mentioned above. Different weight matrices can be used to extract different features from an image. For example, one weight matrix can be used to extract edge information, another to extract specific colors, and yet another to blur unwanted noise. These multiple weight matrices have the same size (rows × columns), and the resulting convolutional feature maps are also of the same size. These extracted convolutional feature maps are then merged to form the output of the convolution operation.

[0146] The weight values ​​in these weight matrices need to be obtained through extensive training in practical applications. The weight matrices formed by the weight values ​​obtained through training can be used to extract information from the input image, thereby enabling the convolutional neural network 100 to make correct predictions.

[0147] When a convolutional neural network 100 has multiple convolutional layers, the initial convolutional layers (e.g., 121) tend to extract more general features, which can also be called low-level features. As the depth of the convolutional neural network 100 increases, the features extracted by later convolutional layers (e.g., 126) become more and more complex, such as high-level semantic features. Features with higher semantic levels are more suitable for the problem to be solved.

[0148] Pooling layer:

[0149] Because it is often necessary to reduce the number of training parameters, pooling layers are often introduced periodically after convolutional layers, such as... Figure 2c Layers 121-126 in example 120 can be a convolutional layer followed by a pooling layer, or multiple convolutional layers followed by one or more pooling layers. In image processing, the sole purpose of pooling layers is to reduce the spatial size of the image. Pooling layers can include average pooling and / or max pooling operators to sample the input image to obtain a smaller image size. Average pooling calculates the average value of pixel values ​​within a specific range as the result of average pooling. Max pooling takes the pixel with the largest value within a specific range as the result of max pooling. Furthermore, just as the size of the weight matrix in a convolutional layer should be related to the image size, the operators in a pooling layer should also be related to the image size. The size of the output image after pooling can be smaller than the size of the input image of the pooling layer. Each pixel in the output image represents the average or maximum value of the corresponding sub-region of the input image of the pooling layer.

[0150] Neural network layer 130:

[0151] After processing by the convolutional / pooling layers 120, the convolutional neural network 100 is still insufficient to output the required information. As mentioned earlier, the convolutional / pooling layers 120 only extract features and reduce the parameters introduced by the input image. However, to generate the final output information (the required class information or other relevant information), the convolutional neural network 100 needs to utilize neural network layers 130 to generate one or more outputs representing the required number of classes. Therefore, neural network layers 130 may include multiple hidden layers (such as...). Figure 2cAs shown in 131, 132 to 13n) and output layer 140, the parameters contained in these multi-layer hidden layers can be pre-trained based on relevant training data for specific task types, such as image recognition, image classification, image super-resolution reconstruction, etc.

[0152] After the multiple hidden layers in neural network layer 130, the final layer of the entire convolutional neural network 100 is the output layer 140. This output layer 140 has a loss function similar to classification cross-entropy, specifically used to calculate the prediction error. Once the entire convolutional neural network 100 has undergone forward propagation (e.g., ...), the loss function is applied. Figure 2c Propagation from 120° to 140° is considered forward propagation, while reverse propagation (such as...) is completed. Figure 2c The propagation from 140 to 120 (backpropagation) will begin to update the weight values ​​and biases of the layers mentioned above, in order to reduce the loss of the convolutional neural network 100 and the error between the output of the convolutional neural network 100 through the output layer and the ideal result.

[0153] It should be noted that, as Figure 2c The convolutional neural network 120 shown is only an example of a convolutional neural network. In specific applications, convolutional neural networks can also exist in the form of other network models.

[0154] The image processing method in this application embodiment specifically employs a neural network structure as follows: Figure 2d As shown. In Figure 2d In this example, the convolutional neural network (CNN) 100 may include an input layer 110, convolutional / pooling layers 120 (where pooling layers are optional), and neural network layers 130. Figure 2c compared to, Figure 2d Multiple convolutional / pooling layers in the convolutional / pooling layer 120 are processed in parallel, and the extracted features are input into the full neural network layer 130 for processing.

[0155] It should be noted that, Figure 2c and Figure 2d The convolutional neural network shown is only an example. In specific applications, convolutional neural networks can also exist in the form of other network models.

[0156] (5) Backpropagation algorithm

[0157] Convolutional neural networks can employ backpropagation (BP) to correct the parameters in the initial super-resolution model during training, thereby reducing the reconstruction error loss. Specifically, forward propagation of the input signal to the output generates an error loss; this error loss information is then propagated back to update the parameters in the initial super-resolution model, leading to convergence of the error loss. The backpropagation algorithm is an error-loss-driven backpropagation process aimed at obtaining the optimal parameters of the super-resolution model, such as the weight matrix.

[0158] (6) Loss Function

[0159] In training a deep neural network, to ensure the output closely approximates the desired predicted value, we compare the network's prediction with the target value. Based on the difference, we update the weight vector of each layer (usually pre-configuring parameters before the initial update). For example, if the prediction is too high, the weight vector is adjusted to predict a lower value. This adjustment continues until the deep neural network predicts the target value or a value very close to it. Therefore, we need to predefine "how to compare the difference between the predicted and target values," which is the loss function or objective function. These are important equations used to measure the difference between the predicted and target values. Taking the loss function as an example, a higher output value (loss) indicates a greater difference, and training the deep neural network becomes a process of minimizing this loss.

[0160] (7) Backpropagation algorithm

[0161] Neural networks can employ backpropagation (BP) to correct the parameters of the initial neural network model during training, thereby reducing the reconstruction error loss. Specifically, forward propagation of the input signal to the output generates error loss; this error loss information is then propagated back to update the parameters of the initial neural network model, leading to convergence of the error loss. The backpropagation algorithm is an error-loss-driven backpropagation process aimed at obtaining the optimal parameters of the neural network model, such as the weight matrix.

[0162] The system architecture provided by the embodiments of this application is described below.

[0163] See Figure 3This application provides a system architecture 100. As shown in the system architecture 100, the data acquisition device 160 is used to acquire training data. In this application embodiment, the training data includes: images or image blocks of objects and the categories of objects; and the training data is stored in the database 130. The training device 120 trains a CNN feature extraction network based on the training data maintained in the database 130 (Explanation: The feature extraction network here is the model trained in the training phase as described above, which can be a neural network for feature extraction, etc.). The following will describe in more detail how the training device 120 obtains a CNN feature extraction network based on the training data using Embodiment 1. This CNN feature extraction network can be used to implement the perception network provided in this application embodiment, that is, the image or image block to be identified is input into the CNN feature extraction network after relevant preprocessing. The CNN feature extraction network in this application embodiment can specifically be a CNN convolutional neural network. It should be noted that in actual applications, the training data maintained in the database 130 may not all come from the acquisition by the data acquisition device 160, but may also be received from other devices. It should also be noted that the training device 120 may not necessarily train the CNN feature extraction network entirely based on the training data maintained by the database 130. It may also obtain training data from the cloud or other places for model training. The above description should not be construed as limiting the embodiments of this application.

[0164] The target model / rules trained using training device 120 can be applied to different systems or devices, such as... Figure 3 The execution device 110 shown can be a terminal, such as a mobile phone terminal, tablet computer, laptop computer, augmented reality (AR) / virtual reality (VR) device, vehicle terminal, etc., or it can be a server or cloud service. Figure 3 In the embodiment of this application, the execution device 110 is configured with an input / output (I / O) interface 112 for data interaction with external devices. Users can input data to the I / O interface 112 through the client device 140. The input data may include images or image blocks.

[0165] During the preprocessing of input data by the execution device 120, or during the calculation module 111 of the execution device 120 performing calculations and other related processing (such as implementing the neural network function in this application), the execution device 120 may call data, code, etc. in the data storage system 150 for corresponding processing, or store the data, instructions, etc. obtained from the corresponding processing into the data storage system 150.

[0166] Finally, the I / O interface 112 returns the processing results, such as the 2D, 3D, mask, key points, and other information of the objects of interest in the image or image block obtained above, to the client device 140, thereby providing them to the user.

[0167] Optionally, the customer device 140 may be a planning control unit in an autonomous driving system or a beautification algorithm module in a mobile terminal.

[0168] It is worth noting that the training device 120 can generate corresponding target models / rules based on different training data for different objectives or tasks. These corresponding target models / rules can be used to achieve the above objectives or complete the above tasks, thereby providing the user with the required results.

[0169] exist Figure 3 In the scenario shown, the user can manually provide input data, which can be done through the interface provided by I / O interface 112. Alternatively, the client device 140 can automatically send input data to I / O interface 112. If user authorization is required for the client device 140 to automatically send input data, the user can set the corresponding permissions in the client device 140. The user can view the output results of the execution device 110 on the client device 140, which can be presented in various forms such as display, sound, or animation. The client device 140 can also act as a data acquisition terminal, collecting the input data and output results of the input I / O interface 112 as new sample data and storing them in the database 130. Alternatively, data can be collected directly from the I / O interface 112 without going through the client device 140, using the input data and output results of the input I / O interface 112 as new sample data and storing them in the database 130.

[0170] It is worth noting that, Figure 3 This is merely a schematic diagram of a system architecture provided in an embodiment of this application. The positional relationships between the devices, components, modules, etc., shown in the diagram do not constitute any limitation. For example, in Figure 3 In this context, the data storage system 150 is an external memory relative to the execution device 110. In other cases, the data storage system 150 may also be placed within the execution device 110.

[0171] First, we will take the model inference stage as an example to introduce the perceptual network provided in the embodiments of this application.

[0172] Reference Figure 6 , Figure 6 This application provides a schematic diagram of the structure of a sensing network, as shown in the embodiments. Figure 6As shown, the perception network provided in this application embodiment includes: a feature extraction network, the feature extraction network including a first block, at least one second block connected in series, a target operation, and a splicing operation, wherein the first block and the M second blocks are blocks within the same stage in the feature extraction network. Figure 6 The target stage is shown in the image.

[0173] Feature extraction networks typically consist of multiple stages, each of which can include multiple blocks. Each block can be composed of at least one convolutional operation. Figure 5a ,in, Figure 5a The diagram illustrates the structure of a perceptual network in the prior art. The feature extraction network comprises four stages (stage1, stage2, stage3, and stage4). Within each stage, the input and output feature maps of each block are of the same size. "Same size" means that the number of channels in the feature map and the size of each channel's feature map are identical. For example, in stage1, the input and output feature maps of each block are 56*56 pixels with 24 channels; in stage2, they are 28*28 pixels with 40 channels; in stage1, they are 14*14 pixels with 80 channels; and in stage2, they are 7*7 pixels with 160 channels. Each block can include at least one convolutional operation. Figure 5a The diagram shows three convolution operations (1x1 convolution, 3x3 convolution, and 1x1 convolution), and there are also residual connection operations that connect the input to the output.

[0174] For details, please refer to Figure 5b ,in, Figure 5b In this context, a convolutional module can be represented as a stage, and convolutional module 1 can correspond to... Figure 5a In the stage 1, convolutional module 2 can correspond to Figure 5a In stage 2, convolutional module 3 can correspond to Figure 5a In stage 3, convolutional module 4 can correspond to Figure 5a In stage 4, feature map C1 is the feature map output from stage 1, feature map C2 is the feature map output from stage 2, feature map C3 is the feature map output from stage 3, and feature map C4 is the feature map output from stage 4. Taking stage 1 as an example, refer to... Figure 5cThe convolutional module 1 can include multiple convolutional layers (or blocks).

[0175] Feature extraction networks can extract features from input images to obtain output feature maps. These output feature maps can be input into task networks, which then perform corresponding task processing to obtain processing results. For example, if the task is object detection, the processing result can be the detection box containing the object in the image; if the task is image segmentation, the processing result can be the segmented region containing the object in the image.

[0176] Reference Figure 5d Within the same stage of a feature extraction network, the output features of different blocks exhibit high similarity, while the output features of blocks in different stages show low similarity. Therefore, this embodiment utilizes cross-layer target operations within the same stage to enable the perceptual network to generate features highly similar to key features, reducing the number of model parameters and thereby improving the model's running speed on GPU, TPU, and NPU devices.

[0177] The perception network in the embodiments of this application will now be described in detail.

[0178] In this embodiment of the application, the feature extraction network includes a first block, at least one second block connected in series, a target operation, and a splicing operation. The first block and the M second blocks are blocks within the same stage in the feature extraction network, and the number of parameters of the target operation is less than the number of parameters of the M second blocks.

[0179] Reference Figure 6 The first block is used to perform convolution processing on the input data to obtain M target feature maps, each of which corresponds to one channel.

[0180] Wherein, the at least one second block is used to perform convolution processing on M1 target feature maps out of the M target feature maps to obtain M1 first feature maps, where M1 is less than M; the target operation is used to process M2 target feature maps out of the M target feature maps to obtain M2 second feature maps, where M2 is less than M; and the concatenation operation is used to concatenate the M1 first feature maps and the M2 second feature maps to obtain a concatenated feature map.

[0181] In one possible implementation, the intersection of the M1 target feature maps and the M2 target feature maps is empty, and the sum of the M1 and M2 is the M, and the number of channels of the concatenated feature map is the M.

[0182] Essentially, the output feature map and the number of channels of at least one second block are a portion of the number of channels of the target feature map output from the first block, with the remaining channels of the target feature map being processed by the target operation. Furthermore, since the number of parameters in the target operation is less than the number of parameters in at least one second block, the overall number of parameters in the perceptual network is reduced, thereby improving the running speed of the perceptual network on GPU, TPU, and NPU devices.

[0183] In one possible implementation, the target operation is a convolution operation with fewer parameters than the at least one second block; or, the target operation is a residual concatenation operation between the output of the first block and the output of the concatenation operation.

[0184] For example, if the target operation is a 1*1 convolution, or if the target operation is a residual connection operation between the output of the first block and the output of the concatenation operation, then M2 second feature maps are equivalent to M1 first feature maps, that is, M1 first feature maps are directly used as M2 second feature maps.

[0185] It should be understood, and referenced Figure 7 Alternatively, the output feature map of the first block can be split into multiple sets of feature maps, and multiple target operations can process a set of feature maps. As long as the number of channels of the feature map output by each target operation and the sum of the feature maps output by at least one second block are the same as the number of channels of the feature map output by the first block, the number of channels of the feature maps output by different target operations can be different.

[0186] In one possible implementation, the first block and the M second blocks are blocks within the target stage of the feature extraction network, and the concatenated feature map is used as the output feature map of the target stage in the feature extraction network; or, the target stage further includes at least one third block, which is used to perform a convolution operation on the concatenated feature map to obtain the output feature map of the target stage.

[0187] In this embodiment, the stitched feature map can be used as the output feature map of the target stage, or the stitched feature map can be processed by other blocks (the third block) included in the target stage, for example, referring to... Figure 8 The concatenated feature map can be processed by at least one third block to obtain the output feature map of the target stage.

[0188] In one possible implementation, the first block could be the first block in the feature extraction network, or a block from an intermediate layer, as shown below. Figure 9 If at least one third block can be connected before the first block, then the first block is used to perform convolution processing on the data output by at least one third block.

[0189] In one possible implementation, the at least one second block is used to perform convolution processing on M1 target feature maps out of the M target feature maps to obtain feature maps output by each second block, wherein the output of the second block farthest from the first block in the at least one second block is the M1 first feature maps; a fusion operation is used to fuse the feature maps output by each second block to obtain a fused feature map, the size of the fused feature map is the same as the size of the M2 second feature maps, an addition operation is performed on the fused feature map and the M2 second feature maps to obtain processed M2 second feature maps, and then a concatenation operation is used to concatenate the M1 first feature maps and the processed M2 second feature maps to obtain a concatenated feature map.

[0190] Reference Figure 10 Each second block can output a feature map. The fusion operation targets the feature map output by each second block and the output of the target operation (M2 second feature maps). The fusion operation is used to concatenate and reduce the dimensionality of the output of each second block to obtain the fused feature map with the same size as the M2 second feature maps.

[0191] For details, please refer to Figure 11 In this process, the feature maps output from each second block can be concatenated to obtain a concatenated feature map (the number of channels is the sum of the feature maps output from each second block). Since the number of channels in the concatenated feature map is greater than the number of channels in the feature map output from the target operation (M2), in order to perform addition operations between the concatenated feature map and the output of the target operation (M2 second feature maps), it is necessary to perform a dimensionality reduction operation on the concatenated feature map so that the number of channels in the concatenated feature map is equal to M2. Then, matrix addition operations can be performed between the concatenated feature map and the output of the target operation.

[0192] Taking a second block with 5 elements as an example, there are 5 outputs from the second block that will be merged into the final output. First, these 5 features are concatenated into Z = [Y1, Y2, Y3, Y4, Y5]. Then, feature Z is dimensionality reduced to match M2, and the elements are added together. The specific formula can be as follows:

[0193] τ(Z)=W2σ(W1Pooling(Z)+b1)+b2,

[0194]

[0195] In this embodiment, the feature extraction network in the perceptual network can be stacked based on the stages provided in the above embodiments. For example, a schematic diagram of the network structure is shown in Table 1. Here, output represents the size of the output feature map, and #out represents the number of channels of the output feature map (the number of channels of Block and Cheap is actually 1 / 2 of #out).

[0196] Table 1

[0197]

[0198] In one possible implementation, the feature extraction network is used to acquire the input image, extract features from the input image, and output a feature map of the input image; the perception network further includes a task network, used to perform corresponding task processing based on the feature map of the input image to obtain a processing result. The task includes image processing tasks such as object detection, image segmentation, or image classification.

[0199] The following describes a schematic diagram of a perceptual network structure:

[0200] Reference Figure 11 The feature extraction network in this embodiment can be Figure 11 The backbone network shown in the diagram has a feature pyramid network (FPN) connected to it. The FPN can perform convolution processing on multiple feature maps of different resolutions generated by the backbone network to construct the feature pyramid.

[0201] Reference Figure 12 , Figure 12This is a schematic diagram of an FPN structure, in which convolutional module 1 is used to process the top feature map C4. Convolutional module 1 may include at least one convolutional layer. For example, convolutional module 1 can use dilated convolution and 1×1 convolution to reduce the number of channels of the top feature map C4 to 256, which is the top feature map P4 of the feature pyramid. The output of the next layer feature map C3 is horizontally linked and the number of channels is reduced to 256 using 1×1 convolution (convolutional module 2). Then, it is added to feature map p4 pixel by pixel to obtain feature map p3. And so on, from top to bottom, to construct the feature pyramid Φp = {feature map p4, feature map p3, feature map p2, feature map p1}.

[0202] In this embodiment, the FPN includes multiple convolutional modules, each convolutional module includes multiple convolutional layers, and each convolutional module can perform convolution processing on the input feature map. In this embodiment, the second convolutional layer included in the FPN is one of the multiple convolutional layers included in the FPN.

[0203] It should be noted that, Figure 12 The FPN shown is only one implementation method and does not constitute a limitation of this application.

[0204] In this embodiment of the application, taking the task to be performed by the task network as object detection as an example, the header is connected to the FPN. The header can detect the 2D bounding boxes of a task based on the feature map provided by the FPN, and output the 2D bounding boxes of the objects in this task and the corresponding confidence scores, etc. The following describes a schematic diagram of the header structure. Figure 13 This is an illustration of a header, such as... Figure 13 As shown, the Header consists of three modules: Region Proposal Network (RPN), ROI-ALIGN, and RCNN.

[0205] The RPN module can be used to predict the region where the task object is located on one or more feature maps provided by the FPN, and output candidate 2D bounding boxes matching the region. Alternatively, it can be understood that the RPN predicts regions where the task object may exist on one or more feature maps output by the FPN, and provides bounding boxes for these regions, which are called candidate regions (Proposals). For example, when the Header is responsible for detecting cars, its RPN layer predicts candidate bounding boxes where cars may exist; when the Header is responsible for detecting people, its RPN layer predicts candidate bounding boxes where people may exist. Of course, these proposals are not accurate; on the one hand, they may not contain the task object, and on the other hand, these boxes are not tightly packed.

[0206] The 2D candidate region prediction process can be implemented by the RPN module in the Header. Based on the feature maps provided by the FPN, it predicts the regions where the task object may exist and gives the candidate bounding boxes (also called candidate regions, Proposals) for these regions. In this embodiment, if the Header is responsible for detecting vehicles, its RPN layer predicts the candidate bounding boxes where vehicles may exist.

[0207] Refer to Figure 14 , a feature map RPNHidden is generated on the feature map provided by the FPN through convolutional module 1 (such as a 3*3 convolution). Subsequently, the RPN layer in the Header will predict Proposals from RPN Hidden. Specifically, the RPN layer in the Header predicts the coordinates and confidence of the Proposals at each position in RPN Hidden through convolutional module 2 and convolutional module 3 (such as 1*1 convolutions respectively). The higher this confidence, the greater the probability that this Proposal contains the object of this task. For example, the larger the score of a certain Proposal in the Header, the greater the probability that it contains a vehicle. The Proposals predicted by each RPN layer need to go through the Proposal merging module to remove redundant Proposals according to the overlap degree between the Proposals (this process can use but is not limited to the NMS algorithm), and select the N (N < k) Proposals with the largest score from the remaining K Proposals as the candidate regions where the object may exist. From Figure 14 It can be seen that these Proposals are inaccurate. On the one hand, they may not contain the object of this task, and on the other hand, these bounding boxes are not tight. Therefore, the RPN module is only a rough detection process and requires the subsequent RCNN module for refinement. When the RPN module regresses the coordinates of the Proposals, it does not directly regress the absolute values of the coordinates, but regresses the coordinates relative to the Anchor. The higher the matching degree of these Anchors with the actual object, the greater the probability that the RPN can detect the object. The N (N < k) Proposals are used as the candidate regions where the object may exist.

[0208] The ROI-ALIGN module is used to extract features of the candidate 2D bounding box region from a feature map provided by the FPN based on the region predicted by the RPN module. In other words, the ROI-ALIGN module mainly extracts the features of the region containing each proposal from a certain feature map based on the proposals provided by the RPN module, and resizes it to a fixed size to obtain the features of each proposal. It is understood that the ROI-ALIGN module can use, but is not limited to, feature extraction methods such as ROI-POOLING (Region of Interest Pooling), ROI-ALIGN (Region of Interest Extraction), PS-ROIPOOLING (Position-Sensitive Region of Interest Pooling), and PS-ROIALIGN (Position-Sensitive Region of Interest Extraction).

[0209] The RCNN module performs convolutional processing on the features of the regions containing the candidate 2D bounding boxes using a neural network to obtain the confidence scores of the candidate 2D bounding boxes belonging to each object category. It then adjusts the coordinates of the 2D bounding boxes in the candidate regions using the neural network, making the adjusted 2D bounding boxes better match the shape of the actual objects than the original candidate 2D bounding boxes. Adjusted 2D bounding boxes with confidence scores greater than a preset threshold are selected as the 2D bounding boxes for the region. In other words, the RCNN module mainly refines the features of each proposal proposed by the ROI-ALIGN module, obtaining the confidence scores of each proposal belonging to each category (e.g., for the car task, it provides four scores: Background / Car / Truck / Bus). Simultaneously, it adjusts the coordinates of the 2D bounding boxes of the proposals, outputting more compact 2D bounding boxes. These 2D bounding boxes are then merged using nonmaximum suppression (NMS) to become the final 2D bounding box output.

[0210] 2D candidate region fine classification is mainly composed of Figure 14 The RCNN module in the Header implementation uses the features extracted from each proposal by the ROI-ALIGN module to further regress a more compact 2D bounding box coordinate, and classifies the proposal, outputting the confidence score of its belonging to each category. RCNN can be implemented in many ways, one of which is as follows: Figure 13As shown. The ROI-ALIGN module outputs features of size N*14*14*256 (Feature of proposals). In the RCNN module, it is first processed by the ResNet18 convolutional module 4 (Res18-Conv5), resulting in an output feature size of N*7*7*512. Then, it is processed by a Global Avg Pool (average pooling layer) to average the 7*7 features in each channel of the input features, resulting in N*512 features, where each 1*512-dimensional feature vector represents the features of each proposal. Next, two fully connected layers (FC) regress the precise coordinates of the bounding box (outputting an N*4 vector, where these four values ​​represent the x / y coordinates of the box's center point, the width and height of the box), and the confidence score of the box's class (in Header0, the score for whether the box is Background / Car / Truck / Bus needs to be given). Finally, a box merging operation is performed to select the boxes with the highest scores, and NMS is used to remove duplicate boxes, resulting in a compact box output.

[0211] In some practical applications, this perceptual network can also include other headers to further perform 3D / mask / keypoint detection based on the detected 2D bounding boxes. For example, taking 3D as an example, the ROI-ALIGN module extracts features of the region containing each 2D bounding box from the feature map output by the FPN based on the accurate 2D bounding boxes provided by the header. Assuming the number of 2D bounding boxes is M, the feature size output by the ROI-ALIGN module is M*14*14*256. This feature is first processed by the ResNet18 convolutional module 5 (e.g., Res18-Conv5), resulting in an output feature size of N*7*7*512. Then, it is processed by a Global Avg Pool (average pooling layer) to average the 7*7 features of each channel in the input features, obtaining M*512 features, where each 1*512-dimensional feature vector represents the feature of each 2D bounding box. Next, three fully connected layers (FC) are used to regress the orientation (M*1 vector), centroid coordinates (M*2 vector, these two values ​​represent the x / y coordinates of the centroid), and length, width, and height of the object in the bounding box.

[0212] In this embodiment of the application, the header includes at least one convolutional module, each convolutional module includes at least one convolutional layer, and each convolutional module can perform convolution processing on the input feature map. In this embodiment of the application, the third convolutional layer included in the header is one of the multiple convolutional layers included in the header.

[0213] It should be noted that, Figure 13 and Figure 14 The header shown is only one implementation method and does not constitute a limitation of this application.

[0214] The following is a description of the effects of the perception network provided in the embodiments of this application:

[0215] The results of this application embodiment on the image classification dataset CIFAR10 are shown in Tables 2 and 3. Compared with other methods such as ResNet, the perceptual network structure provided in this application embodiment achieves high accuracy with minimal computation and parameter quantity. Table 2 shows the results on the image classification dataset CIFAR10, and Table 3 shows the comparison with existing lightweight networks on the image classification dataset CIFAR10.

[0216] Table 2

[0217]

[0218] Table 3

[0219]

[0220] The results of the perceptual network provided in this embodiment on the ImageNet image classification dataset are shown in Tables 4, 5, and 6. Compared with the baseline network ResNet, which has the same inference speed, the perceptual network provided in this embodiment can improve the inference accuracy by 1.4% on this large classification dataset. Compared with other lightweight inference networks, the perceptual network provided in this embodiment can achieve both the fastest inference speed and high inference accuracy. Table 4 shows the comparison with the baseline network ResNet on the ImageNet image classification dataset, Table 5 shows the comparison with another baseline network RegNet on the ImageNet image classification dataset, and Table 6 shows the comparison with other lightweight networks on the ImageNet image classification dataset.

[0221] Table 4

[0222]

[0223] Table 5

[0224]

[0225] Table 6

[0226]

[0227] Taking the task-oriented network for target detection as an example, the experimental results are shown in Table 7. The perceptual network provided in this embodiment achieves the highest mAP while also having the fastest inference speed, reaching 25.9 frames per second.

[0228] Table 7

[0229]

[0230] This application provides a perceptual network comprising: a feature extraction network, the feature extraction network including a first block, at least one second block connected in series, a target operation, and a concatenation operation; the first block and the M second blocks being blocks within the same stage of the feature extraction network, and the number of parameters of the target operation being less than the number of parameters of the M second blocks; the first block being used to perform convolution processing on input data to obtain M target feature maps, each target feature map corresponding to one channel; the at least one second block being used to perform convolution processing on M1 target feature maps among the M target feature maps to obtain M1 first feature maps, where M1 is less than M; the target operation being used to process M2 target feature maps among the M target feature maps to obtain M2 second feature maps, where M2 is less than M; and the concatenation operation being used to concatenate the M1 first feature maps and the M2 second feature maps to obtain a concatenated feature map. By utilizing the above method, cross-layer target operations between the same stages are used to enable the perceptual network to generate features that are highly similar to key features, thereby reducing the number of model parameters and improving the running speed of the model on GPU devices, TPU devices, and NPU devices.

[0231] Next, from the perspective of product application, we will introduce several application scenarios of the embodiments of this application.

[0232] This application primarily applies to computer vision-based fields such as mobile phones, cloud services, detection, segmentation, and low-level vision. Faced with the parallel computation of massive amounts of data, computer vision and related tasks are increasingly demanding higher computing power, placing greater emphasis on hardware processing speed and power consumption. Currently, more and more terminal devices, such as mobile phone chips, are deploying GPUs and NPG processing units extensively. The lightweight network model proposed in this application can significantly improve inference speed on GPUs and can directly replace existing basic feature extraction networks, such as those used in object detectors, as a foundational network, deployed in scenarios such as autonomous driving. In practical applications, it can be adapted to a wide range of application scenarios and devices (such as mobile terminals and cloud servers), utilizing this fast inference network for tasks such as data processing and image retrieval.

[0233] In one scenario, embodiments of this application can be deployed on a mobile terminal to provide users with efficient and accurate inference. Examples include image processing after taking a photo with a mobile phone, image recognition, and mobile target detection.

[0234] In one scenario, this invention can be lightweightly deployed on cloud services, providing users with efficient data processing services and accelerating deep learning. Users can upload data to be processed, and the data can be quickly processed through the inference model on the cloud service.

[0235] In one scenario, during autonomous driving tasks, real-time detection of targets such as pedestrians and vehicles within the field of vision is crucial for the vehicle to make correct driving decisions. This invention can directly replace the feature extraction module of existing target detectors, compressing and accelerating the detector's inference process.

[0236] Reference Figure 4 This application also provides a data processing method, the method comprising:

[0237] 1401. Obtain a feature extraction network, wherein the feature extraction network includes a first block, at least one second block connected in series, a target operation, and a splicing operation, wherein the first block and the M second blocks are blocks within the same stage in the feature extraction network, and the number of parameters of the target operation is less than the number of parameters of the M second blocks.

[0238] The specific description of step 1401 can be found in the description of the feature extraction network in the above embodiments, and will not be repeated here.

[0239] 1402. The input data is convolved through the first block to obtain M target feature maps, each of which corresponds to one channel;

[0240] The specific description of step 1402 can be found in the description of the first block in the above embodiments, and will not be repeated here.

[0241] 1403. Through the at least one second block, convolution processing is performed on M1 target feature maps among the M target feature maps to obtain M1 first feature maps, where M1 is less than M;

[0242] The specific description of step 1403 can be found in the description of at least one second block in the above embodiments, and will not be repeated here.

[0243] 1404. Through the target operation, M2 target feature maps among the M target feature maps are processed to obtain M2 second feature maps, where M2 is less than M;

[0244] The specific description of step 1404 can be found in the description of the target operation in the above embodiments, and will not be repeated here.

[0245] 1405. The M1 first feature maps and the M2 second feature maps are spliced ​​together through the splicing operation to obtain the spliced ​​feature map.

[0246] The specific description of step 1405 can be found in the description of the splicing operation in the above embodiments, and will not be repeated here.

[0247] In one possible implementation, the intersection of the M1 target feature maps and the M2 target feature maps is empty, and the sum of the M1 and M2 is the M, and the number of channels of the concatenated feature map is the M.

[0248] In one possible implementation, the target operation is a convolution operation with fewer parameters than the at least one second block; or,

[0249] The target operation is a residual connection operation between the output of the first block and the output of the splicing operation.

[0250] In one possible implementation, the at least one second block is used to perform convolution processing on M1 target feature maps among the M target feature maps to obtain feature maps output by each second block, wherein the output of the second block that is farthest from the first block among the at least one second block is the M1 first feature maps;

[0251] The method further includes:

[0252] The feature maps output from each second block are fused through a fusion operation to obtain a fused feature map. The size of the fused feature map is the same as the size of the M2 second feature maps. An addition operation is performed on the fused feature map and the M2 second feature maps to obtain processed M2 second feature maps.

[0253] The process of stitching together the M1 first feature maps and the M2 second feature maps through the stitching operation includes:

[0254] The M1 first feature maps and the processed M2 second feature maps are concatenated to obtain the concatenated feature map.

[0255] In one possible implementation, the fusion operation, which fuses the feature maps output by each second block, includes:

[0256] By performing a fusion operation, the outputs of each second block are concatenated and dimensionality reduced to obtain a fused feature map with the same size as the M2 second feature maps.

[0257] In one possible implementation, the first block and the M second blocks are blocks within the target stage of the feature extraction network, and the concatenated feature map is used as the output feature map of the target stage in the feature extraction network; or...

[0258] The target stage further includes at least one third block, which is used to perform a convolution operation on the concatenated feature map to obtain the output feature map of the target stage.

[0259] In one possible implementation, the feature extraction network is used to acquire an input image, extract features from the input image, and output a feature map of the input image;

[0260] The method further includes:

[0261] The task network is used to process the corresponding task based on the feature map of the input image to obtain the processing result.

[0262] In one possible implementation, the task includes object detection, image segmentation, or image classification.

[0263] Reference Figure 15 , Figure 15 This is an illustration of a data processing apparatus 1500 provided in an embodiment of this application, as shown below. Figure 15 As shown in the figure, the data processing apparatus 1500 provided in this application includes:

[0264] The acquisition module 1501 is used to acquire a feature extraction network, the feature extraction network including a first block, at least one second block connected in series, a target operation and a splicing operation, wherein the first block and the M second blocks are blocks within the same stage in the feature extraction network, and the number of parameters of the target operation is less than the number of parameters of the M second blocks.

[0265] The convolution processing module 1502 performs convolution processing on the input data through the first block to obtain M target feature maps, each target feature map corresponding to one channel;

[0266] Through the at least one second block, convolution processing is performed on M1 target feature maps out of the M target feature maps to obtain M1 first feature maps, where M1 is less than M;

[0267] Through the target operation, M2 target feature maps out of the M target feature maps are processed to obtain M2 second feature maps, where M2 is less than M;

[0268] The splicing module 1503 is used to splice the M1 first feature maps and the M2 second feature maps through the splicing operation to obtain the spliced ​​feature map.

[0269] In one possible implementation, the intersection of the M1 target feature maps and the M2 target feature maps is empty, and the sum of the M1 and M2 is the M, and the number of channels of the concatenated feature map is the M.

[0270] In one possible implementation, the target operation is a convolution operation with fewer parameters than the at least one second block; or,

[0271] The target operation is a residual connection operation between the output of the first block and the output of the splicing operation.

[0272] In one possible implementation, the at least one second block is used to perform convolution processing on M1 target feature maps among the M target feature maps to obtain feature maps output by each second block, wherein the output of the second block that is farthest from the first block among the at least one second block is the M1 first feature maps;

[0273] The device further includes:

[0274] The fusion module is used to fuse the feature maps output by each second block through a fusion operation to obtain a fused feature map, the size of which is the same as the size of the M2 second feature maps; and to perform an addition operation on the fused feature map and the M2 second feature maps to obtain processed M2 second feature maps.

[0275] The splicing module is used to splice the M1 first feature maps and the processed M2 second feature maps to obtain a spliced ​​feature map.

[0276] In one possible implementation, the fusion module is used to perform a fusion operation on the output of each second block, concatenating and reducing the dimensionality to obtain a fused feature map with the same size as the M2 second feature maps.

[0277] In one possible implementation, the first block and the M second blocks are blocks within the target stage of the feature extraction network, and the concatenated feature map is used as the output feature map of the target stage in the feature extraction network; or...

[0278] The target stage further includes at least one third block, which is used to perform a convolution operation on the concatenated feature map to obtain the output feature map of the target stage.

[0279] In one possible implementation, the feature extraction network is used to acquire an input image, extract features from the input image, and output a feature map of the input image;

[0280] The device further includes:

[0281] The task processing module is used to process the corresponding task based on the feature map of the input image through a task network to obtain the processing result.

[0282] In one possible implementation, the task includes object detection, image segmentation, or image classification.

[0283] The following describes an execution device provided in an embodiment of this application. Please refer to [link / reference]. Figure 16 , Figure 16 This is a schematic diagram of an execution device provided in an embodiment of this application. The execution device 1600 can specifically be a mobile phone, tablet, laptop, smart wearable device, server, etc., and is not limited thereto. The execution device 1600 may be equipped with... Figure 15 The data processing apparatus described in the corresponding embodiment is used to implement Figure 15 The corresponding embodiment describes the data processing function. Specifically, the execution device 1600 includes: a receiver 1601, a transmitter 1602, a processor 1603, and a memory 1604 (wherein the execution device 1600 may have one or more processors 1603). Figure 11 (Taking a processor as an example), processor 1603 may include application processor 16031 and communication processor 16032. In some embodiments of this application, receiver 1601, transmitter 1602, processor 1603 and memory 1604 may be connected via bus or other means.

[0284] Memory 1604 may include read-only memory and random access memory, and provides instructions and data to processor 1603. A portion of memory 1604 may also include non-volatile random access memory (NVRAM). Memory 1604 stores processor and operation instructions, executable modules, or data structures, or subsets thereof, or extended sets thereof, wherein the operation instructions may include various operation instructions for implementing various operations.

[0285] Processor 1603 controls the operation of the execution device. In specific applications, the various components of the execution device are coupled together through a bus system, which may include not only the data bus, but also power buses, control buses, and status signal buses. However, for clarity, all buses are referred to as the bus system in the diagram.

[0286] The methods disclosed in the embodiments of this application can be applied to or implemented by processor 1603. Processor 1603 can be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above methods can be completed by integrated logic circuits in the hardware of processor 1603 or by instructions in software form. Processor 1603 can be a general-purpose processor, digital signal processor (DSP), microprocessor or microcontroller, as well as a vision processing unit (VPU), tensor processing unit (TPU), or other processors suitable for AI computation. It may further include application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. Processor 1603 can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in the embodiments of this application can be directly implemented by a hardware decoding processor, or implemented by a combination of hardware and software modules in the decoding processor. The software modules can reside in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. This storage medium is located in memory 1604, and processor 1603 reads information from memory 1604 and, in conjunction with its hardware, completes the steps of the above method.

[0287] Receiver 1601 can be used to receive input digital or character information, and to generate signal inputs related to the settings and function control of the execution device. Transmitter 1602 can be used to output digital or character information through the first interface; transmitter 1602 can also be used to send instructions to the disk group through the first interface to modify the data in the disk group; transmitter 1602 may also include a display device such as a display screen.

[0288] The execution device can operate Figure 6 The described perception network, or execution Figure 14 The data processing method in the corresponding embodiment.

[0289] This application also provides a training device; please refer to [link / reference]. Figure 17 , Figure 17 This is a schematic diagram of a training device provided in an embodiment of this application. Specifically, the training device 1700 is implemented by one or more servers. The training device 1700 can vary significantly due to different configurations or performance. It may include one or more central processing units (CPUs) 1717 (e.g., one or more processors) and memory 1732, and one or more storage media 1730 (e.g., one or more mass storage devices) for storing application programs 1742 or data 1744. The memory 1732 and storage media 1730 can be temporary or persistent storage. The program stored in the storage media 1730 may include one or more modules (not shown in the diagram), each module may include a series of instruction operations on the training device. Furthermore, the CPU 1717 may be configured to communicate with the storage media 1730 and execute the series of instruction operations in the storage media 1730 on the training device 1700.

[0290] The training device 1700 may also include one or more power supplies 1726, one or more wired or wireless network interfaces 1750, one or more input / output interfaces 1758; or, one or more operating systems 1741, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, etc.

[0291] Figure 17 The model training device 1700 described herein can be a module in a training device, and the processor in the training device can execute the training obtained through model training. Figure 6 The described perceptual network.

[0292] This application also provides a computer program product that, when run on a computer, causes the computer to perform steps as performed by the aforementioned execution device, or causes the computer to perform steps as performed by the aforementioned training device.

[0293] This application also provides a computer-readable storage medium storing a program for signal processing, which, when run on a computer, causes the computer to perform steps as performed by the aforementioned execution device, or causes the computer to perform steps as performed by the aforementioned training device.

[0294] The execution device, training device, or terminal device provided in this application embodiment can specifically be a chip. The chip includes a processing unit and a communication unit. The processing unit can be, for example, a processor, and the communication unit can be, for example, an input / output interface, pins, or circuits. The processing unit can execute computer execution instructions stored in the storage unit to cause the chip within the execution device to execute the data processing method described in the above embodiments, or to cause the chip within the training device to execute the data processing method described in the above embodiments. Optionally, the storage unit can be a storage unit within the chip, such as a register or cache. Alternatively, the storage unit can be a storage unit located outside the chip within the wireless access device, such as a read-only memory (ROM) or other types of static storage devices capable of storing static information and instructions, such as random access memory (RAM).

[0295] For details, please refer to Figure 18 , Figure 18 This is a schematic diagram of a chip provided in an embodiment of this application. The chip can be represented as a neural network processor (NPU) 1800. The NPU 1800 is mounted as a coprocessor on the host CPU, and tasks are assigned by the host CPU. The core part of the NPU is the arithmetic circuit 1803, which is controlled by the controller 1804 to extract matrix data from the memory and perform multiplication operations.

[0296] The NPU 1800 achieves this through the cooperation of its various internal components. Figure 6 The described embodiments provide a model training method, or perform inference on the trained model.

[0297] The computing circuit 1803 in the NPU 1800 can perform the steps of acquiring a first neural network model and training the first neural network model.

[0298] More specifically, in some implementations, the arithmetic circuitry 1803 within the NPU 1800 includes multiple processing engines (PEs). In some implementations, the arithmetic circuitry 1803 is a two-dimensional pulsating array. The arithmetic circuitry 1803 can also be a one-dimensional pulsating array or other electronic circuitry capable of performing mathematical operations such as multiplication and addition. In some implementations, the arithmetic circuitry 1803 is a general-purpose matrix processor.

[0299] For example, suppose we have an input matrix A, a weight matrix B, and an output matrix C. The arithmetic circuit retrieves the corresponding data of matrix B from the weight memory 1802 and caches it in each PE of the arithmetic circuit. The arithmetic circuit retrieves the data of matrix A from the input memory 1801 and performs matrix operations with matrix B. The partial result or the final result of the obtained matrix is ​​stored in the accumulator 1808.

[0300] Unified memory 1806 is used to store input and output data. Weight data is directly transferred to weight memory 1802 via Direct Memory Access Controller (DMAC) 1805. Input data is also transferred to unified memory 1806 via DMAC.

[0301] BIU stands for Bus Interface Unit, which is used for interaction between the AXI bus and the DMAC and the Instruction Fetch Buffer (IFB) 1809.

[0302] The Bus Interface Unit (BIU) 1810 is used by the instruction fetch memory 1809 to fetch instructions from external memory, and also by the memory access controller 1805 to fetch the original data of the input matrix A or the weight matrix B from external memory.

[0303] The DMAC is mainly used to move input data from external memory DDR to unified memory 1806, or to weight data to weight memory 1802, or to input data to input memory 1801.

[0304] The vector computation unit 1807 includes multiple processing units that further process the output of the computation circuit 1803 when needed, such as vector multiplication, vector addition, exponential operations, logarithmic operations, size comparisons, etc. It is mainly used for computation in non-convolutional / fully connected layers of neural networks, such as batch normalization, pixel-level summation, and upsampling of feature planes.

[0305] In some implementations, the vector computation unit 1807 can store the processed output vector in the unified memory 1806. For example, the vector computation unit 1807 can apply a linear function, or a nonlinear function, to the output of the computation circuit 1803, such as performing linear interpolation on feature planes extracted by a convolutional layer, or, for example, accumulating a vector of values ​​to generate activation values. In some implementations, the vector computation unit 1807 generates normalized values, pixel-level summed values, or both. In some implementations, the processed output vector can be used as an activation input to the computation circuit 1803, for example, for use in subsequent layers of the neural network.

[0306] The instruction fetch buffer 1809 connected to the controller 1804 is used to store the instructions used by the controller 1804;

[0307] Unified memory 1806, input memory 1801, weighted memory 1802, and instruction fetch memory 1809 are all on-chip memories. External memory is proprietary to this NPU hardware architecture.

[0308] The processor mentioned above can be a general-purpose central processing unit, a microprocessor, an ASIC, or one or more integrated circuits used to control the execution of the above program.

[0309] It should also be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. In addition, in the device embodiment drawings provided in this application, the connection relationship between modules indicates that they have a communication connection, which can be implemented as one or more communication buses or signal lines.

[0310] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware, or it can be implemented by special-purpose hardware including application-specific integrated circuits, special-purpose CPUs, special-purpose memory, special-purpose components, etc. Generally, any function performed by a computer program can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be diverse, such as analog circuits, digital circuits, or special-purpose circuits. However, for this application, software program implementation is more often the preferred implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium, such as a computer floppy disk, USB flash drive, mobile hard disk, ROM, RAM, magnetic disk, or optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, training equipment, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0311] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product.

[0312] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, training device, or data center to another website, computer, training device, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device or data center that integrates one or more available media. The available media may be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state drives (SSDs)).

Claims

1. A perception network device, comprising: The sensing network device includes: a feature extraction network, the feature extraction network comprising: The first block is used to perform convolution processing on the input data to obtain M target feature maps, each target feature map corresponding to one channel; At least one second block is serially connected, and the at least one second block is used to perform convolution processing on M1 target feature maps out of the M target feature maps to obtain M1 first feature maps, where M1 is less than M; the first block and the at least one second block are blocks within the same stage in the feature extraction network; A unit for performing a target operation, wherein the target operation is used to process M2 target feature maps out of the M target feature maps to obtain M2 second feature maps, wherein M2 is less than M, and the number of parameters of the target operation is less than the number of parameters of the at least one second block; A unit for performing a splicing operation, wherein the splicing operation is used to splice the M1 first feature maps and the M2 second feature maps to obtain a spliced ​​feature map.

2. The sensing network device according to claim 1, characterized in that, The intersection of the M1 target feature maps and the M2 target feature maps is empty, and the sum of the M1 and M2 is the M.

3. The sensing network device according to any one of claims 1-2, characterized in that, The target operation is a convolution operation with fewer parameters than the at least one second block; or, The target operation is a residual connection operation between the output of the first block and the output of the splicing operation.

4. The sensing network device according to claim 1, characterized in that, The feature extraction network further includes a unit for performing a fusion operation, which is used to fuse the feature maps output by each second block to obtain a fused feature map, and to perform an addition operation on the fused feature map and the M2 second feature maps to obtain processed M2 second feature maps, wherein the size of the fused feature map is the same as the size of the M2 second feature maps; The spliced ​​feature map is obtained by splicing the M1 first feature maps and the processed M2 second feature maps.

5. The sensing network device according to any one of claims 1-2, characterized in that, The M1 first feature maps are the outputs of the second block that is furthest from the first block among the at least one second block.

6. The sensing network device according to claim 4, characterized in that, The fusion operation is used to concatenate and reduce the dimensions of the output of each second block to obtain the fused feature map with the same size as the M2 second feature maps.

7. The sensing network device according to any one of claims 1-2, characterized in that, The first block and the at least one second block are blocks within the target stage of the feature extraction network, and the concatenated feature map is used as the output feature map of the target stage in the feature extraction network; or, The target stage further includes at least one third block, which is used to perform a convolution operation on the concatenated feature map to obtain the output feature map of the target stage.

8. The sensing network device according to any one of claims 1-2, characterized in that, The feature extraction network is used to acquire the input image, extract features from the input image, and output the feature map of the input image; The sensing network device also includes: The task network is used to process the corresponding task based on the feature map of the input image to obtain the processing result.

9. The sensing network device according to claim 8, characterized in that, The tasks include object detection, image segmentation, or image classification.

10. A data processing method, characterized in that, Applied to a feature extraction network, the feature extraction network comprising: The first block is used to perform convolution processing on the input data to obtain M target feature maps, each target feature map corresponding to one channel; At least one second block is serially connected, and the at least one second block is used to perform convolution processing on M1 target feature maps out of the M target feature maps to obtain M1 first feature maps; the first block and the at least one second block are blocks within the same stage in the feature extraction network; A unit for performing a target operation, wherein the target operation is used to process M2 target feature maps out of the M target feature maps to obtain M2 second feature maps, and the number of parameters of the target operation is less than the number of parameters of the at least one second block; A unit for performing a splicing operation, wherein the splicing operation is used to splice the M1 first feature maps and the M2 second feature maps; The method includes: The input data is convolved to obtain M target feature maps, each of which corresponds to one channel; Convolution processing is performed on M1 target feature maps out of the M target feature maps to obtain M1 first feature maps, where M1 is smaller than M; M2 target feature maps out of the M target feature maps are processed to obtain M2 second feature maps, where M2 is less than M; The M1 first feature maps and the M2 second feature maps are concatenated to obtain the concatenated feature map.

11. The method according to claim 10, characterized in that, The intersection of the M1 target feature maps and the M2 target feature maps is empty, and the sum of the M1 and M2 is the M.

12. The method according to any one of claims 10-11, characterized in that, The target operation is a convolution operation with fewer parameters than the at least one second block; or, The target operation is a residual connection operation between the output of the first block and the output of the splicing operation.

13. The method according to any one of claims 10-11, characterized in that, The method further includes: The feature maps output from each second block are fused to obtain a fused feature map; An addition operation is performed on the fused feature map and the M2 second feature maps to obtain the processed M2 second feature maps; the size of the fused feature map is the same as the size of the M2 second feature maps; The step of concatenating the M1 first feature maps and the M2 second feature maps includes: The M1 first feature maps and the processed M2 second feature maps are concatenated to obtain the concatenated feature map.

14. The method according to any one of claims 10-11, characterized in that, The output of the second block that is furthest from the first block in the at least one second block is the M1 first feature maps.

15. The method according to claim 13, characterized in that, The fusion of the feature maps output from each second block includes: The outputs of each second block are concatenated and subjected to dimensionality reduction to obtain the fused feature map with the same size as the M2 second feature maps.

16. The method according to any one of claims 10-11, characterized in that, The first block and the at least one second block are blocks within the target stage of the feature extraction network, and the concatenated feature map is used as the output feature map of the target stage in the feature extraction network; or, The target stage further includes at least one third block, which is used to perform a convolution operation on the concatenated feature map to obtain the output feature map of the target stage.

17. The method according to any one of claims 10-11, characterized in that, The feature extraction network is used to acquire the input image, extract features from the input image, and output the feature map of the input image; The method further includes: The task network is used to process the corresponding task based on the feature map of the input image to obtain the processing result.

18. The method according to claim 17, characterized in that, The tasks include object detection, image segmentation, or image classification.

19. A data processing apparatus, characterized in that, The apparatus includes a memory and a processor; the memory stores code, and the processor is configured to retrieve the code to run the data processing method as described in any one of claims 10 to 18.

20. A computer-readable storage medium for storing a computer program, characterized in that, The computer program is used to execute instructions for the method according to any one of claims 10 to 18.

Images