Methods for determining the encoder architecture of a neural network

Abstract

This invention relates to a method for determining the encoder (2) architecture of a convolutional neural network (1), wherein the neural network (1) is configured to process multiple different image processing tasks (t1, t2, t3), wherein the method comprises the following steps: - for each image processing task (t1, t2, t3), calculating a feature scale distribution based on training data (S10); - generating multiple candidate encoder architectures, wherein each of the multiple candidate encoder architectures includes at least one shared encoder layer (SL) that provides computational operations for the multiple image processing tasks, and each encoder architecture further includes multiple branches (b1, b2, b3) spanning one or more encoder layers, which provide at least partially different computational operations for the image processing tasks, wherein each branch (b1, b2, b3) are associated with a specific image processing task (t1, t2, t3) (S11); - Calculate the receptive field size of the encoder layer of multiple encoder architectures (S12); - Calculate multiple evaluation metrics, wherein each evaluation metric relates to a combination of a specific encoder architecture and a specific image processing task (t1, t2, t3) among the multiple encoder architectures, wherein each evaluation metric includes matching quality information about the feature scale distribution of the image processing task (t1, t2, t3) associated with the evaluation metric and the receptive field size of the encoder layer of the encoder architecture associated with the evaluation metric (S13); - Compare the calculated evaluation metrics and establish a comparison result (S14); and - Select an encoder architecture based on the comparison result (S15).

Description

Technical Field

[0001] This invention generally relates to the field of convolutional neural networks. More specifically, this invention relates to a method for determining a shared encoder architecture for a convolutional neural network configured to handle multiple different image processing tasks. Background Technology

[0002] Convolutional neural networks are very powerful in solving pattern-based recognition tasks such as pedestrian detection, lane detection, and sign recognition. Instead of implementing separate networks for each task, using shared networks, especially shared encoders for multiple tasks, offers several advantages in terms of performance, computation, and / or memory.

[0003] However, determining a shared encoder architecture that can effectively handle multiple tasks is not an easy task. Summary of the Invention

[0004] One objective of this invention is to provide a method for determining the encoder architecture or structure of a convolutional neural network configured to handle multiple different image processing tasks, wherein the computational workload of the encoder structure is reduced, and the encoder architecture is resource-efficient and computationally efficient.

[0005] This task is solved by the features of the independent claim. Preferred embodiments are given in the dependent claims. Unless otherwise expressly indicated, embodiments of the invention can be freely combined with each other.

[0006] According to one aspect, the present invention relates to a method for determining the encoder architecture of a convolutional neural network, particularly a shared encoder architecture, wherein the neural network is configured to process multiple different image processing tasks. More specifically, the neural network may be an artificial neural network that includes a shared encoder, wherein the encoder architecture is tailored for processing multiple image processing tasks. After determining the encoder architecture, the neural network is trained to enable it to process the image processing tasks. The method may be at least partially computer-implemented. The method includes the following steps:

[0007] First, for each image processing task, a feature scale distribution based on the training data is calculated. The feature scale distribution indicates the size of the image portion to be provided to the task-specific decoder so that the task-specific decoder can provide task-specific information.

[0008] In addition, multiple candidate encoder architectures are generated. Each candidate encoder architecture (also referred to as an encoder structure) includes at least one shared encoder layer that provides computational operations for multiple image processing tasks. Furthermore, each encoder architecture includes multiple branches spanning one or more encoder layers. Each branch is associated with a specific image processing task and provides at least some task-specific computational operations for that task. More specifically, a task-specific decoder, providing computational specificity suitable for the corresponding image processing task, receives feature maps from one or more encoder layers of the branch associated with the corresponding image processing task.

[0009] Furthermore, the receptive field size of encoder layers in multiple encoder architectures is calculated. The receptive field size associated with a specific encoder layer is indicative of the image portion that influences the encoder layer's output. The receptive field size can be calculated by investigating the size of an image portion of the input image that influences the features of the feature map and thus affects the output of a specific encoder layer. The receptive field size of an encoder layer can be calculated, for example, based on convolutional properties, such as kernel size and stride factor.

[0010] Subsequently, multiple evaluation metrics are computed. An evaluation metric can be an indicator of the quality of a candidate encoder architecture's match to a specific image processing task. More specifically, an evaluation metric can indicate how well the receptive field size of the encoder layer in a candidate encoder architecture matches the feature scale distribution of a specific image processing task. Each evaluation metric involves a combination of a specific encoder architecture and a specific image processing task among multiple encoder architectures. Each evaluation metric includes information about the quality of the match between the feature scale distribution of the image processing task associated with the evaluation metric and the receptive field size of the encoder layer in the encoder architecture associated with the evaluation metric. In other words, the evaluation metric provides information on how well the receptive field size of the branch encoder layer associated with the image processing task matches the feature scale distribution determined by the image processing task.

[0011] As an example, the evaluation metric associated with the combination of encoder architecture and image processing task can be calculated by summing the squares of the differences between the receptive field size of the encoder layer and the percentiles of the feature scale distribution required by the task-specific decoder.

[0012] After determining the evaluation metrics, the calculated evaluation metrics are compared, and the comparison results are established. Specifically, the comparison results lead to an encoder architecture in which the receptive field size of the encoder layer is best matched with the feature scale distribution of the specific image processing task.

[0013] Finally, the encoder architecture is selected based on the comparison results.

[0014] The method is highly beneficial because it enables the development of shared encoders, which, especially in limited hardware environments, reduce computational workload and improve resource efficiency, providing encoding operations for multiple image processing tasks.

[0015] By considering the optimal receptive field based on specific task attributes, this patent application, published herein, proposes improved multi-task neural architectures and task-specific decoders for a given set of image processing tasks. In other architecture search methods, all possible architectures must be stored in memory and trained in parallel, or all architectures must be trained to some extent. In contrast, the method proposed in this patent application provides an architecture based on static attributes, which can be computed without requiring neural network training. Since the number of possible architectures grows exponentially with the number of tasks that the multi-task network must solve, the proposed architecture search method saves significant time, making the principled architectural design of embedded hardware feasible.

[0016] The proposed method is preferably used to determine an encoder architecture for image processing tasks based on images provided by a front-facing camera in an automotive vehicle. In automotive applications, multiple tasks must be addressed simultaneously. Therefore, this patent application disclosed herein relates to two-dimensional (2D) convolutional neural networks.

[0017] In addition to the most common deep learning tasks such as object detection, lane detection, and semantic segmentation, the proposed method can be extended to other two-dimensional (2D) image processing tasks, such as depth estimation based on monocular cameras, surface normal estimation, and keypoint detection for, for example, human pose estimation.

[0018] In addition to further two-dimensional (2D) image processing tasks, this invention can also be extended to one-dimensional (1D), three-dimensional (3D), and even higher-dimensional convolutional neural networks:

[0019] - Applications of three-dimensional (3D) convolutional networks include point cloud processing for radar or lidar, or video feeds for camera devices;

[0020] - One-dimensional (1D) convolutional networks are useful in time series processing, such as for processing other sensor measurements like temperature and acceleration;

[0021] - Four-dimensional (4D) convolutional networks can be applied to process point cloud sequences originating from radar or lidar sensors.

[0022] According to an exemplary implementation, the image processing task includes one or more of the following tasks: object detection, lane detection, and semantic segmentation.

[0023] According to an exemplary embodiment, convolutional neural networks, particularly encoders, are configured and / or specifically tailored for processing on embedded hardware, especially embedded computer modules or system-on-chips (SoCs) included in vehicles. Such embedded hardware typically includes limited computational resources. Therefore, convolutional neural networks, particularly encoders, must be designed for these limited computational resources to provide real-time or near-real-time image processing results, respectively, within given runtime constraints. Runtime constraints can range from 10 milliseconds to 50 milliseconds, particularly 10 milliseconds, 15 milliseconds, 20 milliseconds, 25 milliseconds, 30 milliseconds, 35 milliseconds, 40 milliseconds, 45 milliseconds, or 50 milliseconds.

[0024] According to one embodiment, for each image processing task, one or more percentiles are determined, wherein the values ​​of the one or more percentiles depend on the interface of the task-specific decoder. The interface of the task-specific decoder may, for example, include one or more input connections. Based on the interface of the task-specific decoder, the values ​​of one or more percentiles are selected so that the decoder can receive information required for the task-specific decoding operation (e.g., size information of a specific image portion).

[0025] According to one implementation, the number of percentiles is selected based on the number of input connections required by the task-specific decoder. For example, if the task-specific decoder has only one input connection, a maximum percentile value, such as 99%, can be selected. If the task-specific decoder has multiple input connections (e.g., two, three, or more), multiple different percentile values ​​(e.g., two, three, or more) can be selected to provide different image portion size information to the decoder.

[0026] According to one embodiment, the percentile values ​​are distributed within a percentile range defined by a minimum percentile value and a maximum percentile value. The maximum percentile value could be, for example, the 99th percentile, and the minimum percentile value could be, for example, the 50th percentile or lower. If the decoder provides only two input connections, the minimum percentile value (e.g., 50%) and the maximum percentile value (e.g., 99%) can be selected. If the decoder provides three input connections, the minimum percentile value (e.g., 50%), a middle percentile value (e.g., 75%), and the maximum percentile value (e.g., 99%) can be selected. If the decoder provides four input connections, the minimum percentile value (e.g., 50%), two middle percentile values ​​(e.g., 66% and 82%), and the maximum percentile value (e.g., 99%) can be selected. Thus, different image portion sizes can be provided to a task-specific decoder depending on the decoder input interface.

[0027] According to one embodiment, the percentile values ​​are evenly distributed within a percentile range defined by the minimum and maximum percentile values. This allows for the provision of interleaved information for different image portion sizes to a task-specific decoder.

[0028] According to one embodiment, at least some encoder layers of a task-specific branch of the encoder include feature maps with a resolution that matches the feature resolution of the input connections of the task-specific decoder. Thus, the output provided by the task-specific encoder layer can be coupled to an input connection of the task-specific decoder with an appropriate resolution.

[0029] According to one embodiment, the step of determining multiple candidate encoder architectures includes: selecting specific building blocks from a set of building blocks, the specific building blocks being suitable for an encoder of a neural network, and concatenating the selected building blocks to obtain an encoder architecture. The set of building blocks can be predefined based on the specific computing hardware on which the neural network should run. More specifically, the set of building blocks can be predefined by considering the computing resources provided by the embedded computing hardware. For some embedded hardware, these building blocks are very simple, containing only simple convolutional layers, pooling operations, and activations. For other embedded hardware, these building blocks can be Residual Network (ResNet) or DarkNet building blocks, or even units of architecture search methods such as Differentiable Neural Network Architecture Search (DARTS). By stacking these building blocks, deeper encoders can be built, and each building block has a specific width (or channel number) available for adjusting the network width.

[0030] According to one embodiment, the encoder architecture of a candidate encoder architecture is generated by defining runtime constraints (e.g., in an automatic or semi-automatic manner) and constructing the encoder architecture such that, under given hardware conditions, the encoder architecture provides computational results within 90% to 100% of the runtime constraints. For example, the width (i.e., the number of channels) of a layer can be selected based on the number of encoder layers (i.e., convolutional layers) to take advantage of the given runtime constraints.

[0031] According to one embodiment, the step of calculating multiple evaluation metrics includes: comparing the receptive field size of an encoder layer that matches the resolution of a task-specific decoder with a determined percentile value, and determining the distance between the receptive field size and the percentile value. Therefore, if the receptive field size is X and the percentile value is Y, the calculation of the evaluation metrics includes calculating the difference between X and Y.

[0032] According to one embodiment, particularly if a task-specific decoder includes multiple input connections, the step of calculating multiple evaluation metrics includes: comparing multiple receptive field sizes of an encoder layer matching the resolution of the task-specific decoder with multiple determined percentile values, and determining the sum of distances between the receptive field sizes and the percentile values. Here, the overall distance between the multiple receptive field size / percentile value pairs may be considered for evaluating the encoder architecture.

[0033] According to one implementation, the step of calculating multiple evaluation metrics uses a loss function with the minimum absolute deviation distance metric, namely, an L1 distance metric or a cosine distance metric.

[0034] According to one embodiment, the step of calculating multiple evaluation metrics uses a loss function with a penalty factor, wherein the penalty factor is adapted to increase the evaluation metrics as the number of shared layers among multiple image processing tasks in the encoder architecture decreases. Therefore, encoder architectures with more shared layers are more advantageous than those with fewer shared layers because more shared layers improve computational efficiency.

[0035] According to one embodiment, the step of comparing the calculated evaluation metrics and establishing the comparison results includes: determining the encoder architecture with the lowest evaluation metric for each image processing task. Thus, improved encoder architectures for processing multiple image processing tasks can be determined based on image processing task attributes and encoder / decoder attributes of the neural network architecture, which can be obtained without training the relevant neural network architecture. Therefore, improved encoder architectures can be found with reduced computational effort.

[0036] According to one embodiment, the step of selecting an encoder architecture based on comparison results includes: selecting an encoder architecture with the lowest evaluation metric for each image processing task. For example, the selected encoder architectures are combined by sharing encoder layers with the same attributes (e.g., encoder layer width), and by branching the encoder layers into different encoder branches if the encoder layers have different encoder layer attributes.

[0037] According to another aspect, the present invention relates to a computer program product for determining an encoder architecture. The computer program product includes a computer-readable storage medium containing program instructions executable by a processor to cause the processor to perform the method according to any of the preceding claims.

[0038] The term "vehicle" as used in this invention may refer to automobiles, trucks, buses, rail vehicles or any other means of transportation.

[0039] The term "feature scale" as used in this patent application published herein can refer to the number of pixels or image portions received by a task-specific decoder in order to identify details in an image for a specific task. Therefore, the "feature scale" is highly dependent on the specific image processing task.

[0040] As used in this patent application published herein, the term "receptive field" can refer to a set of image pixels provided as input to an encoder, wherein the set of pixels affects a specific neuron (also called a feature) in the feature map of the encoder layer. In other words, the receptive field refers to the portion of the image that influences the features of the feature map, and thereby affects the output of a particular encoder layer.

[0041] The term "receptive field size" as used in this patent application disclosed herein may refer to the size of a plurality of pixels or image portions received by a particular encoder layer. The receptive field may be rectangular, wherein the length and height of the rectangle constitute the receptive field size. Due to the cascading of encoder layers, the receptive field size of the first encoder layer is smaller than the receptive field size of the hidden encoder layer.

[0042] The term “task-specific decoder” as used in this patent application published herein may refer to a decoder configured to provide decoding operations for a single image processing task, i.e., each image processing task includes its own task-specific decoder.

[0043] The term "feature map" as used in the patent application published herein can refer to a set of features provided by a specific encoder layer that can be transmitted to a task-specific decoder. The feature map may include a specific resolution. The resolution of the feature map must match the resolution required by the task-specific decoder so that the task-specific decoder can process the feature map.

[0044] The term “percentile” as used in the patent application published herein may refer to a number in which a given percentage of numbers fall at or below that number in its frequency distribution.

[0045] The term "image processing task" can refer to any task applied to images or videos provided by a camera device. Additionally, the term "image processing task" can also refer to a task applied to information provided by other sensors, such as radar or lidar.

[0046] The terms “substantially” or “approximately” as used in this invention refer to deviations from the exact value by + / -10%, preferably + / -5%, and / or from variations that are not important to function and / or to traffic rules. Attached Figure Description

[0047] Different aspects of the invention, including its specific features and advantages, will be more readily understood from the following detailed description and accompanying drawings, wherein:

[0048] Figure 1 A schematic diagram of a neural network consisting of a shared encoder and multiple task-specific decoders is shown.

[0049] Figure 2 The diagram illustrates an example of the structure of a shared encoder, which includes a shared layer and multiple branches for performing computational operations for specific tasks.

[0050] Figure 3 The schematic diagram illustrates the characteristic scale distribution of the first and second tasks, as well as their 50th or 99th percentiles.

[0051] Figure 4 The schematic diagram illustrates the characteristic scale distributions of the first and third tasks, as well as their 50th or 90th percentiles.

[0052] Figure 5 The table shows the 50th, 60th, 80th, and 99th percentile values ​​of the characteristic scale distribution for the first and second tasks.

[0053] Figure 6 Multiple encoder architectures with different lengths, coding layer widths, and coding layer receptive fields are shown.

[0054] Figure 7 The calculation of the evaluation metric is shown, which indicates the evaluation metric based on... Figure 6 Architecture A and based on Figure 5 The matching quality of image processing task t1;

[0055] Figure 8 The calculation of the evaluation metric is shown, which indicates the evaluation metric based on... Figure 6 Architecture A and based on Figure 5 The matching quality of image processing task t2;

[0056] Figure 9 The calculation of the evaluation metric is shown, which indicates the evaluation metric based on... Figure 6 The architecture of C and based on Figure 5 The matching quality of image processing task t1;

[0057] Figure 10 The calculation of the evaluation metric is shown, which indicates the evaluation metric based on... Figure 6 The architecture of C and based on Figure 5 The matching quality of image processing task t2;

[0058] Figure 11 It shows that according to Figures 5 to 10 The example results of the encoder structure; and

[0059] Figure 12A schematic block diagram illustrating the steps of a method for determining the shared encoder architecture of a convolutional neural network is shown. Detailed Implementation

[0060] The invention will now be described in more detail with reference to the accompanying drawings, which illustrate exemplary embodiments. The embodiments in the drawings relate to preferred embodiments, and all elements and features described in connection with these embodiments can be used as much as possible in conjunction with any other embodiments and features discussed herein, particularly in relation to any other embodiments further discussed above. However, the invention should not be construed as being limited to the embodiments described herein. Throughout the following description, similar reference numerals denote similar elements, parts, items, or features, where applicable.

[0061] The features of the invention disclosed in the description, claims, embodiments and / or drawings can be used individually or in any combination to implement the invention.

[0062] Figure 1 A schematic block diagram of neural network 1 is shown. Neural network 1 includes encoder 2. Encoder 2 is configured to process multiple image processing tasks; in the illustrated embodiment, image processing tasks t1, t2, and t3 are processed. Encoder 2 provides information to multiple task-specific decoders 3.1, 3.2, and 3.3. Each task-specific decoder 3.1, 3.2, and 3.3 provides output information related to a specific image processing task t1, t2, or t3.

[0063] Figure 2 A schematic block diagram illustrating an example encoder architecture of encoder 2 is provided. Encoder 2 comprises multiple layers, L1 to L5. In the illustrated embodiment, layer L1 is a shared layer that provides computational operations for all image processing tasks t1, t2, and t3. Starting from the shared layer L1, encoder 2 includes multiple branches, b1, b2, and b3. Each branch b1, b2, and b3 is associated with a specific image processing task t1, t2, or t3. More specifically, the encoder architecture provides functional modules in each branch specifically configured for each corresponding image processing task t1, t2, or t3. Therefore, the further layers L2 to L5 at least partially provide different computational operations for image processing tasks t1, t2, and t3 to provide computational results for specific tasks.

[0064] like Figure 2 As shown, subsets of tasks can share functional modules of a layer. For example, branches b1 and b2 in layer L2 use the same functional module, while branch b3 uses another functional module in layer 2.

[0065] In addition, each branch can have a different length, meaning that each branch can span different levels.

[0066] The following provides an exemplary implementation of a method for determining an encoder architecture, which includes optimized branches to provide improved computational resources for processing multiple image processing tasks t1, t2, t3.

[0067] Figure 3 This section presents the feature scale distributions for two different tasks, t1 and t2. The feature scale distributions illustrate the size distribution of the objects to be detected in specific image processing tasks t1 and t2. Therefore, even when considering the same input image information, the feature scale distributions for each task can differ significantly.

[0068] The feature scale distributions of tasks t1 and t2 have significant overlap, meaning that the object sizes required by image processing tasks t1 and t2 are similar, and therefore the object sizes required by the decoders for specific tasks are similar. In other words, the encoders for tasks t1 and t2 can share multiple layers.

[0069] On the contrary, Figure 4 The feature scale distributions of tasks t1 and t3 shown have low overlap, meaning that the object sizes required by image processing tasks t1 and t3 are quite different, and therefore the object sizes required by the decoder for a particular task are quite different. Therefore, compared to... Figure 3 Compared to the previous example, the encoders of tasks t1 and t3 can share fewer layers, meaning they need to branch earlier.

[0070] To determine the improved encoder architecture, each image processing task was analyzed separately to determine the feature scale distribution for each task.

[0071] The following section describes different algorithms for determining the feature scale distribution of some common image processing tasks in driver assistance systems:

[0072] The first image processing task can be object detection. Object detection is a deep learning task that is crucial for assistive functions such as emergency braking assist (traffic participant detection), sign recognition, and traffic signal recognition. An object detection decoder typically receives input from multiple different encoder layers. These different encoder layers have progressively increasing receptive field sizes.

[0073] To determine the feature scale distribution for a specific object detection task, it is necessary to determine the bounding box size distribution that covers the training data for each corresponding object. The encoder's receptive field should ideally match the bounding box size distribution so that the receptive field size covers the median of the feature scale distribution and a reasonable maximum size, such as the 99th percentile of the feature scale distribution. The training data for object detection can be selected to include object-related items at different sizes and / or from different angles to simulate real-world car detection scenarios.

[0074] It is worth noting that the frame size distribution varies significantly across different driving functions. For example, for a specific camera device, the frame size for vehicles and pedestrians is generally larger than that for traffic signs or traffic signals, so their median and 99th percentile distributions are significantly different.

[0075] The second image processing task can be lane detection. Lane and road edge detection are crucial for lane departure warning and lane keeping assist systems. Task-specific decoders configured for lane detection fall into two main categories: point-based solutions and line / anchor box-based solutions. These two types of solutions differ significantly in their feature scale distribution.

[0076] For point-based methods, the feature scale can be determined by the distance between two keypoints: this distance distribution can be calculated based on the training dataset. For line- or anchor-box-based solutions, the feature scale should be determined based on the full length of the lane: in this case, the distribution can also be determined based on the training dataset. For a task-specific decoder connected to only one encoder layer, the receptive field size should match the 99th percentile of the feature scale distribution. For a task-specific decoder connected to multiple encoder layers, the receptive field size of the encoder layer should match the median of the feature scale distribution and a reasonable maximum size such as the 99th percentile of the feature scale distribution.

[0077] The training data for lane detection can be selected by including lane boundaries and / or road boundaries (lane markings, curbs, road edges, etc.) at different sizes and / or from different angles, thereby simulating real car detection scenarios.

[0078] The third image processing task is semantic segmentation. Semantic segmentation is used for free space estimation and low-speed maneuvering functions such as parking assistance. In semantic segmentation, each pixel of an image is classified as a road, vehicle, pedestrian, vegetation, etc. To determine the feature scale of semantic segmentation, the distribution of object mask size or background mask size can be determined based on the training data. Since semantic segmentation includes broad categories such as "road," "building," and "sky," the maximum receptive field size can cover the entire image.

[0079] The selection method for semantic segmentation training data can be to include the aforementioned semantic segmentation-related items in different sizes and / or from different perspectives, thereby simulating real car detection scenarios.

[0080] After determining the feature scale distribution for the image processing task, the relevant percentiles must be determined based on the interfaces of the task-specific decoder 3.1, 3.2, and 3.3. For example, if the task-specific decoder is connected to encoder 2 with a specific feature map resolution in one layer, the maximum object size, for example, the 99th percentile, should be considered. If the task-specific decoder receives input from two, three, or four layers, the relevant percentile can be selected, for example, between the median (50th percentile) and the 99th percentile. For example, if the task-specific decoder is connected to encoder 2 in two different layers, the 50th and 99th percentiles can be selected. If the task-specific decoder is connected to encoder 2 in three different layers, the 50th, 75th, and 99th percentiles can be selected. If the task-specific decoder is connected to encoder 2 in four different layers, the 50th, 66th, 82nd, and 99th percentiles can be selected, and so on.

[0081] If more than four input layers are defined, or if smaller objects need to be detected, percentiles smaller than the 50th percentile can also be considered.

[0082] After calculating the feature scale distribution and relevant percentiles for all tasks, the next step is to determine the possible encoder architectures and the receptive field sizes of the corresponding encoder layers.

[0083] Specifically, in automotive applications, computing resources are very limited. Therefore, a technically feasible encoder architecture largely depends on the specific computing hardware and its limitations.

[0084] The most common constraint is that any feasible encoder architecture must meet a specific runtime budget; that is, there is an extremely strict upper limit on runtime. Typically, if the complexity and / or runtime of a neural network decreases, its performance will also degrade. Therefore, when determining the set of possible encoder architectures, the goal is to find multiple or all feasible encoder architectures that are constrained by 90% to 100% of the predetermined runtime. In other words, feasible candidate encoder architectures should exhaust the predetermined runtime constraint as much as possible to achieve high performance for the neural network.

[0085] Considering predetermined runtime constraints, feasible candidate encoder architectures are constructed automatically or semi-automatically using multiple building blocks. Another limitation for computational hardware, especially embedded hardware, is the availability of specific neural network layers. Many SoCs (System-on-a-Chip) support only a small subset of multiple layers, and the list of available layers varies significantly depending on the specific hardware. Therefore, a set of encoder building blocks is predefined based on the specific hardware. Depending on the hardware, these building blocks can be extremely simple (e.g., containing only simple convolutional layers, pooling operations, and activations), or they can be more complex, such as Residual Network (ResNet) or DarkNet building blocks, or even architecture search method units such as Differential Neural Network Architecture Search (DARTS).

[0086] Once the building blocks, dependent on the given hardware, are determined, these building blocks can be stacked to build deeper encoders. Each block has a specific block width (also known as a channel number). Based on the block width, the network width can be adjusted. The runtime of each building block depends on its width and the input size. In this way, the runtime of a specific block on the target hardware can be calculated. Therefore, once the encoder building blocks are defined, a set of architectures can be determined automatically or semi-automatically to fit within a narrow range of acceptable runtime constraints.

[0087] It's worth noting that the number of convolutional layers that maintain resolution can be adjusted to fine-tune the receptive field size. Since encoders are designed to run on limited computing hardware, especially embedded hardware, the number of channels (also known as the width of encoder layers) should be reduced as the number of convolutional layers increases to meet the aforementioned runtime budget. Furthermore, the branching strategy for multi-task encoders is to set the branch point between two tasks to the most recent layer, as the most resource-efficient encoder scheme shares as many layers as possible.

[0088] After identifying multiple feasible encoder architectures with specific receptive fields or specific branch receptive fields in each corresponding encoder layer, the matching quality of the encoder layer receptive fields of each corresponding encoder architecture relative to the determined corresponding task-specific decoder is evaluated.

[0089] The matching evaluation of feature scale distribution and possible receptive fields can be accomplished by searching for an encoder architecture that minimizes the bias between the percentile of the determined task-specific feature scale distribution and the receptive field size of the encoder layer. The encoder architecture with the minimum bias can be determined by searching for an encoder architecture containing the minimum loss function value. An example of such a loss function is the simple mean squared error between the relevant percentile defined by the task-specific feature scale distribution and the receptive field size of each candidate architecture.

[0090] The loss function can be implemented using different distance metrics. For example, L1 (minimum absolute deviation) or cosine distance can be used as the distance metric.

[0091] Alternatively, the loss function can include a penalty factor that increases the loss value if the number of shared layers in an encoder architecture decreases. In this way, the loss function achieves resource conservation by sharing encoder layers across multiple image processing tasks.

[0092] In the following text, based on references Figures 5 to 11 The example describes a published method for determining the encoder architecture of a convolutional neural network.

[0093] Figure 5 The diagram illustrates multiple feature scale values ​​associated with two different image processing tasks, task t1 and task t2. These tasks have different feature scale distributions. Therefore, the pixel sizes associated with the 50th, 60th, 80th, and 99th percentiles of their respective feature scale distributions are different for tasks t1 and t2.

[0094] Configuring a task-specific decoder to process tasks t1 and t2 separately may require four input connections, meaning each task-specific decoder is connected to four different encoder layers of the encoder. These four input connections for each task-specific decoder have different resolutions. These resolutions are represented by different shaded cells, corresponding to columns of the table. Therefore, each task-specific decoder must be connected to the encoder layer that provides the required resolution.

[0095] Figure 6 This section presents an example of the results from the encoder architecture search. As mentioned above, the architecture search is performed based on the key factor that each feasible candidate encoder architecture meets runtime constraints, such as meeting 90% or more of the runtime limit for a given hardware, with the aim of achieving high performance for the neural network.

[0096] Each loop in each encoder architecture represents an encoder layer, where the leftmost first loop can represent one or more layers. However, when comparing... Figure 5 Cell shadows in the table and Figure 6 When working in a loop, the leftmost first encoder layer is not needed because the resolution of the encoder layer does not match the resolution required by the decoder for a specific task.

[0097] The numbers inside the circle indicate the width of the encoder layer. It's worth noting that as the encoder length increases (i.e., the number of encoder layers increases), the width of the encoder layer must decrease to meet runtime constraints.

[0098] Each encoder layer includes a specific receptive field. The receptive field of each encoder layer is given by the numbers below the turns representing the respective encoder layer. The receptive field of an encoder layer increases with the depth of the encoder; that is, an encoder layer at a greater depth has a larger receptive field compared to an encoder layer at a smaller depth.

[0099] based on Figures 7 to 10 The table describes the steps involved in calculating evaluation metrics to assess the matching quality of the encoder architecture for a specific task. More specifically, it examines... Figure 6 Is encoder architecture A or C shown more suitable for...? Figure 5 For task t1 or task t2, a comparison of feature scale distributions is performed for each task, especially a comparison of the determined percentiles of the feature scale distributions and the receptive field values ​​of the encoder layer that match the required decoder resolution.

[0100] Figure 5 In this context, the first input connection of the task-specific decoder includes, for example, a pixel size of 110 for the feature scale. The pixel size of the receptive field of the encoder layer, which matches the resolution of the first input connection of the task-specific decoder, includes a value of 122. Therefore, the difference is 12.

[0101] To determine the matching between encoder architecture A and image processing task t1, in this embodiment, the mean squared error between the pixel size of the feature scale of each corresponding input connection of the task-specific decoder and the pixel size of the receptive field of the matching encoder layer is calculated.

[0102] like Figure 7 As shown, the sum of the differences between the feature scale pixel size of each corresponding input connection of the task-specific decoder and the receptive field pixel size of the matched encoder layer is 182. This value represents an evaluation metric (referred to as the sum of losses in this embodiment) that can be used to evaluate which encoder architecture is best suited for the corresponding task and should therefore be at least considered as the task-specific branch in the encoder.

[0103] Figure 9 Display according to Figure 6 The encoder architecture C described above is based on Figure 5 The matching evaluation of the image processing task t1. When comparing the evaluation metric or the total loss, it can be determined that encoder architecture A matches the image processing task t1 better than encoder architecture C because the evaluation metric or the total loss is smaller.

[0104] Figure 8 and Figure 10 It shows Figure 6 The encoder architecture A and C are based on Figure 5A similar evaluation of the matching performance for image processing task t2 is performed. When comparing the evaluation metric or the total loss, it can be determined that encoder architecture C matches image processing task t2 better than encoder architecture A because the evaluation metric or the total loss is smaller.

[0105] like Figure 9 and Figure 10 As shown, if there are multiple encoder layers that provide the same resolution, and the resolution is required by the decoder for a specific task, then it can be determined which encoder layer should be connected to the decoder for the specific task by comparing the receptive field of each encoder layer with a determined percentile of the feature scale of the corresponding task, and selecting the encoder layer that provides the receptive field closest to the determined percentile.

[0106] Figure 11 The architecture of the resulting shared encoder is illustrated. This encoder is configured to process image processing tasks t1 and t2 based on the results of a matching evaluation and the coupling between the shared encoder and a task-specific decoder. The encoder includes one or more shared encoder layers that provide computational operations for image processing tasks t1 and t2, as well as two branches, where each branch provides task-specific computational operations for a specific task. Figure 11 In the diagram, the left branch is associated with task t1, meaning that one or more encoder layers in the left branch provide information to a task-specific decoder configured for image processing task t1. Similarly, the right branch is associated with task t2, meaning that one or more encoder layers in the right branch provide information to a task-specific decoder configured for image processing task t2.

[0107] Figure 12 A flowchart is shown, illustrating the steps involved in determining the encoder architecture of a convolutional neural network.

[0108] First, for each image processing task, the feature scale distribution is calculated based on the training data (S10).

[0109] Furthermore, multiple candidate encoder architectures are determined (S11). Each of the multiple candidate encoder architectures includes at least one shared encoder layer that provides computational operations for multiple image processing tasks, and multiple branches spanning one or more encoder layers. The branches provide at least partially different computational operations for the image processing tasks. Each branch is associated with a specific image processing task.

[0110] After determining feasible candidate encoder architectures, the receptive field size of the encoder layer of the encoder architecture is calculated (S12).

[0111] In addition, multiple evaluation metrics are computed (S13). Each evaluation metric involves a combination of a specific encoder architecture and a specific image processing task among the multiple encoder architectures. Each evaluation metric includes matching quality information regarding the feature scale distribution of the image processing task associated with the evaluation metric and the receptive field size of the encoder layer of the encoder architecture associated with the evaluation metric.

[0112] The next step is to compare the calculated evaluation metrics and determine the comparison results (S14).

[0113] Finally, the encoder architecture is selected based on the comparison results (S15).

[0114] It should be noted that the description and accompanying drawings are merely illustrative of the principles of the invention. Those skilled in the art will be able to implement various arrangements that are not explicitly described or shown herein but embody the principles of the invention.

[0115] List of reference numerals

[0116] 1 Neural Network

[0117] 2 encoders

[0118] 3.1, 3.2, 3.3 Task-Specific Decoders

[0119] branches b1, b2, b3

[0120] L1 to L5 encoder layers

[0121] Image processing tasks t1, t2, and t3

[0122] SL Shared Encoder Layer

Claims

1. A method for determining the encoder (2) architecture of a convolutional neural network (1), wherein, The neural network (1) is configured to handle multiple different image processing tasks (t1, t2, t3), the method comprising the following steps: For each image processing task (t1, t2, t3), a feature scale distribution is calculated based on the training data, wherein the feature scale distribution indicates the size distribution of the objects detected by each image processing task (S10). Multiple candidate encoder architectures are generated, wherein each of the multiple candidate encoder architectures includes at least one shared encoder layer (SL) that provides computational operations for multiple image processing tasks and multiple branches (b1, b2, b3) spanning one or more encoder layers that provide at least partially different computational operations for the multiple image processing tasks, wherein each branch (b1, b2, b3) is associated with a specific image processing task (t1, t2, t3) (S11); Calculate the receptive field size of the encoder layer for multiple encoder architectures (S12); Calculate multiple evaluation metrics, wherein each evaluation metric involves a combination of a specific encoder architecture and a specific image processing task (t1, t2, t3) in multiple encoder architectures, wherein each evaluation metric includes information on the feature scale distribution of the image processing task (t1, t2, t3) associated with the evaluation metric and the matching quality of the receptive field size of the encoder layer of the encoder architecture associated with the evaluation metric (S13). Compare the calculated evaluation metrics and establish the comparison results (S14); and The encoder structure is selected based on the comparison results (S15); Specifically, for the feature scale distribution of each image processing task (t1, t2, t3), one or more percentile values ​​are selected according to the interface of the specific task decoder (3.1, 3.2, 3.3) corresponding to the image processing task, so that the specific task decoder can receive the information required for the specific task decoding operation. The step of calculating multiple evaluation metrics includes: comparing the receptive field size of the encoder layer that matches the resolution of the task-specific decoder (3.1, 3.2, 3.3) with a determined percentile value, and determining the distance between the receptive field size and the percentile value.

2. The method according to claim 1, wherein, The determined number of percentiles is selected based on the number of input connections required by the specific task decoder (3.1, 3.2, 3.3).

3. The method according to claim 1, wherein, The percentile values ​​are distributed within a percentile range defined by the minimum and maximum percentile values.

4. The method according to claim 3, wherein, The percentile values ​​are evenly distributed within the percentile range.

5. The method according to claim 1, wherein, At least some encoder layers of a task-specific branch (b1, b2, b3) of encoder (2) include feature maps whose resolution matches the feature resolution of the input connection of the task-specific decoder (3.1, 3.2, 3.3).

6. The method according to claim 1, wherein, The steps for generating multiple candidate encoder architectures include: selecting the building blocks of the encoder (2) of the appropriate neural network (1) from a set of building blocks, and connecting the selected building blocks to obtain the encoder architecture.

7. The method according to claim 1, wherein, The encoder architecture of the candidate encoder architecture is determined by defining a runtime constraint, and the encoder architecture is constructed so that, given hardware conditions, the encoder architecture provides computational results over a runtime range of 90% to 100% of the runtime constraint.

8. The method according to claim 1, wherein, The steps for calculating multiple evaluation metrics include comparing multiple receptive field sizes of encoder layers that match the resolution of the task-specific decoder (3.1, 3.2, 3.3) with multiple determined percentile values, and determining the sum of the distances between the receptive field sizes and the percentile values.

9. The method according to claim 1 or 8, wherein, The step of calculating multiple evaluation metrics utilizes a loss function with the minimum absolute deviation distance metric, namely, the L1 distance metric or the cosine distance metric.

10. The method according to claim 1, wherein, The step of calculating multiple evaluation metrics utilizes a loss function with a penalty factor, wherein the penalty factor is adapted to increase the evaluation metrics as the number of shared layers among multiple image processing tasks (t1, t2, t3) of the encoder architecture decreases.

11. The method according to claim 1, wherein, The steps for comparing the computed evaluation metrics and establishing the comparison results include: determining the encoder architecture with the lowest evaluation metric for each image processing task (t1, t2, t3).

12. The method according to claim 1, wherein, The steps for selecting an encoder architecture based on the comparison results include: selecting the encoder architecture with the lowest evaluation metric for each image processing task (t1, t2, t3).

13. A computer program product for determining the encoder architecture of a convolutional neural network, wherein, The computer program product includes a computer-readable storage medium containing program instructions, wherein the program instructions are executable by a processor to cause the processor to perform the method according to any of the preceding claims.

Images