Image processing method and system based on branch multi-task neural network structure search

Abstract

The invention discloses an image processing method and system based on branch multi-task neural network structure search, and the method comprises the steps: obtaining image data, and determining a search space which comprises all possible branch structures of a given task number in an encoder network; mask matrixes corresponding to individuals are coded through a matrix gene coding strategy to represent different network structures, and a promising branch multi-task network structure is obtained by adopting an evolutionary operator and an individual fitness evaluation strategy; a high-quality individual group is obtained through a genetic algorithm, and an optimal branch multi-task network is searched by adopting an evolutionary neural network search algorithm; training the searched neural network, and adjusting the specific weight of the task; and an optimal branch multi-task neural network structure is obtained to process the image. The evolutionary computing thought is applied to the network structure search of the branch multi-task network, the algorithm can be effectively helped to find the network structure suitable for all tasks, and it is guaranteed that multiple tasks obtain good results at the same time.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and particularly relates to an image processing method and system based on branch multi-task neural network architecture search. Background Art

[0002] The multi-task learning network architecture (MTL) network architecture can predict all expected tasks of a given input image. Compared with learning tasks separately, MTL has the potential to improve the generalization ability of all tasks in a single model and can reduce latency. The multi-task learning architecture plays a crucial role in multi-task learning. Generally, multi-task learning architectures are usually divided into two categories: hard parameter sharing and soft parameter sharing.

[0003] In the hard parameter sharing method, multiple tasks share the same hidden layer in the early stage, and each task points to its specific output layer. Different from the hard parameter sharing method, the soft parameter sharing method has task-specific networks for each task, sharing parameters or features among parallel tasks. The branch multi-task network follows the hard parameter sharing scheme. Generally speaking, deep neural networks tend to share low-level image features such as edges, corners, etc. in the early processing layers of the network, while deeper layers tend to extract specific high-level features. The branch multi-task network usually starts from multiple shared layers and then assigns different tasks to their own sequences. Going deeper, the branch groups gradually become more specific. However, due to the huge number of possible combinations of network configurations, determining which layers to share and where to branch becomes cumbersome. Existing branch multi-task network architecture design algorithms can be divided into two categories: manual network architecture design and automatic network architecture search. Manual design of the network architecture relies on prior methods to determine the shared layers and branch layers, and determines the task grouping at each branch position based on the measurement of the "task relevance" of the training tasks. As mentioned above, in order to manually design a multi-task network with good results, prior knowledge of the relevant information and constraints between tasks is required. Existing automatic network architecture search usually utilizes reinforcement learning and gradient methods. Reinforcement learning network search searches for a branch multi-task network architecture suitable for the current task by designing a penalty term. The gradient-based method requires certain prior knowledge to design a super network and searches for the optimal network architecture through gradient differentiation. However, experiments show that reinforcement learning-based designs usually require more extensive computing resources than evolution algorithm-based designs. The gradient-based algorithm design requires certain prior knowledge of the training tasks and the network, which may not be mastered by every user interested in multi-task learning. The above two design methods both require in-depth understanding of network design.

[0004] At present, designing a branched multi-task learning framework suitable for the current task needs to solve the following problems: on the one hand, the performance of the multi-task network should be comparable to that of the single-task network. This is not trivial because the interference between tasks may significantly affect the performance of individual tasks. On the other hand, while ensuring task balance as much as possible, the network must have design simplicity. The relative importance of task performance and design simplicity depends on the appropriate network design method, which highlights the necessity of a reasonable search for the multi-task search network structure. Summary of the Invention

[0005] In view of the above existing technical problems, the object of the present invention is to provide an image processing method and system based on the search of the branched multi-task neural network structure. This method applies the idea of evolutionary computation to the search of the network structure of the branched multi-task network, and designs an effective multi-task adaptive weight strategy, which can effectively help the algorithm find a network structure suitable for all tasks and ensure that multiple tasks obtain good results simultaneously.

[0006] The technical solution of the present invention is as follows:

[0007] An image processing method based on the search of the branched multi-task neural network structure, comprising the following steps:

[0008] S01: Obtain image data and determine the search space. The search space includes the branch structures in all possible encoder networks with a given number of tasks;

[0009] S02: Encode the mask matrix corresponding to the individual through the matrix gene encoding strategy to represent different network structures, and use the evolutionary operator and the individual fitness evaluation strategy to obtain a promising branched multi-task network structure;

[0010] S03: Obtain a population of high-quality individuals through the genetic algorithm, and use the evolutionary neural network search algorithm to find the optimal branched multi-task network;

[0011] S04: Train the searched neural network and adjust the task-specific weights;

[0012] S05: Obtain the optimal branched multi-task neural network structure to process the image.

[0013] In the preferred technical solution, the search space in step S01 is described by a directed cyclic graph, where the vertices represent the layers in the network, and the paths represent the routing of data in the network. For an encoder with L1 layers, the graph has a length of L1 and a width of T between consecutive vertices, where T is the given number of tasks.

[0014] In the preferred technical solution, step S02 further includes initializing the population. The method for initializing the population includes:

[0015] S21: Define the population size N1, the length L of the backbone network, and the number M of tasks;

[0016] S22: Randomly generate a matrix, create N1 zero matrices of L×M, generate the initial population, and represent each individual with a matrix;

[0017] S23: Fill the matrix, traverse each element of the matrix, randomly generate an integer between 0 and M - 1, and fill it into the matrix to generate the initial gene encoding, where each number represents a specific layer of a specific task.

[0018] In the preferred technical solution, in step S02, the method for obtaining a promising branch multi-task network structure by using an evolutionary operator and an individual fitness evaluation strategy includes:

[0019] Evaluate the fitness of each individual in the initial population for the parental selection stage. Select parental individuals from the current population Pt through a binary tournament selection operation. Generate offspring individuals from the selected parental individuals through genetic operations. Evaluate the fitness of the offspring individuals Gt using the proposed evaluation strategy. If the maximum number of iterations D is reached, end the evolutionary process;

[0020] Perform environmental selection from the combination of the current population Pt and the new offspring Gt to select the parental individuals for the next generation. Select the optimal individual from the last generation population Pt and map it to the corresponding branch neural network structure to obtain a promising branch multi-task network structure.

[0021] In the preferred technical solution, in step S03, the method for obtaining a population of high-quality individuals by using a genetic algorithm includes:

[0022] Initialize the population;

[0023] Select parent individuals through a binary tournament selection operation. After selecting the parent individuals, perform a crossover operation to generate new individuals, and then perform a mutation operation to obtain offspring; repeat the selection, crossover, and mutation operations until the number of generated offspring individuals reaches the expected size;

[0024] Use the elitist strategy to ensure that the best individuals can be directly selected into the next generation, ensure population convergence, and use the binary tournament algorithm to increase population diversity.

[0025] In the preferred technical solution, in step S04, the method for adjusting task-specific weights includes:

[0026] First, initialize the task weights, and preset a series of initial weights r for each task i based on prior knowledge i ;

[0027] Define a joint loss function according to the goal of multi-task learning where Li is a specific task loss function, θ represents the shared parameters, and N is the number of specific tasks;

[0028] Then, evolutionary computation is carried out. In each generation of the genetic algorithm, the individuals in the population are used for training, and the loss of each task is calculated;

[0029] After each generation of evolution, the weights are adjusted according to the performance of the tasks.

[0030] In the preferred technical solution, the formula for adjusting the weights is:

[0031] |r i,j -r i,j-1 | = m × r i,j-1 × M i,j

[0032]

[0033] where i represents the task number, j represents the evolution number, m is a fixed value that determines the weight change, and M i,j is a defined metric, which is the loss change of the task during evolution, and r i,j is the weight of the i-th task in the j-th evolution, and L ij is the loss function of the i-th task in the j-th evolution.

[0034] The present invention also discloses an image processing system based on the search of a branched multi-task neural network structure, including:

[0035] An image and search space determination module, which acquires image data, determines the search space, and the search space includes the branch structures in all possible encoder networks with a given number of tasks;

[0036] A matrix gene encoding module, which encodes the mask matrix corresponding to the individual through the matrix gene encoding strategy, represents different network structures, and adopts an evolutionary operator and an individual fitness evaluation strategy to obtain a promising branched multi-task network structure;

[0037] A network search module, which obtains a population of high-quality individuals through the genetic algorithm and uses the evolutionary neural network search algorithm to find the optimal branched multi-task network;

[0038] A weight adjustment module, which trains the searched neural network and adjusts the task-specific weights;

[0039] An image processing module, which processes the image with the optimal branched multi-task neural network structure.

[0040] In the preferred technical solution, the matrix gene encoding module further includes initializing the population, and the method for initializing the population includes:

[0041] S21: Define the population size N1, the length L of the backbone network, and the number M of tasks;

[0042] S22: Randomly generate matrices, create N1 zero matrices of L×M, generate the initial population, and represent each individual with a matrix;

[0043] S23: Fill the matrices, traverse each element of the matrices, randomly generate an integer between 0 and M - 1, and fill it into the matrices to generate the initial gene encoding, where each number represents a specific layer for a specific task.

[0044] The present invention also discloses a computer storage medium, on which a computer program is stored, and when the computer program is executed, it implements the above-mentioned image processing method based on the search of the branched multi-task neural network structure.

[0045] Compared with the prior art, the beneficial effects of the present invention are:

[0046] 1. The present invention uses Neural Architecture Search (ENAS) to design a branched multi-task network structure, reducing the dependence on professional knowledge and domain-specific knowledge. Compared with the existing search algorithms based on reinforcement learning, the design based on genetic algorithms performs better in terms of computational resource requirements and search speed. And it can automatically design a branched multi-task learning framework suitable for the current task, solving the complexity and low efficiency problems in manual design. Experimental results show that the proposed algorithm is verified on MobileNetV2 and VGG-16 backbone networks, and its performance is superior to existing automatic design algorithms.

[0047] 2. The method designs an effective multi-task adaptive weight strategy, which helps the algorithm search for a network structure suitable for all tasks, ensuring good results for multiple tasks simultaneously. It can, to a certain extent, solve the problem that the network weights are dominated by a single task, promoting the collaborative optimization of each task.

[0048] 3. The method proposes a matrix gene encoding strategy. Different from traditional fixed-length or variable-length linear encoding strategies, this strategy is specifically designed for the branched multi-task network structure to better represent task sharing and connection methods. And it can effectively represent the changes in the branched multi-task model during the evolution process, which is crucial for automatically designing high-performance neural network structures. An encoding strategy adapted to multi-modal data. This strategy generates the initial population through random initialization and specific rules, helping to solve the complexity problem of determining shared layers and branch layers due to the huge number of network configuration combinations. Through this encoding strategy, the evolutionary algorithm can explore and develop more effectively in the search space, quickly helping the algorithm discover promising branched multi-task network structures, thus evolving better multi-task learning models. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] The present invention will be further described below in conjunction with the accompanying drawings and embodiments:

[0050] Figure 1 It is a flowchart of an image processing method based on the search of a branched multi-task neural network structure for this embodiment;

[0051] Figure 2 It is a schematic block diagram of an image processing system based on the search of a branched multi-task neural network structure for this embodiment;

[0052] Figure 3 It is a working flowchart of an image processing system based on the search of a branched multi-task neural network structure for this embodiment. Specific embodiments

[0053] To make the objectives, technical solutions and advantages of the present invention clearer and more understandable, the present invention will be further described in detail below in conjunction with specific embodiments and with reference to the accompanying drawings. It should be understood that these descriptions are exemplary and are not intended to limit the scope of the present invention. In addition, in the following description, the descriptions of well-known structures and technologies are omitted to avoid unnecessarily confusing the concepts of the present invention.

[0054] Embodiment

[0055] As Figure 1 shown, an image processing method based on the search of a branched multi-task neural network structure includes the following steps:

[0056] S01: Obtain image data, determine the search space, where the search space includes the branch structures in all possible encoder networks with a given number of tasks;

[0057] S02: Encode the mask matrix corresponding to the individual through the matrix gene encoding strategy to represent different network structures, and adopt the evolutionary operator and the individual fitness evaluation strategy to obtain a promising branched multi-task network structure;

[0058] S03: Obtain a population of high-quality individuals through the genetic algorithm, and adopt the evolutionary neural network search algorithm to find the optimal branched multi-task network;

[0059] S04: Train the searched neural network and adjust the task-specific weights;

[0060] S05: Obtain the optimal branched multi-task neural network structure to process the image.

[0061] In a preferred embodiment, the search space in step S01 is described using a directed cyclic graph, where the vertices represent the layers in the network and the paths represent the routing of data in the network. For an encoder with L1 layers, the graph has a length of L1 and a width of T between consecutive vertices, where T is the given number of tasks.

[0062] In a preferred embodiment, the initialization of the population is further included in step S02. The method for initializing the population includes:

[0063] S21: Define the population size N1, the length L of the backbone network, and the number M of tasks;

[0064] S22: Randomly generate a matrix, create N1 zero matrices of L×M, generate the initial population, and represent each individual with a matrix;

[0065] S23: Fill the matrix, traverse each element of the matrix, randomly generate an integer between 0 and M - 1, and fill it into the matrix to generate the initial gene encoding. Each number represents a specific layer of a specific task.

[0066] In a preferred embodiment, an evolutionary operator and an individual fitness evaluation strategy are adopted in step S02. The method for obtaining a promising branch multi-task network structure includes:

[0067] Evaluate the fitness of each individual in the initial population for the parent selection stage. Select parent individuals from the current population Pt through the binary tournament selection operation. Generate offspring individuals from the selected parent individuals through genetic operations. Evaluate the fitness of the offspring individuals Gt using the proposed evaluation strategy. If the maximum number of iterations D is reached, end the evolutionary process;

[0068] Perform environmental selection from the combination of the current population Pt and the new offspring Gt to select the parent individuals for the next generation. Select the optimal individual from the last generation population Pt and map it to the corresponding branch neural network structure to obtain a promising branch multi-task network structure.

[0069] In a preferred embodiment, the method for obtaining a group of high-quality individuals through a genetic algorithm in step S03 includes:

[0070] Initialize the population;

[0071] Select parent individuals through the binary tournament selection operation. After the selected parent individuals are determined, perform crossover operations to generate new individuals, and then perform mutation operations to obtain offspring; Repeat the selection, crossover, and mutation operations until the number of generated offspring individuals reaches the expected size;

[0072] Use the elitist strategy to ensure that the best individuals can be directly selected into the next generation, ensure the convergence of the population, and use the binary tournament algorithm to increase the population diversity.

[0073] In a preferred embodiment, the method for adjusting task-specific weights in step S04 includes:

[0074] First, initialize the task weights, and preset a series of initial weights r for each task i based on prior knowledge i;

[0075] According to the goal of multi-task learning, a joint loss function is defined. where L i is a specific task loss function, θ represents the shared parameters, and N is the number of specific tasks;

[0076] Then, evolutionary computation is carried out. In each generation of the genetic algorithm, the individuals in the population are used for training, and the loss of each task is calculated.

[0077] After each generation of evolution, the weights are adjusted according to the performance of the tasks.

[0078] In a preferred embodiment, the formula for adjusting the weights is:

[0079] |r i,j - r i,j-1 | = m × r i,j-1 × M i,j

[0080]

[0081] where i represents the task number, j represents the evolution number, m is a fixed value determining the weight change, and M i,j is the defined metric, which is the loss change of the task during evolution. r i,j is the weight of the i-th task in the j-th evolution, and L ij is the loss function of the i-th task in the j-th evolution.

[0082] In another embodiment, a computer storage medium stores a computer program, and when the computer program is executed, it implements the above-mentioned image processing method based on the search of the branch multi-task neural network structure. The specific method is the same as the above-described image processing method based on the search of the branch multi-task neural network structure and will not be elaborated here.

[0083] In another embodiment, as Figure 2 shown, an image processing system based on the search of the branch multi-task neural network structure includes:

[0084] An image and search space determination module 10 that obtains image data, determines the search space, and the search space includes the branch structures in all possible encoder networks with a given number of tasks;

[0085] A matrix gene encoding module 20 that encodes the mask matrix corresponding to the individual through the matrix gene encoding strategy, represents different network structures, and adopts an evolutionary operator and an individual fitness evaluation strategy to obtain a promising branch multi-task network structure;

[0086] The network search module 30 obtains a population of high-quality individuals through a genetic algorithm and uses an evolutionary neural network search algorithm to find the optimal branched multi-task network;

[0087] The weight adjustment module 40 trains the searched neural network and adjusts the task-specific weights;

[0088] The image processing module 50 processes the image using the obtained optimal branched multi-task neural network structure.

[0089] The following takes an optimal example to illustrate in detail the working process of the image processing system based on the search of the branched multi-task neural network structure. As Figure 3 shown, it includes the following steps:

[0090] Step 1: Input the data set and perform preprocessing of the data;

[0091] Step 2: Define the search space;

[0092] Step 3: Initialize the population using the matrix gene encoding strategy;

[0093] Step 4: Generate offspring.

[0094] Loop judgment ①: Evaluate the fitness of each individual in the initial population to perform the parent selection stage. Select parent individuals from the current population Pt through the binary tournament selection operation, generate offspring individuals from the selected parent individuals through genetic operations (crossover and mutation), and evaluate the fitness of the offspring individuals Gt using the proposed evaluation strategy. If the maximum iteration number D is reached, end the evolutionary process;

[0095] Step 5: Environmental selection. Perform environmental selection from the combination of the current population Pt and the new offspring Gt to select the parent individuals of the next generation, select the optimal individual from the last generation population Pt, and map it to the corresponding branched neural network structure;

[0096] Step 6: Enter the GA-based evolutionary computation and set predefined stopping conditions;

[0097] Loop judgment ②: First, predefined the population size (N), the length of the task skeleton (L), and the number of tasks (M). Then, perform random initialization of the population and create individuals until the population size reaches the predefined N. Perform selection, crossover, and mutation operations. Introduce diversity in the population until the predefined stopping conditions are met;

[0098] Step 7: Design a matrix gene encoding strategy to encode the mask matrix corresponding to the individuals in the algorithm;

[0099] Step 8: Generate offspring. Select parents, perform crossover operations to generate new individuals, perform mutation operations to obtain offspring, and repeat until the expected number of offspring is reached;

[0100] Step 9: Environmental selection. Use the elitist strategy to ensure population convergence and use the binary tournament algorithm to improve population diversity;

[0101] Step 10: Train the searched neural network;

[0102] Step 11: Adjust task-specific weights;

[0103] Step 12: Output the high-performance branch multi-task network for the current task and end.

[0104] In Step 1, the PASCAL-Context dataset is mainly used. This dataset contains 4,998 training images and 5,105 test images, which are densely annotated and mainly used for semantic segmentation, saliency detection, and surface normal estimation. To accelerate the network structure search, the present invention uses resized input images. Specifically, the resolution of the PASCAL-Context dataset is scaled down to half of the original size. And during training, the input images are enhanced by randomly scaling the image values between 0.5 and 2.0 (in increments of 0.25) and rescaling the image intensity to the range of [-1, 1] to enhance the data.

[0105] Step 2 defines the search space to effectively process multi-task data and defines the encoding space. The search space directly includes the branch structures in all possible encoder networks for a given number of tasks T. This means that the algorithm needs to search in a huge space of possible network structures to find the optimal branch multi-task network structure suitable for the current task. The search space is described using a directed acyclic graph, where vertices represent the layers in the network and paths represent the routing of data in the network. For an encoder with L1 layers, the graph has a length of L1 and a width of T between consecutive vertices. Before the search, a set of random matrix masks are initialized to generate a set of random branch structures. For task t ∈ T, its routing in the supernetwork is determined by the matrix mask. The algorithm defines a complex search space and uses an evolutionary computing strategy to traverse this space to find the optimal branch multi-task network structure.

[0106] The gene encoding strategy adopted in step 3 is used to encode the mask matrix corresponding to an individual. In this strategy, some necessary parameters are predefined first, including the population size (denoted as N1), the length of the initial network (denoted as L), and the number of tasks (denoted as M). Next, the population is randomly initialized to generate individuals until the predefined size N1 is reached. Specifically, each individual is initialized with an L×M zero matrix. Then we generate a random number from (0, M - 1) and iterate through each row of each matrix. Finally, the initialized population P0 is returned.

[0107] It should be noted that in order to prevent the generation of directed closed-loop structures, we will impose some restrictions on the random generation of numbers to prevent the generation of closed-loop multi-task architectures. The gene encoding strategy can flexibly represent various different network structures to adapt to different task requirements. And through random initialization, the diversity of the population can be ensured, providing a broad exploration basis for the subsequent evolutionary process.

[0108] In step 4, to generate offspring, first, parent individuals are selected through the binary tournament selection operation. After the parent individuals are selected, the crossover operation is carried out next. The purpose of the crossover operation is to combine the excellent features of two parent individuals to produce new offspring. Then the mutation operation is performed. The mutation operation is used in the genetic algorithm to introduce new genetic diversity and prevent the algorithm from falling into a local optimum. The selection, crossover, and mutation operations are repeated until the number of generated offspring individuals reaches the expected size, thus completing the formation of a new generation of the population.

[0109] In step 5, environmental selection is carried out from the current generation of the population and the new offspring generated through the crossover and mutation operations. Different selection operations are used in this paper to achieve environmental selection, including the elite strategy and the binary tournament algorithm. The elite strategy ensures that the best individuals can be directly selected into the next generation, thus ensuring the convergence of the population, which helps to increase the diversity of the population. In this paper, the environmental selection step is part of the genetic algorithm framework and is used to automatically design the branch multi-task network structure. Through the carefully designed environmental selection strategy, the algorithm can effectively explore and exploit in the search space and finally find a high-performance branch multi-task model suitable for the current task.

[0110] In step 6, the GA-based evolutionary computation is used to automatically design the branch multi-task network structure. By setting predefined stopping conditions, setting a maximum number of generations or iterations, when this value is reached, the algorithm stops the evolutionary process. If the predefined performance criteria (such as accuracy, the value of the loss function, etc.) are found, the algorithm can terminate in advance. When the performance improvement of the optimal solution in several consecutive generations is very small or there is no improvement, it can be considered that the algorithm has converged, and at this time, the evolution can be stopped. This enables the algorithm to effectively search for the optimal or near-optimal network structure while maintaining high efficiency.

[0111] In step 7, a design matrix gene coding strategy is used to code the mask matrix corresponding to the individuals in the algorithm. First, initialize the population, define the population size (N1), the length of the task backbone (L), and the number of tasks (M); then randomly generate a matrix, create N1 zero matrices of L×M. This step is to generate the initial population, and each individual is represented in this matrix; then fill the matrix, traverse each element of the matrix, randomly generate an integer between 0 and M-1, and fill it into the matrix. This step is to generate the initial gene coding, and each number represents a specific layer of a specific task; finally, perform checks and adjustments.

[0112] Parent selection: Select parent individuals from the current population through the binary tournament strategy. Binary tournament is a common selection mechanism, which randomly selects a pair of individuals for comparison, and the winner becomes the parent.

[0113] Generate a new offspring population from the given population through crossover and mutation operators. The crossover operation can retain favorable genes in the parental individuals and ensure the effectiveness of evolution. The mutation operation helps the random generation of the offspring population, thus ensuring its diversity. Among them, before performing crossover and mutation, it is necessary to pre-define the crossover rate W and the mutation rate V. These two parameters are between (0,1) and are used to control the probabilities of the crossover and mutation operations. For crossover, we randomly generate an integer X between (0, N-1) (where N is the number of rows in the matrix), and let X be the crossover point. Next, a random floating-point number in the range of (0,1) is generated. When the floating-point number is greater than the pre-defined crossover rate W, the rows with row labels greater than X will perform the crossover operation, that is, the corresponding rows will be swapped. For mutation, we randomly generate an integer Y between (0, N-1), and set Y as the mutation point. Next, a random floating-point number between (0,1) will be generated for all matrix elements with row numbers greater than Y. When the floating-point number is greater than the pre-defined mutation rate V, the elements of the matrix will mutate. Finally, repeat the crossover and mutation operations until a sufficient number of offspring individuals are generated to meet the size requirements of the next generation population.

[0114] Environmental selection After the offspring individuals are generated, not all the generated offspring will be included in the next generation. Instead, the next generation population is selected from the parents and offspring through the environmental selection process.

[0115] After the rough and transfer search in the first stage, the optimal individual is obtained, that is, an efficient branch network structure suitable for multi-task learning. When the iteration of the transfer search ends, it enters the training network stage in step 11. In the training stage, the optimal network is trained separately for each sub-dataset.

[0116] Step 11 Adjusting task-specific weights is the key to solving the problem of task gradient conflict in multi-task learning. In multi-task learning, different tasks may have different difficulty levels and performance requirements. Task-specific weights are used to balance the influence of each task on the update of network weights, thereby preventing certain tasks from dominating the entire learning process and ensuring that all tasks can be fully learned. This paper proposes a novel adaptive weight strategy. First, initialize the task weights, and preset a series of initial weights r for each task based on prior knowledge i ; then, according to the goal of multi-task learning, define a joint loss function where L i is the specific task loss function, r i is the specific task weight, and θ represents the shared parameters; then perform evolutionary computation. In each generation of the genetic algorithm, use the individuals (network structures) in the population for training and calculate the loss of each task; after each generation of evolution, adjust the weights according to the performance of the tasks. Tasks with better performance will reduce their weights, while tasks with poorer performance will increase their weights.

[0117] The weight adjustment formula in the present invention is as follows:

[0118] |r i,j -r i,j-1 |=m×r i,j-1 ×M i,j

[0119]

[0120] where i represents the task number, j represents the evolution number, m is a fixed value determining the weight change, M i,j is the defined metric, which is the loss change of the task during evolution, r i,j is the weight of the i-th task at the j-th evolution, and L ij is the loss function of the i-th task at the j-th evolution. Tasks with better performance in evolution will have a corresponding decrease in r i value, while tasks with poorer performance in evolution will have a corresponding increase in r i value.

[0121] By adjusting the weights, it is possible to promote the performance balance between different tasks, prevent certain tasks from dominating the learning process due to large gradients, and the dynamic adjustment of weights allows the algorithm to adapt to the learning progress and difficulty of different tasks, improving the overall efficiency of multi-task learning.

[0122] Finally, obtain and output a high-performance branch multi-task network for the current task and end.

[0123] In the experimental design, the PASCAL-Context dataset was used. This is a dataset widely used in multi-task learning, containing various dense prediction tasks, such as semantic segmentation, human part segmentation, saliency detection, surface normal estimation, and edge detection. Structurally, the experiments were conducted on two different backbone networks, MobileNetV2 and VGG16, to demonstrate the generality of the proposed method. A baseline model was adopted and compared with single-task models and multi-task shared encoder models to eliminate the influence of training setting differences. And metrics such as average task performance degradation (△m) and edge detection (Edge) were adopted to evaluate the performance of the multi-task model. Table 1 shows the breakdown of single-task metrics for the MobileNetV2 backbone on PASCAL-Context.

[0124] Table 1 Experimental setting parameters

[0125]

[0126]

[0127] We compared the proposed framework BMT-ENAS with state-of-the-art methods, including C.S., NDDR-CNN, MTAN, PHN, and various baselines, including single-task and shared bottom. The single-task baselines independently trained each task using task-specific encoders and task-specific heads for each task. The shared bottom baselines simultaneously trained multiple tasks using a shared encoder and separate task-specific heads. The peer competitors used in this experiment can be divided into two categories. One category is manually designed, including C.S., NDDR-CNN. The other type is the state-of-the-art NAS-based multi-task networks in the latest literature.

[0128] In terms of training details, the training and test code for this project was written using PyTorch. We enhanced the input images during training for data augmentation by randomly scaling values between 0.5 and 2.0 (in increments of 0.25), randomly cropping the input size, and randomly horizontally flipping. The image intensity was rescaled to the range of [-1,1]. For semantic segmentation and human part segmentation, we used cross-entropy loss. For saliency detection, we used balanced cross-entropy loss. For normal estimation, we used L1 loss normalized by the unit vector. For edge detection, we used weighted cross-entropy loss. For edge testing, in PASCAL-Context, the weight of positive pixels is 0.95 and the weight of negative pixels is 0.05. The initial r for each task can be found by performing a logarithmic grid search on candidate values using a single-task network. iThe cross mutation rate was set, and an evolutionary process consisting of 100 individuals over 10 generations was designed. After the search, the obtained branched multi-task network was retrained from scratch for 40,000 iterations. The weights of the encoder network were initialized using ImageNet weights. For all operations shared among multiple tasks, we divide the learning rate by the number of shared tasks because these operations receive more updates. Specifically, the task breakdowns of other competitors in VGG16 and MobileNetV2 are shown in Tables 2 and 3 below.

[0129] Table 2 Breakdown of single-task metrics of the VGG16 backbone on PASCAL-Context

[0130] Method SemSeg↑ PartSeg↑ Sal↑ Norm↓ Edge↑ <![CDATA[△ m [%]↑]]> OURS 64.25 59.05 64.99 14.15 68.32 -0.45 W / ores 65.59 58.53 65.88 14.09 67.94 -0.28

[0131] Table 3 Training details

[0132]

[0133]

[0134] The experimental results verified the effectiveness of the algorithm of the present invention on different backbone networks, demonstrating its high performance and task balance in multi-task learning. It is between the shared baseline and the single baseline in terms of multi-task performance, showing good performance. Compared with networks that require a large amount of prior knowledge for manual design, it has significant performance advantages and design simplicity on different backbone networks.

[0135] It should be understood that the above specific embodiments of the present invention are only used for exemplary illustration or explanation of the principles of the present invention, and do not constitute a limitation to the present invention. Therefore, any modifications, equivalent replacements, improvements, etc. made without departing from the spirit and scope of the present invention shall be included within the protection scope of the present invention. In addition, the appended claims of the present invention are intended to cover all variations and modification examples that fall within the scope and boundaries of the appended claims, or equivalent forms of such scope and boundaries.

Claims

1. An image processing method based on the search of a branched multi-task neural network structure, characterized in that It includes the following steps: S01: Obtain image data and determine the search space. The search space includes the branch structures in all possible encoder networks for a given number of tasks; S02: Encode the mask matrix corresponding to an individual through the matrix gene encoding strategy to represent different network structures. Adopt the evolutionary operator and the individual fitness evaluation strategy to obtain a promising branch multi-task network structure; S03: Obtain a population of high-quality individuals through the genetic algorithm, and use the evolutionary neural network search algorithm to find the optimal branch multi-task network; S04: Train the searched neural network and adjust the task-specific weights; S05: Obtain the optimal branch multi-task neural network structure to process the image.

2. The image processing method based on the search of the branched multi-task neural network structure according to claim 1, characterized in that, In step S01, the search space is described using a directed cyclic graph, where vertices represent the layers in the network and paths represent the routing of data in the network. For an encoder with L1 layers, the graph has a length of L1 and a width of T between consecutive vertices, where T is the given number of tasks.

3. The image processing method based on the search of the branched multi-task neural network structure according to claim 1, characterized in that Step S02 also includes initializing the population. The method for initializing the population includes: S21: Define the population size N1, the length L of the backbone network, and the number M of tasks; S22: Randomly generate matrices, create N1 zero matrices of L×M, generate the initial population, and represent each individual with a matrix; S23: Fill the matrices. Traverse each element of the matrices, randomly generate an integer between 0 and M - 1, and fill it into the matrices to generate the initial gene encoding. Each number represents a specific layer for a specific task.

4. The image processing method based on branch multi-task neural network architecture search according to claim 1, wherein, In step S02, the method for adopting the evolutionary operator and the individual fitness evaluation strategy to obtain a promising branch multi-task network structure includes: Evaluate the fitness of each individual in the initial population for the parent selection phase. Select parent individuals from the current population Pt through the binary tournament selection operation. Generate offspring individuals from the selected parent individuals through genetic operations. Evaluate the fitness of the offspring individuals Gt using the proposed evaluation strategy. If the maximum iteration number D is reached, end the evolutionary process; Perform environmental selection from the combination of the current population Pt and the new offspring Gt to select the parent individuals for the next generation. Select the optimal individual from the last generation population Pt and map it to the corresponding branch neural network structure to obtain a promising branch multi-task network structure.

5. The image processing method based on the search of the branch multi-task neural network structure according to claim 1, wherein In step S03, obtaining a population of high-quality individuals through the genetic algorithm includes: Initializing the population; Selecting parent individuals through the binary tournament selection operation. After the selected parent individuals are determined, perform the crossover operation to generate new individuals, and then perform the mutation operation to obtain offspring; Repeat the selection, crossover, and mutation operations until the number of generated offspring individuals reaches the expected size; Use the elitist strategy to ensure that the best individuals can be directly selected into the next generation, ensure the convergence of the population, and use the binary tournament algorithm to increase the population diversity.

6. The image processing method based on branch multi-task neural network architecture search according to claim 5, characterized in that, The method for adjusting the task-specific weights in step S04 includes: First, initialize the task weights, and preset a series of initial weights r for each task i based on prior knowledge i ; Define a joint loss function according to the goal of multi-task learning where L i is the specific task loss function, θ represents the shared parameters, and N is the number of specific tasks; Then perform evolutionary computation. In each generation of the genetic algorithm, use the individuals in the population for training and calculate the loss for each task; After each generation of evolution, adjust the weights according to the performance of the tasks.

7. The image processing method based on branch multi-task neural network structure search according to claim 6, wherein, The formula for adjusting the weights is: |r i,j -r i,j-1 | = m × r i,j-1 × M i,j Among them, i represents the task number, j represents the evolution number, m is a fixed value for determining the weight change, and M i,j is the defined metric, which is the loss change of the task during the evolution process, and r i,j is the weight of the i-th task at the j-th evolution, and L ij is the loss function of the i-th task at the j-th evolution.

8. An image processing system based on branch multi-task neural network architecture search, characterized in that, It includes: An image and search space determination module, which acquires image data and determines a search space. The search space includes branch structures in all possible encoder networks for a given number of tasks; A matrix gene encoding module, which encodes the mask matrix corresponding to an individual through a matrix gene encoding strategy to represent different network structures, and adopts an evolutionary operator and an individual fitness evaluation strategy to obtain a promising branch multi-task network structure; A network search module, which obtains a population of high-quality individuals through a genetic algorithm and uses an evolutionary neural network search algorithm to find an optimal branch multi-task network; A weight adjustment module, which trains the searched neural network and adjusts task-specific weights; An image processing module, which processes an image with the obtained optimal branch multi-task neural network structure.

9. The image processing system based on branch multi-task neural network architecture search according to claim 8, wherein, The matrix gene encoding module further includes initializing a population. The method for initializing the population includes: S21: Define the population size N1, the length L of the backbone network, and the number M of tasks; S22: Randomly generate matrices, create N1 zero matrices of L×M, generate an initial population, and represent each individual with a matrix; S23: Fill the matrices. Traverse each element of the matrices, randomly generate an integer between 0 and M - 1, and fill it into the matrices to generate an initial gene encoding. Each number represents a specific layer for a specific task.

10. A computer storage medium, on which a computer program is stored, characterized in that, When the computer program is executed, it implements the image processing method based on the search of the branch multi-task neural network structure according to any one of claims 1-7.

Images