Backbone network training method, image processing method, device and equipment

Abstract

The invention discloses a training method of a backbone network, an image processing method, an image processing device and equipment. Adjacent neural network layers in the backbone network are associated and have associated weights. A weighting cycle is set, which may include at least one backbone network training cycle. And in the current weight selection period, training the backbone network by adopting the sample data, and recording the weight adjustment cumulant of each weight in the backbone network in the current weight selection period. And selecting a target weight of which the weight adjustment cumulant satisfies a preset condition from the backbone network based on the weight adjustment cumulant of each weight, only adjusting the target weight in the backbone network in a next weight selection period, and completing training of the backbone network in the next weight selection period based on the adjusted target weight. Therefore, only a small number of backbone network weights can be adjusted, and other weights are kept unchanged, so that the training efficiency of the backbone network on the electronic equipment is improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a backbone network training method, image processing method, device and equipment. Background Art

[0002] In the field of computer vision, backbone networks are mainly used to complete feature extraction of images in this field to achieve intensive prediction tasks such as semantic segmentation, depth estimation, edge detection, and key point detection. Currently, backbone networks can be pre-trained on large data sets, but the training efficiency of backbone networks on electronic devices is low, which increases the resource usage of electronic devices. Summary of the invention

[0003] In view of this, the present application provides a backbone network training method, image processing method, device and equipment, which can improve the training efficiency of the backbone network on electronic devices and save the resource occupation of the electronic devices.

[0004] To solve the above problems, the technical solutions provided by this application are as follows:

[0005] In a first aspect, the present application provides a training method for a backbone network, wherein adjacent neural network layers in the backbone network are associated with each other and have associated weights, the method comprising:

[0006] Setting a weight selection cycle; the weight selection cycle includes at least one backbone network training cycle;

[0007] In a current weight selection cycle, the backbone network is trained using sample data, and the accumulated amount of weight adjustment of each weight in the backbone network in the current weight selection cycle is recorded; the sample data includes sample images in the field of computer vision;

[0008] The weight whose accumulated amount of weight adjustment satisfies a preset condition is determined as the target weight in the backbone network;

[0009] The target weights in the backbone network are adjusted, and based on the adjusted target weights, the training of the backbone network in the next weight selection cycle is completed.

[0010] In a second aspect, the present application provides an image processing method, the method comprising:

[0011] Using the backbone network generated by training using the method described in the first aspect to generate an image processing model for the field of computer vision;

[0012] The computer vision image to be processed is input into the image processing model to obtain an image processing result.

[0013] In a third aspect, the present application provides a training device for a backbone network, wherein adjacent neural network layers in the backbone network are associated with each other and have associated weights, and the device comprises:

[0014] A setting unit, used to set a weight selection cycle; the weight selection cycle includes at least one backbone network training cycle;

[0015] A recording unit, used to train the backbone network using sample data in a current weight selection cycle, and record the cumulative amount of weight adjustment of each weight in the backbone network in the current weight selection cycle; the sample data includes sample images in the field of computer vision;

[0016] A determination unit, configured to determine the weight whose weight adjustment cumulative amount satisfies a preset condition as a target weight in the backbone network;

[0017] An adjustment unit is used to adjust the target weight in the backbone network, and complete the training of the backbone network in the next weight selection cycle based on the adjusted target weight.

[0018] In a fourth aspect, the present application provides an image processing device, the device comprising:

[0019] A generating unit, configured to generate an image processing model for the field of computer vision by training a backbone network generated by the method of the first aspect;

[0020] The processing unit is used to input the computer vision image to be processed into the image processing model to obtain an image processing result.

[0021] In a fifth aspect, the present application provides an electronic device, including:

[0022] one or more processors;

[0023] a storage device having one or more programs stored thereon,

[0024] When the one or more programs are executed by the one or more processors, the one or more processors implement any one of the methods described.

[0025] In a sixth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program implements any of the methods described above when executed by a processor.

[0026] It can be seen that this application has the following beneficial effects:

[0027] The present application provides a training method, an image processing method, an apparatus and a device for a backbone network, wherein adjacent neural network layers in the backbone network are associated with each other and have associated weights. During the training process of the backbone network, it is necessary to adjust the weights in the backbone network to improve the performance of the backbone network. In order to improve the training efficiency of the backbone network on the electronic device, in the training method, a weight selection cycle is first set, and the weight selection cycle may include at least one backbone network training cycle. Taking the current weight selection cycle as an example, within the current weight selection cycle, the backbone network is trained using sample data (i.e., sample images in the field of computer vision), that is, the backbone network is trained based on sample data in each backbone network training cycle within the current weight selection cycle. Afterwards, the cumulative amount of weight adjustment of each weight in the backbone network within the current weight selection cycle is recorded. The larger the cumulative amount of weight adjustment, the greater the change in the weight, which can reflect the importance of the weight in the backbone network training process. Based on this, it is determined whether the cumulative amount of weight adjustment of each weight meets the preset conditions, and the weight that meets the preset conditions is selected and determined as the target weight in the backbone network. Then, in the next weight selection cycle, only the target weight in the backbone network is adjusted. Based on the adjusted target weights, the training of the backbone network in the next weight selection cycle is completed.

[0028] It can be seen that, compared with adjusting all weights in the backbone network in each backbone network training cycle, the present application can select the target weight whose cumulative weight adjustment meets the preset conditions from the backbone network based on the cumulative weight adjustment of each weight in the current weight selection cycle, and only adjust the target weight in the next weight selection cycle. In this way, only a small number of backbone network weights can be adjusted, and other weights can be kept unchanged, so as to save the training resources of the electronic equipment for the backbone network and improve the training efficiency of the backbone network on the electronic equipment. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 A schematic diagram of a framework of an exemplary application scenario provided in an embodiment of the present application;

[0030] Figure 2 A flowchart of a backbone network training method provided in an embodiment of the present application;

[0031] Figure 3 A schematic diagram of the structure of a backbone network provided in an embodiment of the present application;

[0032] Figure 4 A schematic diagram of backbone network weight selection and backbone network training provided in an embodiment of the present application;

[0033] Figure 5 A schematic diagram of the structure of a characteristic regulator provided in an embodiment of the present application;

[0034] Figure 6 A flowchart of an image processing method provided in an embodiment of the present application;

[0035] Figure 7 A schematic diagram of the structure of a backbone network training device provided in an embodiment of the present application;

[0036] Figure 8 A schematic diagram of the structure of an image processing device provided in an embodiment of the present application;

[0037] Fig. 9 A schematic diagram of the basic structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0038] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, the embodiments of the present application are further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0039] In order to facilitate the understanding and explanation of the technical solutions provided by the embodiments of the present application, the background technology of the present application will be described below.

[0040] In the field of computer vision, backbone networks can be used to construct large models, such as the visual Transformer model. The visual Transformer model is used to implement dense prediction tasks, which are an important type of tasks in the field of computer vision, such as semantic segmentation, depth estimation, edge detection, and key point detection.

[0041] In the visual Transformer model, the backbone network can be used to complete feature extraction. For example, image feature extraction. In dense prediction tasks, the image can be a medical image, a vehicle condition image, a face image, an autonomous driving scene image, and other application scenarios, which are not limited here.

[0042] Currently, backbone networks can be pre-trained on large data sets, but the training efficiency of backbone networks on electronic devices is low, which increases the resource usage of electronic devices.

[0043] Based on this, the embodiment of the present application provides a training method, an image processing method, an apparatus and a device for a backbone network, wherein adjacent neural network layers in the backbone network are associated with each other and have associated weights. In the training method, a weight selection cycle is first set, and the weight selection cycle may include at least one backbone network training cycle. Taking the current weight selection cycle as an example, in the current weight selection cycle, sample data (i.e., sample images in the field of computer vision) are used to train the backbone network, that is, the backbone network is trained based on sample data in each backbone network training cycle in the current weight selection cycle. Afterwards, the weight adjustment cumulative amount of each weight in the backbone network in the current weight selection cycle is recorded. The larger the weight adjustment cumulative amount, the greater the change in the weight, which can reflect the importance of the weight in the backbone network training process. Based on this, the weight that meets the preset conditions is selected and determined as the target weight in the backbone network. Then, in the next weight selection cycle, only the target weight in the backbone network is adjusted. Based on the adjusted target weight, the training of the backbone network in the next weight selection cycle is completed.

[0044] It can be seen that, compared with adjusting all weights in the backbone network in each backbone network training cycle, the present application can select the target weight whose cumulative weight adjustment meets the preset conditions from the backbone network based on the cumulative weight adjustment of each weight in the current weight selection cycle, and only adjust the target weight in the next weight selection cycle. In this way, only a small number of backbone network weights can be adjusted, and other weights can be kept unchanged, so as to save the training resources of the electronic equipment for the backbone network and improve the training efficiency of the backbone network on the electronic equipment.

[0045] It is understandable that the defects in the above solutions are the results obtained by the applicant after practice and careful research. Therefore, the discovery process of the above problems and the solutions proposed in the embodiments of the present application for the above problems below should be the contributions made by the applicant to the embodiments of the present application during the application process.

[0046] The training method of the backbone network provided in the embodiment of the present application can be applied to computer vision tasks such as semantic segmentation, depth estimation, edge detection, key point detection, image classification, and target detection, and is used to train the backbone network in machine learning models such as semantic segmentation models, depth estimation models, edge detection models, key point detection models, image classification models, and target detection models, so that the backbone network obtained after training can be deployed to lightweight electronic devices. The lightweight electronic device includes a portable device with limited storage space and computing power resources, such as a mobile terminal, an embedded terminal, and other end-side devices. The lightweight electronic device may also include an edge computing device in an edge environment. The edge environment indicates a cluster of edge computing devices that are geographically close to the terminal (i.e., the end-side device) for providing computing, storage, and communication resources. The edge computing device cluster includes one or more edge computing devices, which may be a server, a computing box, and the like.

[0047] In order to facilitate understanding of the backbone network training method provided in the embodiment of the present application, Figure 1 See the example scenario shown. Figure 1 As shown, this figure is a framework diagram of an exemplary application scenario provided in an embodiment of the present application.

[0048] The framework consists of a terminal device 101, a network 102, and a server 103. The terminal device 101 may be a terminal device such as a smart phone, a tablet computer, a laptop computer, a desktop computer, etc., which is not limited here. The network 102 is used to provide a communication link between the terminal device 101 and the server 103, which may be a wired communication link, a wireless communication link, etc., which is not limited here. The server 103 may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, and cloud communications, which is not limited here.

[0049] In actual applications, the terminal device 101 can respond to the user's request operation and send a backbone network training request to the server 103 through the network 102, so that the server 103 completes the backbone network training and deployment by executing the backbone network training method provided in the embodiment of the present application.

[0050] Those skilled in the art will understand that Figure 1 The framework diagram shown is only an example in which the embodiments of the present application can be implemented. The scope of application of the embodiments of the present application is not limited by any aspect of the framework.

[0051] To facilitate understanding of the present application, a backbone network training method provided in an embodiment of the present application is described below in conjunction with the accompanying drawings.

[0052] See also Figure 2 As shown in FIG. 1 , the figure is a flow chart of a backbone network training method provided in an embodiment of the present application, and the method can be applied to electronic devices deployed with a backbone network. Figure 2 As shown, the method may include S201-S204:

[0053] S201: Setting a weight selection cycle; the weight selection cycle includes at least one backbone network training cycle.

[0054] Usually, after the backbone network is constructed and the hyperparameters in the backbone network are determined, the backbone network can be trained. The hyperparameters in the backbone network are not limited here and can be set according to actual conditions. The backbone network is composed of a neural network, and adjacent neural network layers in the backbone network are associated with each other and have a set of associated weights. Specifically, the neurons in each neural network layer are associated with the neurons in the adjacent neural network layer by a weight. For example, the first neuron in the neural network layer of the i-th layer is associated with the first neuron in the neural network layer of the i+1-th layer by a weight, and the remaining neurons are similar, which will not be repeated here. During the training process of the backbone network, the performance of the backbone network is improved by adjusting the weights in the backbone network.

[0055] For example, the weights in the backbone network can be updated by a gradient descent method (such as stochastic gradient descent method SGD). Specifically, each weight in the backbone network is updated based on the gradient descent method. However, the applicant has found that the update method of updating each weight will make the training cost of the electronic device higher and the training efficiency lower.

[0056] Based on this, a weight selection cycle is provided in the embodiment of the present application. When the weight selection cycle is reached, some weights in the backbone network are selected. In the subsequent training process of the backbone network, only the selected weights are adjusted, and the weights of the backbone network that are not selected remain unchanged. In this way, the training cost of the backbone network can be saved and the training efficiency of the backbone network can be improved.

[0057] The weight selection cycle includes at least one backbone network training cycle, and the backbone network training cycle can be understood as the time for each backbone network training. For example, the weight selection cycle may include 20 backbone network training cycles, and after training the backbone network 20 times, one weight selection cycle is reached.

[0058] In a possible implementation, the embodiment of the present application provides a specific implementation method for setting a weight selection period, including:

[0059] A1: Determine the total number of backbone network training cycles and the number of weight selections.

[0060] The total number of backbone network training cycles can be understood as the total number of times the backbone network is trained during the overall training process of the backbone network. The number of weight selections can be understood as the total number of times weights are selected during the overall training process of the backbone network.

[0061] For example, the total number of backbone network training cycles can be 100, that is, after training the backbone network 100 times, the backbone network training is terminated to obtain a trained backbone network. The number of weight selections can be 5 times, that is, during the 100 times of backbone network training, the backbone network weights are selected 5 times, and after each 20 times of training the backbone network, the backbone network weights are selected once.

[0062] A2: Determine the weight selection cycle based on the total number of backbone network training cycles and the number of weight selections.

[0063] For example, the quotient of the total number of backbone network training cycles and the number of weight selection times is determined as the weight selection cycle. For example, the weight selection cycle may be 100 / 5=20 times.

[0064] It is understandable that the backbone network training cycle can be the same as the weight selection cycle. In this case, each time the backbone network is trained, the backbone network weight is selected. It can be seen that the embodiment of the present application does not limit the weight selection cycle, and it can be set according to the actual training requirements of the backbone network.

[0065] S202: In the current weight selection cycle, sample data is used to train the backbone network, and the accumulated amount of weight adjustment of each weight in the backbone network in the current weight selection cycle is recorded; the sample data includes sample images in the field of computer vision.

[0066] The backbone network is trained in each backbone network training cycle until the total number of backbone network training cycles is reached, and the training of the backbone network is terminated. Specifically, the backbone network is trained using sample data. The sample data is a large-scale data set, and the sample data includes sample images in the field of computer vision and sample features of the sample images. The sample image is the input image of the backbone network, and the backbone network is used to extract and output image features in the sample image. The sample feature is the expected output of the backbone network, which is the label value.

[0067] In the current weight selection cycle, the backbone network is trained using sample data. That is, the backbone network is trained in each backbone network training cycle in the current weight selection cycle. Moreover, after the backbone network is trained in each backbone network training cycle, the weight adjustment amount corresponding to each weight in the backbone network can be recorded. Thus, after the backbone network training in the current weight selection cycle is completed, the cumulative amount of weight adjustment for each weight can be counted. Among them, the cumulative amount of weight adjustment of the weight is obtained by accumulating at least one weight adjustment amount, and at least one weight adjustment amount is obtained by training the backbone network through at least one backbone network training cycle, and the at least one backbone network training cycle is a backbone network training cycle in the current weight selection cycle. For example, at least one backbone network training cycle is all backbone network training cycles in the current weight selection cycle.

[0068] As an optional example, if the gradient descent method is used to adjust the weight of the backbone network, the weight adjustment amount corresponding to each weight can be obtained by the gradient descent method, such as Among them, w is the weight, η is the learning rate, and L is the loss function / objective function constructed by training the backbone network. is the gradient of the loss function with respect to the weight, It can be understood as the weight adjustment amount. It can be seen that there is no limitation here, and the weight adjustment amount can also be calculated according to other calculation methods.

[0069] For example, w1 is a weight in the backbone network, and the current weight selection cycle includes 20 backbone network training cycles. Then, after the backbone network training in each backbone network training cycle, the weight adjustment amount of w1 can be calculated based on the gradient descent method. Therefore, the 20 weight adjustment amounts of w1 obtained in the current weight selection cycle are accumulated to obtain the cumulative weight adjustment amount of w1 in the current weight selection cycle.

[0070] It is understandable that after the backbone network training of each backbone network training cycle, the weight adjustment amount corresponding to each weight can be calculated, but it does not mean that the weight is actually adjusted based on the weight adjustment amount. Instead, after a weight selection cycle, some weights are selected from all the weights of the backbone network based on the accumulated weight adjustment amount of each weight, and then the weights are adjusted in the subsequent backbone network training process.

[0071] See also Figure 3 , Figure 3 (a) and Figure 3 (b) are schematic diagrams of the structures of two backbone networks provided in the embodiments of the present application.

[0072] As an optional example, the backbone network is used to build a machine learning model in the field of computer vision. The machine learning model can be an image processing model, a semantic segmentation model, a target detection model, etc., which is not limited here and can be determined according to the specific application scenario in the field of computer vision. Figure 3 As shown in (a), the machine learning model includes a head network and at least one backbone network. The head network can also be called a dense prediction head layer Dense Head, which is used to obtain the prediction results of the model. The internal structure of the head network is not limited here. The number of backbone networks can be N, and the structures of the N backbone networks are the same.

[0073] As an alternative example, Figure 3 As shown in (a), the backbone network includes a first structural block and a second structural block. The first structural block includes a normalization layer Layer Norm and a multi-head attention layer MH-Attention, and the second structural block includes a normalization layer Layer Norm and a multi-layer perceptron MLP.

[0074] In addition, the first structural block and the second structural block of the backbone network may further include a residual layer, and the function of the residual layer is to pass the output of the previous layer in the backbone network backward.

[0075] The sample image is input into the backbone network as an input image, and first passes through the normalization layer and the multi-head attention layer of the first structural block, and then the residual layer inputs the output of the multi-head attention layer and the sample image into the second structural block. In the second structural block, it is first processed by the normalization layer and then by the multi-layer perceptron.

[0076] like Figure 3 As shown in (a), another residual layer in the backbone network will input the output of the multi-layer perceptron, the output of the multi-head attention layer, and the sample image into the head network, so that the head network can perform model prediction and obtain the predicted output. If it is a dense prediction task, the output is specifically the dense prediction output.

[0077] For example, multiple normalization layers, multi-head attention layers, and multi-layer perceptrons can all be composed of neural networks. Therefore, the weights in the backbone network in the embodiment of the present application may refer to the weights in multiple neural network layers within the backbone network, such as the weights in multiple normalization layers, multi-head attention layers, and multi-layer perceptrons.

[0078] In addition, during the training of the backbone network, the head network is trained synchronously. In specific implementation, the weights in the head network are trained synchronously. For example, any weight in the head network can be trained synchronously without a weight selection process.

[0079] S203: Determine the weight whose accumulated weight adjustment amount meets the preset condition as the target weight in the backbone network.

[0080] It can be seen that the larger the weight adjustment amount and the larger the cumulative weight adjustment amount, the greater the change in the weight. The training process of the backbone network includes the forward transfer process and the reverse gradient derivation process. The weight of the backbone network is adjusted according to the constructed objective function. The larger the change in weight, the greater the importance of the weight in the backbone network training process.

[0081] Then, after obtaining the cumulative amount of weight adjustment of each weight in the current weight selection cycle, it is determined whether the cumulative amount of weight adjustment of each weight meets the preset conditions. The weight that meets the preset conditions is determined as the target weight in the backbone network. The number of target weights is one or more. That is, based on the cumulative amount of weight adjustment of the weight (which can reflect the importance of the weight in the backbone network training process) as the selection basis, the weight is selected from the backbone network weight to obtain the target weight. It can be considered that during the training process of the backbone network, the adjustment of the target weight has a greater impact on the performance of the backbone network.

[0082] In a possible implementation, the embodiment of the present application provides a specific implementation method of determining a weight whose weight adjustment cumulative amount satisfies a preset condition as a target weight in a backbone network, including B1-B2:

[0083] B1: Get the target number of weights with the largest cumulative weight adjustment; the target number is the single-cycle weight adjustment number.

[0084] It can be seen that the target number weight with the largest cumulative weight adjustment has a greater impact on the performance of the backbone network. The target number can be expressed as K.

[0085] B2: Determine the target number of weights as the target weights in the backbone network.

[0086] For example, if the target number K is 1, the weight with the largest cumulative weight adjustment is determined as the target weight.

[0087] In a possible implementation, the embodiment of the present application provides a specific implementation method for determining the number of single-cycle weight adjustments in B1, including C1-C3:

[0088] C1: Determine the total weight of the backbone network and the total weight adjustment ratio; the total weight adjustment ratio is the proportion of the number of weights adjusted during the overall training process of the backbone network to the total weight.

[0089] The total amount of weights in the backbone network is related to the internal structure of the backbone network. For example, it is related to multiple normalization layers, multi-head attention layers, and multi-layer perceptrons in the backbone network. After determining the internal structure of the backbone network, the total number of all weights in the backbone network can be determined.

[0090] The number of weights for adjusting weights during the overall training of the backbone network can be set according to actual conditions and is not limited here.

[0091] For example, during the overall training process of the backbone network, the number of weights adjusted is 5, and the total number of weights is 100, then the total weight adjustment ratio is 5%, which is only used as an example for explanation here.

[0092] C2: Determine the total weight adjustment ratio for a single period within the weight selection period based on the number of weight selections and the total weight adjustment ratio.

[0093] For example, the quotient of the total weight adjustment ratio and the number of weight selections is determined as the total weight adjustment ratio for a single cycle within the weight selection cycle. The total weight adjustment ratio for a single cycle represents the proportion of the number of weights selected in the total weights each time the weights are selected.

[0094] For example, if the number of weight selections is 5, the total weight adjustment ratio in a single cycle is 5% / 5=1%.

[0095] C3: Determine the number of single-period weight adjustments within the weight selection period based on the total proportion of single-period weight adjustments and the total weight amount.

[0096] For example, the product of the total weight adjustment ratio of a single cycle and the total weight is determined as the single cycle weight adjustment quantity K in the weight selection cycle. For example, the single cycle weight adjustment quantity K is 1%×100=1.

[0097] Based on C1-C3, it can be seen that if the backbone network weights are selected 5 times during 100 trainings of the backbone network, then the backbone network weights are selected once after each 20 trainings of the backbone network, and each time the backbone network weights are selected, a target weight is selected from all the weights of the backbone network. The target weight can be the backbone network weight with the largest cumulative weight adjustment.

[0098] S204: Adjust the target weights in the backbone network, and complete the training of the backbone network in the next weight selection cycle based on the adjusted target weights.

[0099] After the current weight selection cycle, the target weight is selected, and the target weight is determined to be the trainable backbone network weight, and the other weights are non-trainable backbone network weights. Then, in the next weight selection cycle, only the target weight in the backbone network is adjusted to complete the training of the backbone network in the next weight selection cycle based on the adjusted target weight. For example, the target weight can be adjusted based on the adjustment amount of the target weight, and the adjustment amount of the weight can be obtained according to the gradient descent method.

[0100] When the backbone network is trained in the next weight selection cycle, the next weight selection cycle is the new current weight selection cycle, and a new target weight will be reselected based on the cumulative amount of weight adjustment of each weight in the next weight selection cycle. Then, in the subsequent backbone network training process, the target weight in the previously determined backbone network will be adjusted.

[0101] It can be understood that the target weights in the adjusted backbone network are the target weights determined after all previous weight selection cycles.

[0102] For example, if the current weight selection cycle is the initial weight selection cycle, after the current weight selection cycle, the target weight determined is w1. Then after the next weight selection cycle, the target weight w1 of the backbone network will be trained. After the next weight selection cycle, the target weight determined is w2, and the target weights of the backbone network, i.e. w1 and w2, will continue to be trained in the next weight selection cycle.

[0103] See also Figure 4 , Figure 4 A schematic diagram of backbone network weight selection and backbone network training provided in an embodiment of the present application. Figure 4 As shown, T1 can be 20 backbone network training cycles, T2 can be 40 backbone network training cycles, and T3 can be 60 backbone network training cycles, that is, a target weight is selected from the backbone network weights once every 20 backbone network training cycles, and the number of target weights selected each time is 1.

[0104] The initial weight adjustment period is weight adjustment period T1. During weight adjustment period T1, the backbone network is trained. After reaching T1, the weight adjustment cumulative amount △ is determined. 3 The corresponding weight w3 is determined as the target weight.

[0105] In the weight adjustment period T2, the backbone network is trained to adjust the target weight w3. After reaching T2, the cumulative weight adjustment of each weight in T2 is determined △ 5 The corresponding weight w5 is determined as the target weight.

[0106] In the weight adjustment period T3, the backbone network is trained to adjust the target weights w3 and w5. After reaching T3, the cumulative weight adjustment of each weight in T3 is determined △ 1 The corresponding weight w1 is determined as the target weight. In the next weight adjustment cycle, the backbone network is trained, and the target weights w3, w5, and w1 are adjusted. The subsequent weight adjustment cycles are similar and will not be repeated here.

[0107] In addition, there is no need to train and store a set of backbone network weight parameters for each dense prediction task. Instead, after obtaining the target backbone network based on the backbone network training method provided in the embodiment of the present application, the weight of the target backbone network can be shared among different dense prediction tasks. This not only maintains a high network performance, but also greatly reduces the model storage overhead.

[0108] Based on the relevant contents of S201-S204, it can be known that in the training method of the backbone network provided in the embodiment of the present application, non-trainable weights with higher importance in the backbone network are gradually selected, and the selected non-trainable weights are converted into trainable weights and participate in the subsequent backbone network training. In this way, the backbone network contains trainable weights and non-trainable weights, and only a small number of selected backbone network weights are adjusted, and other weights are kept unchanged, which can save the training resources of the electronic equipment for the backbone network and improve the training efficiency of the backbone network on the electronic equipment.

[0109] As an optional example, the backbone network may further include a feature adapter, which is used to adjust the output of the previous layer network.

[0110] For example, Figure 3 The backbone network shown in (b) includes two feature regulators. Figure 3 As shown in (b), the first structure block and the second structure block of the backbone network can be connected via a feature regulator. The feature regulator is used to adjust the output of the first structure block. Specifically, the feature regulator is connected after the residual layer (the residual layer between the first structure block and the second structure block) to adjust the output of the residual layer, that is, the output feature of the first structure block and the input image.

[0111] like Figure 3 As shown in (b), the second structure block and the head network can be connected through another feature regulator, and the feature regulator is used to adjust the output of the second structure block. Specifically, the feature regulator is connected after another residual layer (the residual layer between the second structure block and the head network) to adjust the output of another residual layer, that is, the output of the multi-layer perceptron, the output of the multi-head attention layer, and the sample image.

[0112] The feature regulator is trainable, and during the training process of the backbone network, the feature regulator is trained synchronously. Specifically, any weight in the feature regulator is trained synchronously, without a weight selection process.

[0113] Taking the feature regulator between the first structural block and the second structural block as an example, it should be understood that after a weight selection cycle, if any weight in the first structural block is not a target weight and is a non-trainable weight, then by connecting a feature regulator with adjustable weights between the first structural block and the second structural block, the feature expression of the backbone network can be further made more accurate by adjusting the weights in the feature regulator. In this way, on the basis of only adjusting the trainable target weights to improve the training efficiency of the backbone network, the feature expression of the backbone network can also be made more accurate by adding a feature regulator to maintain a high performance of intensive prediction tasks.

[0114] In addition, two feature regulators are set behind each structural block in the backbone network, which can improve the performance without increasing the number of parameters to be trained too much.

[0115] See also Figure 5 , Figure 5 A structural diagram of a characteristic regulator provided in an embodiment of the present application is shown in FIG. Figure 5 As shown, the feature conditioner may include a first fully connected layer (such as Figure 5 The fully connected layer 1), the second fully connected layer (such as Figure 5 The first fully connected layer is used to realize feature extraction and feature dimension reduction, i.e., downsampling, and the second fully connected layer is used to realize feature recovery and feature dimension increase, i.e., upsampling. In this structure, the fully connected layer is used to realize feature remapping, and the features are adjusted by adjusting the weights of the fully connected layer to make the feature expression more accurate.

[0116] See also Figure 6 , Figure 6 This is a flow chart of an image processing method provided in an embodiment of the present application. Figure 6 As shown, the embodiment of the present application provides an image processing method, which includes S601-S602:

[0117] S601: Use the backbone network training method described in any of the above embodiments to train a generated backbone network to generate an image processing model for the field of computer vision.

[0118] Among them, the image processing model belongs to the machine learning model.

[0119] S602: Input the computer vision image to be processed into the image processing model to obtain the image processing result.

[0120] It is understandable that the computer vision images to be processed are different depending on the application scenarios in the field of computer vision, which is not limited here.

[0121] Those skilled in the art will appreciate that, in the above method of specific implementation, the order in which the steps are written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of the steps should be determined by their functions and possible internal logic.

[0122] Based on the training method of a backbone network provided in the above method embodiment, the present application embodiment also provides a training device for a backbone network, and the training device for the backbone network will be described below in conjunction with the accompanying drawings. Since the principle of solving the problem by the device in the embodiment of the present application is similar to the training method for the backbone network in the embodiment of the present application, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be repeated.

[0123] See also Figure 7 , which is a schematic diagram of the structure of a backbone network training device provided in an embodiment of the present application. The device can be applied to an electronic device deployed with a backbone network, where adjacent neural network layers in the backbone network are associated with each other and have associated weights, such as Figure 7 As shown, the training device of the backbone network includes:

[0124] A setting unit 701 is used to set a weight selection cycle; the weight selection cycle includes at least one backbone network training cycle;

[0125] A recording unit 702 is used to train the backbone network using sample data in a current weight selection cycle, and record the cumulative amount of weight adjustment of each weight in the backbone network in the current weight selection cycle; the sample data includes sample images in the field of computer vision;

[0126] A determining unit 703 is used to determine the weight whose weight adjustment cumulative amount meets a preset condition as a target weight in the backbone network;

[0127] The adjusting unit 704 is used to adjust the target weight in the backbone network, and complete the training of the backbone network in the next weight selection cycle based on the adjusted target weight.

[0128] In a possible implementation, the weight adjustment cumulative amount is obtained by accumulating at least one weight adjustment amount; and the at least one weight adjustment amount is obtained by training the backbone network through at least one backbone network training cycle.

[0129] In a possible implementation, the determining unit 703 includes:

[0130] An acquisition subunit is used to acquire a target number of weights with the largest cumulative amount of weight adjustment; the target number is a single-cycle weight adjustment number;

[0131] The first determination subunit is used to determine the target number of weights as target weights in the backbone network.

[0132] In a possible implementation, the backbone network includes a first structure block and a second structure block, the first structure block and the second structure block are connected via a feature regulator; the feature regulator is used to adjust the output of the first structure block; during the training process of the backbone network, the feature regulator performs synchronous training;

[0133] The first structural block includes a normalization layer and a multi-head attention layer, and the second structural block includes a normalization layer and a multi-layer perceptron.

[0134] In a possible implementation, the backbone network is used to construct a machine learning model in the field of computer vision; the machine learning model includes a head network and at least one backbone network; during the training process of the backbone network, the head network is synchronously trained;

[0135] The second structure block and the head network are connected via a feature regulator, and the feature regulator is used to adjust the output of the second structure block.

[0136] In one possible implementation, the feature adjuster includes a first fully connected layer, a second fully connected layer, and a residual layer that are interconnected; the first fully connected layer is used to achieve feature dimensionality reduction, and the second fully connected layer is used to achieve feature dimensionality increase.

[0137] In a possible implementation, the setting unit 701 includes:

[0138] A second determination subunit is used to determine the total number of backbone network training cycles and the number of weight selection times;

[0139] A third determining subunit is used to determine a weight selection cycle according to the total number of backbone network training cycles and the number of weight selection times;

[0140] The process of obtaining the single-cycle weight adjustment quantity includes:

[0141] Determine the total weight of the backbone network and the total weight adjustment ratio; the total weight adjustment ratio is the proportion of the number of weights adjusted during the overall training process of the backbone network to the total weight;

[0142] Determining the total weight adjustment ratio of a single cycle within the weight selection cycle according to the number of weight selections and the total weight adjustment ratio;

[0143] The number of single-cycle weight adjustments within the weight selection cycle is determined according to the total proportion of the single-cycle weight adjustments and the total weight amount.

[0144] See also Figure 8 , Figure 8 Schematic diagram of the structure of an image processing device provided in an embodiment of the present application. Figure 8 As shown, the image processing device comprises:

[0145] A generating unit 801 is used to train a generated backbone network using the backbone network training method described in any of the above embodiments to generate an image processing model for the field of computer vision;

[0146] The processing unit 802 is used to input the computer vision image to be processed into the image processing model to obtain an image processing result.

[0147] Based on the implementations provided in the above aspects, this application can also be further combined to provide more implementations.

[0148] It should be noted that the specific implementation of each unit in this embodiment can refer to the relevant description in the above method embodiment. The division of units in the embodiment of the present application is schematic and is only a logical function division. There may be other division methods in actual implementation. The functional units in the embodiment of the present application can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. For example, in the above embodiment, the processing unit and the sending unit can be the same unit or different units. The above integrated units can be implemented in the form of hardware or in the form of software functional units.

[0149] Based on the backbone network training method provided in the above method embodiment, the present application also provides an electronic device, including: one or more processors; a storage device, on which one or more programs are stored, when the one or more programs are executed by the one or more processors, the one or more processors implement the backbone network training method described in any of the above embodiments.

[0150] Reference below Fig. 9, which shows a schematic diagram of the structure of an electronic device 900 suitable for implementing the embodiment of the present application. The terminal device in the embodiment of the present application may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (portable android devices), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs (televisions), desktop computers, etc. Fig. 9 The electronic device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0151] like Fig. 9 As shown, the electronic device 900 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 901, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 902 or a program loaded from a storage device 908 into a random access memory (RAM) 903. In the RAM 903, various programs and data required for the operation of the electronic device 900 are also stored. The processing device 901, the ROM 902, and the RAM 903 are connected to each other via a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0152] Typically, the following devices may be connected to the I / O interface 905: an input device 906 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 907 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 908 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 909. The communication device 909 may allow the electronic device 900 to communicate with other devices wirelessly or by wire to exchange data. Although Fig. 9 The electronic device 900 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0153] In particular, according to an embodiment of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication device 909, or installed from the storage device 908, or installed from the ROM 902. When the computer program is executed by the processing device 901, the above-mentioned functions defined in the method of the embodiment of the present application are executed.

[0154] The electronic device provided in the embodiment of the present application and the training method of the backbone network provided in the above embodiment belong to the same inventive concept. The technical details not fully described in this embodiment can be referred to the above embodiment, and this embodiment has the same beneficial effects as the above embodiment.

[0155] Based on a backbone network training method provided in the above method embodiment, an embodiment of the present application provides a computer-readable medium having a computer program stored thereon, wherein when the program is executed by a processor, the backbone network training method as described in any of the above embodiments is implemented.

[0156] It should be noted that the computer-readable medium mentioned above in the present application may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0157] In some embodiments, the client and the server may communicate using any currently known or future developed network protocol such as HTTP (Hyper Text Transfer Protocol), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

[0158] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0159] The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device executes the backbone network training method.

[0160] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including, but not limited to, object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0161] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0162] The units involved in the embodiments described in this application may be implemented by software or hardware. The name of a unit / module does not limit the unit itself in some cases. For example, a voice data acquisition module may also be described as a "data acquisition module".

[0163] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0164] In the context of the present application, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0165] It should be noted that the various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments, and the same or similar parts between the various embodiments can be referred to each other. For the system or device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part description.

[0166] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0167] It should also be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0168] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0169] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A backbone network training method, characterized in that: Adjacent neural network layers in the backbone network are associated with each other and have associated weights, and the method includes: Setting a weight selection cycle; the weight selection cycle includes at least one backbone network training cycle; In a current weight selection cycle, the backbone network is trained using sample data, and the accumulated amount of weight adjustment of each weight in the backbone network in the current weight selection cycle is recorded; the sample data includes sample images in the field of computer vision; The weight whose accumulated amount of weight adjustment satisfies a preset condition is determined as the target weight in the backbone network; The target weights in the backbone network are adjusted, and based on the adjusted target weights, the training of the backbone network in the next weight selection cycle is completed.

2. The method according to claim 1, characterized in that The weight adjustment cumulative amount is obtained by accumulating at least one weight adjustment amount; and the at least one weight adjustment amount is obtained by training the backbone network through at least one backbone network training cycle.

3. The method according to claim 1 or 2, characterized in that: The step of determining the weight whose weight adjustment cumulative amount satisfies a preset condition as the target weight in the backbone network includes: Obtain the target number of weights with the largest cumulative amount of weight adjustment; the target number is the single-cycle weight adjustment number; The target number of weights are all determined as target weights in the backbone network.

4. The method according to claim 1, characterized in that: The backbone network includes a first structure block and a second structure block, wherein the first structure block and the second structure block are connected via a feature regulator; the feature regulator is used to adjust the output of the first structure block; during the training process of the backbone network, the feature regulator performs synchronous training; The first structural block includes a normalization layer and a multi-head attention layer, and the second structural block includes a normalization layer and a multi-layer perceptron.

5. The method according to claim 4, characterized in that The backbone network is used to construct a machine learning model in the field of computer vision; the machine learning model includes a head network and at least one backbone network; during the training process of the backbone network, the head network is synchronously trained; The second structure block and the head network are connected via a feature regulator, and the feature regulator is used to adjust the output of the second structure block.

6. The method according to claim 4, characterized in that The feature regulator includes a first fully connected layer, a second fully connected layer and a residual layer which are interconnected; the first fully connected layer is used to achieve feature dimensionality reduction, and the second fully connected layer is used to achieve feature dimensionality increase.

7. The method according to claim 3, characterized in that The step of setting a weight selection cycle includes: Determine the total number of backbone network training cycles and the number of weight selections; Determining a weight selection cycle according to the total number of backbone network training cycles and the number of weight selection times; The process of obtaining the single-cycle weight adjustment quantity includes: Determine the total weight of the backbone network and the total weight adjustment ratio; the total weight adjustment ratio is the proportion of the number of weights adjusted during the overall training process of the backbone network to the total weight; Determining the total weight adjustment ratio of a single cycle within the weight selection cycle according to the number of weight selections and the total weight adjustment ratio; The number of single-cycle weight adjustments within the weight selection cycle is determined according to the total proportion of the single-cycle weight adjustments and the total weight amount.

8. An image processing method, characterized in that: The method comprises: Using the backbone network generated by training using the method described in any one of claims 1 to 7 to generate an image processing model for the field of computer vision; The computer vision image to be processed is input into the image processing model to obtain an image processing result.

9. A training device for a backbone network, characterized in that: Adjacent neural network layers in the backbone network are associated with each other and have associated weights, and the device comprises: A setting unit, used to set a weight selection cycle; the weight selection cycle includes at least one backbone network training cycle; A recording unit, used to train the backbone network using sample data in a current weight selection cycle, and record the cumulative amount of weight adjustment of each weight in the backbone network in the current weight selection cycle; the sample data includes sample images in the field of computer vision; A determination unit, configured to determine the weight whose weight adjustment cumulative amount satisfies a preset condition as a target weight in the backbone network; An adjustment unit is used to adjust the target weight in the backbone network, and complete the training of the backbone network in the next weight selection cycle based on the adjusted target weight.

10. An image processing device, characterized in that: The device comprises: A generating unit, configured to generate an image processing model for the field of computer vision by using a backbone network generated by training using the method according to any one of claims 1 to 7; The processing unit is used to input the computer vision image to be processed into the image processing model to obtain an image processing result.

11. An electronic device, characterized in that: include: one or more processors; a storage device having one or more programs stored thereon, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 8.

12. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and when the computer program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

Images