Image processing method, device, equipment, medium and computer program product

Abstract

The present invention relates to the field of image processing technology, and specifically discloses an image processing method, device, equipment, medium and computer program product. By adding a multi-resolution generalization layer for adjusting the size of input image features in a visual perception model, when using multiple resolution image samples to train the visual perception model, a meta-learner is used to generate parameters of the multi-resolution generalization layer according to the image feature size corresponding to the resolution, the corresponding image samples are input into the visual perception model at this time and a perception loss value is calculated, and the parameters of the visual perception model and the parameters of the meta-learner are updated using each perception loss value. The visual perception model is obtained by iterative training, which reduces the optimization difficulty caused by the data distribution difference of image samples with different resolutions, improves the resolution generalization ability of the visual perception model to process input images with different resolutions, and further improves the resolution generalization ability of computer vision to adapt to input images with different resolutions.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to an image processing method, device, equipment, medium and computer program product. Background Art

[0002] Computer vision is an important branch of artificial intelligence technology. One of its main tasks is to obtain corresponding scene information by processing the collected images. It is widely used in autonomous driving, intelligent robots, smart homes, smart terminals, as well as industrial production such as transportation, aerial remote sensing, and industrial product assembly.

[0003] The visual perception model is one of the core technologies of computer vision. The perception ability of the visual perception model determines the visual understanding ability of artificial intelligence. Due to the differences in requirements of different application scenarios, different image samples need to be used when training visual perception models for different application scenarios. For example, for application scenarios with limited computing resources, low-resolution image samples are usually used to train visual perception models, while for perception scenarios of small targets and long-distance targets, high-resolution image samples are used to train visual perception models. This results in that in practical applications, once the visual perception model is deployed, it can only process input images with fixed resolution.

[0004] In order to enable the visual perception model to process input images of different resolutions, it is necessary to use image samples of different resolutions to train the visual perception model. However, due to the large data distribution differences between image samples of different resolutions, the optimization of the visual perception model is extremely difficult and cannot be directly applied to visual perception tasks, thus failing to actually solve the problem of improving the resolution generalization ability of computer vision.

[0005] How to improve the resolution generalization capability of computer vision to adapt to input images of different resolutions is a technical problem that technicians in this field need to solve. Summary of the invention

[0006] The object of the present invention is to provide an image processing method, apparatus, device, medium and computer program product for improving the resolution generalization capability of computer vision to adapt to input images of different resolutions.

[0007] In order to solve the above technical problems, the present invention provides an image processing method, comprising:

[0008] Using an image sample set including image samples of multiple resolutions to train a visual perception model to which a multi-resolution generalization layer is added;

[0009] In iterative training, for each of the resolutions, after updating the parameters of the multi-resolution generalization layer in the visual perception model using a meta-learner corresponding to the resolution, the image samples corresponding to the resolution are input into the visual perception model and the perception loss value is calculated;

[0010] Using the perceptual loss values ​​corresponding to the resolutions, the parameters of the visual perception model and the parameters of the meta-learner corresponding to the resolutions are updated to obtain the visual perception model corresponding to the current iterative training;

[0011] After reaching the iterative training end condition, obtaining the trained visual perception model;

[0012] After using the meta-learner to update the parameters of the multi-resolution generalization layer of the visual perception model according to the resolution corresponding to the visual perception task to be processed, the input image of the visual perception task to be processed is input into the updated visual perception model to obtain a visual perception result of the perception target of the visual perception task to be processed;

[0013] The multi-resolution generalization layer is used to resize the input image features; and the meta-learner is used to generate parameters of the multi-resolution generalization layer according to the corresponding image feature size of the resolution.

[0014] On the one hand, updating the parameters of the multi-resolution generalization layer in the visual perception model using the meta-learner corresponding to the resolution includes:

[0015] The corresponding meta-learner is used to generate parameters of the multi-resolution generalization layer according to the image feature size corresponding to the resolution and the image feature size corresponding to the multi-resolution generalization layer.

[0016] On the other hand, using the corresponding meta learner to generate parameters of the multi-resolution generalization layer according to the image feature size corresponding to the resolution and the image feature size corresponding to the multi-resolution generalization layer, including:

[0017] Generate an image feature vector according to the image feature size corresponding to the resolution and the first image feature size;

[0018] Generating parameters of the multi-resolution generalization layer according to the image feature vector and the image feature size corresponding to the multi-resolution generalization layer;

[0019] The first image feature size is the image feature size after the image sample of the resolution corresponding to the meta-learner is processed by the network layer before the first multi-resolution generalization layer in the visual perception model.

[0020] On the other hand, generating an image feature vector according to the image feature size corresponding to the resolution and the first image feature size includes:

[0021] The reference image feature size, the first image feature size and the second image feature size are concatenated to obtain the image feature vector;

[0022] Among them, the benchmark image feature size is the image feature size of the benchmark image samples in the image sample set after being processed by the visual perception model; the second image feature size is the image feature size of the image samples of the resolution corresponding to the meta-learner after being input into the visual perception model for processing.

[0023] On the other hand, the step of determining the first image feature size comprises:

[0024] After sequentially performing a depth separation convolution operation and a global mean pooling operation on the first image feature, the first image feature size is obtained.

[0025] On the other hand, the calculation step of the multi-resolution generalization layer includes:

[0026] Performing deep feature extraction on input image features and generating attention weights based on the resolution respectively;

[0027] The deep feature extraction results are concatenated with the generated attention weights and then output.

[0028] On the other hand, deep feature extraction is performed on the input image features, including:

[0029] The input image features are sequentially subjected to a first depth separation convolution operation, a normalization operation, and a second depth separation convolution operation to obtain the depth feature extraction result.

[0030] On the other hand, generating attention weights based on the resolution for the input image features includes:

[0031] The input image features are sequentially subjected to global mean pooling operations and linear layer operations to obtain the attention weights.

[0032] On the other hand, updating the parameters of the multi-resolution generalization layer using the meta-learner includes:

[0033] The meta-learner is used to update parameters of the linear layer of the multi-resolution generalization layer.

[0034] On the other hand, updating the parameters of the visual perception model and the parameters of the meta-learner corresponding to each resolution using the perceptual loss value corresponding to each resolution includes:

[0035] The model learning loss value of the visual perception model is calculated by the sum of the perception loss values ​​corresponding to the resolutions;

[0036] The model learning loss value is used to update the parameters of the visual perception model and the parameters of the meta-learner corresponding to each of the resolutions.

[0037] On the other hand, the parameter updating step of the meta-learner includes:

[0038] Before iterative training of the visual perception model, randomly initializing the parameters of the meta-learner corresponding to each of the resolutions;

[0039] In the iterative training of the visual perception model, the parameters of the meta-learner are updated according to the perceptual loss values ​​corresponding to each of the resolutions.

[0040] On the other hand, updating the parameters of the meta-learner according to the perceptual loss values ​​corresponding to the resolutions includes:

[0041] The parameters of the meta-learner are updated according to the current parameters of the meta-learner, the learning rate corresponding to the meta-learner, and the perceptual loss value corresponding to each of the resolutions.

[0042] On the other hand, it also includes:

[0043] The learning rate corresponding to the meta-learner is updated during the iterative training of the visual perception model.

[0044] On the other hand, the parameter updating step of the visual perception model includes:

[0045] Before iterative training of the visual perception model, randomly initializing parameters of the visual perception model;

[0046] In the iterative training of the visual perception model, the parameters of the visual perception model are updated according to the perceptual loss values ​​corresponding to each of the resolutions.

[0047] On the other hand, updating the parameters of the visual perception model according to the perception loss values ​​corresponding to the resolutions includes:

[0048] The parameters of the visual perception model are updated according to the current parameters of the visual perception model, the learning rate corresponding to the visual perception model, and the perceptual loss value corresponding to each of the resolutions.

[0049] On the other hand, it also includes:

[0050] The learning rate corresponding to the visual perception model is updated during the iterative training of the visual perception model.

[0051] In order to solve the above technical problems, the present invention further provides an image processing device, comprising:

[0052] A model training unit is used to train a visual perception model with a multi-resolution generalization layer added using an image sample set including image samples of multiple resolutions; in iterative training, for each resolution, after updating the parameters of the multi-resolution generalization layer in the visual perception model using a meta-learner corresponding to the resolution, input the image samples corresponding to the resolution into the visual perception model and calculate the perceptual loss value; update the parameters of the visual perception model and the parameters of the meta-learner corresponding to each resolution using the perceptual loss value corresponding to each resolution, and obtain the visual perception model corresponding to the current iterative training; after the iterative training end condition is met, obtain the trained visual perception model;

[0053] A model fine-tuning unit, configured to use the meta-learner to update the parameters of the multi-resolution generalization layer of the visual perception model according to the resolution corresponding to the visual perception task to be processed, so as to obtain the updated visual perception model;

[0054] An image processing unit, used for inputting the input image of the visual perception task to be processed into the updated visual perception model to obtain a visual perception result of the perception target of the visual perception task to be processed;

[0055] The multi-resolution generalization layer is used to resize the input image features; and the meta-learner is used to generate parameters of the multi-resolution generalization layer according to the corresponding image feature size of the resolution.

[0056] In order to solve the above technical problems, the present invention further provides an image processing device, comprising:

[0057] Memory for storing computer programs;

[0058] A processor is used to execute the computer program, and when the computer program is executed by the processor, the steps of any one of the image processing methods described above are implemented.

[0059] In order to solve the above technical problem, the present invention also provides a non-volatile storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the image processing method described in any one of the above items are implemented.

[0060] In order to solve the above technical problem, the present invention also provides a computer program product, including a computer program, and when the computer program is executed by a processor, the steps of any one of the above image processing methods are implemented.

[0061] The image processing method provided by the present invention has the beneficial effect of adding a multi-resolution generalization layer for resizing input image features in a visual perception model, and when training the visual perception model using an image sample set including image samples of multiple resolutions, using a meta-learner to generate parameters of the multi-resolution generalization layer according to the image feature size corresponding to the resolution, inputting the image sample corresponding to the resolution into the visual perception model at this time and calculating the perception loss value, and similarly obtaining the perception loss value corresponding to each resolution and using each perception loss value to update the parameters of the visual perception model and the parameters of the meta-learner corresponding to each resolution, so as to obtain the visual perception model corresponding to the current iterative training, thereby performing iterative training to obtain the trained visual perception model, reducing the optimization difficulty caused by the data distribution difference of image samples with different resolutions, thereby improving the resolution generalization ability of the visual perception model to process input images with different resolutions; in the inference process, only the meta-learner needs to be used to update the parameters of the multi-resolution generalization layer of the visual perception model according to the resolution corresponding to the visual perception task to be processed, so as to adapt to the input image resolution of the application scenario, thereby improving the resolution generalization ability of computer vision to adapt to input images with different resolutions.

[0062] The present invention also provides an image processing device, equipment, medium and computer program product, which have the above-mentioned beneficial effects and are not described in detail here. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] In order to more clearly illustrate the embodiments of the present invention or the technical solutions of the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0064] Figure 1 A flowchart of an image processing method provided by an embodiment of the present invention;

[0065] Figure 2 A schematic diagram of the structure of an image processing system provided by an embodiment of the present invention;

[0066] Figure 3 A flowchart of multi-resolution loss optimization provided by an embodiment of the present invention;

[0067] Figure 4 A schematic diagram of the structure of a visual perception model with a multi-resolution generalization layer added provided by an embodiment of the present invention;

[0068] Figure 5 A training flow chart of a visual perception model provided by an embodiment of the present invention;

[0069] Figure 6 A schematic diagram of updating and iterative calculation of a visual perception model based on a meta-learning task provided by an embodiment of the present invention;

[0070] Figure 7 A schematic diagram of the structure of a meta-learner provided by an embodiment of the present invention;

[0071] Figure 8 A schematic diagram of the structure of a parameter predictor provided by an embodiment of the present invention;

[0072] Fig. 9 A schematic diagram of the structure of a multi-resolution generalization layer provided by an embodiment of the present invention;

[0073] Fig.10 A schematic diagram of the structure of an image processing device provided by an embodiment of the present invention;

[0074] Fig.11 A schematic diagram of the structure of an image processing device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0075] The core of the present invention is to provide an image processing method, apparatus, device, medium and computer program product for improving the resolution generalization capability of computer vision to adapt to input images of different resolutions.

[0076] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0077] Resolution is used to describe the clarity of an image, video or display device. It indicates the number of pixels that can be displayed per unit length. Resolution is usually expressed in the following forms:

[0078] Pixel resolution: This is the most intuitive way of expressing it, which is expressed directly in pixels. For example, 1920x1080 means there are 1920 pixels in the horizontal direction (image width) and 1080 pixels in the vertical direction (image height).

[0079] Screen resolution: Usually used to describe the resolution of a monitor, TV or other display device. For example, Full HD usually refers to a resolution of 1920x1080, while 4K resolution usually refers to 3840x2160.

[0080] Print resolution: In the printing field, resolution is usually expressed in dots per inch (DPI). For example, 300 DPI means there are 300 dots per inch.

[0081] Image scanning resolution: The resolution of the scanner is also expressed in DPI, which determines the clarity of the scanned image.

[0082] Video resolution: The resolution of a video is usually expressed in pixels, such as 720p (1280x720), 1080p (1920x1080), and 4K (3840x2160 or 4096x2160).

[0083] The image recognition method provided in the embodiment of the present invention can be used to process images captured by a visual sensor such as a camera, or to process video frames captured from a captured video. In the embodiment of the present invention, the resolution described can be pixel resolution.

[0084] The resolution generalization ability of a visual perception model refers to the ability of the visual perception model to process input images with resolutions that are inconsistent with the training resolution during the testing phase.

[0085] Traditional computer vision solutions usually use the same resolution in the model training and testing stages. This is because the resolution generalization ability of visual perception models is poor. Specifically, when the test resolution is inconsistent with the training resolution, the model accuracy will be seriously reduced. The greater the difference between the test resolution and the training resolution, the more serious the reduction in model accuracy, which limits the application of visual perception models in different scenarios. The fundamental reason for this phenomenon is that the distribution of image data with different resolutions is different, and current visual perception models have difficulty in handling such distribution differences.

[0086] However, in practical applications, the visual perception model after deployment may need to process input images with higher resolutions. For example, a visual perception model trained with a resolution of (256x704) may need to process input images with a resolution of (512x1408) or a resolution of (128x352) after deployment. To meet the application needs of processing images with higher resolutions, one current solution is to adjust the input image to the same size as the resolution of the image sample in the training phase during the application phase, that is, adjust the input image with a resolution of (512x1408) and the input image with a resolution of (128x352) to a resolution of (256x704).

[0087] Although this solution of unifying the resolution of input images is feasible, in some practical application scenarios, there is a high demand for different resolution settings. For some application scenarios with limited computing resources, the visual perception model is required to process low-resolution input images (such as 128x352 resolution in the above example) to reduce the computational burden and ensure real-time performance. If the input image resolution is increased to align it to the training resolution, the computational burden will increase and affect the real-time performance of the system. For some application scenarios with sufficient computing resources, it is hoped that a higher image resolution (such as 512x1408 resolution in the above example) will be used to improve the accuracy of 3D target detection. By reducing the high resolution of the image to align it to the training resolution, a lot of detail information will be lost, which may be detrimental to the detection of small and distant targets.

[0088] It can be seen that in actual computer vision applications, there is still a great demand for the resolution generalization capability of the visual perception model itself. However, the resolution generalization capability of the current visual perception model is very poor and it is difficult to directly apply it to scenes with different resolutions.

[0089] In order to improve the resolution generalization ability of the visual perception model, it is necessary to use image samples of different resolutions to train the visual perception model. However, due to the large differences in the feature distribution of image samples of different resolutions, model optimization is difficult to succeed.

[0090] In order to meet the needs of computer vision applications for processing input images of different resolutions, an embodiment of the present invention provides an image processing solution, which adds a multi-resolution generalization layer for resizing input image features in a visual perception model. When the visual perception model is trained using an image sample set including image samples of multiple resolutions, a meta-learner is used to generate parameters of the multi-resolution generalization layer according to the image feature size corresponding to the resolution, the image sample corresponding to the resolution is input into the visual perception model at this time and a perceptual loss value is calculated. Similarly, a perceptual loss value corresponding to each resolution is obtained and each perceptual loss value is used to update the parameters of the visual perception model and the parameters of the meta-learner corresponding to each resolution, so as to obtain a visual perception model corresponding to the current iterative training. Thus, iterative training is performed to obtain a trained visual perception model, which reduces the optimization difficulty caused by the data distribution difference of image samples of different resolutions, thereby improving the resolution generalization ability of the visual perception model to process input images of different resolutions; in the inference process, only the meta-learner needs to be used to update the parameters of the multi-resolution generalization layer of the visual perception model according to the resolution corresponding to the visual perception task to be processed, so as to adapt to the input image resolution of the application scenario, thereby improving the resolution generalization ability of computer vision to adapt to input images of different resolutions.

[0091] The image processing method provided by the embodiment of the present invention is described below with reference to the accompanying drawings.

[0092] Figure 1 A flowchart of an image processing method provided by an embodiment of the present invention; Figure 2 A schematic diagram of the structure of an image processing system provided by an embodiment of the present invention; Figure 3 A flowchart of multi-resolution loss optimization is provided in an embodiment of the present invention.

[0093] like Figure 1 As shown, the image processing method provided by the embodiment of the present invention includes:

[0094] S101: training a visual perception model with a multi-resolution generalization layer using an image sample set including image samples of multiple resolutions;

[0095] S102: In iterative training, for each resolution, after using a meta-learner corresponding to the resolution to update the parameters of the multi-resolution generalization layer in the visual perception model, the image sample corresponding to the resolution is input into the visual perception model and the perception loss value is calculated;

[0096] S103: using the perceptual loss values ​​corresponding to each resolution to update the parameters of the visual perception model and the parameters of the meta-learner corresponding to each resolution, to obtain the visual perception model corresponding to the current iterative training;

[0097] S104: after reaching the iterative training end condition, obtaining a trained visual perception model;

[0098] S105: after updating the parameters of the multi-resolution generalization layer of the visual perception model according to the resolution corresponding to the visual perception task to be processed by the meta-learner, inputting the input image of the visual perception task to be processed into the updated visual perception model to obtain a visual perception result of the perception target of the visual perception task to be processed;

[0099] Among them, the multi-resolution generalization layer is used to resize the input image features; the meta-learner is used to generate parameters of the multi-resolution generalization layer according to the image feature size of the corresponding resolution.

[0100] The image processing method provided by the embodiment of the present invention mainly includes two major links: training of visual perception model and application of visual perception model. In this regard, the image processing method provided by the embodiment of the present invention can be applied to Figure 2 The image processing system shown includes two parts: a model training system and a perception computing device.

[0101] The model training system may be composed of one or more computing devices for training a visual perception model based on an image sample set composed of image samples of multiple resolutions. Figure 2As shown, the model training system can be composed of multiple threads of multiple artificial intelligence devices to form a thread pool, and the image samples corresponding to each resolution are assigned to a thread pool, so as to achieve parallel accelerated learning of multiple resolution image sets. It should be noted that m, n, and k only represent the meaning of multiple, and do not represent a specific number or the same number of threads of each artificial intelligence processor, nor do they mean that each thread pool is composed of one thread in each artificial intelligence device. The model training system can be deployed in a cloud computing center. The device structure of the cloud computing center is similar Figure 2 The perceptual computing devices in the system include computing, storage, network and other infrastructure, operating system, database, middleware and other software environments and application software. The model training system receives image data sets of various resolutions, uses the stored pre-trained models, combined with manual annotation, to annotate visual perception targets (such as the category, location, size of the bounding box, target orientation information, etc.) and stores the annotated data.

[0102] The perception computing device can be a computing device deployed in a cloud computing center or an edge computing device deployed in an edge device. Its structure is as follows: Figure 2 As shown in the perception computing device shown. If the computing device of the cloud computing center is used, after the trained visual perception model is deployed, the edge device collects the image to be detected through a visual sensor such as a camera and uploads it to the cloud computing center, and uses the perception computing device that deploys the visual perception model to perform visual perception calculation on the image to be detected to obtain the visual perception result of the visual perception target. If the perception computing device uses an edge computing device deployed on the edge device, the visual perception model is trained by the model training system and sent to the edge computing device through the communication unit, and deployed in the memory of the edge computing device. After the edge computing device collects the image to be detected through a visual sensor such as a camera, it performs visual perception calculation through the local visual perception model to obtain the visual perception result of the visual perception target.

[0103] Applied to Figure 2 The multi-thread pool shown, in an embodiment of the present invention, the step of calculating the perceptual loss value of the visual perception model for image samples of each resolution may include: inputting the image samples corresponding to each resolution into different thread pools one by one, so as to obtain the perceptual loss value of the visual perception model for the image samples of each resolution by parallel calculation.

[0104] like Figure 3 As shown, in an embodiment of the present invention, when training a visual perception model, all images of each resolution can be stacked to obtain a set of input images of different resolutions, so that the training speed can be improved through batch processing.

[0105] In some optional implementations of the embodiments of the present invention, using the perceptual loss value corresponding to each resolution to update the parameters of the visual perception model and the parameters of the meta-learner corresponding to each resolution in S103 may include: calculating the model learning loss value of the current iteration number according to the perceptual loss value corresponding to each resolution, and updating the parameters of the visual perception model and the parameters of the meta-learner corresponding to each resolution according to the model learning loss value.

[0106] Among them, calculating the model learning loss value of the current iteration number according to the perceptual loss value corresponding to each resolution may include: taking the average of the loss values ​​calculated by the loss function corresponding to each resolution as the model learning loss value.

[0107] like Figure 3 As shown, for each resolution, all image samples are input into the visual perception model to obtain the corresponding visual perception results, and the loss value of the visual perception model for each resolution compared with the true value (which can be manually labeled) is calculated through the loss function. Finally, the model learning loss value of the visual perception model can be the average of the perceptual loss values ​​of the visual perception model for image samples of all resolutions, as shown in the following formula:

[0108] ;

[0109] in, represents the loss function of the visual perception model, represents the perceptual loss value corresponding to the r-th resolution, represents the loss function corresponding to the rth resolution, and k+1 represents the number of resolutions used in visual perception model training.

[0110] It is very challenging to directly use input images of multiple resolutions to train visual perception models. This is because there are large data distribution differences between input images of different resolutions, which makes it more difficult to optimize the perception model. To solve this problem, an embodiment of the present invention provides a multi-resolution visual perception model based on meta-learning, which uses a meta-learning optimization method to improve the generalization ability of the visual perception model for multiple resolutions.

[0111] Figure 4 A schematic diagram of the structure of a visual perception model with a multi-resolution generalization layer added provided by an embodiment of the present invention; Figure 5 A training flow chart of a visual perception model provided by an embodiment of the present invention; Figure 6 A schematic diagram of updating and iterative calculation of a visual perception model based on a meta-learning task provided in an embodiment of the present invention.

[0112] like Figure 4As shown, the embodiment of the present invention proposes a multi-resolution generalization layer for adjusting the feature size of the input image, and the parameters of the multi-resolution generalization layer are generated by the meta-learner according to the image feature size of the corresponding resolution, so that the multi-resolution generalization layer is used to adjust the image feature size of the input image according to the resolution information of the current input image, so that the visual perception model can adapt to the resolution of the current input image. In the iterative training of the visual perception model, the parameters of the meta-learner corresponding to each resolution are updated, so that the meta-learner also acquires the ability to adjust the visual perception model through continuous learning, so that the visual perception model is more adapted to the resolution of the current input image.

[0113] In an embodiment of the present invention, one or more multi-resolution generalization layers may be added to a traditional visual perception model.

[0114] The network structure of traditional visual perception models usually includes convolutional neural networks (CNN), residual networks (ResNet), etc., among which convolutional neural networks are used to extract image features of input images. Some visual perception models design a multi-layer network structure to continuously extract image features of input images, compress and map image features, etc., to obtain the information required for identifying targets on the input images. Taking the ResNet-50 network as an example, it contains a 50-layer network structure and is a common visual perception model. Applying the image processing method provided in the embodiment of the present invention, such as Figure 4 As shown, the ResNet-50 network is divided into five stages, and the multi-resolution generalization layer provided by the embodiment of the present invention can be added after the first, second, third, and fourth stages respectively.

[0115] It should be noted that the visual perception model can also use models other than the ResNet-50 network, and the location of adding the multi-resolution generalization layer can be set as needed, and does not necessarily have to be added to the Figure 4 Position shown.

[0116] It is understandable that the model structure and model parameters of the multi-resolution generalization layer are related to its position in the visual perception model. Then, in S102, the use of a meta-learner corresponding to the resolution to update the parameters of the multi-resolution generalization layer in the visual perception model may include: using a corresponding meta-learner to generate the parameters of the multi-resolution generalization layer according to the image feature size corresponding to the resolution and the image feature size corresponding to the multi-resolution generalization layer. That is, by designing a meta-learner corresponding to the resolution, when the visual perception model learns the image sample corresponding to the resolution, the parameters of the multi-resolution generalization layer predicted by the meta-learner carry the information of the resolution and the position information of the multi-resolution generalization layer in the visual perception model, so that the image features of the input image of the resolution can be optimized in a targeted manner to solve the problem of difficulty in model optimization caused by large data distribution differences between input images of different resolutions.

[0117] like Figure 5 As shown, the meta-learner corresponding to each resolution can be constructed as a meta-learning task, thereby constructing a multi-resolution meta-learning task. Through cross-task meta-learning training, the multi-resolution generalization ability of the perception model can be improved. Then the number N of multi-resolution meta-learning tasks is equal to the number of resolution types in the image sample set. As introduced in embodiments S102 and S103 of the present invention, in one iterative training of the visual perception model, each multi-resolution meta-learning task corresponds to the visual perception model output perceptual loss value, which is explained as follows: For one of the multi-resolution meta-learning tasks, the meta-learner generates the parameters of the multi-resolution generalization layer according to the image feature size corresponding to the resolution. At this time, Figure 3 As shown in one branch, the visual perception model is used to learn the image samples of the resolution and calculate the perceptual loss value corresponding to the resolution.

[0118] For a multi-resolution meta-learning task, its execution principle can be as follows Figure 6 As shown, the input images are input into the meta-learner respectively, and the meta-learner extracts the image feature size corresponding to the resolution from the input image to update the parameters of the multi-resolution generalization layer in the visual perception model, and obtains the updated visual perception model. The input image is input into the updated visual perception model and the perception result is output. This process can be applied to the training process of the visual perception model, and can also be applied to the model migration process of deploying the visual perception model to the actual application environment, that is, the input image can be an image sample or an input image in the actual application scenario. When the resolution of the input image no longer changes, there is no need to update the parameters of the visual perception model and the parameters of the meta-learner.

[0119] For S103, the perceptual loss value corresponding to each multi-resolution meta-learning task is collected as a basis for updating the parameters of the visual perception model and the meta-learner. It is used to update the model parameters of the visual perception model and the model parameters of the meta-learner during the training process of the visual perception model and the model migration process of deploying the visual perception model to the actual application environment.

[0120] like Figure 2 As shown, in order to accelerate the training of the visual perception model, m meta-learning tasks can be deployed in m thread pools for execution to achieve parallel accelerated training.

[0121] For S104, the iterative training end condition may be that the number of iterative training times of the visual perception model reaches a preset number of iterations, or the model learning loss value of the visual perception model is less than a preset loss value.

[0122] For S105, when the trained visual perception model is deployed in an actual application scenario, the parameters of the multi-resolution generalization layer of the visual perception model are updated according to the resolution corresponding to the visual perception task to be processed, and the visual perception task to be processed is performed using the updated visual perception model. When the resolution of the input image of the actual application scenario changes, the parameters of the multi-resolution generalization layer of the visual perception model are updated again according to the new resolution.

[0123] The image processing method provided by the embodiment of the present invention has the beneficial effect of adding a multi-resolution generalization layer for resizing input image features in a visual perception model, and when training the visual perception model using an image sample set including image samples of multiple resolutions, using a meta-learner to generate parameters of the multi-resolution generalization layer according to the image feature size corresponding to the resolution, inputting the image sample corresponding to the resolution into the visual perception model at this time and calculating the perceptual loss value, and similarly obtaining the perceptual loss value corresponding to each resolution and using each perceptual loss value to update the parameters of the visual perception model and the parameters of the meta-learner corresponding to each resolution, so as to obtain the visual perception model corresponding to the current iterative training, thereby performing iterative training to obtain the trained visual perception model, reducing the optimization difficulty caused by the data distribution difference of image samples with different resolutions, thereby improving the resolution generalization ability of the visual perception model to process input images with different resolutions; in the inference process, only the meta-learner needs to be used to update the parameters of the multi-resolution generalization layer of the visual perception model according to the resolution corresponding to the visual perception task to be processed, so as to adapt to the input image resolution of the application scenario, thereby improving the resolution generalization ability of computer vision to adapt to input images with different resolutions.

[0124] Based on the above embodiment, the embodiment of the present invention further illustrates the steps of collecting the image sample set.

[0125] The image sample set used for training the visual perception model in the embodiment of the present invention includes image samples of multiple resolutions. When collecting the image sample set, a visual sensor such as a camera can be used to directly collect image samples of different resolutions through set resolution parameters to form an image sample set; when collecting the image sample set, image samples of one resolution can also be collected first, and image samples of multiple resolutions can be generated by scaling or enlarging to form an image sample set.

[0126] In some optional implementations of the embodiments of the present invention, the step of collecting the image sample set may include: obtaining an original image set; processing each original image in the original image set using different scaling factors to obtain scaled images of different resolutions corresponding to each original image; and using each original image and the scaled image corresponding to each original image as the image sample set.

[0127] In practical applications, each original image in the original image set is processed with different scaling factors to obtain scaled images of different resolutions corresponding to each original image, which may include: processing each original image with a reference scaling factor to obtain a reference scaled image corresponding to the original image; sampling multiple scaling factors from a preset distribution range, and processing each original image with each scaling factor to obtain scaled images of multiple resolutions corresponding to each original image; using the corresponding reference scaled image and each corresponding scaled image as an image set corresponding to the original image to obtain an image set corresponding to each original image. That is, the generated image sample set is composed of image sets corresponding to different original images, which include image samples of different resolutions, and different image samples of the same resolution may be obtained by processing different original images.

[0128] As described in the above embodiments of the present invention, the resolution used for processing by the visual perception model of computer vision tasks is usually expressed in pixel resolution, that is, expressed as the product of the number of pixels in the horizontal direction and the number of pixels in the vertical direction. In the embodiments of the present invention, the scaling factor may include a scaling factor acting on the image height and / or a scaling factor on the image width. In order to obtain image samples of multiple resolutions with greater resolution differences, the scaling factor may include an image height scaling factor and an image width scaling factor.

[0129] To reduce the computational difficulty, the scaling factor can also be a scaling ratio for the image height and image width. For example, for an original image with a resolution of H×W , the base scaling factor can be used The processed resolution is After the image is distributed in Uniform distribution Sample k scaling factors , and process the image using each scaling factor , and finally obtain a collection of images with multiple resolutions ,in The resolution is . Where H represents the vertical size of the image, and W represents the horizontal size of the image.

[0130] When applied to the field of autonomous driving, cameras and other visual sensors are usually set up in different directions of the vehicle, such as left front (fl), front (f), right front (fr), left rear (bl), rear (b), right rear (br), etc., so that images of multiple perspectives of the road scene are collected each time. In the actual training process, in order to use the batch method to accelerate model training, the embodiment of the present invention samples a set of scaling factors at each iteration, and , the above-obtained scaling coefficient set is used for processing, that is, a multi-resolution image set is obtained for each viewing angle .

[0131] Since the resolution difference between the multi-resolution image samples used in training the visual perception model is small, it is not conducive to the training of the model's resolution generalization ability. In order to obtain as many resolutions with large differences as possible and improve the training effect of the model's resolution generalization ability, in other optional implementations of the embodiments of the present invention, the difference between the scaling factors can be set to be no less than the scaling factor difference threshold, that is, when determining the scaling factor, the difference between the scaling factor and other scaling factors in the collected scaling factor set is judged, and only when the difference is greater than the set scaling factor difference threshold is the scaling factor added to the scaling factor set, thereby ensuring the diversity between the scaling factors obtained by sampling and improving the training effect of the model's resolution generalization ability.

[0132] Furthermore, the scaling factors may be designed to all belong to a preset distribution range and the difference between any two scaling factors may not be less than a scaling factor difference threshold.

[0133] In some optional implementations of the embodiments of the present invention, the scaling factor difference threshold is calculated based on the interval length of the preset distribution range divided by twice the number of scaling factors. That is, the scaling factor difference threshold can be expressed by the following formula:

[0134] ;

[0135] Where M represents the scaling factor difference threshold, Indicates the lower limit of the preset distribution range. represents the upper limit of the preset distribution range, and k represents the total number of scaling factors to be collected from the preset distribution range.

[0136] In some other alternative embodiments of the embodiments of the present invention, for each original image in the original image set, different scaling factors are used for processing to obtain scaled images with different resolutions corresponding to each original image, which may also include: processing each original image with a reference scaling factor to obtain a reference scaled image corresponding to the original image; sampling a plurality of scaling factors from a preset distribution range until a preset number of scaling factors are collected; during the process of collecting scaling factors, for each newly collected scaling factor, calculating the difference between the new scaling factor and the existing scaling factors, if the difference is not less than the scaling factor difference threshold, adding the new scaling factor to the set of existing scaling factors; if the difference is less than the scaling factor difference threshold, discarding the new scaling factor; processing each original image with each scaling factor respectively to obtain scaled images with multiple resolutions corresponding to each original image; using the corresponding reference scaled image and the corresponding scaled images as the image set corresponding to the original image to obtain the image set corresponding to each original image.

[0137] That is, the following algorithm can be designed to obtain the set of scaling factors:

[0138] Input: reference scaling factor , uniformly distributed in , set of scaling factors A, scaling factor difference threshold M;

[0139] Add the reference scaling factor to the set of scaling factors A ;

[0140] If the length of the set of scaling factors A < k + 1, then:

[0141] Randomly sample from the uniform distribution to obtain ;

[0142] Check the difference between and all the scaling factors in the current set of scaling factors A. If the difference is greater than the threshold scaling factor difference M, then add to the set of scaling factors A;

[0143] Output: set of scaling factors A.

[0144] To further improve the ability of the visual perception model to process input images with different resolutions, image sample sets can be generated multiple times during the model training process. That is to say, new image sample sets can be generated by re-sampling scaling factors in each iteration training, or new image sample sets can be generated by re-sampling scaling factors after every few iteration trainings.

[0145] ​In the embodiment of the present invention, since a meta-learner is used to generate parameters of a multi-resolution generalization layer according to resolution so that the visual perception model can acquire the ability to process multi-resolution input images, if the image sample set is regenerated by the above method in each iterative training, it may cause the model to be difficult to converge.

[0146] To ensure model convergence, in some optional implementations of the embodiments of the present invention, a method of generating an image sample set once before training the visual perception model can be adopted, and the image sample set is no longer updated in subsequent iterative training. In this way, the same batch of resolutions are learned in each iterative training of the visual perception model, and the diversity of samples can be ensured by collecting different image samples corresponding to the resolutions. If computing resources and time permit, a method of generating an image sample set once before training the visual perception model can be adopted, but multiple rounds of iterative training are performed, and the image sample set is regenerated before the start of each round of iterative training, and the same image sample set is used in different iterative training steps in each round of iterative training.

[0147] In some other optional implementations of the embodiments of the present invention, the image sample set may be regenerated every multiple iterations of training, and the frequency of generating the image sample set may be determined according to the convergence of the visual perception model.

[0148] In some other optional implementations of the embodiments of the present invention, if the image sample set is regenerated in each iterative training or every few iterative trainings, the multiple resolutions corresponding to the resampling scaling factors can arrange the resolutions in a fixed order (such as from small to large or from large to small) according to the scaling factors, thereby ensuring the correspondence between the arrangement of the resolutions and the arrangement of the loss function, so that the model training can inherit the experience of the previous iterative training or the previous few iterative trainings.

[0149] In each embodiment of the present invention, the loss function corresponding to the resolution of the visual perception model in each iterative training number inherits the parameters of the loss function of the corresponding position in the previous iterative training, but the corresponding resolution can be the resolution in the previous iterative training or the updated resolution.

[0150] On the basis of the above embodiment, in order to adapt to the problem that the multi-resolution model is difficult to optimize, the embodiment of the present invention continues to explain the steps of using a meta-learner to predict the parameters of the multi-resolution generalization layer in the visual perception model.

[0151] Figure 7 A schematic diagram of the structure of a meta-learner provided by an embodiment of the present invention; Figure 8 A schematic diagram of the structure of a parameter predictor provided by an embodiment of the present invention.

[0152] In the above embodiment, the use of a meta-learner corresponding to the resolution in S102 to update the parameters of the multi-resolution generalization layer in the visual perception model may include: using the corresponding meta-learner to generate the parameters of the multi-resolution generalization layer according to the image feature size corresponding to the resolution and the image feature size corresponding to the multi-resolution generalization layer. That is, it is necessary to design the network structure of the meta-learner to generate the parameters of the multi-resolution generalization layer according to the image feature size corresponding to the resolution and the image feature size corresponding to the multi-resolution generalization layer.

[0153] To this end, the embodiment of the present invention provides a structure of a meta-learner as follows: Figure 7 As shown in the figure, the design principle is to first calculate the image feature vector according to the image feature size information of the resolution, and then predict the parameters of each multi-resolution generalization layer according to the image feature size information corresponding to each multi-resolution generalization layer. In order to predict the parameters of each multi-resolution generalization layer, a parameter predictor corresponding to each multi-resolution generalization layer is designed.

[0154] Then in some optional implementations of the embodiments of the present invention, using the corresponding meta-learner to generate the parameters of the multi-resolution generalization layer according to the image feature size corresponding to the resolution and the image feature size corresponding to the multi-resolution generalization layer may include: generating an image feature vector according to the image feature size corresponding to the resolution and the first image feature size; generating the parameters of the multi-resolution generalization layer according to the image feature vector and the image feature size corresponding to the multi-resolution generalization layer; wherein the first image feature size is the image feature size of the image sample of the resolution corresponding to the meta-learner after being processed by the network layer before the first multi-resolution generalization layer in the visual perception model. That is to say, when learning an image sample of a certain resolution, the image feature size information before the first multi-resolution generalization layer in the visual perception model can be predicted by the meta-learner to generate an image feature vector, so that each parameter predictor generates the parameters of the multi-resolution generalization layer according to the image feature vector and the image feature size corresponding to the multi-resolution generalization layer.

[0155] like Figure 7 As shown, generating an image feature vector according to the image feature size corresponding to the resolution and the first image feature size can include: concatenating the reference image feature size, the first image feature size and the second image feature size to obtain the image feature vector; wherein the reference image feature size is the image feature size of the reference image sample in the image sample set after being processed by the visual perception model; and the second image feature size is the image feature size of the image sample of the resolution corresponding to the meta-learner after being input into the visual perception model for processing.

[0156] That is to say, after the input image passes through the network layer before the first multi-resolution generalization layer of the visual perception model to extract image features, the image features are obtained. ,in, represents the number of feature channels, Indicates the feature downsampling multiple, represents the scaling factor corresponding to the i-th resolution, H represents the vertical size of the original image corresponding to the input image, and W represents the horizontal size of the original image.

[0157] In an embodiment of the present invention, the step of determining the first image feature size may include: performing a depth separation convolution operation and a global mean pooling operation on the first image feature in sequence to obtain the first image feature size. Figure 7 As shown in the figure, in the meta-learner, the input image is passed through the network layer before the first multi-resolution generalization layer of the visual perception model to extract the image features. After the deep separation convolution operation and the global mean pooling operation, the image feature size of the image feature obtained at this time is recorded as the first image feature size , and its calculation process is expressed as .in, represents the depthwise separation convolution operation, Represents a global mean pooling operation.

[0158] As described in the above embodiment of the present invention, when generating an image sample set, the original image is first processed with a reference scaling factor to obtain a reference scaling image, which is recorded as the reference image sample. The image feature size of the reference scaling image after being processed by the visual perception model is recorded as the reference image feature size. The image feature size of the image sample corresponding to the resolution after being processed by the visual perception model is .

[0159] If Figure 7 As shown, in the meta-learner, the reference image feature size, the first image feature size, and the second image feature size are concatenated to obtain the image feature vector , can be expressed by the following formula:

[0160] ;

[0161] in, Represents the concatenation function.

[0162] The image feature vector Input each parameter predictor to obtain the parameter prediction results corresponding to each multi-resolution generalization layer .

[0163] like Figure 7 As shown, there are n multi-resolution generalization layers, and n parameter predictors are designed. The structure of a parameter predictor provided by the embodiment of the present invention is as follows Figure 8As shown, the parameter predictor can adopt a linear layer. The i-th parameter predictor is obtained by transforming the image feature vector , benchmark image feature size and the size of the input image features of the i-th multi-resolution generalization layer After splicing, the parameters of the i-th multi-resolution generalization layer are calculated through the linear layer . Then the calculation formula of the i-th parameter predictor can be expressed as follows:

[0164] ;

[0165] in, represents the downsampling multiple of the network layer before the i-th multi-resolution generalization layer in the image feature extraction network of the visual perception model, Represents the overall downsampling multiple of the visual perception model, represents the concatenation function, represents the parameter prediction function of the ith parameter predictor.

[0166] In summary, by introducing the relationship between the image features of the resolution and the reference image features into the parameter predictor, the parameter predictor generates the parameters of the multi-resolution generalization layer according to the resolution information and the position information of the multi-resolution generalization layer in the visual perception model, thereby optimizing the image features of different resolutions and improving the resolution generalization ability of the visual perception model.

[0167] On the basis of the above embodiments, in order to adapt to the problem that the multi-resolution model is difficult to optimize, the embodiment of the present invention continues to explain the structure of the multi-resolution generalization layer.

[0168] Fig. 9 A schematic diagram of the structure of a multi-resolution generalization layer provided in an embodiment of the present invention.

[0169] The function of the multi-resolution generalization layer is to optimize and adjust the image features of input images of different resolutions. In some optional implementations of the embodiments of the present invention, the calculation steps of the multi-resolution generalization layer may include: performing deep feature extraction on the input image features and generating attention weights based on resolution respectively; splicing the deep feature extraction results with the generated attention weights and outputting them. By performing deep feature extraction on the input image features, the hierarchical features of the input image can be further learned. By generating attention weights based on resolution, the generalization ability for different resolutions is improved.

[0170] To this end, the embodiment of the present invention designs a structure of a multi-resolution generalization layer as follows: Fig. 9 shown.

[0171] In the multi-resolution generalization layer, the input image features can be processed through two depth separation convolutions and a multi-resolution normalization layer to achieve deep feature extraction of the input image features. The deep feature extraction of the input image features can include: performing a first depth separation convolution operation, a normalization operation, and a second depth separation convolution operation on the input image features in sequence to obtain a deep feature extraction result.

[0172] like Fig. 9 As shown, in the multi-resolution generalization layer, the input image features can be processed by global mean pooling and linear layer to realize resolution-based attention weight generation. Then, the resolution-based attention weight generation for the input image features can include: performing global mean pooling operations and linear layer operations on the input image features in sequence to obtain attention weights.

[0173] Then, using a meta-learner to update the parameters of the multi-resolution generalization layer may include: using a meta-learner to update the parameters of the linear layer of the multi-resolution generalization layer. That is, only the parameters of the linear layer of the multi-resolution generalization layer can be predicted by the meta-learner, and the parameters of other network layers of the multi-resolution generalization layer can use preset parameters, thereby simplifying calculations and accelerating model convergence.

[0174] In the above embodiment, for the i-th multi-resolution generalization layer, a meta-learner is used to predict its parameters. The structure of the multi-resolution generalization layer introduced in the embodiment of the present invention is applied. In the first calculation branch of the multi-resolution generalization layer, the input image features are , To use the first depth-wise separable convolution Processing, using a multi-resolution normalization layer Perform a normalization operation followed by a second depthwise separable convolution Processing to obtain image features , the process can be expressed by the following formula:

[0175] ;

[0176] in, represents the first depth-wise separation convolution operation, represents the normalization operation, Represents the second depth-wise separable convolution operation.

[0177] In the second computation branch (attention branch) of the multi-resolution generalization layer, global mean pooling is used Processing input image features , followed by a linear layer Processing, get attention weight , the process can be expressed by the following formula:

[0178] ;

[0179] in, represents the global mean pooling calculation, Represents a linear layer computation.

[0180] Finally, the image features are weighted to obtain the image features output by the multi-resolution generalization layer. :

[0181] .

[0182] Based on the method of using a meta-learner to update the parameters of a multi-resolution generalization layer of a visual perception model introduced in the above embodiment, the embodiment of the present invention continues to explain the training steps and reasoning steps of the visual perception model.

[0183] As described in the above embodiments of the present invention, the training task of the visual perception model can be executed in a cloud computing center.

[0184] For example, when applied to the field of autonomous driving, a visual perception model can be trained on collected and labeled road scene image data. The multi-resolution meta-learning optimization method provided by an embodiment of the present invention is used to construct m multi-resolution meta-learning tasks in each training batch. In the i-th meta-learning task, the meta-learner is first used to predict the parameters of the multi-resolution generalization layer. After the parameters of the multi-resolution generalization layer are updated, the updated visual perception model is used to input the image sample corresponding to the i-th resolution to obtain the perception result and the perception loss value is calculated according to the true value. , collect all the perception loss values ​​in the training batch and substitute them into the loss function of the visual perception model to obtain the model learning loss value, and then optimize the model through the back propagation algorithm. Among them, the optimizer can adopt the gradient descent (GD) optimizer, the adaptive moment estimation (Adam) optimizer, the adaptive weight decay regularization (Adam Weight Decay Regularization, AdamW) optimizer, etc. During training, the multi-resolution parallel acceleration method proposed in the present invention is used to improve the training efficiency. After completing the model training, the perception model obtained by training is deployed to the edge computing platform of the unmanned vehicle.

[0185] The image processing method provided by the embodiment of the present invention can be applied to, but not limited to, the field of autonomous driving, and can also be applied to intelligent terminals such as intelligent robots, smart homes, and mobile phones, and can also be applied to visual perception models involved in industrial production fields such as transportation, aerial remote sensing, and industrial product assembly. For example, the visual perception model can be an autonomous driving perception model; for this, the image sample set can be constructed based on image samples collected by vehicle-mounted sensors from different perspectives, and the visual perception model can be trained on the collected and annotated road scene image data set when executing model training in the cloud data center.

[0186] In an embodiment of the present invention, using the perceptual loss value corresponding to each resolution to update the parameters of the visual perception model and the parameters of the meta-learner corresponding to each resolution may include: calculating the model learning loss value of the visual perception model by the sum of the perceptual loss values ​​corresponding to each resolution; and using the model learning loss value to update the parameters of the visual perception model and the parameters of the meta-learner corresponding to each resolution. For example, a loss function of a visual perception model provided in the above embodiment of the present invention may be used: .

[0187] In some optional implementations of the embodiments of the present invention, the parameter updating step of the meta-learner may include: before the iterative training of the visual perception model, randomly initializing the parameters of the meta-learner corresponding to each resolution; during the iterative training of the visual perception model, updating the parameters of the meta-learner according to the perceptual loss value corresponding to each resolution.

[0188] Among them, updating the parameters of the meta-learner according to the perceptual loss values ​​corresponding to each resolution may include: updating the parameters of the meta-learner according to the current parameters of the meta-learner, the learning rate corresponding to the meta-learner, and the perceptual loss values ​​corresponding to each resolution.

[0189] Based on this, the parameter update formula of the meta-learner can be shown as follows:

[0190] ;

[0191] in, represents the updated parameters of the meta-learner, represents the parameters of the meta-learner, represents the learning rate corresponding to the meta-learner, Represents the gradient of the meta-learner parameters calculated using the model learning loss value calculated at the current iteration.

[0192] The learning rate corresponding to the meta-learner may be a fixed parameter or a self-learning parameter. The image processing method provided by the embodiment of the present invention may further include: updating the learning rate corresponding to the meta-learner in the iterative training of the visual perception model.

[0193] In some optional implementations of the embodiments of the present invention, the parameter updating step of the visual perception model may include: before the iterative training of the visual perception model, randomly initializing the parameters of the visual perception model; during the iterative training of the visual perception model, updating the parameters of the visual perception model according to the perceptual loss values ​​corresponding to each resolution.

[0194] Among them, updating the parameters of the visual perception model according to the perceptual loss values ​​corresponding to each resolution may include: updating the parameters of the visual perception model according to the current parameters of the visual perception model, the learning rate corresponding to the visual perception model, and the perceptual loss values ​​corresponding to each resolution.

[0195] Based on this, the parameter update formula of the visual perception model can be shown as follows:

[0196] ;

[0197] in, represents the parameters of the updated visual perception model, represents the parameters of the visual perception model, represents the learning rate of the visual perception model, Represents the gradient of the parameters of the visual perception model calculated using the model learning loss value trained at the current iteration.

[0198] The learning rate corresponding to the visual perception model may be a fixed parameter or a self-learning parameter. The image processing method provided by the embodiment of the present invention may further include: updating the learning rate corresponding to the visual perception model during the iterative training of the visual perception model.

[0199] During the execution of the inference task using the visual perception model, a meta-learner is used to generate parameters of a multi-resolution generalization layer according to the resolution corresponding to the visual perception task to be processed. After updating the parameters of the multi-resolution generalization layer, the updated visual perception model is used to execute the visual perception task to be processed, and the visual perception result of the perception target of the visual perception task to be processed is output.

[0200] For example, when applied in the field of autonomous driving, since the image resolution obtained by the camera sensor of each autonomous driving vehicle is usually fixed, for each autonomous driving vehicle, the parameter update of the multi-resolution generalization layer is performed only once after the visual perception model is deployed. There is no need to perform this step again afterwards, and the visual perception model can be used directly for prediction. Only when the image resolution changes again, the process of updating the parameters of the multi-resolution generalization layer needs to be executed again, so it will not affect the normal reasoning efficiency.

[0201] It should be noted that in the embodiments of the image processing methods of the present invention, some of the steps or features may be ignored or not executed. The hardware or software functional modules divided for the convenience of description are not the only implementation forms of the image processing methods provided in the embodiments of the present invention.

[0202] The above describes in detail various embodiments corresponding to the image processing method. On this basis, the present invention also discloses an image processing apparatus, device, non-volatile storage medium and computer program product corresponding to the above method.

[0203] Fig.10 A schematic diagram of the structure of an image processing device provided by an embodiment of the present invention.

[0204] like Fig.10 As shown, the image processing device provided by the embodiment of the present invention may include:

[0205] The model training unit 1001 is used to train a visual perception model with a multi-resolution generalization layer added using an image sample set including image samples of multiple resolutions; in iterative training, for each resolution, after updating the parameters of the multi-resolution generalization layer in the visual perception model using a meta-learner corresponding to the resolution, input the image samples corresponding to the resolution into the visual perception model and calculate the perceptual loss value; update the parameters of the visual perception model and the parameters of the meta-learner corresponding to each resolution using the perceptual loss value corresponding to each resolution, and obtain the visual perception model corresponding to the current iterative training; after the iterative training end condition is met, obtain the trained visual perception model;

[0206] A model fine-tuning unit 1002 is used to update the parameters of the multi-resolution generalization layer of the visual perception model according to the resolution corresponding to the visual perception task to be processed by using a meta-learner to obtain an updated visual perception model;

[0207] An image processing unit 1003 is used to input the input image of the visual perception task to be processed into the updated visual perception model to obtain a visual perception result of the perception target of the visual perception task to be processed;

[0208] Among them, the multi-resolution generalization layer is used to resize the input image features; the meta-learner is used to generate parameters of the multi-resolution generalization layer according to the image feature size of the corresponding resolution.

[0209] It should be noted that in each implementation of the image processing device provided by the embodiment of the present invention, the division of units is only a logical functional division, and other division methods can be used. The connection method between different units can be electrical, mechanical or other connection methods. The separated units can be located in the same physical location or distributed on multiple network nodes. Each unit can be implemented in the form of hardware or in the form of a software functional unit. That is, part or all of the units provided by the embodiment of the present invention can be selected according to actual needs and the corresponding connection method or integration method can be adopted to achieve the purpose of the embodiment of the present invention.

[0210] Since the embodiments of the apparatus part correspond to the embodiments of the method part, please refer to the description of the embodiments of the method part for the embodiments of the apparatus part, which will not be repeated here.

[0211] Fig.11 A schematic diagram of the structure of an image processing device provided by an embodiment of the present invention.

[0212] like Fig.11 As shown, the image processing device provided by the embodiment of the present invention includes: a memory 1110, used to store a computer program 1111; a processor 1120, used to execute the computer program 1111, and when the computer program 1111 is executed by the processor 1120, the steps of the image processing method provided by any of the above embodiments are implemented.

[0213] Among them, the processor 1120 may include one or more processing cores, such as a 3-core processor, an 8-core processor, etc. The processor 1120 may be implemented in at least one hardware form of digital signal processing (DSP), field-programmable gate array (FPGA), and programmable logic array (PLA). The processor 1120 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a central processing unit (CPU); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 1120 may be integrated with a graphics processing unit (GPU), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 1120 may also include an artificial intelligence (AI) processor, which is used to process computing operations related to machine learning.

[0214] The memory 1110 may include one or more non-volatile storage media, which may be non-transitory. The memory 1110 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In this embodiment, the memory 1110 is at least used to store the following computer program 1111, wherein the computer program 1111, after being loaded and executed by the processor 1120, can implement the relevant steps in the image processing method disclosed in any of the aforementioned embodiments. In addition, the resources stored in the memory 1110 may also include an operating system 1112 and data 1113, etc., and the storage method may be temporary storage or permanent storage. Among them, the operating system 1112 may be Windows or other types of operating systems. Data 1113 may include, but is not limited to, the data involved in the above method.

[0215] In some embodiments, the image processing device may further include a display screen 1130 , a power supply 1140 , a communication interface 1150 , an input / output interface 1160 , a sensor 1170 , and a communication bus 1180 .

[0216] Those skilled in the art will understand that Fig.11 The structure shown in the figure does not constitute a limitation on the image processing device, and may include more or fewer components than those shown in the figure.

[0217] The image processing device provided by the embodiment of the present invention includes a memory and a processor. When the processor executes the program stored in the memory, it can implement the steps of the image processing method provided by the above embodiment, and the effect is the same as above.

[0218] An embodiment of the present invention provides a non-volatile storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the image processing method provided in any one of the above embodiments can be implemented.

[0219] The non-volatile storage medium may include: a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and other media that can store program codes.

[0220] For the introduction of the non-volatile storage medium provided in the embodiment of the present invention, please refer to the above method embodiment, and the effect thereof is the same as the image processing method provided in the embodiment of the present invention, and the present invention will not elaborate on it here.

[0221] An embodiment of the present invention provides a computer program product, including a computer program, which, when executed by a processor, implements the steps of the image processing method provided in any one of the above embodiments.

[0222] For an introduction to the computer program product provided by the embodiment of the present invention, please refer to the above method embodiment, and the effect thereof is the same as the image processing method provided by the embodiment of the present invention, and the present invention will not elaborate on it here.

[0223] The above is a detailed introduction to an image processing method, device, equipment, medium and computer program product provided by the present invention. The various embodiments in the specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same and similar parts between the embodiments can be referenced to each other. For the devices, equipment, non-volatile storage media and computer program products disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple, and the relevant parts can be referred to the method part description. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of the present invention, the present invention can also be improved and modified in a number of ways, and these improvements and modifications also fall within the scope of protection of the present invention.

[0224] It should also be noted that, in this specification, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the statement "comprises a ..." does not exclude the presence of other identical elements in the process, method, article or device including the element.

Claims

1. An image processing method, characterized in that: include: Using an image sample set including image samples of multiple resolutions to train a visual perception model to which a multi-resolution generalization layer is added; In iterative training, for each of the resolutions, after updating the parameters of the multi-resolution generalization layer in the visual perception model using a meta-learner corresponding to the resolution, the image samples corresponding to the resolution are input into the visual perception model and the perception loss value is calculated; Using the perceptual loss values ​​corresponding to the resolutions, the parameters of the visual perception model and the parameters of the meta-learner corresponding to the resolutions are updated to obtain the visual perception model corresponding to the current iterative training; After reaching the iterative training end condition, obtaining the trained visual perception model; After using the meta-learner to update the parameters of the multi-resolution generalization layer of the visual perception model according to the resolution corresponding to the visual perception task to be processed, the input image of the visual perception task to be processed is input into the updated visual perception model to obtain a visual perception result of the perception target of the visual perception task to be processed; The multi-resolution generalization layer is used to resize the input image features; the meta-learner is used to generate parameters of the multi-resolution generalization layer according to the image feature size of the corresponding resolution; Wherein, using the meta-learner corresponding to the resolution to update the parameters of the multi-resolution generalization layer in the visual perception model includes: Generate an image feature vector according to the image feature size corresponding to the resolution and the first image feature size; Generating parameters of the multi-resolution generalization layer according to the image feature vector and the image feature size corresponding to the multi-resolution generalization layer; The first image feature size is the image feature size after the image sample of the resolution corresponding to the meta-learner is processed by the network layer before the first multi-resolution generalization layer in the visual perception model; Wherein, using the perceptual loss value corresponding to each of the resolutions to update the parameters of the visual perception model and the parameters of the meta-learner corresponding to each of the resolutions includes: The model learning loss value of the current iterative training is calculated according to the perceptual loss value corresponding to each of the resolutions; the perceptual loss value is calculated according to the loss function corresponding to the resolution; The model learning loss value is used to update the parameters of the visual perception model and the parameters of the meta-learner corresponding to each of the resolutions.

2. The image processing method according to claim 1, characterized in that: Generating an image feature vector according to the image feature size corresponding to the resolution and the first image feature size includes: The reference image feature size, the first image feature size and the second image feature size are concatenated to obtain the image feature vector; Among them, the benchmark image feature size is the image feature size of the benchmark image samples in the image sample set after being processed by the visual perception model; the second image feature size is the image feature size of the image samples of the resolution corresponding to the meta-learner after being input into the visual perception model for processing.

3. The image processing method according to claim 1, characterized in that: The step of determining the first image feature size comprises: After sequentially performing a depth separation convolution operation and a global mean pooling operation on the first image feature, the first image feature size is obtained.

4. The image processing method according to claim 1, characterized in that: The calculation steps of the multi-resolution generalization layer include: Performing deep feature extraction on input image features and generating attention weights based on the resolution respectively; The deep feature extraction results are concatenated with the generated attention weights and then output.

5. The image processing method according to claim 4, characterized in that: Perform deep feature extraction on the input image features, including: The input image features are sequentially subjected to a first depth separation convolution operation, a normalization operation, and a second depth separation convolution operation to obtain the depth feature extraction result.

6. The image processing method according to claim 4, characterized in that: Generating attention weights based on the resolution for input image features, including: The input image features are sequentially subjected to global mean pooling operations and linear layer operations to obtain the attention weights.

7. The image processing method according to claim 6, characterized in that: Updating the parameters of the multi-resolution generalization layer using the meta-learner includes: The meta-learner is used to update parameters of the linear layer of the multi-resolution generalization layer.

8. The image processing method according to claim 1, characterized in that: Updating the parameters of the visual perception model and the parameters of the meta-learner corresponding to each of the resolutions using the perceptual loss values ​​corresponding to each of the resolutions includes: The model learning loss value is calculated by taking the sum of the perception loss values ​​corresponding to the resolutions; The model learning loss value is used to update the parameters of the visual perception model and the parameters of the meta-learner corresponding to each of the resolutions.

9. The image processing method according to claim 1, characterized in that: The parameter updating step of the meta-learner includes: Before iterative training of the visual perception model, randomly initializing the parameters of the meta-learner corresponding to each of the resolutions; In the iterative training of the visual perception model, the parameters of the meta-learner are updated according to the perceptual loss values ​​corresponding to each of the resolutions.

10. The image processing method according to claim 9, characterized in that: Updating the parameters of the meta-learner according to the perceptual loss values ​​corresponding to the resolutions includes: The parameters of the meta-learner are updated according to the current parameters of the meta-learner, the learning rate corresponding to the meta-learner, and the perceptual loss value corresponding to each of the resolutions.

11. The image processing method according to claim 10, characterized in that: Also includes: The learning rate corresponding to the meta-learner is updated during the iterative training of the visual perception model.

12. The image processing method according to claim 1, characterized in that: The parameter updating step of the visual perception model comprises: Before iterative training of the visual perception model, randomly initializing parameters of the visual perception model; In the iterative training of the visual perception model, the parameters of the visual perception model are updated according to the perceptual loss values ​​corresponding to each of the resolutions.

13. The image processing method according to claim 12, characterized in that: Updating the parameters of the visual perception model according to the perception loss values ​​corresponding to the resolutions includes: The parameters of the visual perception model are updated according to the current parameters of the visual perception model, the learning rate corresponding to the visual perception model, and the perceptual loss value corresponding to each of the resolutions.

14. The image processing method according to claim 13, characterized in that: Also includes: The learning rate corresponding to the visual perception model is updated during the iterative training of the visual perception model.

15. An image processing device, characterized in that: include: A model training unit, for training a visual perception model with a multi-resolution generalization layer added using an image sample set including image samples of multiple resolutions; In the iterative training, for each of the resolutions, after using the meta-learner corresponding to the resolution to update the parameters of the multi-resolution generalization layer in the visual perception model, the image samples corresponding to the resolution are input into the visual perception model and the perceptual loss value is calculated; the perceptual loss value corresponding to each of the resolutions is used to update the parameters of the visual perception model and the parameters of the meta-learner corresponding to each of the resolutions, so as to obtain the visual perception model corresponding to the current iterative training; After reaching the iterative training end condition, obtaining the trained visual perception model; A model fine-tuning unit, configured to use the meta-learner to update the parameters of the multi-resolution generalization layer of the visual perception model according to the resolution corresponding to the visual perception task to be processed, so as to obtain the updated visual perception model; An image processing unit, used for inputting the input image of the visual perception task to be processed into the updated visual perception model to obtain a visual perception result of the perception target of the visual perception task to be processed; The multi-resolution generalization layer is used to resize the input image features; the meta-learner is used to generate parameters of the multi-resolution generalization layer according to the image feature size of the corresponding resolution; Wherein, using the meta-learner corresponding to the resolution to update the parameters of the multi-resolution generalization layer in the visual perception model includes: Generate an image feature vector according to the image feature size corresponding to the resolution and the first image feature size; Generating parameters of the multi-resolution generalization layer according to the image feature vector and the image feature size corresponding to the multi-resolution generalization layer; The first image feature size is the image feature size after the image sample of the resolution corresponding to the meta-learner is processed by the network layer before the first multi-resolution generalization layer in the visual perception model; Wherein, using the perceptual loss value corresponding to each of the resolutions to update the parameters of the visual perception model and the parameters of the meta-learner corresponding to each of the resolutions includes: The model learning loss value of the current iterative training is calculated according to the perceptual loss value corresponding to each of the resolutions; the perceptual loss value is calculated according to the loss function corresponding to the resolution; The model learning loss value is used to update the parameters of the visual perception model and the parameters of the meta-learner corresponding to each of the resolutions.

16. An image processing device, characterized in that: include: Memory for storing computer programs; A processor is used to execute the computer program, and when the computer program is executed by the processor, the steps of the image processing method according to any one of claims 1 to 14 are implemented.

17. A non-volatile storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the image processing method according to any one of claims 1 to 14 are implemented.

18. A computer program product storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the image processing method according to any one of claims 1 to 14 are implemented.

Images