Image recognition method, device, equipment, medium and computer program product

Abstract

The present invention relates to the field of image processing technology, and specifically discloses an image recognition method, device, equipment, medium and computer program product. By adding an image feature adjustment network to a visual perception model, in the process of iteratively training the visual perception model using multi-resolution image samples, the image feature adjustment network is used to adjust the image features of different resolutions to align with the reference image features from the spatial dimension and the channel dimension, and the parameters of the image feature adjustment network are optimized as the visual perception model perceives the loss values ​​of different resolutions. This can not only reduce the interference between different resolutions, but also enable the visual perception model to compensate for the loss of detail information caused by the image feature adjustment during the iterative training process, thereby obtaining the ability to process images with multiple resolutions, and obtaining a visual perception model that can process multi-resolution input images. When performing image recognition tasks, high-precision image recognition results can be guaranteed for images with different resolutions.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to an image recognition method, device, equipment, medium and computer program product. Background Art

[0002] Image recognition is a common application of computer vision. Its main task is to obtain corresponding scene information by performing information recognition on the collected images. It is widely used in autonomous driving, intelligent robots, smart homes, smart terminals, as well as industrial production such as transportation, aerial remote sensing, and industrial product assembly.

[0003] When performing image recognition tasks, the visual perception model determines the accuracy of detection. In practical applications, it is sometimes necessary for the visual perception model to be able to process input images of different resolutions. For example, for application scenarios with limited computing resources, lower-resolution image samples are usually used to train the visual perception model, while for perception scenarios of small targets and long-distance targets, higher-resolution image samples are used to train the visual perception model. However, if the resolution of the input image of the visual perception model in the inference calculation stage is different from the resolution of the image samples in the training stage, the detection accuracy of the visual perception model will be significantly reduced. As a result, in practical applications, once the visual perception model is deployed, it can only process input images with a fixed resolution.

[0004] How to improve the resolution generalization capability of adapting to input images of different resolutions in image recognition tasks of computer vision is a technical problem that technicians in this field need to solve. Summary of the invention

[0005] The purpose of the present invention is to provide an image recognition method, apparatus, device, medium and computer program product for improving the resolution generalization capability of adapting to input images of different resolutions in image recognition tasks of computer vision.

[0006] In order to solve the above technical problems, the present invention provides an image recognition method, comprising:

[0007] A visual perception model for adding an image feature adjustment network is trained using an image sample set including image samples of various resolutions;

[0008] In iterative training, for each resolution, an image sample of the resolution is input into the visual perception model and a perception loss value is calculated, and in the image feature adjustment network, the spatial size of the input image feature is aligned with the reference image feature size, and according to the mapping relationship between the spatial size of the input image feature and the reference image feature size, a channel attention weighted calculation is performed on the input image feature, and the adjusted image feature is output;

[0009] Using the perceptual loss values ​​corresponding to each of the resolutions, the parameters of the visual perception model and the parameters of the image feature adjustment network are updated to obtain the visual perception model corresponding to the current iterative training batch;

[0010] After reaching the iterative training end condition, obtaining the trained visual perception model;

[0011] The input image corresponding to the image recognition task to be processed is input into the trained visual perception model to obtain an image recognition result for the image recognition task to be processed.

[0012] On the one hand, the spatial size of the input image features is aligned with the reference image feature size, including:

[0013] If the spatial size corresponding to the resolution is larger than the reference image feature size, downsampling the input image feature according to the reference image feature size to obtain a downsampled image feature;

[0014] If the spatial size corresponding to the resolution is less than or equal to the reference image feature size, the input image feature is upsampled according to the reference image feature size to obtain an upsampled image feature.

[0015] On the other hand, downsampling the input image features according to the reference image feature size to obtain downsampled image features includes:

[0016] The image feature size of the input image feature is changed to the reference image feature size, and the corresponding mean feature and the corresponding maximum feature are respectively calculated;

[0017] Perform convolution calculation on the mean feature and the maximum feature respectively to obtain a mean convolution feature and a maximum convolution feature;

[0018] The down-sampled image feature is calculated based on the mean convolution feature and the maximum convolution feature.

[0019] On the other hand, the downsampled image feature is calculated according to the mean convolution feature and the maximum convolution feature, including:

[0020] Calculating a downsampled feature size ratio between the image feature size corresponding to the resolution and the reference image feature size;

[0021] Perform feature mapping calculation according to the down-sampling feature size ratio to obtain a down-sampling mean weight and a down-sampling maximum weight;

[0022] Optimizing the mean convolution feature by using the down-sampling mean weight to obtain an optimized mean convolution feature;

[0023] Optimizing the maximum value convolution feature using the down-sampling maximum value weight to obtain an optimized maximum value convolution feature;

[0024] The optimized mean convolution feature and the optimized maximum convolution feature are concatenated along the channel dimension and a fusion convolution calculation is performed to obtain the downsampled image feature.

[0025] On the other hand, the mean convolution feature is optimized by using the down-sampling mean weight to obtain the optimized mean convolution feature, including:

[0026] After calculating the product of the down-sampling mean weight and the mean convolution feature, the sum of the product and the mean feature is taken as the optimized mean convolution feature.

[0027] On the other hand, the maximum value convolution feature is optimized by using the down-sampling maximum value weight to obtain the optimized maximum value convolution feature, including:

[0028] After calculating the product of the down-sampling maximum weight and the maximum convolution feature, the sum of the product and the maximum feature is used as the optimized maximum convolution feature.

[0029] On the other hand, upsampling the input image features according to the reference image feature size to obtain upsampled image features includes:

[0030] Performing interpolation processing on the input image feature to change the image feature size of the input image feature to the reference image feature size, thereby obtaining an interpolation feature;

[0031] Performing convolution calculation on the interpolation feature to obtain an interpolation convolution feature;

[0032] The up-sampled image features are calculated based on the interpolated convolution features.

[0033] On the other hand, the upsampled image feature is calculated according to the interpolated convolution feature, including:

[0034] Calculating an up-sampled feature size ratio between the image feature size corresponding to the resolution and the reference image feature size;

[0035] Performing feature mapping calculation according to the up-sampling feature size ratio to obtain up-sampling interpolation weights;

[0036] Optimizing the interpolation convolution feature by using the upsampling interpolation weight to obtain an optimized interpolation convolution feature;

[0037] The optimized interpolation convolution feature is subjected to up-sampling convolution calculation to obtain the up-sampled image feature.

[0038] On the other hand, the interpolation convolution feature is optimized by using the upsampling interpolation weight to obtain the optimized interpolation convolution feature, including:

[0039] After calculating the product of the up-sampling interpolation weight and the interpolation convolution feature, the sum of the product and the interpolation feature is used as the optimized interpolation convolution feature.

[0040] On the other hand, channel attention weighted calculation is performed on the input image features according to the mapping relationship between the spatial size of the input image features and the reference image feature size, including:

[0041] Performing feature vector mapping processing on the image feature size corresponding to the resolution and the reference image feature size to obtain a feature size feature vector;

[0042] The channel attention weight is calculated based on the image features after spatial size alignment and the feature size feature vector;

[0043] The channel attention weights are used to perform weighted calculation on the image features after the spatial size alignment, and the adjusted image features are output.

[0044] On the other hand, the channel attention weight is calculated based on the image features after spatial size alignment and the feature size feature vector, including:

[0045] Performing global mean pooling processing and global maximum pooling processing on the image features after the spatial size alignment respectively to obtain a global mean pooling feature vector and a global maximum pooling feature vector;

[0046] Concatenating the global mean pooling feature vector, the global maximum pooling feature vector, and the feature size feature vector along the channel dimension to obtain a concatenated feature vector;

[0047] The channel attention weight is calculated according to the concatenated feature vector.

[0048] On the other hand, the channel attention weight is calculated according to the concatenated feature vector, including:

[0049] Processing the concatenated feature vector using a channel attention weight mapping unit to obtain an initial channel attention weight consistent with the image features after the channel dimension is aligned with the spatial dimension;

[0050] The initial channel attention weight is processed by an activation function and a normalization function in turn to obtain the channel attention weight.

[0051] On the other hand, the parameters of the visual perception model and the parameters of the image feature adjustment network are updated using the perceptual loss values ​​corresponding to each of the resolutions to obtain the visual perception model corresponding to the current iterative training batch, including:

[0052] The regularization constraint item corresponding to each of the resolutions is calculated according to the image feature adjustment coefficient of each of the resolutions compared with the reference image feature size;

[0053] According to the perceptual loss value corresponding to each of the resolutions and the regularization constraint item corresponding to each of the resolutions, the parameters of the visual perception model and the parameters of the image feature adjustment network are updated to obtain the visual perception model corresponding to the current iterative training batch.

[0054] On the other hand, the image feature adjustment coefficient corresponding to the resolution makes the transformation of the image feature size of the resolution smaller, and the regularization constraint item corresponding to the resolution approaches 0.

[0055] On the other hand, the image feature adjustment coefficient includes: a spatial size transformation coefficient of the resolution compared to the reference image feature size and a channel attention weight of the resolution corresponding to the reference image feature size;

[0056] Among them, the spatial size transformation coefficient is the downsampling coefficient or upsampling coefficient of the resolution compared to the characteristic size of the benchmark image, the downsampling coefficient is the downsampling mean weight of the resolution compared to the characteristic size of the benchmark image, the downsampling maximum weight of the resolution compared to the characteristic size of the benchmark image, and the upsampling coefficient is the upsampling interpolation weight of the resolution compared to the characteristic size of the benchmark image.

[0057] On the other hand, it also includes:

[0058] The weight coefficient of the regularization constraint term is updated during the iterative training of the visual perception model.

[0059] In order to solve the above technical problems, the present invention further provides an image recognition device, comprising:

[0060] A model training unit is used to train a visual perception model with an image feature adjustment network added using an image sample set including image samples of multiple resolutions; in iterative training, for each resolution, input the image sample of the resolution into the visual perception model and calculate the perception loss value, and in the image feature adjustment network, align the spatial size of the input image feature with the reference image feature size, and perform channel attention weighted calculation on the input image feature according to the mapping relationship between the spatial size of the input image feature and the reference image feature size, and output the adjusted image feature; update the parameters of the visual perception model and the parameters of the image feature adjustment network using the perception loss value corresponding to each resolution, and obtain the visual perception model corresponding to the current iterative training batch; after the iterative training end condition is met, obtain the trained visual perception model;

[0061] The image recognition unit is used to input the input image corresponding to the image recognition task to be processed into the trained visual perception model to obtain the image recognition result of the image recognition task to be processed.

[0062] In order to solve the above technical problems, the present invention further provides an image recognition device, comprising:

[0063] Memory for storing computer programs;

[0064] A processor is used to execute the computer program, and when the computer program is executed by the processor, the steps of any one of the image recognition methods described above are implemented.

[0065] In order to solve the above technical problems, the present invention also provides a non-volatile storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the image recognition method as described in any one of the above items are implemented.

[0066] In order to solve the above technical problems, the present invention also provides a computer program product, including a computer program, which implements the steps of any one of the above-mentioned image recognition methods when executed by a processor.

[0067] The image recognition method provided by the present invention has the beneficial effect of adding an image feature adjustment network to a visual perception model. In the process of iteratively training the visual perception model using multi-resolution image samples, the image feature adjustment network is used to adjust the image features of different resolutions to align with the reference image features in terms of spatial dimension and channel dimension, and the parameters of the image feature adjustment network are optimized as the visual perception model perceives the loss values ​​of different resolutions. This not only reduces the interference between different resolutions, but also enables the visual perception model to compensate for the loss of detail information caused by such image feature adjustment during iterative training, thereby obtaining the ability to process images of multiple resolutions, and obtaining a visual perception model that can process multi-resolution input images, which can ensure high-precision image recognition results for input images of different resolutions when performing image recognition tasks.

[0068] The present invention also provides an image recognition device, equipment, medium and computer program product, which have the above-mentioned beneficial effects and are not described in detail here. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] In order to more clearly illustrate the embodiments of the present invention or the technical solutions of the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0070] Figure 1 A flowchart of an image recognition method provided by an embodiment of the present invention;

[0071] Figure 2 A schematic diagram of the structure of an image recognition system provided by an embodiment of the present invention;

[0072] Figure 3 A flowchart of multi-resolution loss optimization provided by an embodiment of the present invention;

[0073] Figure 4 A schematic diagram of the structure of a visual perception model with an image feature adjustment network added provided by an embodiment of the present invention;

[0074] Figure 5 A training flow chart of a visual perception model provided by an embodiment of the present invention;

[0075] Figure 6 A flowchart of an image feature downsampling process provided by an embodiment of the present invention;

[0076] Figure 7 A flowchart of an image feature upsampling process provided by an embodiment of the present invention;

[0077] Figure 8 A flowchart of image feature channel attention weighted calculation provided by an embodiment of the present invention;

[0078] Fig. 9 A schematic diagram of the structure of an image recognition device provided by an embodiment of the present invention;

[0079] Fig.10 A schematic diagram of the structure of an image recognition device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0080] The core of the present invention is to provide an image recognition method, apparatus, device, medium and computer program product for improving the resolution generalization capability of adapting to input images of different resolutions in image recognition tasks of computer vision.

[0081] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0082] Resolution is used to describe the clarity of an image, video or display device. It indicates the number of pixels that can be displayed per unit length. Resolution is usually expressed in the following forms:

[0083] Pixel resolution: This is the most intuitive way of expressing it, which is expressed directly in pixels. For example, 1920x1080 means there are 1920 pixels in the horizontal direction (image width) and 1080 pixels in the vertical direction (image height).

[0084] Screen resolution: Usually used to describe the resolution of a monitor, TV or other display device. For example, Full HD usually refers to a resolution of 1920x1080, while 4K resolution usually refers to 3840x2160.

[0085] Print resolution: In the printing field, resolution is usually expressed in dots per inch (DPI). For example, 300 DPI means there are 300 dots per inch.

[0086] Image scanning resolution: The resolution of the scanner is also expressed in DPI, which determines the clarity of the scanned image.

[0087] Video resolution: The resolution of a video is usually expressed in pixels, such as 720p (1280x720), 1080p (1920x1080), and 4K (3840x2160 or 4096x2160).

[0088] The image recognition method provided in the embodiment of the present invention can be used to process images captured by a visual sensor such as a camera, or to process video frames captured from a captured video. In the embodiment of the present invention, the resolution described can be pixel resolution.

[0089] The resolution generalization ability of a visual perception model refers to the ability of the visual perception model to process input images with resolutions that are inconsistent with the training resolution during the testing phase.

[0090] Traditional image recognition solutions usually use the same resolution in the model training and testing stages. This is because the resolution generalization ability of visual perception models is poor. Specifically, when the test resolution is inconsistent with the training resolution, the model accuracy will be seriously reduced. The greater the difference between the test resolution and the training resolution, the more serious the reduction in model accuracy, which limits the application of visual perception models in different scenarios. The fundamental reason for this phenomenon is that the distribution of image data with different resolutions is different, and current visual perception models have difficulty in handling such distribution differences.

[0091] However, in practical applications, the visual perception model after deployment may need to process input images with higher resolutions. For example, a visual perception model trained with a resolution of (256x704) may need to process input images with a resolution of (512x1408) or a resolution of (128x352) after deployment. To meet the application needs of processing images with higher resolutions, one current solution is to adjust the input image to the same size as the resolution of the image sample in the training phase during the application phase, that is, adjust the input image with a resolution of (512x1408) and the input image with a resolution of (128x352) to a resolution of (256x704).

[0092] Although this solution of unifying the resolution of input images is feasible, in some practical application scenarios, there is a high demand for different resolution settings. For some application scenarios with limited computing resources, the visual perception model is required to process low-resolution input images (such as 128x352 resolution in the above example) to reduce the computational burden and ensure real-time performance. If the input image resolution is increased to align it to the training resolution, the computational burden will increase and affect the real-time performance of the system. For some application scenarios with sufficient computing resources, it is hoped that a higher image resolution (such as 512x1408 resolution in the above example) will be used to improve the accuracy of three-dimensional image recognition. By reducing the high resolution of the image to align it to the training resolution, a lot of detail information will be lost, which may be detrimental to the detection of small and distant targets.

[0093] It can be seen that although this method of processing images of different resolutions to training resolution before inputting the images to be recognized into the visual perception model solves the problem of the difference in different resolutions, it causes the aligned and adjusted images to lose a lot of detail information, which is not conducive to practical applications. That is, in practical applications, in order to solve the problem, the visual perception model with the same training resolution can only be selected according to the resolution of the images to be recognized, which results in the inability to process images of different resolutions to be recognized, poor flexibility, and high deployment cost.

[0094] Therefore, in actual image recognition applications, there is still a great demand for the resolution generalization capability of the visual perception model itself. In order to improve the resolution generalization capability of the visual perception model, it is necessary to use image samples of different resolutions to train the visual perception model. In order to solve the problem of large differences in feature distribution of image samples of different resolutions, the present invention provides an image recognition solution, which adds an image feature adjustment network to the visual perception model to adjust the image features of different resolutions to align with the reference image features in terms of spatial and channel dimensions, thereby solving the interference problem between different resolutions.

[0095] Simply adjusting the feature sizes of images with different resolutions may result in a large loss of image detail information. However, unlike the above-mentioned traditional solutions, the image recognition solution provided by the embodiment of the present invention adjusts the spatial dimensions and channel dimensions of the input image features by adding an image feature adjustment network to the visual perception model, and updates the parameters of the image feature adjustment network during the iterative training of the visual perception model, so that the visual perception model can eliminate interference between different resolutions and avoid convergence failure due to interference between different resolutions in the process of extracting image features, and compensate for the loss of image details caused by image features.

[0096] Therefore, the image recognition solution provided by the embodiment of the present invention can obtain a visual perception model that can process multi-resolution input images during the model training stage, and can ensure high-precision image recognition results for input images of different resolutions when performing image recognition tasks.

[0097] The image recognition method provided by the embodiment of the present invention is described below with reference to the accompanying drawings.

[0098] Figure 1 A flowchart of an image recognition method provided by an embodiment of the present invention; Figure 2 A schematic diagram of the structure of an image recognition system provided by an embodiment of the present invention; Figure 3 A flowchart of multi-resolution loss optimization is provided in an embodiment of the present invention.

[0099] like Figure 1 As shown, the image recognition method provided by the embodiment of the present invention includes:

[0100] S101: using an image sample set including image samples of various resolutions to train a visual perception model that adds an image feature adjustment network;

[0101] S102: In iterative training, for each resolution, an image sample of the resolution is input into a visual perception model and a perception loss value is calculated, and in an image feature adjustment network, a spatial size of the input image feature is aligned with a reference image feature size, and a channel attention weighted calculation is performed on the input image feature according to a mapping relationship between the spatial size of the input image feature and the reference image feature size, and the adjusted image feature is output;

[0102] S103: using the perceptual loss values ​​corresponding to each resolution to update the parameters of the visual perception model and the parameters of the image feature adjustment network, to obtain the visual perception model corresponding to the current iterative training batch;

[0103] S104: after reaching the iterative training end condition, obtaining a trained visual perception model;

[0104] S105: Inputting the input image corresponding to the image recognition task to be processed into the trained visual perception model to obtain an image recognition result of the image recognition task to be processed.

[0105] The image recognition method provided by the embodiment of the present invention mainly includes two major links: training of the visual perception model and application of the visual perception model. In this regard, the image recognition method provided by the embodiment of the present invention can be applied to Figure 2 The image recognition system shown includes two parts: a model training system and a perception computing device.

[0106] The model training system may be composed of one or more computing devices for training a visual perception model based on an image sample set composed of image samples of multiple resolutions. Figure 2 As shown, the model training system can be composed of multiple threads of multiple artificial intelligence devices to form a thread pool, and the image samples corresponding to each resolution are assigned to a thread pool, so as to achieve parallel accelerated learning of multiple resolution image sets. It should be noted that m, n, and k only represent the meaning of multiple, and do not represent a specific number or the same number of threads of each artificial intelligence processor, nor do they mean that each thread pool is composed of one thread in each artificial intelligence device. The model training system can be deployed in a cloud computing center. The device structure of the cloud computing center is similar Figure 2 The perceptual computing devices in the system include computing, storage, network and other infrastructure, operating system, database, middleware and other software environments and application software. The model training system receives image data sets of various resolutions, uses the stored pre-trained models, combined with manual annotation, to annotate visual perception targets (such as the category, location, size of the bounding box, target orientation information, etc.) and stores the annotated data.

[0107] The perception computing device can be a computing device deployed in a cloud computing center or an edge computing device deployed in an edge device. Its structure is as follows: Figure 2 As shown in the perception computing device shown. If the computing device of the cloud computing center is used, after the trained visual perception model is deployed, the edge device collects the image to be recognized through the visual sensor such as the camera and uploads it to the cloud computing center, and uses the perception computing device that deploys the visual perception model to perform visual perception calculation on the image to be recognized to obtain the image recognition result of the visual perception target. If the perception computing device uses the edge computing device deployed on the edge device, the visual perception model is trained by the model training system and sent to the edge computing device through the communication unit, and deployed in the memory of the edge computing device. After the edge computing device collects the image to be recognized through the visual sensor such as the camera, it performs visual perception calculation through the local visual perception model to obtain the image recognition result of the visual perception target.

[0108] Applied to Figure 2 The multi-thread pool shown, in an embodiment of the present invention, the step of calculating the perceptual loss value of the visual perception model for image samples of each resolution may include: inputting the image samples corresponding to each resolution into different thread pools one by one, so as to obtain the perceptual loss value of the visual perception model for the image samples of each resolution by parallel calculation.

[0109] like Figure 3 As shown, in an embodiment of the present invention, when training a visual perception model, all images of each resolution can be stacked to obtain a set of input images of different resolutions, so that the training speed can be improved through batch processing.

[0110] In some optional implementations of the embodiments of the present invention, the parameters of the visual perception model and the parameters of the image feature adjustment network are updated using the perceptual loss values ​​corresponding to each resolution in S103, which may include: calculating the model learning loss value of the current iteration number according to the perceptual loss values ​​corresponding to each resolution, and updating the parameters of the visual perception model and the parameters of the image feature adjustment network according to the model learning loss value.

[0111] Among them, calculating the model learning loss value of the current iteration number according to the perceptual loss value corresponding to each resolution may include: taking the average of the loss values ​​calculated by the loss function corresponding to each resolution as the model learning loss value.

[0112] like Figure 3 As shown, for each resolution, all image samples are input into the visual perception model to obtain the corresponding visual perception results, and the loss value of the visual perception model for each resolution compared with the true value (which can be manually labeled) is calculated through the loss function. Finally, the model learning loss value of the visual perception model can be the average of the perceptual loss values ​​of the visual perception model for image samples of all resolutions, as shown in the following formula:

[0113] ;

[0114] in, represents the loss function of the visual perception model, represents the perceptual loss value corresponding to the r-th resolution, represents the loss function corresponding to the rth resolution, and k+1 represents the number of resolutions used in visual perception model training.

[0115] It is very challenging to directly use input images of multiple resolutions to train a visual perception model because there are large differences in data distribution between input images of different resolutions, which increases the difficulty of optimizing the perception model. To solve this problem, an embodiment of the present invention provides a visual perception model with an added image feature adjustment network, which uses the image feature adjustment network to adjust the image features of different resolutions to be consistent with the reference image features, thereby improving the generalization ability of the visual perception model for multiple resolutions.

[0116] Figure 4 A schematic diagram of the structure of a visual perception model that adds an image feature adjustment network provided in an embodiment of the present invention.

[0117] like Figure 4As shown, an embodiment of the present invention proposes an image feature adjustment network for adjusting input image features, and in the iterative training of the visual perception model, the visual perception model uses the perceptual loss values ​​of image samples of each resolution to simultaneously update the parameters of the image feature adjustment network, so that the image features of different resolutions are adjusted to be consistent with the reference image features through the image feature adjustment network, thereby solving the interference problem between different resolutions when the visual perception model learns image samples of multiple resolutions.

[0118] In an embodiment of the present invention, one or more image feature adjustment networks may be added to a traditional visual perception model. A visual perception model is usually composed of multiple image feature extraction layers. The input of the image feature adjustment network is the image feature, and the output is the corrected image feature, so the multi-resolution image correction network may be added at any position after the first image feature extraction layer of the visual perception model.

[0119] For example, if the visual perception model adopts the ResNet-50 network, the image feature adjustment network provided by the embodiment of the present invention can be added after the image feature extraction network therein.

[0120] It should be noted that the visual perception model can also use models other than the ResNet-50 network, and the location of adding image feature adjustment network can be set as needed, and does not necessarily have to be added to the Figure 4 Position shown.

[0121] It is understandable that the model parameters of the image feature adjustment network are related to the position in the visual perception model in which they are located. Then in S102, in the image feature adjustment network, the spatial size of the input image feature is aligned with the reference image feature size, and the input image feature is subjected to channel attention weighted calculation according to the mapping relationship between the spatial size of the input image feature and the reference image feature size, and the adjusted image feature is output, which may include: according to the relationship between the image feature size corresponding to the resolution and the reference image feature size and the image feature size of the image feature input by the image feature adjustment network, the spatial size of the input image feature is aligned with the reference image feature size to obtain the image feature after the spatial size is aligned, and the channel attention weighted calculation is performed on the image feature after the spatial size is aligned according to the mapping relationship between the spatial size of the input image feature and the reference image feature size, and the adjusted image feature is output.

[0122] The image feature adjustment network provided by the embodiment of the present invention is used to adjust the image features of multi-resolution image samples from the spatial dimension and the channel dimension respectively. In this regard, the image feature adjustment network provided by the embodiment of the present invention can be composed of two parts: a spatial size adjustment network and a channel attention weighted calculation network.

[0123] In terms of spatial dimension, the difference in spatial dimensions between image features of different resolutions is obvious. In view of the difference in spatial dimension, in the image feature adjustment network provided in the embodiment of the present invention, by setting a reference image feature size, the spatial size adjustment network upsamples or downsamples the image feature sizes of different resolutions to align them to the reference image feature size, and finally makes the spatial size of the image feature of the current resolution consistent with the spatial size of the reference image feature size.

[0124] In addition, there are large differences in the distribution of image features of different resolutions in the channel dimension. In view of the differences in channel dimensions, in the image feature adjustment network provided in the embodiment of the present invention, the channel attention weight is calculated by the channel attention weighted calculation network according to the mapping relationship between the image feature size corresponding to the resolution and the reference image feature size, and the channel attention weight is used to perform weighted calculation on the image features after spatial size alignment, and the adjusted image features are output.

[0125] Therefore, each time the image feature adjustment network provided by an embodiment of the present invention passes through, the image features corresponding to the current resolution can be aligned with the reference image features from the spatial dimension and the channel dimension. This alignment may cause the loss of image detail information, but this loss will be reflected in the model learning loss of the visual perception model for different resolutions. Ultimately, the loss is optimized to achieve the processing accuracy of images with different resolutions, thereby solving the problem of difficulty in model optimization caused by large data distribution differences between input images of different resolutions.

[0126] It should be noted that the "input image features" in S102 refer to image features of the image samples of the current resolution that are extracted by the image feature extraction network of the visual perception model before the image feature adjustment network and input into the image feature adjustment network. They are usually image features that are downsampled compared to the image samples of the current resolution. When aligning with the benchmark image feature size, the downsampling multiple of the image features corresponding to the current resolution should be controlled to be the same as the downsampling multiple of the benchmark image features.

[0127] Figure 5 A training flow chart of a visual perception model provided by an embodiment of the present invention.

[0128] like Figure 5As shown, for image samples corresponding to resolutions 1, 2, ..., m, they are respectively input into the visual perception model to obtain corresponding perceptual loss values ​​1, 2, ..., m. As described in Embodiment S102 of the present invention, in one iterative training of the visual perception model, for a resolution, an image sample of the resolution is first input into the visual perception model, and in the image feature adjustment network of the visual perception model, the image features corresponding to the resolution are adjusted, and then the adjusted image features are input into the subsequent network layer, and the perceptual loss value corresponding to the resolution is calculated using the perceptual result finally output by the visual perception model.

[0129] In S103, in one iterative training of the visual perception model, the image samples of each resolution are subjected to the operation of S102, the perception loss values ​​corresponding to all resolutions are summarized to calculate the model learning loss value of the current iterative training, and the original model parameters of the visual perception model are updated according to the model learning loss value while updating the model parameters of the image feature adjustment network in the visual perception model.

[0130] like Figure 2 As shown in the figure, in order to accelerate the training of the visual perception model, the self-supervised loss optimization tasks corresponding to m resolutions can be deployed in m thread pools for execution to achieve parallel accelerated training.

[0131] For S104, the iterative training end condition may be that the number of iterative training times of the visual perception model reaches a preset number of iterations, or the model learning loss value of the visual perception model is less than a preset loss value.

[0132] For S105, after the trained visual perception model is deployed in an actual application scenario, it can be directly used to perform image recognition tasks. When performing image recognition tasks, in the process of using the visual perception model to perceive the input image corresponding to the image recognition task to be processed, regardless of the resolution of the input image, the image feature adjustment network in the visual perception model can be used to correct the input image features to ensure the perception accuracy of the visual perception model.

[0133] The image recognition method provided by the embodiment of the present invention adds an image feature adjustment network to the visual perception model. In the process of iteratively training the visual perception model using multi-resolution image samples, the image feature adjustment network is used to adjust the image features of different resolutions to align with the reference image features from the spatial dimension and the channel dimension, and the parameters of the image feature adjustment network are optimized as the visual perception model perceives the loss values ​​of different resolutions. This can not only reduce the interference between different resolutions, but also enable the visual perception model to compensate for the loss of detail information caused by such image feature adjustment during the iterative training process, thereby obtaining the ability to process images of multiple resolutions, and obtaining a visual perception model that can process multi-resolution input images. When performing image recognition tasks, high-precision image recognition results can be guaranteed for input images of different resolutions.

[0134] Based on the above embodiment, the embodiment of the present invention further illustrates the steps of collecting the image sample set.

[0135] The image sample set used for training the visual perception model in the embodiment of the present invention includes image samples of multiple resolutions. When collecting the image sample set, a visual sensor such as a camera can be used to directly collect image samples of different resolutions through set resolution parameters to form an image sample set; when collecting the image sample set, image samples of one resolution can also be collected first, and image samples of multiple resolutions can be generated by scaling or enlarging to form an image sample set.

[0136] In some optional implementations of the embodiments of the present invention, the step of collecting the image sample set may include: obtaining an original image set; processing each original image in the original image set using different scaling factors to obtain scaled images of different resolutions corresponding to each original image; and using each original image and the scaled image corresponding to each original image as the image sample set.

[0137] In practical applications, each original image in the original image set is processed with different scaling factors to obtain scaled images of different resolutions corresponding to each original image, which may include: processing each original image with a reference scaling factor to obtain a reference scaled image corresponding to the original image; sampling multiple scaling factors from a preset distribution range, and processing each original image with each scaling factor to obtain scaled images of multiple resolutions corresponding to each original image; using the corresponding reference scaled image and each corresponding scaled image as an image set corresponding to the original image to obtain an image set corresponding to each original image. That is, the generated image sample set is composed of image sets corresponding to different original images, which include image samples of different resolutions, and different image samples of the same resolution may be obtained by processing different original images.

[0138] As described in the above embodiment of the present invention, the resolution used for processing by the visual perception model of the image recognition task is usually expressed in pixel resolution, that is, expressed as the product of the number of pixels in the horizontal direction and the number of pixels in the vertical direction. In the embodiment of the present invention, the scaling factor may include a scaling factor acting on the image height and / or a scaling factor on the image width. In order to obtain image samples of multiple resolutions with greater resolution differences, the scaling factor may include an image height scaling factor and an image width scaling factor.

[0139] To reduce the computational difficulty, the scaling factor can also be a scaling ratio for the image height and image width. For example, for an original image with a resolution of H×W , the base scaling factor can be used The processed resolution is After the image is distributed in Uniform distribution Sample k scaling factors , and process the image using each scaling factor , and finally obtain a collection of images with multiple resolutions ,in The resolution is . Where H represents the vertical size of the image, and W represents the horizontal size of the image.

[0140] When applied to the field of autonomous driving, cameras and other visual sensors are usually set up in different directions of the vehicle, such as left front (fl), front (f), right front (fr), left rear (bl), rear (b), right rear (br), etc., so that images of multiple perspectives of the road scene are collected each time. In the actual training process, in order to use the batch method to accelerate model training, the embodiment of the present invention samples a set of scaling factors at each iteration, and , the above-obtained scaling coefficient set is used for processing, that is, a multi-resolution image set is obtained for each viewing angle .

[0141] Since the resolution difference between the multi-resolution image samples used in training the visual perception model is small, it is not conducive to the training of the model's resolution generalization ability. In order to obtain as many resolutions with large differences as possible and improve the training effect of the model's resolution generalization ability, in other optional implementations of the embodiments of the present invention, the difference between the scaling factors can be set to be no less than the scaling factor difference threshold, that is, when determining the scaling factor, the difference between the scaling factor and other scaling factors in the collected scaling factor set is judged, and only when the difference is greater than the set scaling factor difference threshold is the scaling factor added to the scaling factor set, thereby ensuring the diversity between the scaling factors obtained by sampling and improving the training effect of the model's resolution generalization ability.

[0142] Furthermore, the scaling factors may be designed to all belong to a preset distribution range and the difference between any two scaling factors may not be less than a scaling factor difference threshold.

[0143] In some optional implementations of the embodiments of the present invention, the scaling factor difference threshold is calculated based on the interval length of the preset distribution range divided by twice the number of scaling factors. That is, the scaling factor difference threshold can be expressed by the following formula:

[0144] ;

[0145] Where M represents the scaling factor difference threshold, Indicates the lower limit of the preset distribution range. represents the upper limit of the preset distribution range, and k represents the total number of scaling factors to be collected from the preset distribution range.

[0146] In some other optional implementations of the embodiments of the present invention, each original image in the original image set is processed using a different scaling factor to obtain a scaled image of different resolutions corresponding to each original image, which may also include: using a benchmark scaling factor to process each original image to obtain a benchmark scaled image corresponding to the original image; sampling multiple scaling factors from a preset distribution range until a preset number of scaling factors are collected; in the process of collecting scaling factors, each time a new scaling factor is collected, the difference between the new scaling factor and the existing scaling factor is calculated, and if the difference is not less than the scaling factor difference threshold, the new scaling factor is added to the set of existing scaling factors; if the difference is less than the scaling factor difference threshold, the new scaling factor is discarded; using each scaling factor to process each original image respectively to obtain scaled images of multiple resolutions corresponding to each original image; using the corresponding benchmark scaling image and the corresponding scaled images as the image set corresponding to the original image to obtain the image set corresponding to each original image.

[0147] That is, the following algorithm can be designed to obtain the set of scaling factors:

[0148] Input: reference scaling factor , distributed in uniform distribution , scaling factor set A, scaling factor difference threshold M;

[0149] Add the reference scaling factor to the scaling factor set A ;

[0150] If the length of the scaling factor set A < k + 1, then:

[0151] Randomly sample from the uniform distribution to obtain ;

[0152] Check the difference between and all scaling factors in the current scaling factor set A. If the difference is greater than the threshold scaling factor difference M, then add to the scaling factor set A;

[0153] Output: scaling factor set A.

[0154] In various embodiments of the present invention, the reference scaling image obtained by the above method can be used as the reference image in the image sample set to obtain the reference image features. Alternatively, an image sample of one of the resolutions can also be used as the reference image.

[0155] To further improve the ability of the visual perception model to process input images of different resolutions, an image sample set can be generated multiple times during the model training process. That is to say, the scaling factors can be re-collected to generate a new image sample set in each iteration training, or the scaling factors can be re-collected to generate a new image sample set after every few iteration trainings.

[0156] Since if a new image sample set is regenerated by the above method in each iteration training, it may cause the visual perception model to be difficult to converge.

[0157] To ensure model convergence, in some optional implementation manners of the embodiments of the present invention, an image sample set can be generated once before the training of the visual perception model, and the image sample set is not updated during the subsequent iteration training. Then, in each iteration training of the visual perception model, the same batch of resolutions are learned, and the diversity of samples can be ensured by collecting different image samples corresponding to the resolutions. If computing resources and time permit, an image sample set can be generated once before the training of the visual perception model, but multiple rounds of iteration training are performed. A new image sample set is regenerated before the start of each round of iteration training, and the same image sample set is used in different iteration training steps in each round of iteration training.

[0158] In some other optional implementations of the embodiments of the present invention, the image sample set may be regenerated every multiple iterations of training, and the frequency of generating the image sample set may be determined according to the convergence of the visual perception model.

[0159] In some other optional implementations of the embodiments of the present invention, if the image sample set is regenerated in each iterative training or every few iterative trainings, the multiple resolutions corresponding to the resampling scaling factors can arrange the resolutions in a fixed order (such as from small to large or from large to small) according to the scaling factors, thereby ensuring the correspondence between the arrangement of the resolutions and the arrangement of the loss function, so that the model training can inherit the experience of the previous iterative training or the previous few iterative trainings.

[0160] In each embodiment of the present invention, the loss function corresponding to the resolution of the visual perception model in each iterative training number inherits the parameters of the loss function of the corresponding position in the previous iterative training, but the corresponding resolution can be the resolution in the previous iterative training or the updated resolution.

[0161] On the basis of the above-mentioned embodiment, in order to realize the alignment of the image features of the current resolution with the reference image features from the spatial size dimension, the embodiment of the present invention continues to explain the spatial size adjustment network in the image feature adjustment network.

[0162] In an embodiment of the present invention, aligning the spatial size of the input image feature with the benchmark image feature size in S102 may include: if the spatial size corresponding to the resolution is larger than the benchmark image feature size, downsampling the input image feature according to the benchmark image feature size to obtain a downsampled image feature; if the spatial size corresponding to the resolution is less than or equal to the benchmark image feature size, upsampling the input image feature according to the benchmark image feature size to obtain an upsampled image feature.

[0163] The spatial size adjustment network of the embodiment of the present invention can be composed of a judgment unit, a downsampling processing network and an upsampling processing network. The judgment unit is used to judge the size relationship between the spatial size corresponding to the resolution and the reference image feature size. If the spatial size corresponding to the resolution is larger than the reference image feature size, the image feature corresponding to the resolution is input into the downsampling processing network for downsampling processing; if the spatial size corresponding to the resolution is less than or equal to the reference image feature size, the image feature corresponding to the resolution is input into the upsampling processing network for upsampling processing.

[0164] In some optional implementations of the embodiments of the present invention, downsampling the input image features according to the reference image feature size to obtain the downsampled image features may include: changing the image feature size of the input image features to the reference image feature size, and respectively calculating the corresponding mean feature and the corresponding maximum feature; respectively convolving the mean feature and the maximum feature to obtain the mean convolution feature and the maximum convolution feature; and calculating the downsampled image features based on the mean convolution feature and the maximum convolution feature.

[0165] Among them, the downsampled image features are calculated based on the mean convolution features and the maximum convolution features, which can include: calculating the downsampled feature size ratio between the image feature size corresponding to the resolution and the reference image feature size; performing feature mapping calculation based on the downsampled feature size ratio to obtain the downsampled mean weight and the downsampled maximum weight; optimizing the mean convolution feature using the downsampled mean weight to obtain the optimized mean convolution feature; optimizing the maximum convolution feature using the downsampled maximum weight to obtain the optimized maximum convolution feature; splicing the optimized mean convolution feature and the optimized maximum convolution feature along the channel dimension and performing a fused convolution calculation to obtain the downsampled image feature.

[0166] Among them, optimizing the mean convolution feature by using the down-sampling mean weight to obtain the optimized mean convolution feature can include: calculating the product of the down-sampling mean weight and the mean convolution feature, and taking the sum of the product and the mean feature as the optimized mean convolution feature.

[0167] Optimizing the maximum value convolution feature by using the downsampled maximum value weight to obtain the optimized maximum value convolution feature may include: calculating the product of the downsampled maximum value weight and the maximum value convolution feature, and taking the sum of the product and the maximum value feature as the optimized maximum value convolution feature.

[0168] Figure 6 A flowchart of image feature downsampling processing provided by an embodiment of the present invention.

[0169] To achieve the above image feature downsampling steps, Figure 6 As shown, the downsampling processing network provided by an embodiment of the present invention may include an adaptive mean pooling unit, a mean feature convolution block, an adaptive maximum pooling unit, a maximum feature convolution block, a downsampling mean mapping unit, a downsampling maximum mapping unit, a first channel dimension splicing unit and a first fusion convolution block.

[0170] Assume that the reference image feature size is , the image feature size corresponding to the current resolution is the i-th resolution is For an image feature extraction network with a downsampling factor of s, the size of the baseline image feature space is , the image feature size of the current resolution is ,use represents the image features of the resolution input image feature adjustment network corresponding to the i-th scaling factor. Among them, the image feature extraction network with a downsampling factor of s means that the image features of the input image have been downsampled s times before the input image feature adjustment network.

[0171] For the input image features , when its feature space size Larger than the reference image feature size When the image features of the current resolution are consistent with the reference image features in terms of spatial size, the downsampling processing network provided by the embodiment of the present invention can be used.

[0172] like Figure 6 As shown, the adaptive mean pooling unit is used to pool image features. Perform adaptive mean pooling , and use the adaptive maximum pooling unit to image features Perform adaptive maximum pooling , the image features The feature space size is changed to , respectively get the mean characteristics and the maximum feature , the above process can be expressed by the following formula:

[0173] ;

[0174] .

[0175] Next, use the mean feature convolution block respectively and the maximum feature convolution block Dealing with mean features and the maximum feature , get the mean convolution feature And the maximum convolution feature , as follows:

[0176] ;

[0177] .

[0178] The embodiment of the present invention provides a method for defining a downsampling feature size ratio, wherein the feature size of the reference image is Image feature size corresponding to the current resolution Calculated, the downsampling feature size ratio includes the height ratio Ratio of width to , calculated by the following formula:

[0179] ;

[0180] ;

[0181] It should be noted that in the embodiment provided by the present invention, the length and width of the image resolution are changed according to the same scaling factor, so and Similarly, when the length and width of the image resolution are changed by different scaling factors, and are different.

[0182] Then, we use the downsampled mean mapping unit and downsampled maximum mapping unit Handling downsampled feature size ratios , , as follows:

[0183] ;

[0184] ;

[0185] and Represents the downsampling mean weight and downsampling maximum weight respectively, which can be used and Optimize the mean convolution features separately And the maximum convolution feature Channel distribution.

[0186] Use downsampled mean weights Optimizing mean convolution features , which can include: calculating the downsampling mean weight Convolution feature with mean The product of and the mean feature Add together to get the optimized mean convolution feature .

[0187] Use downsampled maximum weights Optimizing the maximum convolution feature , which can include: calculating the downsampling maximum weight Convolution feature with the maximum value The product of and the maximum feature Add together to get the optimized maximum convolution feature .

[0188] Finally, the optimized mean convolution feature is concatenated using the first channel dimension concatenation unit Convolution feature with the optimized maximum value Concatenate along the channel dimension and use the first fused convolutional block Processing to obtain the final downsampled image features .

[0189] The above process can be expressed as follows:

[0190] ;

[0191] Among them, cat(·) represents concatenation along the channel dimension.

[0192] In some optional implementations of the embodiments of the present invention, upsampling the input image features according to the reference image feature size to obtain the upsampled image features may include: interpolating the input image features to change the image feature size of the input image features to the reference image feature size to obtain the interpolated features; performing convolution calculation on the interpolated features to obtain the interpolated convolution features; and obtaining the upsampled image features based on the interpolated convolution feature calculation.

[0193] Among them, obtaining the upsampled image features according to the interpolation convolution feature calculation can include: calculating the upsampled feature size ratio between the image feature size corresponding to the resolution and the reference image feature size; performing feature mapping calculation according to the upsampled feature size ratio to obtain the upsampled interpolation weight; optimizing the interpolation convolution feature using the upsampled interpolation weight to obtain the optimized interpolation convolution feature; performing upsampled convolution calculation on the optimized interpolation convolution feature to obtain the upsampled image feature.

[0194] Among them, optimizing the interpolation convolution feature by using the upsampling interpolation weight to obtain the optimized interpolation convolution feature can include: calculating the product of the upsampling interpolation weight and the interpolation convolution feature, and taking the sum of the product and the interpolation feature as the optimized interpolation convolution feature.

[0195] Figure 7 A flowchart of image feature upsampling processing provided by an embodiment of the present invention.

[0196] To achieve the above image feature upsampling steps, such as Figure 7 As shown, the upsampling processing network provided by the embodiment of the present invention may include: an interpolation processing unit, an interpolation feature convolution block, an upsampling interpolation mapping unit and a second fusion convolution block.

[0197] Also assume that the reference image feature size is , the image feature size corresponding to the current resolution is the i-th resolution is For an image feature extraction network with a downsampling factor of s, the size of the baseline image feature space is , the image feature size of the current resolution is ,use represents the image features of the resolution input image feature adjustment network corresponding to the i-th scaling factor. Among them, the image feature extraction network with a downsampling factor of s means that the image features of the input image have been downsampled s times before the input image feature adjustment network.

[0198] For the input image features , when its feature space size Smaller than the reference image feature size When the image features of the current resolution are consistent with the reference image features in terms of spatial size, the upsampling processing network provided by the embodiment of the present invention can be used.

[0199] like Figure 7 As shown, using the interpolation processing unit Image features Interpolation processing is performed to transform image features The spatial dimensions are changed to , and get the interpolation feature , the process can be expressed as follows:

[0200] ;

[0201] The interpolation operation used in the interpolation process may be a linear interpolation method, a bilinear interpolation method or the like.

[0202] Next, use the interpolation feature convolution block Handling interpolation features , get the interpolated convolution feature , as shown below:

[0203] .

[0204] The embodiment of the present invention provides a method for defining an upsampling feature size ratio, wherein the feature size of the reference image is Image feature size corresponding to the current resolution Calculated, the upsampling feature size ratio includes the height ratio Ratio of width to , calculated by the following formula:

[0205] ;

[0206] ;

[0207] It should be noted that in the embodiment provided by the present invention, the length and width of the image resolution are changed according to the same scaling factor, so and Similarly, when the length and width of the image resolution are changed by different scaling factors, and are different.

[0208] Then use the upsampling interpolation mapping unit Handling upsampling feature size ratio , , as shown below:

[0209] ;

[0210] in, Represents the upsampling interpolation weight, using Optimizing interpolated convolution features Channel distribution.

[0211] Use upsampling interpolation weights Optimizing interpolated convolution features , which can include: calculating upsampling interpolation weights Convolutional features with interpolation The product of After addition, the optimized interpolation convolution feature is obtained .

[0212] Finally, the second fusion convolution block is used Processing to obtain the final upsampled image features .

[0213] The above process can be expressed as follows:

[0214] .

[0215] The spatial size adjustment network provided in the embodiment of the present invention is applied. When updating the parameters of the image feature adjustment network in S103, it is specifically necessary to update the parameters of the convolution blocks in the downsampling processing network and the upsampling processing network and the parameters of the feature mapping unit.

[0216] On the basis of the above-mentioned embodiment, in order to align the image features of the current resolution with the reference image features from the channel dimension, the embodiment of the present invention continues to explain the channel attention weighted calculation network in the image feature adjustment network.

[0217] In an embodiment of the present invention, in S102, channel attention weighted calculation is performed on the input image features according to the mapping relationship between the spatial size of the input image features and the reference image feature size, which may include: performing feature vector mapping processing on the image feature size corresponding to the resolution and the reference image feature size to obtain a feature size feature vector; calculating the channel attention weight based on the image features after spatial size alignment and the feature size feature vector; using the channel attention weight to perform weighted calculation on the image features after spatial size alignment, and outputting the adjusted image features.

[0218] In some optional implementations of the embodiments of the present invention, calculating the channel attention weight based on the image features and feature size feature vectors after spatial size alignment can include: performing global mean pooling processing and global maximum pooling processing on the image features after spatial size alignment, respectively, to obtain a global mean pooling feature vector and a global maximum pooling feature vector; splicing the global mean pooling feature vector, the global maximum pooling feature vector and the feature size feature vector along the channel dimension to obtain a spliced ​​feature vector; and calculating the channel attention weight based on the spliced ​​feature vector.

[0219] Among them, calculating the channel attention weight based on the spliced ​​feature vector can include: using the channel attention weight mapping unit to process the spliced ​​feature vector to obtain the initial channel attention weight consistent with the image features after the channel dimension is aligned with the spatial dimension; processing the initial channel attention weight through an activation function and a normalization function in turn to obtain the channel attention weight.

[0220] Figure 8 A flowchart of image feature channel attention weighted calculation provided by an embodiment of the present invention.

[0221] To achieve the above channel dimension feature alignment step, such as Figure 8 As shown, the channel attention weighted calculation network provided by the embodiment of the present invention can be composed of an image feature processing network, a channel attention weight generation network and a channel attention weighted unit. Among them, the image feature processing network may include a global mean pooling layer, a global maximum pooling layer, a feature vector mapping unit and a second channel dimension splicing unit. The channel attention weight generation network may include a channel attention weight mapping network, an activation function layer and a normalization function layer.

[0222] For image features after spatial size alignment (downsampled image features Or upsample image features ), respectively, after the global mean pooling layer Perform global mean pooling and global maximum pooling layer Perform global maximum pooling to obtain the global mean pooling feature vector and the global maximum pooling feature vector .

[0223] The embodiment of the present invention provides a characteristic size characteristic vector mapping unit , the image feature size corresponding to the current resolution , and the reference image feature size , As output, we obtain the characteristic dimension feature vector , as shown below:

[0224] .

[0225] Characteristic size and characteristic vector mapping unit provided by the embodiment of the present invention , the input layer contains 4 neurons, the hidden layer contains c / 2 neurons, and the output layer contains c neurons. Where c is the number of channels of the image feature.

[0226] Then, the second channel dimension concatenation unit is used to pool the global mean feature vector , global maximum pooling feature vector and the characteristic size characteristic vector Splice along the channel dimension and map the network with channel attention weights Process the concatenated feature vector to obtain the channel attention weight, whose channel dimension is the same as the downsampled image feature And upsampled image features The initial channel attention weight is then processed by the activation function and the normalization function to obtain the channel attention weight The channel attention weights are used to adjust the downsampled image features. Or upsample image features , adapting its feature distribution to the current resolution.

[0227] The above steps can be expressed by the following formula:

[0228] ;

[0229] in, represents the activation function, Represents the normalization function. ϵ is a small constant, which can be 0.05 or 0.1, etc.

[0230] For the above steps, an embodiment of the present invention provides a channel attention weight mapping unit The input layer contains 3c neurons, the hidden layer contains c / 2 neurons, and the output layer contains c neurons.

[0231] The channel attention weighted calculation network provided in the embodiment of the present invention is applied. When updating the parameters of the image feature adjustment network in S103, it is specifically necessary to update the parameters of the feature vector mapping unit in the image feature processing network and the channel attention weight generation network and the parameters of the channel attention weight mapping network.

[0232] On the basis of the above embodiments, in order to further improve the effect of multi-resolution image feature adjustment, in the image recognition method provided in the embodiment of the present invention, S103 uses the perceptual loss value corresponding to each resolution to update the parameters of the visual perception model and the parameters of the image feature adjustment network to obtain the visual perception model corresponding to the current iterative training batch, which may include: calculating the regularization constraint item corresponding to each resolution according to the image feature adjustment coefficient of each resolution compared with the feature size of the benchmark image; updating the parameters of the visual perception model and the parameters of the image feature adjustment network according to the perceptual loss value corresponding to each resolution and the regularization constraint item corresponding to each resolution to obtain the visual perception model corresponding to the current iterative training batch.

[0233] In an embodiment of the present invention, on the basis of the loss calculated according to the perceptual loss value corresponding to each resolution, a regularization constraint term corresponding to each resolution is calculated according to the size transformation coefficient of each resolution compared with the characteristic size of the reference image to ensure the optimization effect.

[0234] The design principle of the regularization constraint term is that the image feature adjustment coefficient corresponding to the resolution makes the transformation of the image feature size of the resolution smaller, and the regularization constraint term corresponding to the resolution is closer to 0.

[0235] Taking the image feature adjustment coefficient as a spatial size conversion coefficient as an example, referring to the description of the above embodiment of the present invention, according to the size relationship between the image feature size of the current resolution and the feature size of the reference image, the spatial size conversion coefficient is a downsampling coefficient or an upsampling coefficient. The downsampling coefficient may include a downsampling mean weight and a downsampling maximum weight. and downsampled maximum weight , can be constrained according to the following regularization formula:

[0236] ;

[0237] ;

[0238] in, Represents the downsampling mean weight The corresponding regularization constraint is Represents the maximum weight of downsampling The corresponding regularization constraint.

[0239] The upsampling coefficients may include upsampling interpolation weights. The constraints can be made according to the following regularization formula:

[0240] ;

[0241] in, Represents the upsampling interpolation weight The corresponding regularization constraint.

[0242] Taking the image feature adjustment coefficient as the channel attention weight as an example, the channel attention weight The constraints can be made according to the following regularization formula:

[0243] ;

[0244] in, Represents the channel attention weight The corresponding regularization constraint.

[0245] In the image recognition method provided by an embodiment of the present invention, the image feature adjustment coefficient may include: a spatial size transformation coefficient of the resolution compared to the reference image feature size and a channel attention weight of the resolution corresponding to the reference image feature size; wherein the spatial size transformation coefficient is a downsampling coefficient or an upsampling coefficient of the resolution compared to the reference image feature size, the downsampling coefficient is the downsampling mean weight of the resolution compared to the reference image feature size, the downsampling maximum weight of the resolution compared to the reference image feature size, and the upsampling coefficient is the upsampling interpolation weight of the resolution compared to the reference image feature size.

[0246] Loss function for visual perception model It can be expressed as follows:

[0247] ;

[0248] in, is the loss value calculated according to the perceptual loss value corresponding to each resolution. In some optional implementations of the embodiments of the present invention, , , , is the weight coefficient of each regularization constraint term, which can be used as a hyperparameter to balance the loss function and regularization constraints , , , ,For example, , , , They can all be 0.5.

[0249] It should be noted that in one iteration of the visual perception model training, some image features of different resolutions may require downsampling coefficients, while others may require upsampling coefficients, and image features of each resolution have regularization constraints of channel attention weights. In the visual perception model, each regularization constraint can be composed of the sum of regularization constraints corresponding to each resolution.

[0250] In some other optional implementations of the embodiments of the present invention, , , , Self-learning parameters may also be used. The image recognition method provided by the embodiment of the present invention may further include: updating the weight coefficient of the regularization constraint term in the iterative training of the visual perception model.

[0251] Based on the multi-resolution image feature correction method scheme introduced in the above embodiment, the embodiment of the present invention continues to explain the training steps and reasoning steps of the visual perception model.

[0252] As described in the above embodiments of the present invention, the training task of the visual perception model can be executed in a cloud computing center.

[0253] If applied to the field of autonomous driving, the visual perception model can be trained on the collected and annotated road scene image data, and the multi-resolution meta-learning optimization method provided by the embodiment of the present invention is used. In each training batch, the image samples corresponding to m resolutions are respectively input into the visual perception model in the current iterative training, and the multi-resolution image correction network is used in the visual perception model to correct the image features. Finally, the perception result is output through the visual perception model and the perception loss value is calculated according to the true value. All the perception loss values ​​in the training batch are collected and substituted into the loss function of the visual perception model to obtain the model learning loss value, and then the model is optimized by the back propagation algorithm. Among them, the optimizer can use a gradient descent (GD) optimizer, an adaptive moment estimation (Adam) optimizer, an adaptive weight decay regularization (Adam Weight Decay Regularization, AdamW) optimizer, etc. The multi-resolution parallel acceleration method proposed by the present invention is used during training to improve training efficiency. After completing the model training, the perception model obtained by training is deployed to the edge computing platform of the unmanned vehicle.

[0254] The image recognition method provided by the embodiment of the present invention can be applied to, but not limited to, the field of autonomous driving, and can also be applied to intelligent terminals such as intelligent robots, smart homes, and mobile phones, and can also be applied to visual perception models involved in industrial production fields such as transportation, aerial remote sensing, and industrial product assembly. For example, the visual perception model can be an autonomous driving perception model; for this, the image sample set can be constructed based on image samples collected by vehicle-mounted sensors from different perspectives, and the visual perception model can be trained on the collected and annotated road scene image data set when executing model training in the cloud data center.

[0255] In an embodiment of the present invention, updating the parameters of the visual perception model using the perceptual loss values ​​corresponding to each resolution may include: calculating the model learning loss value of the visual perception model using the sum of the perceptual loss values ​​corresponding to each resolution; and updating the parameters of the visual perception model using the model learning loss value. For example, a loss function of a visual perception model provided in the above embodiment of the present invention may be used. If the optimization effect of image feature adjustment is enhanced by adding a regularization constraint term in combination with the above-mentioned embodiment of the present invention, the loss function of the visual perception model can be expressed as:

[0256] .

[0257] When using the visual perception model to perform image recognition tasks, the trained visual perception model can be directly deployed in actual application scenarios to perform image recognition tasks.

[0258] It should be noted that in the embodiments of the image recognition method of the present invention, some of the steps or features may be ignored or not executed. The hardware or software functional modules divided for the convenience of description are not the only implementation form of the image recognition method provided by the embodiment of the present invention.

[0259] The above describes in detail various embodiments corresponding to the image recognition method. On this basis, the present invention also discloses an image recognition device, equipment, non-volatile storage medium and computer program product corresponding to the above method.

[0260] Fig. 9 A schematic diagram of the structure of an image recognition device provided by an embodiment of the present invention.

[0261] like Fig. 9 As shown, the image recognition device provided by the embodiment of the present invention includes:

[0262] The model training unit 901 is used to train a visual perception model with an image feature adjustment network added using an image sample set including image samples of multiple resolutions; in iterative training, for each resolution, the image sample of the resolution is input into the visual perception model and a perceptual loss value is calculated, and in the image feature adjustment network, the spatial size of the input image feature is aligned with the reference image feature size, and the input image feature is subjected to channel attention weighted calculation according to the mapping relationship between the spatial size of the input image feature and the reference image feature size, and the adjusted image feature is output; the parameters of the visual perception model and the parameters of the image feature adjustment network are updated using the perceptual loss value corresponding to each resolution, and the visual perception model corresponding to the current iterative training batch is obtained; after the iterative training end condition is met, the trained visual perception model is obtained;

[0263] The image recognition unit 902 is used to input the input image corresponding to the image recognition task to be processed into the trained visual perception model to obtain the image recognition result of the image recognition task to be processed.

[0264] It should be noted that in each implementation of the image recognition device provided by the embodiment of the present invention, the division of units is only a logical functional division, and other division methods can be used. The connection method between different units can be electrical, mechanical or other connection methods. The separated units can be located in the same physical location or distributed on multiple network nodes. Each unit can be implemented in the form of hardware or in the form of a software functional unit. That is, part or all of the units provided in the embodiment of the present invention can be selected according to actual needs and the corresponding connection method or integration method can be adopted to achieve the purpose of the embodiment of the present invention.

[0265] Since the embodiments of the apparatus part correspond to the embodiments of the method part, please refer to the description of the embodiments of the method part for the embodiments of the apparatus part, which will not be repeated here.

[0266] Fig.10 A schematic diagram of the structure of an image recognition device provided by an embodiment of the present invention.

[0267] like Fig.10 As shown, the image recognition device provided by the embodiment of the present invention includes: a memory 1010, used to store a computer program 1011; a processor 1020, used to execute the computer program 1011, and when the computer program 1011 is executed by the processor 1020, the steps of the image recognition method provided in any of the above embodiments are implemented.

[0268] Among them, the processor 1020 may include one or more processing cores, such as a 3-core processor, an 8-core processor, etc. The processor 1020 may be implemented in at least one hardware form of digital signal processing (DSP), field-programmable gate array (FPGA), and programmable logic array (PLA). The processor 1020 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a central processing unit (CPU); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 1020 may be integrated with a graphics processing unit (GPU), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 1020 may also include an artificial intelligence (AI) processor, which is used to process computing operations related to machine learning.

[0269] The memory 1010 may include one or more non-volatile storage media, which may be non-transitory. The memory 1010 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In this embodiment, the memory 1010 is at least used to store the following computer program 1011, wherein the computer program 1011, after being loaded and executed by the processor 1020, can implement the relevant steps in the image recognition method disclosed in any of the aforementioned embodiments. In addition, the resources stored in the memory 1010 may also include an operating system 1012 and data 1013, etc., and the storage method may be temporary storage or permanent storage. Among them, the operating system 1012 may be Windows or other types of operating systems. Data 1013 may include, but is not limited to, the data involved in the above method.

[0270] In some embodiments, the image recognition device may further include a display screen 1030 , a power supply 1040 , a communication interface 1050 , an input / output interface 1060 , a sensor 1070 , and a communication bus 1080 .

[0271] Those skilled in the art will understand that Fig.10 The structure shown in the figure does not constitute a limitation on the image recognition device, and may include more or fewer components than those shown in the figure.

[0272] The image recognition device provided by the embodiment of the present invention includes a memory and a processor. When the processor executes the program stored in the memory, it can implement the steps of the image recognition method provided by the above embodiment, and the effect is the same as above.

[0273] An embodiment of the present invention provides a non-volatile storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the image recognition method provided in any one of the above embodiments can be implemented.

[0274] The non-volatile storage medium may include: a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and other media that can store program codes.

[0275] For the introduction of the non-volatile storage medium provided in the embodiment of the present invention, please refer to the above method embodiment, and the effect thereof is the same as the image recognition method provided in the embodiment of the present invention, and the present invention will not elaborate on it here.

[0276] An embodiment of the present invention provides a computer program product, including a computer program, which, when executed by a processor, implements the steps of the image recognition method provided in any one of the above embodiments.

[0277] For an introduction to the computer program product provided by the embodiment of the present invention, please refer to the above method embodiment, and the effect thereof is the same as the image recognition method provided by the embodiment of the present invention, and the present invention will not elaborate on it here.

[0278] The above is a detailed introduction to an image recognition method, device, equipment, medium and computer program product provided by the present invention. The various embodiments in the specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same and similar parts between the embodiments can be referenced to each other. For the devices, equipment, non-volatile storage media and computer program products disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple, and the relevant parts can be referred to the method part description. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of the present invention, the present invention can also be improved and modified in a number of ways, and these improvements and modifications also fall within the scope of protection of the present invention.

[0279] It should also be noted that, in this specification, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the statement "comprises a ..." does not exclude the presence of other identical elements in the process, method, article or device including the element.

Claims

1. An image recognition method, characterized in that: include: A visual perception model for adding an image feature adjustment network is trained using an image sample set including image samples of various resolutions; In iterative training, for each resolution, an image sample of the resolution is input into the visual perception model and a perception loss value is calculated, and in the image feature adjustment network, the spatial size of the input image feature is aligned with the reference image feature size, and according to the mapping relationship between the spatial size of the input image feature and the reference image feature size, a channel attention weighted calculation is performed on the input image feature, and the adjusted image feature is output; Using the perceptual loss values ​​corresponding to the resolutions, the parameters of the visual perception model and the parameters of the image feature adjustment network are updated to obtain the visual perception model corresponding to the current iterative training batch; After reaching the iterative training end condition, obtaining the trained visual perception model; The input image corresponding to the image recognition task to be processed is input into the trained visual perception model to obtain an image recognition result for the image recognition task to be processed.

2. The image recognition method according to claim 1, characterized in that: Align the spatial dimensions of the input image features with the reference image feature dimensions, including: If the spatial size corresponding to the resolution is larger than the reference image feature size, downsampling the input image feature according to the reference image feature size to obtain a downsampled image feature; If the spatial size corresponding to the resolution is less than or equal to the reference image feature size, the input image feature is upsampled according to the reference image feature size to obtain an upsampled image feature.

3. The image recognition method according to claim 2, characterized in that: Downsampling the input image features according to the reference image feature size to obtain downsampled image features includes: The image feature size of the input image feature is changed to the reference image feature size, and the corresponding mean feature and the corresponding maximum feature are respectively calculated; Perform convolution calculation on the mean feature and the maximum feature respectively to obtain a mean convolution feature and a maximum convolution feature; The down-sampled image feature is calculated based on the mean convolution feature and the maximum convolution feature.

4. The image recognition method according to claim 3, characterized in that: The down-sampled image feature is calculated according to the mean convolution feature and the maximum convolution feature, comprising: Calculating a downsampled feature size ratio between the image feature size corresponding to the resolution and the reference image feature size; Perform feature mapping calculation according to the down-sampling feature size ratio to obtain a down-sampling mean weight and a down-sampling maximum weight; Optimizing the mean convolution feature by using the down-sampling mean weight to obtain an optimized mean convolution feature; Optimizing the maximum value convolution feature using the down-sampling maximum value weight to obtain an optimized maximum value convolution feature; The optimized mean convolution feature and the optimized maximum convolution feature are concatenated along the channel dimension and a fusion convolution calculation is performed to obtain the downsampled image feature.

5. The image recognition method according to claim 4, characterized in that: The mean convolution feature is optimized by using the down-sampling mean weight to obtain an optimized mean convolution feature, including: After calculating the product of the down-sampling mean weight and the mean convolution feature, the sum of the product and the mean feature is taken as the optimized mean convolution feature.

6. The image recognition method according to claim 4, characterized in that: Optimizing the maximum value convolution feature by using the down-sampling maximum value weight to obtain an optimized maximum value convolution feature, including: After calculating the product of the down-sampling maximum weight and the maximum convolution feature, the sum of the product and the maximum feature is used as the optimized maximum convolution feature.

7. The image recognition method according to claim 2, characterized in that: The input image feature is upsampled according to the reference image feature size to obtain the upsampled image feature, including: Performing interpolation processing on the input image feature to change the image feature size of the input image feature to the reference image feature size, thereby obtaining an interpolation feature; Performing convolution calculation on the interpolation feature to obtain an interpolation convolution feature; The up-sampled image features are calculated based on the interpolated convolution features.

8. The image recognition method according to claim 7, characterized in that: The up-sampled image feature is calculated according to the interpolated convolution feature, comprising: Calculating an up-sampled feature size ratio between the image feature size corresponding to the resolution and the reference image feature size; Performing feature mapping calculation according to the up-sampling feature size ratio to obtain up-sampling interpolation weights; Optimizing the interpolation convolution feature by using the upsampling interpolation weight to obtain an optimized interpolation convolution feature; The optimized interpolation convolution feature is subjected to up-sampling convolution calculation to obtain the up-sampled image feature.

9. The image recognition method according to claim 8, characterized in that: Optimizing the interpolation convolution feature by using the upsampling interpolation weight to obtain an optimized interpolation convolution feature includes: After calculating the product of the up-sampling interpolation weight and the interpolation convolution feature, the sum of the product and the interpolation feature is used as the optimized interpolation convolution feature.

10. The image recognition method according to claim 1, characterized in that: The channel attention weighted calculation is performed on the input image feature according to the mapping relationship between the spatial size of the input image feature and the reference image feature size, including: Performing feature vector mapping processing on the image feature size corresponding to the resolution and the reference image feature size to obtain a feature size feature vector; The channel attention weight is calculated based on the image features after spatial size alignment and the feature size feature vector; The channel attention weights are used to perform weighted calculation on the image features after the spatial size alignment, and the adjusted image features are output.

11. The image recognition method according to claim 10, characterized in that: The channel attention weight is calculated based on the image features after spatial size alignment and the feature size feature vector, including: Performing global mean pooling processing and global maximum pooling processing on the image features after the spatial size alignment respectively to obtain a global mean pooling feature vector and a global maximum pooling feature vector; Concatenating the global mean pooling feature vector, the global maximum pooling feature vector, and the feature size feature vector along the channel dimension to obtain a concatenated feature vector; The channel attention weight is calculated according to the concatenated feature vector.

12. The image recognition method according to claim 11, characterized in that: The channel attention weight is calculated according to the concatenated feature vector, including: Processing the concatenated feature vector using a channel attention weight mapping unit to obtain an initial channel attention weight consistent with the image features after the channel dimension is aligned with the spatial dimension; The initial channel attention weight is processed by an activation function and a normalization function in turn to obtain the channel attention weight.

13. The image recognition method according to claim 1, characterized in that: The parameters of the visual perception model and the parameters of the image feature adjustment network are updated using the perceptual loss values ​​corresponding to each of the resolutions to obtain the visual perception model corresponding to the current iterative training batch, including: The regularization constraint item corresponding to each of the resolutions is calculated according to the image feature adjustment coefficient of each of the resolutions compared with the reference image feature size; According to the perceptual loss value corresponding to each of the resolutions and the regularization constraint item corresponding to each of the resolutions, the parameters of the visual perception model and the parameters of the image feature adjustment network are updated to obtain the visual perception model corresponding to the current iterative training batch.

14. The image recognition method according to claim 13, characterized in that: The image feature adjustment coefficient corresponding to the resolution makes the transformation of the image feature size of the resolution smaller, and the regularization constraint item corresponding to the resolution approaches 0.

15. The image recognition method according to claim 13, characterized in that: The image feature adjustment coefficient includes: a spatial size conversion coefficient of the resolution compared to the reference image feature size and a channel attention weight of the resolution corresponding to the reference image feature size; Among them, the spatial size transformation coefficient is the downsampling coefficient or upsampling coefficient of the resolution compared to the characteristic size of the benchmark image, the downsampling coefficient is the downsampling mean weight of the resolution compared to the characteristic size of the benchmark image, the downsampling maximum weight of the resolution compared to the characteristic size of the benchmark image, and the upsampling coefficient is the upsampling interpolation weight of the resolution compared to the characteristic size of the benchmark image.

16. The image recognition method according to claim 13, characterized in that: Also includes: The weight coefficient of the regularization constraint term is updated during the iterative training of the visual perception model.

17. An image recognition device, characterized in that: include: A model training unit, for training a visual perception model of adding an image feature adjustment network using an image sample set including image samples of multiple resolutions; In iterative training, for each resolution, an image sample of the resolution is input into the visual perception model and a perceptual loss value is calculated, and in the image feature adjustment network, the spatial size of the input image feature is aligned with the reference image feature size, and a channel attention weighted calculation is performed on the input image feature according to a mapping relationship between the spatial size of the input image feature and the reference image feature size, and the adjusted image feature is output; the parameters of the visual perception model and the parameters of the image feature adjustment network are updated using the perceptual loss value corresponding to each resolution, so as to obtain the visual perception model corresponding to the current iterative training batch; After reaching the iterative training end condition, obtaining the trained visual perception model; The image recognition unit is used to input the input image corresponding to the image recognition task to be processed into the trained visual perception model to obtain the image recognition result of the image recognition task to be processed.

18. An image recognition device, characterized in that: include: Memory for storing computer programs; A processor is used to execute the computer program, and when the computer program is executed by the processor, the steps of the image recognition method according to any one of claims 1 to 16 are implemented.

19. A non-volatile storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the image recognition method according to any one of claims 1 to 16 are implemented.

20. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the image recognition method according to any one of claims 1 to 16 are implemented.

Images