A target identification method, device, equipment, medium and computer program product

Abstract

The present invention relates to the field of image processing technology, and specifically discloses a target recognition method, device, equipment, medium and computer program product. A multi-resolution generalization layer for resizing input image features is added to a visual perception model. When the visual perception model is trained using image samples of multiple resolutions, the parameters of the multi-resolution generalization layer are updated using a self-supervised loss method, and then the image samples of different resolutions are input into the visual perception model to calculate the perception loss value; the parameters of the visual perception model are updated using the perception loss values ​​corresponding to each resolution, and iterative training is performed to obtain the trained visual perception model, thereby reducing the optimization difficulty caused by the data distribution differences of image samples of different resolutions, thereby improving the resolution generalization ability of the visual perception model to process input images of different resolutions, and further improving the resolution generalization ability of the target recognition task to adapt to input images of different resolutions.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a target recognition method, device, equipment, medium and computer program product. Background Art

[0002] Object recognition is a common application of computer vision. Its main task is to obtain corresponding scene information by processing the collected images. It is widely used in autonomous driving, intelligent robots, smart homes, smart terminals, as well as industrial production such as transportation, aerial remote sensing, and industrial product assembly.

[0003] When performing target recognition tasks, the visual perception model determines the accuracy of recognition. Due to the differences in requirements of different application scenarios, different image samples need to be used when training visual perception models for different application scenarios. For example, for application scenarios with limited computing resources, low-resolution image samples are usually used to train visual perception models, while for perception scenarios of small targets and long-distance targets, high-resolution image samples are used to train visual perception models. This results in that in practical applications, once the visual perception model is deployed, it can only process input images with fixed resolution.

[0004] How to improve the resolution generalization capability of adapting to input images of different resolutions in target recognition tasks is a technical problem that technicians in this field need to solve. Summary of the invention

[0005] The purpose of the present invention is to provide a target recognition method, device, equipment, medium and computer program product for improving the resolution generalization capability of adapting to input images of different resolutions in target recognition tasks.

[0006] In order to solve the above technical problems, the present invention provides a target recognition method, comprising:

[0007] Using an image sample set including image samples of multiple resolutions to train a visual perception model to which a multi-resolution generalization layer is added;

[0008] In iterative training, for each resolution, an image sample of the resolution is input into the visual perception model, an intermediate image feature output by the last multi-resolution generalization layer of the visual perception model is obtained, the intermediate image feature is input into an image feature decoder, a decoded image is output, a self-supervised loss is calculated based on the decoded image and the corresponding image sample, and after the parameters of the multi-resolution generalization layer are updated using the self-supervised loss, the image sample of the resolution is input into the visual perception model and a perceptual loss value is calculated;

[0009] Using the perceptual loss values ​​corresponding to each of the resolutions, the parameters of the visual perception model are updated to obtain the visual perception model corresponding to the current iterative training batch;

[0010] After reaching the iterative training end condition, obtaining the trained visual perception model;

[0011] After updating the parameters of the multi-resolution generalization layer of the visual perception model according to the resolution corresponding to the target recognition task to be processed, inputting the input image of the target recognition task to be processed into the updated visual perception model to obtain a target recognition result for the target recognition task to be processed;

[0012] The multi-resolution generalization layer is used to resize the input image features.

[0013] On the one hand, inputting the intermediate image features into an image feature decoder and outputting a decoded image comprises:

[0014] According to the downsampling multiples of the last multi-resolution generalization layer and the previous network layers in the visual perception model, convolution operation and feature rearrangement are performed on the intermediate image features to obtain a first intermediate image feature;

[0015] According to the downsampling multiple, upsampling the intermediate image feature by the same multiple is performed to obtain a second intermediate image feature;

[0016] The decoded image is obtained according to the first intermediate image feature and the second intermediate image feature.

[0017] On the other hand, according to the downsampling multiples of the last multi-resolution generalization layer and the previous network layer in the visual perception model, the intermediate image features are subjected to convolution operation and feature rearrangement to obtain the first intermediate image features, including:

[0018] After performing convolution operation on the intermediate image feature using the first convolution layer and the second convolution layer respectively, a third intermediate image feature is obtained;

[0019] Rearranging the features of the third intermediate image so that the size of the features of the third intermediate image is the same as the corresponding image sample of the resolution;

[0020] The number of channels of the convolution kernel of the first convolution layer is the same as the number of channels of the intermediate image features, and the number of channels of the convolution kernel of the second convolution layer is the value rounded down after dividing the number of channels of the convolution kernel of the first convolution layer by 2.

[0021] On the other hand, a self-supervised loss is calculated based on the decoded image and the corresponding image sample, including:

[0022] The self-supervised loss is calculated based on the second intermediate image features of all image feature positions of the image samples of the corresponding resolution and the decoded image.

[0023] On the other hand, the self-supervisory loss is calculated according to the second intermediate image features at all positions of the image samples of the corresponding resolution and the decoded image, including:

[0024] Calculate the self-supervision loss value corresponding to each image feature position respectively;

[0025] The mean of the self-supervised loss values ​​at each of the image feature positions is taken as the self-supervised loss corresponding to the resolution.

[0026] On the other hand, the step of calculating the self-supervised loss value corresponding to the image feature position includes:

[0027] If the absolute value of the difference between the second intermediate image feature corresponding to the image feature position and the image feature at the corresponding position of the decoded image is less than a preset constant, the self-supervised loss value is the square of the second intermediate image feature and the decoded image divided by 2;

[0028] If the absolute value of the difference between the second intermediate image feature corresponding to the image feature position and the image feature at the corresponding position of the decoded image is greater than or equal to the preset constant, the self-supervised loss value is the absolute value of the difference between the second intermediate image feature and the decoded image minus 0.5.

[0029] On the other hand, the parameters of the multi-resolution generalization layer are updated using the self-supervised loss, including:

[0030] The parameters of the multi-resolution generalization layer are updated using the self-supervised loss corresponding to the resolution and the learning rate corresponding to the multi-resolution generalization layer.

[0031] On the other hand, the learning rate corresponding to the multi-resolution generalization layer is a parameter updated by self-learning in the iterative training of the visual perception model.

[0032] On the other hand, the self-learning update step of the learning rate corresponding to the multi-resolution generalization layer includes:

[0033] Extracting image features from image samples of the resolution to obtain image feature vectors;

[0034] For each of the multi-resolution generalization layers, the image feature vector is used as a key vector and a value vector, a query vector corresponding to the multi-resolution generalization layer is adopted, self-attention calculation is performed according to the query vector, the key vector and the value vector, and the self-attention calculation result is processed by a linear layer to obtain a learning rate corresponding to the multi-resolution generalization layer.

[0035] On the other hand, the self-learning update step of the learning rate corresponding to the multi-resolution generalization layer includes:

[0036] Extracting image features from image samples of the resolution to obtain image feature vectors;

[0037] For each of the multi-resolution generalization layers, the image feature vector is used as a key vector and a value vector, a query vector is determined according to the image feature size corresponding to the multi-resolution generalization layer, a self-attention calculation is performed according to the query vector, the key vector and the value vector, and the self-attention calculation result is processed by a linear layer to obtain a learning rate corresponding to the multi-resolution generalization layer.

[0038] On the other hand, the self-learning update step of the learning rate corresponding to the multi-resolution generalization layer includes:

[0039] A meta-learner is used to generate a learning rate for each of the multi-resolution generalization layers according to the image feature size corresponding to the resolution.

[0040] On the other hand, it also includes:

[0041] In the iterative training of the visual perception model, the parameters of the meta-learner are updated according to the perceptual loss values ​​corresponding to each of the resolutions.

[0042] On the other hand, updating the parameters of the meta-learner according to the perceptual loss values ​​corresponding to the resolutions includes:

[0043] The parameters of the meta-learner are updated according to the current parameters of the meta-learner, the learning rate corresponding to the meta-learner, and the perceptual loss value corresponding to each of the resolutions.

[0044] On the other hand, it also includes:

[0045] The learning rate corresponding to the meta-learner is updated during the iterative training of the visual perception model.

[0046] On the other hand, the parameter updating step of the visual perception model includes:

[0047] Before the iterative training of the visual perception model begins, randomly initializing the parameters of the visual perception model;

[0048] In the iterative training of the visual perception model, the parameters of the visual perception model are updated according to the perceptual loss values ​​corresponding to each of the resolutions.

[0049] On the other hand, updating the parameters of the visual perception model according to the perception loss values ​​corresponding to the resolutions includes:

[0050] The parameters of the visual perception model are updated according to the current parameters of the visual perception model, the learning rate corresponding to the visual perception model, and the perceptual loss value corresponding to each of the resolutions.

[0051] On the other hand, it also includes:

[0052] The learning rate corresponding to the visual perception model is updated during the iterative training of the visual perception model.

[0053] In order to solve the above technical problems, the present invention further provides a target recognition device, comprising:

[0054] A model training unit is used to train a visual perception model with a multi-resolution generalization layer added by using an image sample set including image samples of multiple resolutions; in iterative training, for each resolution, input the image sample of the resolution into the visual perception model, obtain the intermediate image features output by the last multi-resolution generalization layer of the visual perception model, input the intermediate image features into an image feature decoder, output a decoded image, calculate a self-supervised loss according to the decoded image and the corresponding image sample, and after updating the parameters of the multi-resolution generalization layer by using the self-supervised loss, input the image sample of the resolution into the visual perception model and calculate the perceptual loss value; update the parameters of the visual perception model by using the perceptual loss values ​​corresponding to each resolution to obtain the visual perception model corresponding to the current iterative training batch; after the iterative training end condition is met, obtain the trained visual perception model;

[0055] A model migration unit, used for updating the parameters of the multi-resolution generalization layer of the visual perception model according to the resolution corresponding to the target recognition task to be processed, so as to obtain the updated visual perception model;

[0056] A target recognition unit, used for inputting the input image of the target recognition task to be processed into the updated visual perception model to obtain a target recognition result for the target recognition task to be processed;

[0057] The multi-resolution generalization layer is used to resize the input image features.

[0058] In order to solve the above technical problems, the present invention further provides a target recognition device, comprising:

[0059] Memory for storing computer programs;

[0060] A processor is used to execute the computer program, and when the computer program is executed by the processor, the steps of the target recognition method as described in any one of the above items are implemented.

[0061] In order to solve the above technical problems, the present invention also provides a non-volatile storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the target identification method as described in any one of the above items are implemented.

[0062] In order to solve the above technical problems, the present invention also provides a computer program product, including a computer program, which implements the steps of any one of the target recognition methods described above when executed by a processor.

[0063] The target recognition method provided by the present invention has the beneficial effect of adding a multi-resolution generalization layer for resizing input image features in a visual perception model, and when training the visual perception model using an image sample set including image samples of multiple resolutions, after updating the parameters of the multi-resolution generalization layer using self-supervised loss, the image samples of the resolutions are input into the visual perception model and the perceptual loss value is calculated; the parameters of the visual perception model are updated using the perceptual loss values ​​corresponding to each resolution to obtain the visual perception model corresponding to the current iterative training batch, thereby performing iterative training to obtain the trained visual perception model, reducing the optimization difficulty caused by the data distribution differences of image samples of different resolutions, thereby improving the resolution generalization ability of the visual perception model to process input images of different resolutions; when performing a target recognition task, it is only necessary to update the parameters of the multi-resolution generalization layer again using self-supervised learning to adapt to the input image resolution of the application scenario, thereby improving the resolution generalization ability of the target recognition task to adapt to input images of different resolutions.

[0064] The present invention also provides a target identification device, equipment, medium and computer program product, which have the above-mentioned beneficial effects and are not described in detail here. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] In order to more clearly illustrate the embodiments of the present invention or the technical solutions of the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0066] Figure 1 A flow chart of a target recognition method provided by an embodiment of the present invention;

[0067] Figure 2 A schematic diagram of the structure of a target recognition system provided by an embodiment of the present invention;

[0068] Figure 3 A flowchart of multi-resolution loss optimization provided by an embodiment of the present invention;

[0069] Figure 4 A schematic diagram of the structure of a visual perception model with a multi-resolution generalization layer added provided by an embodiment of the present invention;

[0070] Figure 5 A training flow chart of a visual perception model provided by an embodiment of the present invention;

[0071] Figure 6 A schematic diagram of updating and iterative calculation of a visual perception model based on self-supervision loss provided by an embodiment of the present invention;

[0072] Figure 7 A schematic diagram of the structure of a multi-resolution generalization layer provided by an embodiment of the present invention;

[0073] Figure 8 A schematic diagram of the structure of another multi-resolution generalization layer provided by an embodiment of the present invention;

[0074] Fig. 9 A flowchart of self-supervisory loss calculation provided by an embodiment of the present invention;

[0075] Fig.10 A schematic diagram of a calculation flow of an image feature decoder provided by an embodiment of the present invention;

[0076] Fig.11 Another schematic diagram of updating and iterative calculation of a visual perception model based on self-supervision loss provided by an embodiment of the present invention;

[0077] Fig.12 A schematic diagram of the structure of a meta-learner provided by an embodiment of the present invention;

[0078] Fig.13 A schematic diagram of the structure of an optimization factor predictor provided by an embodiment of the present invention;

[0079] Fig.14 A schematic diagram of the structure of a target recognition device provided by an embodiment of the present invention;

[0080] Fig.15 A schematic diagram of the structure of a target recognition device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0081] The core of the present invention is to provide a target recognition method, device, equipment, medium and computer program product for improving the resolution generalization capability of adapting to input images of different resolutions in target recognition tasks.

[0082] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0083] Resolution is used to describe the clarity of an image, video or display device. It indicates the number of pixels that can be displayed per unit length. Resolution is usually expressed in the following forms:

[0084] Pixel resolution: This is the most intuitive way of expressing it, which is expressed directly in pixels. For example, 1920x1080 means there are 1920 pixels in the horizontal direction (image width) and 1080 pixels in the vertical direction (image height).

[0085] Screen resolution: Usually used to describe the resolution of a monitor, TV or other display device. For example, Full HD usually refers to a resolution of 1920x1080, while 4K resolution usually refers to 3840x2160.

[0086] Print resolution: In the printing field, resolution is usually expressed in dots per inch (DPI). For example, 300 DPI means there are 300 dots per inch.

[0087] Image scanning resolution: The resolution of the scanner is also expressed in DPI, which determines the clarity of the scanned image.

[0088] Video resolution: The resolution of a video is usually expressed in pixels, such as 720p (1280x720), 1080p (1920x1080), and 4K (3840x2160 or 4096x2160).

[0089] The image recognition method provided in the embodiment of the present invention can be used to process images captured by a visual sensor such as a camera, or to process video frames captured from a captured video. In the embodiment of the present invention, the resolution described can be pixel resolution.

[0090] The resolution generalization ability of a visual perception model refers to the ability of the visual perception model to process input images with resolutions that are inconsistent with the training resolution during the testing phase.

[0091] Traditional object recognition solutions usually use the same resolution in the model training and testing stages. This is because the resolution generalization ability of visual perception models is poor. Specifically, when the test resolution is inconsistent with the training resolution, the model accuracy will be seriously reduced. The greater the difference between the test resolution and the training resolution, the more serious the reduction in model accuracy, which limits the application of visual perception models in different scenarios. The fundamental reason for this phenomenon is that the distribution of image data with different resolutions is different, and current visual perception models have difficulty in handling such distribution differences.

[0092] However, in practical applications, the visual perception model after deployment may need to process input images with higher resolutions. For example, a visual perception model trained with a resolution of (256x704) may need to process input images with a resolution of (512x1408) or a resolution of (128x352) after deployment. To meet the application needs of processing images with higher resolutions, one current solution is to adjust the input image to the same size as the resolution of the image sample in the training phase during the application phase, that is, adjust the input image with a resolution of (512x1408) and the input image with a resolution of (128x352) to a resolution of (256x704).

[0093] Although this solution of unifying the resolution of input images is feasible, in some practical application scenarios, there is a high demand for different resolution settings. For some application scenarios with limited computing resources, the visual perception model is required to process low-resolution input images (such as 128x352 resolution in the above example) to reduce the computational burden and ensure real-time performance. If the input image resolution is increased to align it to the training resolution, the computational burden will increase and affect the real-time performance of the system. For some application scenarios with sufficient computing resources, it is hoped that a higher image resolution (such as 512x1408 resolution in the above example) will be used to improve the accuracy of 3D target detection. By reducing the high resolution of the image to align it to the training resolution, a lot of detail information will be lost, which may be detrimental to the detection of small and distant targets.

[0094] It can be seen that in actual object recognition applications, there is still a great demand for the resolution generalization ability of the visual perception model itself. However, the resolution generalization ability of the current visual perception model is very poor and it is difficult to directly apply it to scenes with different resolutions.

[0095] In order to improve the resolution generalization ability of the visual perception model, it is necessary to use image samples of different resolutions to train the visual perception model. However, due to the large differences in the feature distribution of image samples of different resolutions, model optimization is difficult to succeed.

[0096] In order to meet the requirements of target recognition tasks for processing input images of different resolutions, an embodiment of the present invention provides a target recognition solution, which adds a multi-resolution generalization layer for resizing input image features in a visual perception model. When training the visual perception model using an image sample set including image samples of multiple resolutions, after updating the parameters of the multi-resolution generalization layer using self-supervised loss, the image samples of different resolutions are input into the visual perception model and the perceptual loss value is calculated; the parameters of the visual perception model are updated using the perceptual loss values ​​corresponding to each resolution to obtain the visual perception model corresponding to the current iterative training batch, and iterative training is performed to obtain the trained visual perception model, thereby reducing the optimization difficulty caused by the data distribution difference of image samples of different resolutions, thereby improving the resolution generalization ability of the visual perception model to process input images of different resolutions; when performing the target recognition task, it is only necessary to update the parameters of the multi-resolution generalization layer again using self-supervised learning to adapt to the input image resolution of the application scenario, thereby improving the resolution generalization ability of target recognition to adapt to input images of different resolutions.

[0097] The target recognition method provided by the embodiment of the present invention is described below with reference to the accompanying drawings.

[0098] Figure 1 A flow chart of a target recognition method provided by an embodiment of the present invention; Figure 2 A schematic diagram of the structure of a target recognition system provided by an embodiment of the present invention; Figure 3 A flowchart of multi-resolution loss optimization is provided in an embodiment of the present invention.

[0099] like Figure 1 As shown, the target recognition method provided by the embodiment of the present invention includes:

[0100] S101: training a visual perception model with a multi-resolution generalization layer using an image sample set including image samples of multiple resolutions;

[0101] S102: In iterative training, for each resolution, input the image sample of the resolution into the visual perception model, obtain the intermediate image features output by the last multi-resolution generalization layer of the visual perception model, input the intermediate image features into the image feature decoder, output the decoded image, calculate the self-supervised loss according to the decoded image and the corresponding image sample, update the parameters of the multi-resolution generalization layer by using the self-supervised loss, input the image sample of the resolution into the visual perception model and calculate the perceptual loss value;

[0102] S103: updating the parameters of the visual perception model using the perceptual loss values ​​corresponding to each resolution, to obtain the visual perception model corresponding to the current iterative training batch;

[0103] S104: after reaching the iterative training end condition, obtaining a trained visual perception model;

[0104] S105: after updating the parameters of the multi-resolution generalization layer of the visual perception model according to the resolution corresponding to the target recognition task to be processed, inputting the input image of the target recognition task to be processed into the updated visual perception model to obtain the target recognition result of the target recognition task to be processed;

[0105] Among them, the multi-resolution generalization layer is used to resize the input image features.

[0106] The target recognition method provided by the embodiment of the present invention mainly includes two major links: training of the visual perception model and application of the visual perception model. In this regard, the target recognition method provided by the embodiment of the present invention can be applied to Figure 2 The target recognition system shown in the figure includes two parts: a model training system and a perception computing device.

[0107] The model training system may be composed of one or more computing devices for training a visual perception model based on an image sample set composed of image samples of multiple resolutions. Figure 2 As shown, the model training system can be composed of multiple threads of multiple artificial intelligence devices to form a thread pool, and the image samples corresponding to each resolution are assigned to a thread pool, so as to achieve parallel accelerated learning of multiple resolution image sets. It should be noted that m, n, and k only represent the meaning of multiple, and do not represent a specific number or the same number of threads of each artificial intelligence processor, nor do they mean that each thread pool is composed of one thread in each artificial intelligence device. The model training system can be deployed in a cloud computing center. The device structure of the cloud computing center is similar Figure 2 The perceptual computing devices in the system include computing, storage, network and other infrastructure, operating system, database, middleware and other software environments and application software. The model training system receives image data sets of various resolutions, uses the stored pre-trained models, combined with manual annotation, to annotate visual perception targets (such as the category, location, size of the bounding box, target orientation information, etc.) and stores the annotated data.

[0108] The perception computing device can be a computing device deployed in a cloud computing center or an edge computing device deployed in an edge device. Its structure is as follows: Figure 2As shown in the perception computing device shown. If the computing device of the cloud computing center is used, after the trained visual perception model is deployed, the edge device collects the image to be detected through a camera or other visual sensor and uploads it to the cloud computing center, and uses the perception computing device that deploys the visual perception model to perform visual perception calculation on the image to be detected to obtain the target recognition result of the visual perception target. If the perception computing device uses an edge computing device deployed on the edge device, the visual perception model is trained by the model training system and sent to the edge computing device through the communication unit, and deployed in the memory of the edge computing device. After the edge computing device collects the image to be detected through a camera or other visual sensor, it performs visual perception calculation through the local visual perception model to obtain the target recognition result of the visual perception target.

[0109] Applied to Figure 2 The multi-thread pool shown, in an embodiment of the present invention, the step of calculating the perceptual loss value of the visual perception model for image samples of each resolution may include: inputting the image samples corresponding to each resolution into different thread pools one by one, so as to obtain the perceptual loss value of the visual perception model for the image samples of each resolution by parallel calculation.

[0110] like Figure 3 As shown, in an embodiment of the present invention, when training a visual perception model, all images of each resolution can be stacked to obtain a set of input images of different resolutions, so that the training speed can be improved through batch processing.

[0111] In some optional implementations of the embodiments of the present invention, updating the parameters of the visual perception model using the perceptual loss values ​​corresponding to each resolution in S103 may include: calculating the model learning loss value of the current iteration number according to the perceptual loss values ​​corresponding to each resolution, and updating the parameters of the visual perception model according to the model learning loss value.

[0112] Among them, calculating the model learning loss value of the current iteration number according to the perceptual loss value corresponding to each resolution may include: taking the average of the loss values ​​calculated by the loss function corresponding to each resolution as the model learning loss value.

[0113] like Figure 3 As shown, for each resolution, all image samples are input into the visual perception model to obtain the corresponding visual perception results, and the loss value of the visual perception model for each resolution compared with the true value (which can be manually labeled) is calculated through the loss function. Finally, the model learning loss value of the visual perception model can be the average of the perceptual loss values ​​of the visual perception model for image samples of all resolutions, as shown in the following formula:

[0114] ;

[0115] in, represents the loss function of the visual perception model, represents the perceptual loss value corresponding to the r-th resolution, represents the loss function corresponding to the rth resolution, and k+1 represents the number of resolutions used in visual perception model training.

[0116] It is very challenging to directly use input images of multiple resolutions to train visual perception models. This is because there are large data distribution differences between input images of different resolutions, which increases the difficulty of optimizing the perception model. To solve this problem, an embodiment of the present invention provides a multi-resolution visual perception model based on self-supervised loss update, and improves the generalization ability of the visual perception model for multiple resolutions according to the self-supervised loss update method.

[0117] Figure 4 A schematic diagram of the structure of a visual perception model with a multi-resolution generalization layer added provided by an embodiment of the present invention; Figure 5 A training flow chart of a visual perception model provided by an embodiment of the present invention; Figure 6 A schematic diagram of updating and iterative calculation of a visual perception model based on self-supervision loss is provided in an embodiment of the present invention.

[0118] like Figure 4 As shown, the embodiment of the present invention proposes a multi-resolution generalization layer for adjusting the feature size of the input image, and in the iterative training of the visual perception model, for the image samples of each resolution, the parameters of the multi-resolution generalization layer are updated by self-supervised loss optimization, so that the multi-resolution generalization layer is used to adjust the image feature size of the input image according to the resolution information of the current input image, so that the visual perception model can adapt to the resolution of the current input image. On this basis, the parameters of the visual perception model are updated by the perception loss value of the image samples of each resolution through the updated visual perception model, so that the visual perception model can obtain the image feature optimization capability for input images of different resolutions in iterative training, so as to be suitable for different resolutions.

[0119] In an embodiment of the present invention, one or more multi-resolution generalization layers may be added to a traditional visual perception model.

[0120] The network structure of traditional visual perception models usually includes convolutional neural networks (CNN), residual networks (ResNet), etc., among which convolutional neural networks are used to extract image features of input images. Some visual perception models design a multi-layer network structure to continuously extract image features of input images, compress and map image features, etc., to obtain the information required for identifying targets on the input images. Taking the ResNet-50 network as an example, it contains a 50-layer network structure and is a common visual perception model. Applying the image processing method provided in the embodiment of the present invention, such as Figure 4 As shown, the ResNet-50 network is divided into five stages, and the multi-resolution generalization layer provided by the embodiment of the present invention can be added after the 0th and 1st stages respectively.

[0121] It should be noted that the visual perception model can also use models other than the ResNet-50 network, and the location of adding the multi-resolution generalization layer can be set as needed, and does not necessarily have to be added to the Figure 4 Position shown.

[0122] It is understandable that the model structure and model parameters of the multi-resolution generalization layer are related to its position in the visual perception model. Then, using the self-supervised loss in S102 to update the parameters of the multi-resolution generalization layer may include: generating the parameters of the multi-resolution generalization layer according to the self-supervised loss and the image feature size corresponding to the multi-resolution generalization layer. That is to say, after obtaining the learning loss of the visual perception model when processing the image sample of the current resolution through the self-supervised loss, the parameters of each multi-resolution generalization layer are updated in a targeted manner in combination with the position information of the multi-resolution generalization layer in the visual perception model, so that the image features of the input image of the resolution can be optimized in a targeted manner at different network layers of the visual perception model, solving the problem of difficulty in model optimization caused by large data distribution differences between input images of different resolutions.

[0123] like Figure 5 As shown, for image samples corresponding to resolutions 1, 2, ..., m, they are respectively input into the visual perception model to obtain corresponding perceptual loss values ​​1, 2, ..., m. As described in Embodiment S102 of the present invention, in one iterative training of the visual perception model, for a resolution, the image samples of the resolution are first input into the visual perception model, and the parameters of the multi-resolution generalization layer are updated according to the self-supervised loss, and then the image samples of the resolution are input into the updated visual perception model, and the perceptual loss value is obtained according to the corresponding true value calculation, such as Figure 6 shown.

[0124] Then for S103, the perception loss value corresponding to each resolution is collected as a basis for updating the parameters of the visual perception model. The training process of the visual perception model and the model migration process of deploying the visual perception model to the actual application environment in S105 are both used to update the model parameters of the visual perception model.

[0125] like Figure 2 As shown in the figure, in order to accelerate the training of the visual perception model, the self-supervised loss optimization tasks corresponding to m resolutions can be deployed in m thread pools for execution to achieve parallel accelerated training.

[0126] For S104, the iterative training end condition may be that the number of iterative training times of the visual perception model reaches a preset number of iterations, or the model learning loss value of the visual perception model is less than a preset loss value.

[0127] For S105, when the trained visual perception model is deployed in an actual application scenario, the image required for the target recognition task to be processed is input into the visual perception model to update the parameters of the multi-resolution generalization layer of the visual perception model through self-supervisory loss, and its principle is the same as the training stage in S102. The target recognition task to be processed is performed using the updated visual perception model. When the resolution of the input image of the actual application scenario changes, the parameters of the multi-resolution generalization layer of the visual perception model are updated again according to the new resolution.

[0128] The target recognition method provided by the embodiment of the present invention adds a multi-resolution generalization layer for resizing input image features in a visual perception model. When training the visual perception model using an image sample set including image samples of multiple resolutions, the parameters of the multi-resolution generalization layer are updated using self-supervised loss, and then the image samples of different resolutions are input into the visual perception model and the perceptual loss value is calculated; the parameters of the visual perception model are updated using the perceptual loss values ​​corresponding to each resolution to obtain the visual perception model corresponding to the current iterative training batch, and iterative training is performed to obtain the trained visual perception model, thereby reducing the optimization difficulty caused by the data distribution difference of image samples with different resolutions, thereby improving the resolution generalization ability of the visual perception model to process input images with different resolutions; when performing a target recognition task, it is only necessary to update the parameters of the multi-resolution generalization layer again using self-supervised learning to adapt to the input image resolution of the application scenario, thereby improving the resolution generalization ability of the target recognition task to adapt to input images with different resolutions.

[0129] Based on the above embodiment, the embodiment of the present invention further illustrates the steps of collecting the image sample set.

[0130] The image sample set used for training the visual perception model in the embodiment of the present invention includes image samples of multiple resolutions. When collecting the image sample set, a visual sensor such as a camera can be used to directly collect image samples of different resolutions through set resolution parameters to form an image sample set; when collecting the image sample set, image samples of one resolution can also be collected first, and image samples of multiple resolutions can be generated by scaling or enlarging to form an image sample set.

[0131] In some optional implementations of the embodiments of the present invention, the step of collecting the image sample set may include: obtaining an original image set; processing each original image in the original image set using different scaling factors to obtain scaled images of different resolutions corresponding to each original image; and using each original image and the scaled image corresponding to each original image as the image sample set.

[0132] In practical applications, each original image in the original image set is processed with different scaling factors to obtain scaled images of different resolutions corresponding to each original image, which may include: processing each original image with a reference scaling factor to obtain a reference scaled image corresponding to the original image; sampling multiple scaling factors from a preset distribution range, and processing each original image with each scaling factor to obtain scaled images of multiple resolutions corresponding to each original image; using the corresponding reference scaled image and each corresponding scaled image as an image set corresponding to the original image to obtain an image set corresponding to each original image. That is, the generated image sample set is composed of image sets corresponding to different original images, which include image samples of different resolutions, and different image samples of the same resolution may be obtained by processing different original images.

[0133] As described in the above embodiment of the present invention, the resolution used for processing by the visual perception model of the target recognition task is usually expressed in pixel resolution, that is, expressed as the product of the number of pixels in the horizontal direction and the number of pixels in the vertical direction. In the embodiment of the present invention, the scaling factor may include a scaling factor acting on the image height and / or a scaling factor on the image width. In order to obtain image samples of multiple resolutions with greater resolution differences, the scaling factor may include an image height scaling factor and an image width scaling factor.

[0134] To reduce the computational difficulty, the scaling factor can also be a scaling ratio for the image height and image width. For example, for an original image with a resolution of H×W , the base scaling factor can be used The processed resolution is After the image is distributed in Uniform distribution Sample k scaling factors , and process the image using each scaling factor , and finally obtain a collection of images with multiple resolutions ,in The resolution is . Where H represents the vertical size of the image, and W represents the horizontal size of the image.

[0135] When applied to the field of autonomous driving, cameras and other visual sensors are usually set up in different directions of the vehicle, such as left front (fl), front (f), right front (fr), left rear (bl), rear (b), right rear (br), etc., so that images of multiple perspectives of the road scene are collected each time. In the actual training process, in order to use the batch method to accelerate model training, the embodiment of the present invention samples a set of scaling factors at each iteration, and , the above-obtained scaling coefficient set is used for processing, that is, a multi-resolution image set is obtained for each viewing angle .

[0136] Since the resolution difference between the multi-resolution image samples used in training the visual perception model is small, it is not conducive to the training of the model's resolution generalization ability. In order to obtain as many resolutions with large differences as possible and improve the training effect of the model's resolution generalization ability, in other optional implementations of the embodiments of the present invention, the difference between the scaling factors can be set to be no less than the scaling factor difference threshold, that is, when determining the scaling factor, the difference between the scaling factor and other scaling factors in the collected scaling factor set is judged, and only when the difference is greater than the set scaling factor difference threshold is the scaling factor added to the scaling factor set, thereby ensuring the diversity between the scaling factors obtained by sampling and improving the training effect of the model's resolution generalization ability.

[0137] Furthermore, the scaling factors may be designed to all belong to a preset distribution range and the difference between any two scaling factors may not be less than a scaling factor difference threshold.

[0138] In some optional implementations of the embodiments of the present invention, the scaling factor difference threshold is calculated based on the interval length of the preset distribution range divided by twice the number of scaling factors. That is, the scaling factor difference threshold can be expressed by the following formula:

[0139] ;

[0140] Where M represents the scaling factor difference threshold, Indicates the lower limit of the preset distribution range. represents the upper limit of the preset distribution range, and k represents the total number of scaling factors to be collected from the preset distribution range.

[0141] In some other alternative embodiments of the embodiments of the present invention, for each original image in the original image set, different scaling factors are used for processing to obtain scaled images with different resolutions corresponding to each original image, which may also include: processing each original image with a reference scaling factor to obtain a reference scaled image corresponding to the original image; sampling a plurality of scaling factors from a preset distribution range until a preset number of scaling factors are collected; during the process of collecting the scaling factors, for each newly collected scaling factor, calculating the difference between the new scaling factor and the existing scaling factors, if the difference is not less than the scaling factor difference threshold, adding the new scaling factor to the set of existing scaling factors; if the difference is less than the scaling factor difference threshold, discarding the new scaling factor; respectively processing each original image with each scaling factor to obtain scaled images with multiple resolutions corresponding to each original image; using the corresponding reference scaled image and the corresponding scaled images as the image set corresponding to the original image to obtain the image set corresponding to each original image.

[0142] That is, the following algorithm can be designed to obtain the set of scaling factors:

[0143] Input: reference scaling factor , uniformly distributed in , set A of scaling factors, scaling factor difference threshold M;

[0144] Add the reference scaling factor to the set A of scaling factors ;

[0145] If the length of the set A of scaling factors < k + 1, then:

[0146] Randomly sample from the uniform distribution to obtain ;

[0147] Check the difference between and all the scaling factors in the current set A of scaling factors, if the difference is greater than the threshold scaling factor difference M, then add to the set A of scaling factors;

[0148] Output: set A of scaling factors.

[0149] To further improve the ability of the visual perception model to process input images with different resolutions, image sample sets can be generated multiple times during the model training process. That is to say, new image sample sets can be generated by re-collecting the scaling factors in each iterative training, or new image sample sets can be generated by re-collecting the scaling factors after every few iterative trainings.

[0150] In an embodiment of the present invention, since the parameters of the multi-resolution generalization layer are generated by adopting a self-supervised loss optimization method to enable the visual perception model to acquire the ability to process multi-resolution input images, if the image sample set is regenerated by the above method in each iterative training, it may cause the model to be difficult to converge.

[0151] To ensure model convergence, in some optional implementations of the embodiments of the present invention, a method of generating an image sample set once before training the visual perception model can be adopted, and the image sample set is no longer updated in subsequent iterative training. In this way, the same batch of resolutions are learned in each iterative training of the visual perception model, and the diversity of samples can be ensured by collecting different image samples corresponding to the resolutions. If computing resources and time permit, a method of generating an image sample set once before training the visual perception model can be adopted, but multiple rounds of iterative training are performed, and the image sample set is regenerated before the start of each round of iterative training, and the same image sample set is used in different iterative training steps in each round of iterative training.

[0152] In some other optional implementations of the embodiments of the present invention, the image sample set may be regenerated every multiple iterations of training, and the frequency of generating the image sample set may be determined according to the convergence of the visual perception model.

[0153] In some other optional implementations of the embodiments of the present invention, if the image sample set is regenerated in each iterative training or every few iterative trainings, the multiple resolutions corresponding to the resampling scaling factors can arrange the resolutions in a fixed order (such as from small to large or from large to small) according to the scaling factors, thereby ensuring the correspondence between the arrangement of the resolutions and the arrangement of the loss function, so that the model training can inherit the experience of the previous iterative training or the previous few iterative trainings.

[0154] In each embodiment of the present invention, the loss function corresponding to the resolution of the visual perception model in each iterative training number inherits the parameters of the loss function of the corresponding position in the previous iterative training, but the corresponding resolution can be the resolution in the previous iterative training or the updated resolution.

[0155] On the basis of the above embodiments, in order to adapt to the problem that the multi-resolution model is difficult to optimize, the embodiment of the present invention continues to explain the structure of the multi-resolution generalization layer.

[0156] The function of the multi-resolution generalization layer is to optimize and adjust the image features of input images of different resolutions.

[0157] Figure 7 A schematic diagram of the structure of a multi-resolution generalization layer provided in an embodiment of the present invention.

[0158] In some optional implementations of the embodiments of the present invention, the calculation step of the multi-resolution generalization layer may include: performing a channel feature weighting operation on the input image features, and outputting the adjusted image features.

[0159] The basis (channel weight) for the multi-resolution generalization layer to perform channel feature weighted operations on image features comes from the self-supervised loss obtained in S102. The feature channel weighted operation can be implemented through a linear layer, that is, the parameters of the linear layer of the multi-resolution generalization layer are updated by optimizing the self-supervised loss.

[0160] Figure 8 A schematic diagram of the structure of another multi-resolution generalization layer provided in an embodiment of the present invention.

[0161] In some other optional implementations of the embodiments of the present invention, the calculation steps of the multi-resolution generalization layer may include: performing deep feature extraction on the input image features and generating attention weights based on resolution respectively; splicing the deep feature extraction results with the generated attention weights and outputting them. By performing deep feature extraction on the input image features, the hierarchical features of the input image can be further learned. By generating resolution-based attention weights, the generalization ability for different resolutions can be improved.

[0162] To this end, the embodiment of the present invention designs a structure of a multi-resolution generalization layer as follows: Figure 8 shown.

[0163] In the multi-resolution generalization layer, the input image features can be processed through two depth separation convolutions and a multi-resolution normalization layer to achieve deep feature extraction of the input image features. The deep feature extraction of the input image features can include: performing a first depth separation convolution operation, a normalization operation, and a second depth separation convolution operation on the input image features in sequence to obtain a deep feature extraction result.

[0164] like Figure 8 As shown, in the multi-resolution generalization layer, the input image features can be processed by global mean pooling and linear layer to realize resolution-based attention weight generation. Then, the resolution-based attention weight generation for the input image features can include: performing global mean pooling operations and linear layer operations on the input image features in sequence to obtain attention weights.

[0165] Then, using the self-supervised loss to update the parameters of the multi-resolution generalization layer may include: using the self-supervised loss to update the parameters of the linear layer of the multi-resolution generalization layer. That is, only the parameters of the linear layer of the multi-resolution generalization layer may be updated using the self-supervised loss, and the parameters of other network layers of the multi-resolution generalization layer may use preset parameters, thereby simplifying the calculation and accelerating the convergence of the model.

[0166] The parameters of the i-th multi-resolution generalization layer are Applications such as Figure 8 The structure of the multi-resolution generalization layer shown in the figure, in the first calculation branch of the multi-resolution generalization layer, for the input image features , To use the first depth-wise separable convolution Processing, using a multi-resolution normalization layer Perform a normalization operation followed by a second depthwise separable convolution Processing to obtain image features , the process can be expressed by the following formula:

[0167] ;

[0168] in, represents the first depth-wise separation convolution operation, represents the normalization operation, Represents the second depth-wise separable convolution operation.

[0169] In the second computation branch (attention branch) of the multi-resolution generalization layer, global mean pooling is used Processing input image features , followed by a linear layer Processing, get attention weight , the process can be expressed by the following formula:

[0170] ;

[0171] in, represents the global mean pooling calculation, Represents a linear layer computation.

[0172] Finally, the image features are weighted to obtain the image features output by the multi-resolution generalization layer. :

[0173] .

[0174] On the basis of the above embodiment, in order to adapt to the problem that the multi-resolution model is difficult to optimize, the embodiment of the present invention continues to explain the steps of optimizing the self-supervised loss of the multi-resolution generalization layer.

[0175] Fig. 9 A flowchart of self-supervisory loss calculation provided by an embodiment of the present invention; Fig.10 A schematic diagram of a calculation flow of an image feature decoder provided in an embodiment of the present invention.

[0176] To implement the calculation of the self-supervised loss as described in S102, the embodiment of the present invention provides a process of calculating the self-supervised loss as follows: Fig. 9 As shown in the figure, the last multi-resolution generalization layer and the previous network layers of the visual perception model are denoted as image feature extractors. Image features are extracted from the input image (i.e., image samples) to obtain intermediate image features. An image feature decoder is designed to decode the intermediate image features to obtain a decoded image. The loss of the decoded image and the corresponding image sample is calculated as the self-supervised loss.

[0177] As described in the above embodiments of the present invention, Figure 4 The structure of the visual perception model shown adopts the ResNet-50 network and adds multi-resolution generalization layers to the 0th and 1st stages. The 0th and 1st stages and the two multi-resolution generalization layers are recorded as image feature extractors, and the image samples are input into the image feature extractor to obtain the intermediate image features.

[0178] In some optional implementations of the embodiments of the present invention, inputting intermediate image features into an image feature decoder and outputting a decoded image may include: performing convolution operations and feature rearrangement on the intermediate image features according to the downsampling multiples of the last multi-resolution generalization layer and the previous network layers in the visual perception model to obtain a first intermediate image feature; upsampling the intermediate image features by the same multiple according to the downsampling multiple to obtain a second intermediate image feature; and obtaining a decoded image based on the first intermediate image features and the second intermediate image features.

[0179] Among them, according to the downsampling multiples of the last multi-resolution generalization layer and the previous network layers in the visual perception model, convolution operations and feature rearrangement are performed on the intermediate image features to obtain the first intermediate image features, which may include: respectively using the first convolution layer and the second convolution layer to perform convolution operations on the intermediate image features to obtain the third intermediate image features; performing feature rearrangement on the third intermediate image features so that the size of the third intermediate image features is the same as the image samples of the corresponding resolution; wherein the number of channels of the convolution kernel of the first convolution layer is the same as the number of channels of the intermediate image features, and the number of channels of the convolution kernel of the second convolution layer is the number of channels of the convolution kernel of the first convolution layer divided by 2 and rounded down.

[0180] The intermediate image feature is , where C is the number of channels of the intermediate image features, and s is the feature downsampling multiple of the above image feature extractor. The decoded image is output by the image feature decoder, and the self-supervised loss is calculated for the decoded image and the input image. .

[0181] like Fig.10 As shown, in the first computation branch of the image feature decoder, two convolutional layers can be used to process the intermediate image features. Get image features , the first convolutional layer Include indivual The convolution kernel is , the second convolutional layer Include indivual The convolution kernel, ,in Indicates a round-down operation. , the process can be expressed as the following formula:

[0182] ;

[0183] in, , represents the third intermediate image feature, represents the calculation of the first convolutional layer, Represents the calculation of the second convolutional layer.

[0184] Then the third intermediate image feature Perform feature rearrangement operation to obtain the first image feature .

[0185] In the second computation branch of the image feature decoder, the intermediate image features Perform s-fold upsampling , followed by a third convolutional layer Processing upsampled image features , Contains 3 The convolution kernel of the second intermediate image is obtained :

[0186] ;

[0187] in, represents s times upsampling, Represents the calculation of the third convolutional layer.

[0188] Finally, the first image feature and the second intermediate image feature Add together to obtain the image features of the final decoded image :

[0189] .

[0190] By applying the image feature decoder provided by the embodiment of the present invention, the process of decoding the image features is implemented in the first calculation branch, and the second calculation branch is equivalent to a residual connection, which is used to compensate for the information lost in the decoding process.

[0191] In an embodiment of the present invention, calculating the self-supervised loss according to the decoded image and the corresponding image sample in S102 may include: calculating the self-supervised loss according to the second intermediate image features of all image feature positions of the image sample of the corresponding resolution and the decoded image.

[0192] In some optional implementations of the embodiments of the present invention, the self-supervised loss is calculated based on the second intermediate image features and the decoded image at all positions of the image samples of the corresponding resolution, which may include: respectively calculating the self-supervised loss value corresponding to each image feature position; and taking the average of the self-supervised loss values ​​at each image feature position as the self-supervised loss corresponding to the resolution.

[0193] The self-supervision loss The calculation method can be expressed by the following formula:

[0194] ;

[0195] Among them, H represents the vertical size of the image sample, W represents the horizontal size of the image sample, and C represents the number of channels of the intermediate image features. The second intermediate image feature corresponding to the image feature position with the vertical coordinate h and the horizontal coordinate w of the decoded image in the c-th channel representing the image feature of the decoded image, The image feature representing the image feature of the decoded image in the cth channel corresponding to the image feature position with the vertical coordinate h and the horizontal coordinate w of the decoded image, The self-supervised loss value corresponding to the image feature position with vertical coordinate h and horizontal coordinate w of the decoded image in the c-th channel representing the image feature of the decoded image.

[0196] In some optional implementations of the embodiments of the present invention, the step of calculating the self-supervised loss value corresponding to the image feature position may include: if the absolute value of the difference between the second intermediate image feature corresponding to the image feature position and the image feature at the corresponding position of the decoded image is less than a preset constant, the self-supervised loss value is the square of the second intermediate image feature and the decoded image divided by 2; if the absolute value of the difference between the second intermediate image feature corresponding to the image feature position and the image feature at the corresponding position of the decoded image is greater than or equal to the preset constant, the self-supervised loss value is the absolute value of the difference between the second intermediate image feature and the decoded image minus 0.5. The process can be expressed by the following formula:

[0197] .

[0198] The multi-resolution generalization layer self-supervised update method provided in the embodiment of the present invention does not rely on perception task annotation data, and realizes the adjustment of the multi-resolution generalization layer in the visual perception model. Therefore, it can be used not only in the training stage of the visual perception model, but also during the reasoning period of the visual perception model, which can effectively improve the generalization ability of the visual perception model for unknown resolutions.

[0199] On the basis of the above embodiment, in the target recognition method provided by the embodiment of the present invention, the use of the self-supervised loss in S102 to update the parameters of the multi-resolution generalization layer may also include: using the self-supervised loss corresponding to the resolution and the learning rate corresponding to the multi-resolution generalization layer to update the parameters of the multi-resolution generalization layer. That is, the parameter update process of each multi-resolution generalization layer can be controlled by setting the learning rate, so as to further optimize the multi-resolution generalization layer.

[0200] The learning rate corresponding to the multi-resolution generalization layer may be a preset learning rate. To further improve the resolution generalization capability of the visual perception model, the learning rate corresponding to the multi-resolution generalization layer may also be a parameter updated by self-learning in the iterative training of the visual perception model.

[0201] In some optional implementations of the embodiments of the present invention, the self-learning update step of the learning rate corresponding to the multi-resolution generalization layer may include: using a meta-learner to generate the learning rate of each multi-resolution generalization layer according to the image feature size corresponding to the resolution.

[0202] An embodiment of the present invention provides a solution for generating a learning rate corresponding to a multi-resolution generalization layer based on a meta-learner. Fig.11 Another schematic diagram of updating and iterative calculation of a visual perception model based on self-supervision loss provided by an embodiment of the present invention. Fig.11 As shown in the figure, while constructing a multi-resolution self-supervised task to calculate the self-supervised loss, the meta-learner can be used to generate the learning rate corresponding to the multi-resolution generalization layer according to the input image, and jointly participate in the parameter update of the multi-resolution generalization layer.

[0203] Fig.12 A schematic diagram of the structure of a meta-learner provided by an embodiment of the present invention; Fig.13 A schematic diagram of the structure of an optimization factor predictor provided by an embodiment of the present invention.

[0204] like Fig.12As shown, the structure of the meta-learner provided by the embodiment of the present invention can be divided into two parts: an image feature extraction module and an optimization factor prediction module. Among them, the image feature extraction module may include a convolutional network layer and a global mean pooling layer for extracting image features from an input image (image sample), and output an image feature vector. The optimization factor prediction module may include n optimization factor predictors, which correspond one-to-one to the multi-resolution generalization layer, and are used to predict the learning rate of the corresponding multi-resolution generalization layer.

[0205] In some optional implementations of the embodiments of the present invention, Fig.13 As shown, the self-learning update step of the learning rate corresponding to the multi-resolution generalization layer may include: extracting image features from image samples of resolution to obtain image feature vectors; for each multi-resolution generalization layer, using the image feature vector as the key vector and value vector, using the query vector corresponding to the multi-resolution generalization layer, performing self-attention calculation according to the query vector, key vector and value vector, and performing linear layer processing on the self-attention calculation result to obtain the learning rate corresponding to the multi-resolution generalization layer. Among them, the self-attention calculation according to the query vector, key vector and value vector can be implemented by the self-attention layer of the Transformer model. The query vector corresponding to each multi-resolution generalization layer can be obtained by random initialization and self-learning update during the iterative training process of the visual perception model.

[0206] To speed up the learning of the query vector, in some other optional implementations of the embodiments of the present invention, the self-learning update step of the learning rate corresponding to the multi-resolution generalization layer may also include: extracting image features from image samples of resolution to obtain image feature vectors; for each multi-resolution generalization layer, using the image feature vector as a key vector and a value vector, determining the query vector according to the image feature size corresponding to the multi-resolution generalization layer, performing self-attention calculation according to the query vector, the key vector and the value vector, and performing linear layer processing on the self-attention calculation result to obtain the learning rate corresponding to the multi-resolution generalization layer. That is, the image feature size information corresponding to the multi-resolution generalization layer can be introduced into the generation process of the query vector so as to obtain the query vector adapted to the multi-resolution generalization layer more quickly.

[0207] Based on the scheme provided by the embodiment of the present invention, the learning rate corresponding to the multi-resolution generalization layer is generated by meta-learning and the parameters of the multi-resolution generalization layer are updated by self-supervised learning. An image sample of one resolution is input into the meta-learner to obtain the learning rate of each multi-resolution generalization layer. , represents the learning rate of the i-th multi-resolution generalization layer, and n represents the number of resolution generalization layers; the self-supervised loss is obtained using the multi-resolution self-supervised task. , using self-supervised loss and the parameters of the multi-resolution generalization layer Parameters of each multi-resolution generalization layer of the visual perception model Update the parameters of the i-th multi-resolution generalization layer The update formula can be as follows:

[0208] ;

[0209] in, represents the product of corresponding elements, represents the gradient of the parameters of the i-th multi-resolution generalization layer calculated using the self-supervised loss of the current resolution.

[0210] The target recognition method provided by the embodiment of the present invention may further include: in the iterative training of the visual perception model, updating the parameters of the meta-learner according to the perceptual loss values ​​corresponding to each resolution.

[0211] Updating the parameters of the meta-learner according to the perceptual loss values ​​corresponding to each resolution may include: calculating the model learning loss value of the visual perception model by the sum of the perceptual loss values ​​corresponding to each resolution; and updating the parameters of the meta-learner using the model learning loss value. For example, a loss function of a visual perception model provided in the above embodiment of the present invention may be used. .

[0212] Updating the parameters of the meta-learner according to the perceptual loss values ​​corresponding to each resolution may include: updating the parameters of the meta-learner according to the current parameters of the meta-learner, the learning rate corresponding to the meta-learner, and the perceptual loss values ​​corresponding to each resolution.

[0213] Based on this, the parameter update formula of the meta-learner can be shown as follows:

[0214] ;

[0215] in, represents the updated parameters of the meta-learner, represents the parameters of the meta-learner, represents the learning rate corresponding to the meta-learner, Represents the gradient of the meta-learner parameters calculated using the model learning loss value of the current iteration training.

[0216] The learning rate corresponding to the meta-learner may be a fixed parameter or a self-learning parameter. The image processing method provided by the embodiment of the present invention may further include: updating the learning rate corresponding to the meta-learner in the iterative training of the visual perception model.

[0217] Based on the method of using a meta-learner to update the parameters of a multi-resolution generalization layer of a visual perception model introduced in the above embodiment, the embodiment of the present invention continues to explain the training steps and reasoning steps of the visual perception model.

[0218] As described in the above embodiments of the present invention, the training task of the visual perception model can be executed in a cloud computing center.

[0219] For example, when applied to the field of autonomous driving, a visual perception model can be trained on collected and labeled road scene image data. The multi-resolution meta-learning optimization method provided by an embodiment of the present invention is used to construct m multi-resolution meta-learning tasks in each training batch. In the i-th meta-learning task, the meta-learner is first used to predict the parameters of the multi-resolution generalization layer. After the parameters of the multi-resolution generalization layer are updated, the updated visual perception model is used to input the image sample corresponding to the i-th resolution to obtain the perception result and the perception loss value is calculated according to the true value. , collect all the perception loss values ​​in the training batch and substitute them into the loss function of the visual perception model to obtain the model learning loss value, and then optimize the model through the back propagation algorithm. Among them, the optimizer can adopt the gradient descent (GD) optimizer, the adaptive moment estimation (Adam) optimizer, the adaptive weight decay regularization (Adam Weight Decay Regularization, AdamW) optimizer, etc. During training, the multi-resolution parallel acceleration method proposed in the present invention is used to improve the training efficiency. After completing the model training, the perception model obtained by training is deployed to the edge computing platform of the unmanned vehicle.

[0220] The target recognition method provided by the embodiment of the present invention can be applied to, but not limited to, the field of autonomous driving, and can also be applied to intelligent terminals such as intelligent robots, smart homes, and mobile phones, and can also be applied to visual perception models involved in industrial production fields such as transportation, aerial remote sensing, and industrial product assembly. For example, the visual perception model can be an autonomous driving perception model; for this, the image sample set can be constructed based on image samples collected by vehicle-mounted sensors from different perspectives, and the visual perception model can be trained on the collected and annotated road scene image data set when executing model training in the cloud data center.

[0221] In an embodiment of the present invention, updating the parameters of the visual perception model using the perceptual loss values ​​corresponding to each resolution may include: calculating the model learning loss value of the visual perception model using the sum of the perceptual loss values ​​corresponding to each resolution; and updating the parameters of the visual perception model using the model learning loss value. For example, a loss function of a visual perception model provided in the above embodiment of the present invention may be used. .

[0222] In some optional implementations of the embodiments of the present invention, the parameter updating step of the visual perception model includes: before the iterative training of the visual perception model starts, randomly initializing the parameters of the visual perception model; during the iterative training of the visual perception model, updating the parameters of the visual perception model according to the perceptual loss values ​​corresponding to each resolution.

[0223] Among them, updating the parameters of the visual perception model according to the perceptual loss values ​​corresponding to each resolution may include: updating the parameters of the visual perception model according to the current parameters of the visual perception model, the learning rate corresponding to the visual perception model, and the perceptual loss values ​​corresponding to each resolution.

[0224] Based on this, the parameter update formula of the visual perception model can be shown as follows:

[0225] ;

[0226] in, represents the parameters of the updated visual perception model, represents the parameters of the visual perception model, represents the learning rate of the visual perception model, Represents the gradient of the parameters of the visual perception model calculated using the model learning loss value trained at the current iteration.

[0227] The learning rate corresponding to the visual perception model can be a fixed parameter or a self-learning parameter. Then the image processing method provided by the embodiment of the present invention may also include:

[0228] Update the learning rate corresponding to the visual perception model during the iterative training of the visual perception model.

[0229] During the execution of the target recognition task using the visual perception model, a meta-learner is used to generate a learning rate corresponding to the multi-resolution generalization layer according to the resolution corresponding to the target recognition task to be processed, and after updating the parameters of the multi-resolution generalization layer in combination with the self-supervised loss method provided in an embodiment of the present invention, the target recognition task to be processed is executed using the updated visual perception model, and the target recognition result of the target recognition task to be processed is output.

[0230] For example, when applied in the field of autonomous driving, since the image resolution obtained by the camera sensor of each autonomous driving vehicle is usually fixed, for each autonomous driving vehicle, the parameter update of the multi-resolution generalization layer is performed only once after the visual perception model is deployed. There is no need to perform this step again afterwards, and the visual perception model can be used directly for prediction. Only when the image resolution changes again, the process of updating the parameters of the multi-resolution generalization layer needs to be executed again, so it will not affect the normal reasoning efficiency.

[0231] It should be noted that in the embodiments of the target recognition method of the present invention, some of the steps or features may be ignored or not executed. The hardware or software functional modules divided for the convenience of description are not the only implementation form of the target recognition method provided by the embodiment of the present invention.

[0232] The above describes in detail various embodiments corresponding to the target recognition method. On this basis, the present invention also discloses a target recognition device, equipment, non-volatile storage medium and computer program product corresponding to the above method.

[0233] Fig.14 A schematic diagram of the structure of a target recognition device provided by an embodiment of the present invention.

[0234] like Fig.14 As shown, the target recognition device provided by the embodiment of the present invention includes:

[0235] The model training unit 1401 is used to train a visual perception model with a multi-resolution generalization layer added using an image sample set including image samples of multiple resolutions; in iterative training, for each resolution, input the image sample of the resolution into the visual perception model, obtain the intermediate image features output by the last multi-resolution generalization layer of the visual perception model, input the intermediate image features into the image feature decoder, output the decoded image, calculate the self-supervised loss according to the decoded image and the corresponding image sample, update the parameters of the multi-resolution generalization layer using the self-supervised loss, input the image sample of the resolution into the visual perception model and calculate the perceptual loss value; update the parameters of the visual perception model using the perceptual loss values ​​corresponding to each resolution, and obtain the visual perception model corresponding to the current iterative training batch; after the iterative training end condition is met, obtain the trained visual perception model;

[0236] A model migration unit 1402 is used to update the parameters of the multi-resolution generalization layer of the visual perception model according to the resolution corresponding to the target recognition task to be processed, so as to obtain an updated visual perception model;

[0237] The target recognition unit 1403 is used to input the input image of the target recognition task to be processed into the updated visual perception model to obtain the target recognition result of the target recognition task to be processed;

[0238] Among them, the multi-resolution generalization layer is used to resize the input image features.

[0239] It should be noted that in each implementation of the target identification device provided by the embodiment of the present invention, the division of units is only a logical functional division, and other division methods can be adopted. The connection method between different units can be electrical, mechanical or other connection methods. The separated units can be located in the same physical location or distributed on multiple network nodes. Each unit can be implemented in the form of hardware or in the form of a software functional unit. That is, part or all of the units provided in the embodiment of the present invention can be selected according to actual needs and the corresponding connection method or integration method can be adopted to achieve the purpose of the embodiment of the present invention.

[0240] Since the embodiments of the apparatus part correspond to the embodiments of the method part, please refer to the description of the embodiments of the method part for the embodiments of the apparatus part, which will not be repeated here.

[0241] Fig.15 A schematic diagram of the structure of a target recognition device provided by an embodiment of the present invention.

[0242] like Fig.15 As shown, the target recognition device provided by the embodiment of the present invention includes: a memory 1510, used to store a computer program 1511; a processor 1520, used to execute the computer program 1511, and when the computer program 1511 is executed by the processor 1520, the steps of the target recognition method provided by any of the above embodiments are implemented.

[0243] Among them, the processor 1520 may include one or more processing cores, such as a 3-core processor, an 8-core processor, etc. The processor 1520 may be implemented in at least one hardware form of digital signal processing (DSP), field-programmable gate array (FPGA), and programmable logic array (PLA). The processor 1520 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a central processing unit (CPU); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 1520 may be integrated with a graphics processing unit (GPU), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 1520 may also include an artificial intelligence (AI) processor, which is used to process computing operations related to machine learning.

[0244] The memory 1510 may include one or more non-volatile storage media, which may be non-transitory. The memory 1510 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In this embodiment, the memory 1510 is at least used to store the following computer program 1511, wherein the computer program 1511 can implement the relevant steps in the target recognition method disclosed in any of the aforementioned embodiments after being loaded and executed by the processor 1520. In addition, the resources stored in the memory 1510 may also include an operating system 1512 and data 1513, etc., and the storage method may be temporary storage or permanent storage. Among them, the operating system 1512 may be Windows or other types of operating systems. Data 1513 may include, but is not limited to, the data involved in the above method.

[0245] In some embodiments, the target recognition device may further include a display screen 1530 , a power supply 1540 , a communication interface 1550 , an input / output interface 1560 , a sensor 1570 , and a communication bus 1580 .

[0246] Those skilled in the art will understand that Fig.15 The structure shown in the figure does not constitute a limitation on the target identification device, and may include more or fewer components than those shown in the figure.

[0247] The target recognition device provided by the embodiment of the present invention includes a memory and a processor. When the processor executes the program stored in the memory, it can implement the steps of the target recognition method provided by the above embodiment, and the effect is the same as above.

[0248] An embodiment of the present invention provides a non-volatile storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the target recognition method provided in any one of the above embodiments can be implemented.

[0249] The non-volatile storage medium may include: a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and other media that can store program codes.

[0250] For the introduction of the non-volatile storage medium provided in the embodiment of the present invention, please refer to the above method embodiment, and the effect thereof is the same as the target recognition method provided in the embodiment of the present invention, and the present invention will not elaborate on it here.

[0251] An embodiment of the present invention provides a computer program product, including a computer program, which, when executed by a processor, implements the steps of the target recognition method provided in any one of the above embodiments.

[0252] For an introduction to the computer program product provided by the embodiment of the present invention, please refer to the above method embodiment, and the effect thereof is the same as the target recognition method provided by the embodiment of the present invention, and the present invention will not elaborate on it here.

[0253] The above is a detailed introduction to a target identification method, device, equipment, medium and computer program product provided by the present invention. The various embodiments in the specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same and similar parts between the embodiments can be referred to each other. For the devices, equipment, non-volatile storage media and computer program products disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple, and the relevant parts can be referred to the method part description. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of the present invention, the present invention can also be improved and modified in a number of ways, and these improvements and modifications also fall within the scope of protection of the present invention.

[0254] It should also be noted that, in this specification, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the statement "comprises a ..." does not exclude the presence of other identical elements in the process, method, article or device including the element.

Claims

1. A target recognition method, characterized in that: include: Using an image sample set including image samples of multiple resolutions to train a visual perception model to which a multi-resolution generalization layer is added; In iterative training, for each resolution, an image sample of the resolution is input into the visual perception model, an intermediate image feature output by the last multi-resolution generalization layer of the visual perception model is obtained, the intermediate image feature is input into an image feature decoder, a decoded image is output, a self-supervised loss is calculated based on the decoded image and the corresponding image sample, and after the parameters of the multi-resolution generalization layer are updated using the self-supervised loss, the image sample of the resolution is input into the visual perception model and a perceptual loss value is calculated; Using the perceptual loss values ​​corresponding to each of the resolutions, the parameters of the visual perception model are updated to obtain the visual perception model corresponding to the current iterative training batch; After reaching the iterative training end condition, obtaining the trained visual perception model; After updating the parameters of the multi-resolution generalization layer of the visual perception model according to the resolution corresponding to the target recognition task to be processed, inputting the input image of the target recognition task to be processed into the updated visual perception model to obtain a target recognition result for the target recognition task to be processed; The multi-resolution generalization layer is used to resize the input image features.

2. The target recognition method according to claim 1, characterized in that: Inputting the intermediate image features into an image feature decoder and outputting a decoded image, comprising: According to the downsampling multiples of the last multi-resolution generalization layer and the previous network layers in the visual perception model, convolution operation and feature rearrangement are performed on the intermediate image features to obtain a first intermediate image feature; According to the downsampling multiple, upsampling the intermediate image feature by the same multiple is performed to obtain a second intermediate image feature; The decoded image is obtained according to the first intermediate image feature and the second intermediate image feature.

3. The target recognition method according to claim 2, characterized in that: According to the downsampling multiples of the last multi-resolution generalization layer and the previous network layer in the visual perception model, the intermediate image features are subjected to convolution operation and feature rearrangement to obtain the first intermediate image features, including: After performing convolution operation on the intermediate image feature using the first convolution layer and the second convolution layer respectively, a third intermediate image feature is obtained; Rearranging the features of the third intermediate image so that the size of the features of the third intermediate image is the same as the corresponding image sample of the resolution; The number of channels of the convolution kernel of the first convolution layer is the same as the number of channels of the intermediate image features, and the number of channels of the convolution kernel of the second convolution layer is the value rounded down after dividing the number of channels of the convolution kernel of the first convolution layer by 2.

4. The target recognition method according to claim 2, characterized in that: Calculating a self-supervised loss based on the decoded image and the corresponding image sample, including: The self-supervised loss is calculated based on the second intermediate image features of all image feature positions of the image samples of the corresponding resolution and the decoded image.

5. The target recognition method according to claim 4, characterized in that: The self-supervisory loss is calculated according to the second intermediate image features at all positions of the image samples of the corresponding resolution and the decoded image, including: Calculate the self-supervision loss value corresponding to each image feature position respectively; The mean of the self-supervised loss values ​​at each of the image feature positions is taken as the self-supervised loss corresponding to the resolution.

6. The target recognition method according to claim 5, characterized in that: The step of calculating the self-supervised loss value corresponding to the image feature position includes: If the absolute value of the difference between the second intermediate image feature corresponding to the image feature position and the image feature at the corresponding position of the decoded image is less than a preset constant, the self-supervised loss value is the square of the second intermediate image feature and the decoded image divided by 2; If the absolute value of the difference between the second intermediate image feature corresponding to the image feature position and the image feature at the corresponding position of the decoded image is greater than or equal to the preset constant, the self-supervised loss value is the absolute value of the difference between the second intermediate image feature and the decoded image minus 0.

5.

7. The target recognition method according to claim 1, characterized in that: Updating the parameters of the multi-resolution generalization layer using the self-supervised loss includes: The parameters of the multi-resolution generalization layer are updated using the self-supervised loss corresponding to the resolution and the learning rate corresponding to the multi-resolution generalization layer.

8. The target recognition method according to claim 7, characterized in that: The learning rate corresponding to the multi-resolution generalization layer is a parameter updated by self-learning during the iterative training of the visual perception model.

9. The target recognition method according to claim 8, characterized in that: The self-learning update step of the learning rate corresponding to the multi-resolution generalization layer includes: Extracting image features from image samples of the resolution to obtain image feature vectors; For each of the multi-resolution generalization layers, the image feature vector is used as a key vector and a value vector, a query vector corresponding to the multi-resolution generalization layer is adopted, self-attention calculation is performed according to the query vector, the key vector and the value vector, and the self-attention calculation result is processed by a linear layer to obtain a learning rate corresponding to the multi-resolution generalization layer.

10. The target recognition method according to claim 8, characterized in that: The self-learning update step of the learning rate corresponding to the multi-resolution generalization layer includes: Extracting image features from image samples of the resolution to obtain image feature vectors; For each of the multi-resolution generalization layers, the image feature vector is used as a key vector and a value vector, a query vector is determined according to the image feature size corresponding to the multi-resolution generalization layer, a self-attention calculation is performed according to the query vector, the key vector and the value vector, and the self-attention calculation result is processed by a linear layer to obtain a learning rate corresponding to the multi-resolution generalization layer.

11. The target recognition method according to claim 8, characterized in that: The self-learning update step of the learning rate corresponding to the multi-resolution generalization layer includes: A meta-learner is used to generate a learning rate for each of the multi-resolution generalization layers according to the image feature size corresponding to the resolution.

12. The target recognition method according to claim 11, characterized in that: Also includes: In the iterative training of the visual perception model, the parameters of the meta-learner are updated according to the perceptual loss values ​​corresponding to each of the resolutions.

13. The target recognition method according to claim 12, characterized in that: Updating the parameters of the meta-learner according to the perceptual loss values ​​corresponding to the resolutions includes: The parameters of the meta-learner are updated according to the current parameters of the meta-learner, the learning rate corresponding to the meta-learner, and the perceptual loss value corresponding to each of the resolutions.

14. The target recognition method according to claim 13, characterized in that: Also includes: The learning rate corresponding to the meta-learner is updated during the iterative training of the visual perception model.

15. The target recognition method according to claim 1, characterized in that: The parameter updating step of the visual perception model comprises: Before the iterative training of the visual perception model begins, randomly initializing the parameters of the visual perception model; In the iterative training of the visual perception model, the parameters of the visual perception model are updated according to the perceptual loss values ​​corresponding to each of the resolutions.

16. The target recognition method according to claim 15, characterized in that: Updating the parameters of the visual perception model according to the perception loss values ​​corresponding to the resolutions includes: The parameters of the visual perception model are updated according to the current parameters of the visual perception model, the learning rate corresponding to the visual perception model, and the perceptual loss value corresponding to each of the resolutions.

17. The target recognition method according to claim 16, characterized in that: Also includes: The learning rate corresponding to the visual perception model is updated during the iterative training of the visual perception model.

18. A target recognition device, characterized in that: include: A model training unit, for training a visual perception model with a multi-resolution generalization layer added using an image sample set including image samples of multiple resolutions; In iterative training, for each of the resolutions, image samples of the resolution are input into the visual perception model, intermediate image features output by the last multi-resolution generalization layer of the visual perception model are obtained, the intermediate image features are input into an image feature decoder, a decoded image is output, a self-supervised loss is calculated based on the decoded image and the corresponding image samples, and after the parameters of the multi-resolution generalization layer are updated using the self-supervised loss, image samples of the resolution are input into the visual perception model and a perceptual loss value is calculated; the perceptual loss values ​​corresponding to each of the resolutions are used to update the parameters of the visual perception model, and the visual perception model corresponding to the current iterative training batch is obtained; After reaching the iterative training end condition, obtaining the trained visual perception model; A model migration unit, used for updating the parameters of the multi-resolution generalization layer of the visual perception model according to the resolution corresponding to the target recognition task to be processed, so as to obtain the updated visual perception model; A target recognition unit, used for inputting the input image of the target recognition task to be processed into the updated visual perception model to obtain a target recognition result for the target recognition task to be processed; The multi-resolution generalization layer is used to resize the input image features.

19. A target recognition device, characterized in that: include: Memory for storing computer programs; A processor is used to execute the computer program, and when the computer program is executed by the processor, the steps of the target recognition method as described in any one of claims 1 to 17 are implemented.

20. A non-volatile storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the target recognition method according to any one of claims 1 to 17 are implemented.

21. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the target recognition method according to any one of claims 1 to 17 are implemented.

Images