A target perception method, device, equipment, medium and computer program product

Abstract

The present invention relates to the field of computer vision technology, and specifically discloses a target perception method, device, equipment, medium and computer program product. By adding a resolution-independent feature extraction network for extracting resolution-independent features from input image features to a visual perception model, in an iterative training process, in the process of using the visual perception model to process image samples of different resolutions, resolution-independent features are extracted based on the resolution-independent feature extraction network, and the visual perception model is used to update the parameters of the visual perception model and the parameters of the resolution-independent feature extraction network based on the perception loss value of each resolution. By reducing the interference between different resolutions during the training process, the optimization of the visual perception model that can process multi-resolution input images is achieved, so that when the trained visual perception model is used to perform a target perception task, accurate target perception results can be guaranteed for input images of different resolutions that need to be processed.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a target perception method, device, equipment, medium and computer program product. Background Art

[0002] Object perception is a common application of computer vision. Its main task is to obtain corresponding scene information by processing the collected images. It is widely used in autonomous driving, intelligent robots, smart homes, smart terminals, as well as industrial production such as transportation, aerial remote sensing, and industrial product assembly.

[0003] When performing target perception tasks, the visual perception model determines the accuracy of perception. In practical applications, it is sometimes necessary for the visual perception model to be able to process input images of different resolutions. For example, for application scenarios with limited computing resources, lower-resolution image samples are usually used to train the visual perception model, while for perception scenarios of small targets and long-distance targets, higher-resolution image samples are used to train the visual perception model. However, if the resolution of the input image of the visual perception model in the inference calculation stage is different from the resolution of the image samples in the training stage, the detection accuracy of the visual perception model will be significantly reduced. As a result, in practical applications, once the visual perception model is deployed, it can only process input images with a fixed resolution.

[0004] How to improve the resolution generalization capability of adapting to input images of different resolutions in target perception tasks is a technical problem that technical personnel in this field need to solve. Summary of the invention

[0005] The purpose of the present invention is to provide a target perception method, device, equipment, medium and computer program product for improving the resolution generalization capability of adapting to input images of different resolutions in target perception tasks.

[0006] In order to solve the above technical problems, the present invention provides a target perception method, comprising:

[0007] Using an image sample set including image samples of various resolutions to train a visual perception model to which a resolution-independent feature extraction network is added; the resolution-independent feature extraction network is used to extract resolution-independent features from input image features and then output them;

[0008] In iterative training, for each resolution, an image sample of the resolution is input into the visual perception model and a perceptual loss value is calculated; the parameters of the visual perception model and the parameters of the resolution-independent feature extraction network are updated using the perceptual loss value corresponding to each resolution;

[0009] After reaching the iterative training end condition, obtaining the trained visual perception model;

[0010] The input image corresponding to the target perception task to be processed is input into the trained visual perception model to obtain a perception result of the perception target of the target perception task to be processed.

[0011] In one aspect, the resolution-independent feature extraction network comprises:

[0012] A first resolution-independent feature extraction network, used to extract a first resolution-independent feature from an input image feature of the resolution-independent feature extraction network;

[0013] A resolution-dependent feature extraction network for extracting resolution-dependent features from input image features of the resolution-independent feature extraction network;

[0014] A feature aggregation network is used to aggregate the first resolution-independent feature and the second resolution-independent feature in the resolution-dependent feature, and output the resolution-independent feature.

[0015] On the other hand, extracting a first resolution-independent feature from the input image feature of the resolution-independent feature extraction network includes:

[0016] A first deep convolution calculation, a batch normalization calculation, a second deep convolution calculation, and an activation function calculation are performed on the input image features of the resolution-independent feature extraction network to obtain the first resolution-independent features.

[0017] On the other hand, the resolution-independent feature extraction network further includes:

[0018] A first resolution classification network, used for outputting a first resolution classification result according to the first resolution-independent feature;

[0019] Updating the parameters of the visual perception model and the parameters of the resolution-independent feature extraction network using the perceptual loss values ​​corresponding to the resolutions includes:

[0020] A model learning loss value corresponding to a current iterative training batch is calculated according to the perception loss value corresponding to each of the resolutions and the first resolution classification result corresponding to each of the resolutions, and the parameters of the visual perception model and the parameters of the resolution-independent feature extraction network are updated using the model learning loss value;

[0021] Among them, the model learning loss value is positively correlated with the resolution classification confidence corresponding to the first resolution classification result.

[0022] On the other hand, extracting resolution-dependent features from input image features of the resolution-independent feature extraction network includes:

[0023] A third deep convolution calculation, a batch normalization calculation, a fourth deep convolution calculation, and an activation function calculation are performed on the input image features of the resolution-independent feature extraction network to obtain the resolution-dependent features.

[0024] On the other hand, the resolution-independent feature extraction network further includes:

[0025] A second resolution classification network, used for outputting a second resolution classification result according to the resolution-related features;

[0026] Updating the parameters of the visual perception model and the parameters of the resolution-independent feature extraction network using the perceptual loss values ​​corresponding to the resolutions includes:

[0027] A model learning loss value corresponding to a current iterative training batch is calculated according to the perception loss value corresponding to each of the resolutions and the second resolution classification result corresponding to each of the resolutions, and the parameters of the visual perception model and the parameters of the resolution-independent feature extraction network are updated using the model learning loss value;

[0028] Among them, the model learning loss value is negatively correlated with the resolution classification confidence corresponding to the second resolution classification result.

[0029] On the other hand, aggregating the first resolution-independent feature and the second resolution-independent feature in the resolution-dependent feature to output the resolution-independent feature includes:

[0030] The first resolution-independent feature is used as a query feature, and the resolution-dependent feature is used as a key feature and a value feature. Attention calculation is performed using the query feature, the key feature, and the value feature to output the resolution-independent feature.

[0031] On the other hand, performing attention calculation using the query feature, the key feature, and the value feature to output the resolution-independent feature includes:

[0032] After aggregating features from the resolution-dependent features according to the query features, the key features, and the value features using a cross attention layer, the features are processed sequentially through a first normalization layer, a self-attention layer, a second normalization layer, a feedforward neural network, and a third normalization layer, and the above steps are repeated for a preset number of times to output the resolution-independent features;

[0033] Among them, the cross-attention layer, the self-attention layer and the feedforward neural network processing are all provided with a residual connection for adding output features and output features.

[0034] On the other hand, the resolution-independent feature extraction network further includes:

[0035] A third resolution classification network, used for outputting a third resolution classification result according to the resolution-independent features;

[0036] Updating the parameters of the visual perception model and the parameters of the resolution-independent feature extraction network using the perceptual loss values ​​corresponding to the resolutions includes:

[0037] A model learning loss value corresponding to the current iterative training batch is calculated according to the perception loss value corresponding to each of the resolutions and the third resolution classification result corresponding to each of the resolutions, and the parameters of the visual perception model and the parameters of the resolution-independent feature extraction network are updated using the model learning loss value;

[0038] Among them, the model learning loss value is positively correlated with the resolution classification confidence corresponding to the third resolution classification result.

[0039] On the other hand, the resolution-independent feature extraction network further includes:

[0040] A first resolution classification network, used for outputting a first resolution classification result according to the first resolution-independent feature;

[0041] A second resolution classification network, used for outputting a second resolution classification result according to the resolution-related features;

[0042] A third resolution classification network, used for outputting a third resolution classification result according to the resolution-independent features;

[0043] Updating the parameters of the visual perception model and the parameters of the resolution-independent feature extraction network using the perceptual loss values ​​corresponding to the resolutions includes:

[0044] A model learning loss value corresponding to a current iterative training batch is calculated according to the perception loss value corresponding to each of the resolutions, the first resolution classification result corresponding to each of the resolutions, the second resolution classification result corresponding to each of the resolutions, and the third resolution classification result corresponding to each of the resolutions, and the parameters of the visual perception model and the parameters of the resolution-independent feature extraction network are updated using the model learning loss value;

[0045] Among them, the model learning loss value is positively correlated with the resolution classification confidence corresponding to the first resolution classification result, the model learning loss value is negatively correlated with the resolution classification confidence corresponding to the second resolution classification result, and the model learning loss value is positively correlated with the resolution classification confidence corresponding to the third resolution classification result.

[0046] On the other hand, the value of the first resolution classification result is negatively correlated with the resolution classification confidence corresponding to the first resolution classification result;

[0047] The model learning loss value is negatively correlated with the value of the first resolution classification result.

[0048] On the other hand, the first resolution classification network includes:

[0049] A first gradient reversal layer, used for performing gradient reversal on the first resolution-independent feature;

[0050] The first resolution classifier is used to input the output result of the first gradient reversal layer and then output the first resolution classification result.

[0051] On the other hand, the value of the third resolution classification result is negatively correlated with the resolution classification confidence corresponding to the third resolution classification result;

[0052] The model learning loss value is negatively correlated with the value of the third resolution classification result.

[0053] On the other hand, the third resolution classification network includes:

[0054] A second gradient reversal layer, used for performing gradient reversal on the resolution-independent feature;

[0055] The third resolution classifier is used to input the output result of the second gradient reversal layer and then output the third resolution classification result.

[0056] On the other hand, the visual perception model also includes an orthogonal projection unit arranged at the input end of the resolution-independent feature extraction network.

[0057] On the other hand, it also includes:

[0058] In the iterative training of the visual perception model, the parameters of the orthogonal linear layer of the orthogonal projection unit are updated using the perceptual loss values ​​corresponding to each of the resolutions;

[0059] Determining whether the updated parameters of the orthogonal linear layer satisfy the orthogonality constraint;

[0060] If not, the parameters of the orthogonal linear layer are updated again using the orthogonal matrix corresponding to the updated parameters of the orthogonal linear layer.

[0061] In order to solve the above technical problems, the present invention further provides a target sensing device, comprising:

[0062] A model training unit is used to train a visual perception model with a resolution-independent feature extraction network added using an image sample set including image samples of multiple resolutions; the resolution-independent feature extraction network is used to extract resolution-independent features from input image features and then output them; in iterative training, for each resolution, image samples of the resolution are input into the visual perception model and a perception loss value is calculated; the parameters of the visual perception model and the parameters of the resolution-independent feature extraction network are updated using the perception loss values ​​corresponding to each resolution; after the iterative training end condition is reached, the trained visual perception model is obtained;

[0063] The target perception unit is used to input the input image corresponding to the target perception task to be processed into the trained visual perception model to obtain the perception result of the perception target of the target perception task to be processed.

[0064] In order to solve the above technical problems, the present invention further provides a target sensing device, comprising:

[0065] Memory for storing computer programs;

[0066] A processor is used to execute the computer program, and the computer program, when executed by the processor, implements the steps of the target perception method as described in any one of the above items.

[0067] In order to solve the above technical problems, the present invention also provides a non-volatile storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the target perception method as described in any one of the above items are implemented.

[0068] In order to solve the above technical problems, the present invention also provides a computer program product, including a computer program, which, when executed by a processor, implements the steps of the target perception method as described in any one of the above items.

[0069] The target perception method provided by the present invention has the beneficial effect of adding a resolution-independent feature extraction network for outputting resolution-independent features after extracting them from input image features to the visual perception model; in the process of iteratively training the visual perception model using multi-resolution image samples and in the process of using the visual perception model to process image samples of different resolutions, resolution-independent features are extracted based on the resolution-independent feature extraction network; the parameters of the visual perception model and the parameters of the resolution-independent feature extraction network are updated using the perception loss value of each resolution of the visual perception model; the optimization of the visual perception model that can process multi-resolution input images is achieved by reducing the interference between different resolutions during the training process; thereby, when the trained visual perception model is used to perform target perception tasks, accurate target perception results can be guaranteed for input images of different resolutions that need to be processed.

[0070] The present invention also provides a target sensing device, equipment, medium and computer program product, which have the above-mentioned beneficial effects and will not be described in detail here. BRIEF DESCRIPTION OF THE DRAWINGS

[0071] In order to more clearly illustrate the embodiments of the present invention or the technical solutions of the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0072] Figure 1 A flow chart of a target perception method provided by an embodiment of the present invention;

[0073] Figure 2 A schematic diagram of the structure of a target perception system provided by an embodiment of the present invention;

[0074] Figure 3 A flowchart of multi-resolution loss optimization provided by an embodiment of the present invention;

[0075] Figure 4 A schematic diagram of the structure of a first visual perception model with a resolution-independent feature extraction network added provided by an embodiment of the present invention;

[0076] Figure 5 A training flow chart of a visual perception model provided by an embodiment of the present invention;

[0077] Figure 6 A schematic diagram of the structure of a second visual perception model with a resolution-independent feature extraction network added provided by an embodiment of the present invention;

[0078] Figure 7A schematic diagram of the structure of a first resolution-independent feature extraction network provided by an embodiment of the present invention;

[0079] Figure 8 A schematic diagram of the structure of a first resolution classification network provided by an embodiment of the present invention;

[0080] Fig. 9 A schematic diagram of the structure of a resolution-related feature extraction network provided by an embodiment of the present invention;

[0081] Fig.10 A schematic diagram of the structure of a second resolution classification network provided by an embodiment of the present invention;

[0082] Fig.11 A schematic diagram of the structure of a feature aggregation network provided by an embodiment of the present invention;

[0083] Fig.12 A schematic diagram of the structure of a third resolution classification network provided by an embodiment of the present invention;

[0084] Fig.13 A schematic diagram of the structure of a third visual perception model with a resolution-independent feature extraction network added provided by an embodiment of the present invention;

[0085] Fig.14 A schematic diagram of the structure of an orthogonal projection unit provided by an embodiment of the present invention;

[0086] Fig.15 A schematic diagram of the structure of a target sensing device provided by an embodiment of the present invention;

[0087] Fig.16 A schematic diagram of the structure of a target sensing device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0088] The core of the present invention is to provide a target perception method, device, equipment, medium and computer program product for improving the resolution generalization capability of adapting to input images of different resolutions in target perception tasks.

[0089] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0090] Resolution is used to describe the clarity of an image, video or display device. It indicates the number of pixels that can be displayed per unit length. Resolution is usually expressed in the following forms:

[0091] Pixel resolution: This is the most intuitive way of expressing it, which is expressed directly in pixels. For example, 1920x1080 means there are 1920 pixels in the horizontal direction (image width) and 1080 pixels in the vertical direction (image height).

[0092] Screen resolution: Usually used to describe the resolution of a monitor, TV or other display device. For example, Full HD usually refers to a resolution of 1920x1080, while 4K resolution usually refers to 3840x2160.

[0093] Print resolution: In the printing field, resolution is usually expressed in dots per inch (DPI). For example, 300 DPI means there are 300 dots per inch.

[0094] Image scanning resolution: The resolution of the scanner is also expressed in DPI, which determines the clarity of the scanned image.

[0095] Video resolution: The resolution of a video is usually expressed in pixels, such as 720p (1280x720), 1080p (1920x1080), and 4K (3840x2160 or 4096x2160).

[0096] The target perception method provided in the embodiment of the present invention can be used to process images captured by a visual sensor such as a camera, or to process video frames captured from a captured video. In the embodiment of the present invention, the resolution described can be pixel resolution.

[0097] The resolution generalization ability of a visual perception model refers to the ability of the visual perception model to process input images with resolutions that are inconsistent with the training resolution during the testing phase.

[0098] Traditional object perception solutions usually use the same resolution in the model training and testing stages. This is because the resolution generalization ability of visual perception models is poor. Specifically, when the test resolution is inconsistent with the training resolution, the model accuracy will be seriously reduced. The greater the difference between the test resolution and the training resolution, the more serious the reduction in model accuracy, which limits the application of visual perception models in different scenarios. The fundamental reason for this phenomenon is that the distribution of image data of different resolutions is different, and current visual perception models have difficulty in handling such distribution differences.

[0099] However, in practical applications, the visual perception model after deployment may need to process input images with higher resolutions. For example, a visual perception model trained with a resolution of (256x704) may need to process input images with a resolution of (512x1408) or a resolution of (128x352) after deployment. To meet the application needs of processing images with higher resolutions, one current solution is to adjust the input image to the same size as the resolution of the image sample in the training phase during the application phase, that is, adjust the input image with a resolution of (512x1408) and the input image with a resolution of (128x352) to a resolution of (256x704).

[0100] Although this solution of unifying the resolution of input images is feasible, in some practical application scenarios, there is a high demand for different resolution settings. For some application scenarios with limited computing resources, the visual perception model is required to process low-resolution input images (such as 128x352 resolution in the above example) to reduce the computational burden and ensure real-time performance. If the input image resolution is increased to align it to the training resolution, the computational burden will increase and affect the real-time performance of the system. For some application scenarios with sufficient computing resources, it is hoped that a higher image resolution (such as 512x1408 resolution in the above example) will be used to improve the accuracy of three-dimensional target perception. By reducing the high resolution of the image to align it to the training resolution, a lot of detail information will be lost, which may be detrimental to the perception of small and distant targets.

[0101] It can be seen that in actual target perception applications, there is still a great demand for the resolution generalization capability of the visual perception model itself. However, the resolution generalization capability of the current visual perception model is very poor and it is difficult to directly apply it to scenes with different resolutions.

[0102] In order to improve the resolution generalization ability of the visual perception model, it is necessary to use image samples of different resolutions to train the visual perception model. However, due to the large differences in the feature distribution of image samples of different resolutions, model optimization is difficult to succeed.

[0103] In order to meet the needs of target perception tasks for processing input images of different resolutions, an embodiment of the present invention provides a target perception method, by adding a resolution-independent feature extraction network for extracting resolution-independent features from input image features and then outputting them in a visual perception model, in the process of iteratively training the visual perception model using multi-resolution image samples, in the process of using the visual perception model to process image samples of different resolutions, resolution-independent features are extracted based on the resolution-independent feature extraction network, the visual perception model parameters and the resolution-independent feature extraction network parameters are updated using the perception loss value of each resolution of the visual perception model, and the visual perception model that can process multi-resolution input images is optimized by reducing the interference between different resolutions during the training process, so that when the trained visual perception model is used to perform target perception tasks, accurate target perception results can be guaranteed for input images of different resolutions that need to be processed.

[0104] The target perception method provided by the embodiment of the present invention is described below with reference to the accompanying drawings.

[0105] Figure 1 A flow chart of a target perception method provided by an embodiment of the present invention; Figure 2 A schematic diagram of the structure of a target perception system provided by an embodiment of the present invention; Figure 3 A flowchart of multi-resolution loss optimization is provided in an embodiment of the present invention.

[0106] like Figure 1 As shown, the target perception method provided by the embodiment of the present invention includes:

[0107] S101: using an image sample set including image samples of various resolutions to train a visual perception model to which a resolution-independent feature extraction network is added; the resolution-independent feature extraction network is used to extract resolution-independent features from input image features and then output them;

[0108] S102: In iterative training, for each resolution, input the image sample of the resolution into the visual perception model and calculate the perceptual loss value; update the parameters of the visual perception model and the parameters of the resolution-independent feature extraction network using the perceptual loss value corresponding to each resolution;

[0109] S103: after reaching the iterative training end condition, obtaining a trained visual perception model;

[0110] S104: Inputting the input image corresponding to the target perception task to be processed into the trained visual perception model to obtain a perception result of the perception target of the target perception task to be processed.

[0111] The target perception method provided by the embodiment of the present invention mainly includes two major links: training of a visual perception model and application of the visual perception model. In this regard, the target perception method provided by the embodiment of the present invention can be applied to Figure 2 The target perception system shown in the figure includes two parts: a model training system and a perception computing device.

[0112] The model training system may be composed of one or more computing devices, and is used to train the visual perception model based on a pattern sample set composed of image samples of multiple resolutions. Figure 2 As shown, the model training system can be composed of multiple threads of multiple artificial intelligence devices to form a thread pool, and the image samples corresponding to each resolution are assigned to a thread pool, so as to achieve parallel accelerated learning of multiple resolution image sets. It should be noted that m, n, and k only represent the meaning of multiple, and do not represent a specific number or the same number of threads of each artificial intelligence processor, nor do they mean that each thread pool is composed of one thread in each artificial intelligence device. The model training system can be deployed in a cloud computing center. The device structure of the cloud computing center is similar Figure 2 The perceptual computing devices in the system include computing, storage, network and other infrastructure, operating system, database, middleware and other software environments and application software. The model training system receives image data sets of various resolutions, uses the stored pre-trained models, combined with manual annotation, to annotate visual perception targets (such as the category, location, size of the bounding box, target orientation information, etc.) and stores the annotated data.

[0113] The perception computing device can be a computing device deployed in a cloud computing center or an edge computing device deployed in an edge device. Its structure is as follows: Figure 2 As shown in the perception computing device shown. If the computing device of the cloud computing center is used, after the trained visual perception model is deployed, the edge device collects the image to be perceived through visual sensors such as cameras and uploads it to the cloud computing center, and uses the perception computing device that deploys the visual perception model to perform visual perception calculation on the image to be perceived to obtain the target perception result of the visual perception target. If the perception computing device uses an edge computing device deployed on the edge device, the visual perception model is trained by the model training system and sent to the edge computing device through the communication unit, and deployed in the memory of the edge computing device. After the edge computing device collects the image to be perceived through visual sensors such as cameras, it performs visual perception calculation through the local visual perception model to obtain the target perception result of the visual perception target.

[0114] Applied to Figure 2The multi-thread pool shown, in an embodiment of the present invention, the step of calculating the perceptual loss value of the visual perception model for image samples of each resolution may include: inputting the image samples corresponding to each resolution into different thread pools one by one, so as to obtain the perceptual loss value of the visual perception model for the image samples of each resolution by parallel calculation.

[0115] like Figure 3 As shown, in an embodiment of the present invention, when training a visual perception model, all images of each resolution can be stacked to obtain a set of input images of different resolutions, so that the training speed can be improved through batch processing.

[0116] In some optional implementations of the embodiments of the present invention, the parameters of the visual perception model and the parameters of the resolution-independent feature extraction network are updated using the perceptual loss values ​​corresponding to each resolution in S102, which may include: calculating the model learning loss value of the current iteration number according to the perceptual loss values ​​corresponding to each resolution, and updating the parameters of the visual perception model and the parameters of the resolution-independent feature extraction network according to the model learning loss value.

[0117] Among them, calculating the model learning loss value of the current iteration number according to the perceptual loss value corresponding to each resolution may include: taking the average of the loss values ​​calculated by the loss function corresponding to each resolution as the model learning loss value.

[0118] like Figure 3 As shown, for each resolution, all image samples are input into the visual perception model to obtain the corresponding visual perception results, and the loss value of the visual perception model for each resolution compared with the true value (which can be manually labeled) is calculated through the loss function. Finally, the model learning loss value of the visual perception model can be the average of the perceptual loss values ​​of the visual perception model for image samples of all resolutions, as shown in the following formula:

[0119] ;

[0120] in, represents the loss function of the visual perception model, represents the perceptual loss value corresponding to the i-th resolution, represents the loss function corresponding to the i-th resolution, and k+1 represents the number of resolutions used in visual perception model training.

[0121] It is very challenging to directly use input images of multiple resolutions to train a visual perception model because there are large data distribution differences between input images of different resolutions, which increases the difficulty of optimizing the perception model. To solve this problem, an embodiment of the present invention provides a visual perception model with an added resolution-independent feature extraction network, which uses the resolution-independent feature extraction network to extract resolution-independent features from input image features, reduce interference between different resolutions, and achieve optimization of a multi-resolution visual perception model.

[0122] Figure 4 A schematic diagram of the structure of a first visual perception model with an added resolution-independent feature extraction network provided in an embodiment of the present invention.

[0123] like Figure 4 As shown, the resolution-independent feature extraction network for extracting resolution-independent features from input image features proposed in an embodiment of the present invention can be set after any image feature extraction network in the visual perception model. One or more resolution-independent feature extraction networks can be set as needed. When multiple resolution-independent feature extraction networks are set, they can be set after different image feature extraction networks in the visual perception model. The parameters of the resolution-independent feature extraction network are optimized together with the parameters of the visual perception model, thereby solving the interference problem between different resolutions when the visual perception model learns image samples of multiple resolutions.

[0124] In some optional implementations of the embodiments of the present invention, the resolution-independent feature extraction network may be composed of an image feature extraction network and a resolution classification network. The image feature extraction network is used to extract resolution-independent features from input image features of the resolution-independent feature extraction network. The resolution classification network is used to output a resolution classification result based on the resolution-independent features.

[0125] It is understandable that the less resolution-related information there is in the resolution-independent feature, the less reliable the resolution classifier can get. At this time, the resolution-independent feature extraction network is needed to remove interference between different resolutions. Therefore, the resolution classification result can be added to the model learning loss value of the visual perception model, and the model learning loss value is set to be positively correlated with the resolution classification confidence corresponding to the resolution classification result. Based on this, a resolution classification network can be designed to include a gradient reversal layer and a resolution classifier. The gradient reversal layer is used to perform gradient reversal on the resolution-independent feature output by the resolution-independent feature extraction network, and the resolution classifier outputs the resolution classification result based on the output result of the gradient reversal layer. The larger the value of the resolution classification result, the less the resolution classification network can identify the resolution-related information in the resolution-independent feature. At this time, a smaller loss value should be applied. Therefore, the model learning loss value is set to be negatively correlated with the value of the resolution classification result.

[0126] It can be understood that the parameters of the resolution-independent feature extraction network are related to its position in the visual perception model. For example, the image feature extraction network and the resolution classifier network in the resolution-independent feature extraction network need to initialize the corresponding network parameters according to the feature size and number of channels of the input image features.

[0127] Figure 5 A training flow chart of a visual perception model provided by an embodiment of the present invention.

[0128] like Figure 5 As shown, for image samples corresponding to resolutions 1, 2, ..., m, they are respectively input into the visual perception model to obtain corresponding perception loss values ​​1, 2, ..., m. As described in Embodiment S102 of the present invention, in one iterative training of the visual perception model, for a resolution, an image sample of the resolution is first input into the visual perception model, and in the resolution-independent feature extraction network of the visual perception model, resolution-independent features are extracted from the input image features of the resolution-independent feature extraction network, and then the resolution-independent features are input into the subsequent network layers of the visual perception model (such as Figure 4 The perception task module in the image is used, and finally the perception result output by the visual perception model is used to calculate the perception loss value corresponding to the resolution.

[0129] In one iterative training of the visual perception model, the image samples of each resolution are subjected to the operation of S102, the perception loss values ​​corresponding to all resolutions are summarized to calculate the model learning loss value of the current iterative training, and the original model parameters of the visual perception model are updated according to the model learning loss value while updating the model parameters of the resolution-independent feature extraction network in the visual perception model.

[0130] like Figure 2 As shown, in order to accelerate the training of the visual perception model, the loss optimization tasks corresponding to m resolutions can be deployed in m thread pools for execution to achieve parallel accelerated training.

[0131] For S103, the iterative training end condition may be that the number of iterative training times of the visual perception model reaches a preset number of iterations, or the model learning loss value of the visual perception model is less than a preset loss value.

[0132] For S104, after the trained visual perception model is deployed in an actual application scenario, it can be directly used to perform the target perception task. When performing the target perception task, in the process of using the visual perception model to perceive the input image corresponding to the target perception task, regardless of the resolution of the input image, the image features extracted in the intermediate process of the resolution-independent feature extraction network in the visual perception model can be processed to only propagate the resolution-independent features to the subsequent network, so as to ensure the perception accuracy of the visual perception model for input images with different resolutions.

[0133] The target perception method provided by the embodiment of the present invention adds a resolution-independent feature extraction network for extracting resolution-independent features from input image features and then outputting them to the visual perception model. In the process of iteratively training the visual perception model using multi-resolution image samples, and in the process of using the visual perception model to process image samples of different resolutions, the resolution-independent features are extracted based on the resolution-independent feature extraction network. The parameters of the visual perception model and the parameters of the resolution-independent feature extraction network are updated using the perception loss value of each resolution of the visual perception model. The visual perception model that can process multi-resolution input images is optimized by reducing interference between different resolutions during the training process. Therefore, when the trained visual perception model is used to perform target perception tasks, accurate target perception results can be guaranteed for input images of different resolutions that need to be processed.

[0134] Based on the above embodiment, the embodiment of the present invention further illustrates the steps of collecting the image sample set.

[0135] The image sample set used for training the visual perception model in the embodiment of the present invention includes image samples of multiple resolutions. When collecting the image sample set, a visual sensor such as a camera can be used to directly collect image samples of different resolutions through set resolution parameters to form an image sample set; when collecting the image sample set, image samples of one resolution can also be collected first, and image samples of multiple resolutions can be generated by scaling or enlarging to form an image sample set.

[0136] In some optional implementations of the embodiments of the present invention, the step of collecting the image sample set may include: obtaining an original image set; processing each original image in the original image set using different scaling factors to obtain scaled images of different resolutions corresponding to each original image; and using each original image and the scaled image corresponding to each original image as the image sample set.

[0137] In practical applications, each original image in the original image set is processed with different scaling factors to obtain scaled images of different resolutions corresponding to each original image, which may include: processing each original image with a reference scaling factor to obtain a reference scaled image corresponding to the original image; sampling multiple scaling factors from a preset distribution range, and processing each original image with each scaling factor to obtain scaled images of multiple resolutions corresponding to each original image; using the corresponding reference scaled image and each corresponding scaled image as an image set corresponding to the original image to obtain an image set corresponding to each original image. That is, the generated image sample set is composed of image sets corresponding to different original images, which include image samples of different resolutions, and different image samples of the same resolution may be obtained by processing different original images.

[0138] As described in the above embodiment of the present invention, the resolution used for processing by the visual perception model of the visual perception task is usually expressed in pixel resolution, that is, expressed as the product of the number of pixels in the horizontal direction and the number of pixels in the vertical direction. In the embodiment of the present invention, the scaling factor may include a scaling factor acting on the image height and / or a scaling factor on the image width. In order to obtain image samples of multiple resolutions with greater resolution differences, the scaling factor may include an image height scaling factor and an image width scaling factor.

[0139] To reduce the computational difficulty, the scaling factor can also be a scaling ratio for the image height and image width. For example, for an original image with a resolution of H×W , the base scaling factor can be used The processed resolution is After the image is distributed in Uniform distribution Sample k scaling factors , and process the image using each scaling factor , and finally obtain a collection of images with multiple resolutions ,in The resolution is . Where H represents the vertical size of the image, and W represents the horizontal size of the image.

[0140] When applied to the field of autonomous driving, cameras and other visual sensors are usually set up in different directions of the vehicle, such as left front (fl), front (f), right front (fr), left rear (bl), rear (b), right rear (br), etc., so that images of multiple perspectives of the road scene are collected each time. In the actual training process, in order to use the batch method to accelerate model training, the embodiment of the present invention samples a set of scaling factors at each iteration, and , the above-obtained scaling coefficient set is used for processing, that is, a multi-resolution image set is obtained for each viewing angle .

[0141] Since the resolution difference between the multi-resolution image samples used in training the visual perception model is small, it is not conducive to the training of the model's resolution generalization ability. In order to obtain as many resolutions with large differences as possible and improve the training effect of the model's resolution generalization ability, in other optional implementations of the embodiments of the present invention, the difference between the scaling factors can be set to be no less than the scaling factor difference threshold, that is, when determining the scaling factor, the difference between the scaling factor and other scaling factors in the collected scaling factor set is judged, and only when the difference is greater than the set scaling factor difference threshold is the scaling factor added to the scaling factor set, thereby ensuring the diversity between the scaling factors obtained by sampling and improving the training effect of the model's resolution generalization ability.

[0142] Furthermore, the scaling factors may be designed to all belong to a preset distribution range and the difference between any two scaling factors may not be less than a scaling factor difference threshold.

[0143] In some optional implementations of the embodiments of the present invention, the scaling factor difference threshold is calculated based on the interval length of the preset distribution range divided by twice the number of scaling factors. That is, the scaling factor difference threshold can be expressed by the following formula:

[0144] ;

[0145] Where M represents the scaling factor difference threshold, Indicates the lower limit of the preset distribution range. represents the upper limit of the preset distribution range, and k represents the total number of scaling factors to be collected from the preset distribution range.

[0146] In some other alternative embodiments of the embodiments of the present invention, for each original image in the original image set, different scaling factors are used for processing to obtain scaled images with different resolutions corresponding to each original image. It may also include: processing each original image with a reference scaling factor to obtain a reference scaled image corresponding to the original image; sampling a plurality of scaling factors from a preset distribution range until a preset number of scaling factors are collected; during the process of collecting scaling factors, for each newly collected scaling factor, calculating the difference between the new scaling factor and the existing scaling factors. If the difference is not less than the scaling factor difference threshold, the new scaling factor is added to the set of existing scaling factors; if the difference is less than the scaling factor difference threshold, the new scaling factor is discarded; processing each original image with each scaling factor respectively to obtain scaled images with multiple resolutions corresponding to each original image; using the corresponding reference scaled image and the corresponding scaled images as the image set corresponding to the original image to obtain the image set corresponding to each original image.

[0147] That is, the following algorithm can be designed to obtain the set of scaling factors:

[0148] Input: reference scaling factor , uniformly distributed in , set A of scaling factors, scaling factor difference threshold m;

[0149] Add the reference scaling factor to the set A of scaling factors ;

[0150] If the length of the set A of scaling factors < k + 1, then:

[0151] Randomly sample from the uniform distribution to obtain ;

[0152] Check the difference between and all the scaling factors in the current set A of scaling factors. If the difference is greater than the threshold scaling factor difference M, then add to the set A of scaling factors;

[0153] Output: set A of scaling factors.

[0154] In the embodiments of the present invention, the reference scaled image obtained by the above method can be used as the reference image in the image sample set to obtain the reference image features. Alternatively, an image sample with one of the resolutions can also be used as the reference image.

[0155] In order to further improve the ability of the visual perception model to process input images of different resolutions, image sample sets can be generated multiple times during the model training process. That is, the scaling factor can be re-collected to generate a new image sample set in each iterative training, or the scaling factor can be re-collected to generate a new image sample set after every several iterative trainings.

[0156] If the image sample set is regenerated through the above method in each iterative training, it may make it difficult for the visual perception model to converge.

[0157] To ensure model convergence, in some optional implementations of the embodiments of the present invention, a method of generating an image sample set once before training the visual perception model can be adopted, and the image sample set is no longer updated in subsequent iterative training. In this way, the same batch of resolutions are learned in each iterative training of the visual perception model, and the diversity of samples can be ensured by collecting different image samples corresponding to the resolutions. If computing resources and time permit, a method of generating an image sample set once before training the visual perception model can be adopted, but multiple rounds of iterative training are performed, and the image sample set is regenerated before the start of each round of iterative training, and the same image sample set is used in different iterative training steps in each round of iterative training.

[0158] In some other optional implementations of the embodiments of the present invention, the image sample set may be regenerated every multiple iterations of training, and the frequency of generating the image sample set may be determined according to the convergence of the visual perception model.

[0159] In some other optional implementations of the embodiments of the present invention, if the image sample set is regenerated in each iterative training or every few iterative trainings, the multiple resolutions corresponding to the resampling scaling factors can arrange the resolutions in a fixed order (such as from small to large or from large to small) according to the scaling factors, thereby ensuring the correspondence between the arrangement of the resolutions and the arrangement of the loss function, so that the model training can inherit the experience of the previous iterative training or the previous few iterative trainings.

[0160] In each embodiment of the present invention, the loss function corresponding to the resolution of the visual perception model in each iterative training number inherits the parameters of the loss function of the corresponding position in the previous iterative training, but the corresponding resolution can be the resolution in the previous iterative training or the updated resolution.

[0161] The above embodiment of the present invention introduces that by adding a resolution-independent feature extraction network to the visual perception model to extract resolution-independent features, that is, to filter out resolution-related information, the interference between different resolutions when the visual perception model is learned and multi-resolution input images are processed. However, only using the image feature extraction network to extract resolution-independent features may result in a loss of more image features that are beneficial to target perception. To solve this problem, on the basis of the above embodiment, in other optional implementations of the embodiments of the present invention, the resolution-independent feature extraction network may also include: a first resolution-independent feature extraction network, used to extract a first resolution-independent feature from the input image features of the resolution-independent feature extraction network; a resolution-dependent feature extraction network, used to extract resolution-dependent features from the input image features of the resolution-independent feature extraction network; a feature aggregation network, used to aggregate the first resolution-independent feature and the second resolution-independent feature in the resolution-dependent feature, and output the resolution-independent feature.

[0162] Figure 6 The schematic diagram of the structure of the second visual perception model with a resolution-independent feature extraction network added provided by the embodiment of the present invention. Figure 6 As shown, in an embodiment of the present invention, the resolution-independent feature extraction network is designed to be a network composed of a first resolution-independent feature extraction network, a resolution-dependent feature extraction network and a feature aggregation network. The first resolution-independent feature extraction network is used to extract the first resolution-independent feature, the resolution-dependent feature extraction network is used to extract the resolution-dependent feature, and the feature aggregation network is used to further aggregate the resolution-independent features from the resolution-dependent features into the first resolution-independent features, thereby obtaining resolution-independent features that can reduce interference between different resolutions and avoid losing more image features.

[0163] Based on the above embodiment, the embodiment of the present invention continues to illustrate the structure of the first resolution-independent feature extraction network.

[0164] In an embodiment of the present invention, extracting a first resolution-independent feature from an input image feature of a resolution-independent feature extraction network may include: performing a first deep convolution calculation, a batch normalization calculation, a second deep convolution calculation, and an activation function calculation on the input image feature of the resolution-independent feature extraction network to obtain the first resolution-independent feature.

[0165] Figure 7 A schematic diagram of the structure of a first resolution-independent feature extraction network provided in an embodiment of the present invention.

[0166] like Figure 7As shown, the first resolution-independent feature extraction network provided by the embodiment of the present invention can be composed of a first deep convolutional network, a batch normalization layer, a second deep convolutional network and an activation function. Among them, the activation function can adopt a ReLU activation function. The image feature of the input resolution-independent feature extraction network is , then after the first deep convolutional network, batch normalization layer, second deep convolutional network and activation function, the first resolution-independent feature is obtained .

[0167] In order to enhance the ability of the first resolution-independent feature extraction network to extract the first resolution-independent feature, in an embodiment of the present invention, the resolution-independent feature extraction network may further include: a first resolution classification network, which is used to output a first resolution classification result according to the first resolution-independent feature. In S102, the parameters of the visual perception model and the parameters of the resolution-independent feature extraction network are updated using the perceptual loss values ​​corresponding to each resolution, which may include: calculating the model learning loss value corresponding to the current iterative training batch according to the perceptual loss values ​​corresponding to each resolution and the first resolution classification result corresponding to each resolution, and updating the parameters of the visual perception model and the parameters of the resolution-independent feature extraction network using the model learning loss value; wherein the model learning loss value is positively correlated with the resolution classification confidence corresponding to the first resolution classification result.

[0168] That is to say, a first resolution classifier can be designed for the first resolution-independent feature extraction network to test the quality of the first resolution-independent feature extracted by the first resolution-independent feature extraction network. It can be understood that the less resolution-related information in the first resolution-independent feature, the less likely the resolution classifier can obtain a reliable resolution classification result based on the first resolution-independent feature. At this time, the first resolution-independent feature extraction network is required to remove interference between different resolutions. Therefore, the first resolution classification result can be added to the model learning loss value of the visual perception model, and the model learning loss value is set to be positively correlated with the resolution classification confidence corresponding to the first resolution classification result.

[0169] Figure 8 A schematic diagram of the structure of a first resolution classification network provided in an embodiment of the present invention.

[0170] Based on this design principle, Figure 8As shown, in some optional implementations of the embodiments of the present invention, the first resolution classification network may include: a first gradient reversal layer, which is used to perform gradient reversal on the first resolution-independent feature; and a first resolution classifier, which is used to output the first resolution classification result after inputting the output result of the first gradient reversal layer. By using the first gradient reversal layer to perform gradient reversal on the first resolution-independent feature output by the first resolution-independent feature extraction network, and then the first resolution classifier outputs the first resolution classification result based on the output result of the first gradient reversal layer, the larger the value of the first resolution classification result, the less able the first resolution classification network is to recognize the resolution-related information in the first resolution-independent feature, and a smaller loss value should be corresponding to this. Therefore, the model learning loss value is set to be negatively correlated with the value of the first resolution classification result.

[0171] like Figure 8 As shown, in an embodiment of the present invention, the first resolution classifier may include a global mean pooling layer, a linear layer and a classifier function. The classifier function may use a Softmax activation function.

[0172] Then we can design the first resolution classification loss function corresponding to the first resolution-independent feature extraction network. As shown below:

[0173] ;

[0174] in, Indicates the total number of resolutions included in the image sample set, represents the output result of the first resolution classifier corresponding to the i-th resolution, Indicates the true value of the label of the current image sample belonging to the i-th resolution, Indicates that the current image sample is an image sample of the i-th resolution, Indicates that the current image sample is not an image sample of the i-th resolution.

[0175] Through the constraints of the first resolution classification loss function and the first gradient reversal layer, the first resolution-independent features is optimized in the direction that the first resolution classifier cannot effectively identify the image resolution. Separate resolution-independent features from the image.

[0176] In some other optional implementations of the embodiments of the present invention, the first resolution classification network may also only include a first resolution classifier, which is used to output a first resolution classification result based on the first resolution-independent feature, and set the model learning loss value to be positively correlated with the value of the first resolution classification result.

[0177] In the embodiment of the present invention, the parameters of the first resolution classifier may adopt pre-trained parameters, or may be updated along with the iterative training of the visual perception model.

[0178] Based on the above embodiments, the embodiments of the present invention continue to illustrate the structure of the resolution-related feature extraction network.

[0179] In an embodiment of the present invention, extracting resolution-related features from input image features of a resolution-independent feature extraction network may include: performing a third deep convolution calculation, a batch normalization calculation, a fourth deep convolution calculation, and an activation function calculation on the input image features of the resolution-independent feature extraction network to obtain resolution-related features.

[0180] Fig. 9 A schematic diagram of the structure of a resolution-related feature extraction network provided in an embodiment of the present invention.

[0181] like Fig. 9 As shown, the resolution-dependent feature extraction network provided by the embodiment of the present invention can be composed of a third deep convolutional network, a batch normalization layer, a fourth deep convolutional network and an activation function. Among them, the activation function can adopt a ReLU activation function. The image feature of the input resolution-independent feature extraction network is , then after the first deep convolutional network, batch normalization layer, second deep convolutional network and activation function, the resolution-related features are obtained. .

[0182] In order to enhance the ability of the resolution-dependent feature extraction network to extract resolution-dependent features, in an embodiment of the present invention, the resolution-independent feature extraction network further includes: a second resolution classification network, which is used to output a second resolution classification result according to the resolution-dependent features. In S102, the parameters of the visual perception model and the parameters of the resolution-independent feature extraction network are updated using the perceptual loss values ​​corresponding to each resolution, which may include: calculating the model learning loss value corresponding to the current iterative training batch according to the perceptual loss values ​​corresponding to each resolution and the second resolution classification result corresponding to each resolution, and updating the parameters of the visual perception model and the parameters of the resolution-independent feature extraction network using the model learning loss value; wherein the model learning loss value is negatively correlated with the resolution classification confidence corresponding to the second resolution classification result.

[0183] That is to say, a second resolution classifier can be designed for the resolution-related feature extraction network to test the quality of the resolution-related features extracted by the resolution-related feature extraction network. It can be understood that the more resolution-related information there is in the resolution-related features, the more reliable the second resolution classifier can obtain from the resolution-related features. At this time, the resolution-related feature extraction network is required to extract resolution-related information. Therefore, the second resolution classification result can be added to the model learning loss value of the visual perception model, and the model learning loss value is set to be negatively correlated with the resolution classification confidence corresponding to the second resolution classification result.

[0184] Fig.10 A schematic diagram of the structure of a second resolution classification network provided in an embodiment of the present invention.

[0185] like Fig.10 As shown, the second resolution classifier may also include a global mean pooling layer, a linear layer, and a classifier function. The classifier function may use a Softmax activation function.

[0186] Then we can design the second resolution classification loss function corresponding to the resolution-dependent feature extraction network. As shown below:

[0187] ;

[0188] in, Indicates the total number of resolutions included in the image sample set, represents the output result of the second resolution classifier corresponding to the i-th resolution, Indicates the true value of the label of the current image sample belonging to the i-th resolution, Indicates that the current image sample is an image sample of the i-th resolution, Indicates that the current image sample is not an image sample of the i-th resolution.

[0189] Through the constraint of the second resolution classification loss function, resolution-dependent features is optimized towards making the second resolution classifier effectively identify the image resolution, so it can be derived from the input image features Separate resolution-dependent features from the image.

[0190] In the embodiment of the present invention, the parameters of the second resolution classifier can be pre-trained parameters, or can be updated with the iterative training of the visual perception model. The second resolution classifier can share parameters with the first resolution classifier introduced in the above embodiment.

[0191] Based on the above embodiment, the embodiment of the present invention continues to illustrate the structure of the feature aggregation network.

[0192] In an embodiment of the present invention, aggregating the first resolution-independent feature and the second resolution-independent feature in the resolution-dependent feature to output the resolution-independent feature may include: taking the first resolution-independent feature as a query feature, taking the resolution-dependent feature as a key feature and a value feature, performing attention calculation using the query feature, the key feature and the value feature, and outputting the resolution-independent feature.

[0193] Fig.11 A schematic diagram of the structure of a feature aggregation network provided in an embodiment of the present invention.

[0194] like Fig.11 As shown, the feature aggregation network provided by the embodiment of the present invention can be composed of a cross attention layer, a first normalization layer, a self-attention layer, a second normalization layer, a feedforward neural network processing and a third normalization layer. Among them, the cross attention layer can be implemented by the cross attention layer of the transformer model (Transformer model). The self-attention layer can be implemented by the self-attention layer of the transformer model.

[0195] Then, attention calculation is performed using query features, key features and value features, and resolution-independent features are output, which may include: using a cross-attention layer to aggregate features from resolution-related features according to query features, key features and value features, and then sequentially passing through a first normalization layer, a self-attention layer, a second normalization layer, a feedforward neural network processing and a third normalization layer processing, repeating the above steps for a preset number of times, and outputting resolution-independent features; wherein the cross-attention layer, the self-attention layer and the feedforward neural network processing are all provided with a residual connection for adding output features to output features.

[0196] like Fig.11 As shown, the first resolution-independent feature Based on the resolution-related features Aggregate features, first with the first resolution-independent features To query features, resolution-dependent features For key features and value features, a cross-attention layer is used to extract resolution-related features After the features are aggregated in the network, they are processed by the normalization layer, then processed by the self-attention layer, input into the feedforward neural network, and finally processed by the normalization layer. Residual connections are added to the cross attention layer, self-attention layer, and feedforward neural network, and the output features of these network layers are added to the input features to enhance the performance and generalization ability of these network layers. The above process is repeated N times, N can be 6, and the resolution-independent features are finally aggregated. .

[0197] Similar to the first resolution classification network designed to enhance the ability of the first resolution-independent feature extraction network to extract first resolution-independent features, the resolution-independent features output by the feature aggregation network can also be Whether it is truly resolution-independent is tested. In an embodiment of the present invention, the resolution-independent feature extraction network may further include: a third resolution classification network, which is used to output a third resolution classification result based on the resolution-independent feature. Using the perceptual loss value corresponding to each resolution in S102 to update the parameters of the visual perception model and the parameters of the resolution-independent feature extraction network may include: calculating the model learning loss value corresponding to the current iterative training batch based on the perceptual loss value corresponding to each resolution and the third resolution classification result corresponding to each resolution, and using the model learning loss value to update the parameters of the visual perception model and the parameters of the resolution-independent feature extraction network; wherein the model learning loss value is positively correlated with the resolution classification confidence corresponding to the third resolution classification result.

[0198] That is to say, a third resolution classifier can be designed for the feature aggregation network to test the quality of the resolution-independent features output by the feature aggregation network. It can be understood that the less resolution-related information there is in the resolution-independent features, the less likely the resolution classifier will be able to obtain a reliable resolution classification result based on the resolution-independent features. At this time, the resolution-independent features are required to remove interference between different resolutions. Therefore, the third resolution classification result can be added to the model learning loss value of the visual perception model, and the value of the third resolution classification result is set to be negatively correlated with the resolution classification confidence corresponding to the third resolution classification result; the model learning loss value is negatively correlated with the value of the third resolution classification result.

[0199] Fig.12 A schematic diagram of the structure of a third-resolution classification network provided in an embodiment of the present invention.

[0200] Based on this design principle, Fig.12 As shown, in some optional implementations of the embodiments of the present invention, the third resolution classification network may include: a second gradient reversal layer, used to perform gradient reversal on resolution-independent features; and a third resolution classifier, used to output a third resolution classification result after inputting the output result of the second gradient reversal layer.

[0201] By using the second gradient reversal layer to perform gradient reversal on the resolution-independent features output by the feature aggregation network, and then the third resolution classifier outputs the third resolution classification result for the output result of the second gradient reversal layer, the larger the value of the third resolution classification result, the more the third resolution classification network is essentially unable to recognize the resolution-related information in the resolution-independent features, and a smaller loss value should be corresponding to this. Therefore, the model learning loss value is set to be negatively correlated with the value of the third resolution classification result.

[0202] like Fig.12 As shown, in an embodiment of the present invention, the third resolution classifier may include a global mean pooling layer, a linear layer and a classifier function. The classifier function may use a Softmax activation function.

[0203] Then we can design the third resolution classification loss function corresponding to the feature aggregation network As shown below:

[0204] ;

[0205] in, Indicates the total number of resolutions included in the image sample set, represents the output result of the third resolution classifier corresponding to the i-th resolution, Indicates the true value of the label of the current image sample belonging to the i-th resolution, Indicates that the current image sample is an image sample of the i-th resolution, Indicates that the current image sample is not an image sample of the i-th resolution.

[0206] The resolution-independent features are ensured by the third resolution classification loss function and the constraints of the second gradient reversal layer. Maintain resolution-independent characteristics. By fusing the perceptual information in resolution-dependent features, the accuracy of the visual perception model is further improved, and by constraining the resolution-independent nature of resolution-independent features, the resolution generalization capability of the visual perception model is guaranteed.

[0207] In some other optional implementations of the embodiments of the present invention, the third resolution classification network may also only include a third resolution classifier, which is used to output a third resolution classification result based on the third resolution-independent features, and set the model learning loss value to be positively correlated with the value of the third resolution classification result.

[0208] In the embodiment of the present invention, the parameters of the third resolution classifier can be pre-trained parameters, or can be updated with the iterative training of the visual perception model. It should be noted that the third resolution classifier is different from the first resolution classifier and the second resolution classifier introduced in the above embodiment.

[0209] On the basis of the above embodiments, a resolution-independent feature extraction network can be designed to also include: a first resolution classification network, used to output a first resolution classification result according to the first resolution-independent feature; a second resolution classification network, used to output a second resolution classification result according to the resolution-dependent feature; and a third resolution classification network, used to output a third resolution classification result according to the resolution-independent feature.

[0210] Then, in S102, the parameters of the visual perception model and the parameters of the resolution-independent feature extraction network are updated using the perceptual loss value corresponding to each resolution, which may include: calculating the model learning loss value corresponding to the current iterative training batch according to the perceptual loss value corresponding to each resolution, the first resolution classification result corresponding to each resolution, the second resolution classification result corresponding to each resolution, and the third resolution classification result corresponding to each resolution, and using the model learning loss value to update the parameters of the visual perception model and the parameters of the resolution-independent feature extraction network; wherein the model learning loss value is positively correlated with the resolution classification confidence corresponding to the first resolution classification result, the model learning loss value is negatively correlated with the resolution classification confidence corresponding to the second resolution classification result, and the model learning loss value is positively correlated with the resolution classification confidence corresponding to the third resolution classification result.

[0211] Among them, the value of the first resolution classification result can be negatively correlated with the resolution classification confidence corresponding to the first resolution classification result; the model learning loss value can be negatively correlated with the value of the first resolution classification result.

[0212] The first resolution classification network may include: a first gradient reversal layer, used to perform gradient reversal on the first resolution-independent feature; and a first resolution classifier, used to output a first resolution classification result after inputting an output result of the first gradient reversal layer.

[0213] Among them, a second resolution classifier can be designed for the resolution-related feature extraction network to test the quality of the resolution-related features extracted by the resolution-related feature extraction network. The model learning loss value can be negatively correlated with the resolution classification confidence corresponding to the second resolution classification result.

[0214] Among them, the value of the third resolution classification result can be negatively correlated with the resolution classification confidence corresponding to the third resolution classification result; the model learning loss value can be negatively correlated with the value of the third resolution classification result.

[0215] The third resolution classification network may include a second gradient reversal layer, used to perform gradient reversal on resolution-independent features; and a third resolution classifier, used to output a third resolution classification result after inputting the output result of the second gradient reversal layer.

[0216] Then, in the embodiment of the present invention, the loss function of the visual perception model can be expressed by the following formula:

[0217] ;

[0218] in, Represents the loss value calculated based on the perceptual loss value corresponding to each resolution, represents the first resolution classification loss function corresponding to the first resolution-independent feature extraction network, represents the weight coefficient of the first resolution classification loss function, represents the second resolution classification loss function corresponding to the resolution-dependent feature extraction network, represents the weight coefficient of the second resolution classification loss function, represents the third-resolution classification loss function corresponding to the feature aggregation network, Represents the weight coefficient of the third resolution classification loss function.

[0219] Weight coefficient , , A preset value may be used, or a coefficient updated through self-learning during the iterative training process of the visual perception model may be used.

[0220] Fig.13 A schematic diagram of the structure of a third visual perception model that adds a resolution-independent feature extraction network provided in an embodiment of the present invention.

[0221] On the basis of the above embodiments, in order to further enhance the extraction effect of resolution-independent features, as Fig.13 As shown, in the target perception method provided by the embodiment of the present invention, the visual perception model may further include an orthogonal projection unit provided at the input end of the resolution-independent feature extraction network. The image features are projected into the orthogonal space by the orthogonal projection unit, which reduces the correlation between features of different channels and is conducive to decoupling resolution-dependent features and resolution-independent features.

[0222] Fig.14 A schematic structural diagram of an orthogonal projection unit provided in an embodiment of the present invention.

[0223] like Fig.14 As shown, the orthogonal projection unit provided in the embodiment of the present invention can be composed of a fifth deep convolutional network, a global mean pooling layer, a linear layer, an activation function and an orthogonal linear layer. Among them, the activation function can adopt a sigmoid activation function.

[0224] The image features of the input orthogonal projection unit are recorded as , represents the image features corresponding to the i-th resolution, and s represents the downsampling multiple of the input image in the network before the orthogonal projection unit in the visual perception model.

[0225] Applying the orthogonal projection unit provided by the embodiment of the present invention, in the main branch, using the fifth deep convolutional network The processed image features are , while in the bypass branch, the image features are Perform global mean pooling Processing, Linear Layer Processed and activated by sigmoid function, the weight of the channel dimension is obtained , the process can be expressed by the following formula:

[0226] .

[0227] Using the fifth depth convolutional network with channel-dimensional weights The output results are weighted and finally passed through the orthogonal linear layer The process can be expressed by the following formula:

[0228] ;

[0229] Among them, the orthogonal linear layer Parameters It is randomly orthogonally initialized and can be initialized using the following formula:

[0230] ;

[0231] in, is the identity matrix.

[0232] In the embodiment of the present invention, the orthogonal projection unit may adopt fixed parameters, or be optimized along with the iterative training of the visual perception model.

[0233] If the parameters of the orthogonal projection unit are optimized along with the iterative training of the visual perception model, the following update method can be used:

[0234] First calculate the parameters of the orthogonal projection unit The gradient is:

[0235] ;

[0236] Then the parameters of the orthogonal projection unit are updated using the following formula:

[0237] ;

[0238] in, is the model learning loss value for the visual perception model, represents the learning rate, Represents the updated parameters of the orthographic projection unit.

[0239] When the parameters of the orthogonal projection unit are updated with the iterative training of the visual perception model, in order to ensure that they still maintain the orthogonal constraint, the target perception method provided by the embodiment of the present invention may also include: in the iterative training of the visual perception model, using the perception loss value corresponding to each resolution to update the parameters of the orthogonal linear layer of the orthogonal projection unit; judging whether the updated parameters of the orthogonal linear layer meet the orthogonal constraint; if not, updating the parameters of the orthogonal linear layer again using the orthogonal matrix corresponding to the updated parameters of the orthogonal linear layer.

[0240] The above-mentioned judgment and re-updating process of the orthogonal linear layer can be implemented by the orthogonal projection optimization unit provided by the embodiment of the present invention, such as Fig.13 shown.

[0241] In the embodiment of the present invention, judging whether the updated parameters of the orthogonal linear layer satisfy the orthogonal constraint may include:

[0242] use Compute the orthogonal linear layer of the orthogonal projection unit The orthogonality constraint score of ; where represents the square of the Frobenius norm;

[0243] If the orthogonal linear layer of the orthogonal projection unit The orthogonality constraint score Less than or equal to threshold , then determine the updated orthogonal linear layer Satisfy orthogonality constraints;

[0244] If the orthogonal linear layer of the orthogonal projection unit The orthogonality constraint score Greater than , then determine the updated orthogonal linear layer The orthogonality constraint is not satisfied.

[0245] If the updated orthogonal linear layer To satisfy the orthogonality constraint, an orthogonal linear layer can be used Updated parameters As an orthogonal linear layer The new parameters.

[0246] If the updated orthogonal linear layer If the orthogonal constraint is not satisfied, the embodiment of the present invention provides an orthogonal linear layer parameter approximation optimization method for the orthogonal linear layer. Updated parameters Perform QR decomposition:

[0247] ;

[0248] in, is an orthogonal matrix, is an upper triangular matrix.

[0249] Use the orthogonal matrix Q as the orthogonal linear layer Updated parameters to ensure the orthogonality constraint of the updated parameters, that is:

[0250] ;

[0251] in, Represents an orthogonal linear layer The parameters after updating again.

[0252] The orthogonal projection optimization method proposed in the embodiment of the present invention can ensure that the orthogonal linear layer in the orthogonal projection unit is optimized during the model training of the visual perception model. The orthogonal constraint of is beneficial to decoupling the resolution-dependent features and the first resolution-independent features.

[0253] Based on the visual perception model parameter adjustment scheme based on resolution-based image feature information introduced in the above embodiment, the embodiment of the present invention continues to explain the training steps and reasoning steps of the visual perception model.

[0254] As described in the above embodiments of the present invention, the training task of the visual perception model can be executed in a cloud computing center.

[0255] If applied to the field of autonomous driving, the visual perception model can be trained on the collected and annotated road scene image data, and the multi-resolution meta-learning optimization method provided by the embodiment of the present invention is used. In each training batch, the image samples corresponding to m resolutions are respectively input into the visual perception model in the current iterative training, and the multi-resolution image correction network is used in the visual perception model to correct the image features. Finally, the perception result is output through the visual perception model and the perception loss value is calculated according to the true value. All the perception loss values ​​in the training batch are collected and substituted into the loss function of the visual perception model to obtain the model learning loss value, and then the model is optimized by the back propagation algorithm. Among them, the optimizer can use a gradient descent (GD) optimizer, an adaptive moment estimation (Adam) optimizer, an adaptive weight decay regularization (Adam Weight Decay Regularization, AdamW) optimizer, etc. The model parameter adjustment method of the visual perception model proposed by the present invention is used during training to improve the training efficiency. After the model training is completed, the visual perception model obtained by training is deployed to the edge computing platform of the unmanned vehicle, and the parameters of the visual perception model are updated according to the resolution required to be processed by the edge computing platform and the trained model parameter prediction network.

[0256] The visual perception method provided by the embodiment of the present invention can be applied to, but not limited to, the field of autonomous driving, and can also be applied to intelligent terminals such as intelligent robots, smart homes, and mobile phones, and can also be applied to visual perception models involved in industrial production fields such as transportation, aerial remote sensing, and industrial product assembly. For example, the visual perception model can be an autonomous driving perception model; for this, the image sample set can be constructed based on image samples collected by vehicle-mounted sensors from different perspectives, and the visual perception model can be trained on the collected and annotated road scene image data set when executing model training in the cloud data center.

[0257] In an embodiment of the present invention, the parameters of the visual perception model and the parameters of the resolution-independent feature extraction network are updated using the perceptual loss values ​​corresponding to each resolution, which may include: calculating the model learning loss value of the visual perception model with the sum of the perceptual loss values ​​corresponding to each resolution; and updating the parameters of the visual perception model and the parameters of the resolution-independent feature extraction network using the model learning loss value. For example, a loss function of a visual perception model provided in the above embodiment of the present invention may be used. If the above embodiment of the present invention is combined with the method of adding resolution classification loss to enhance the extraction effect of resolution-independent features and resolution-dependent features, the loss function of the visual perception model is It can be expressed as:

[0258] ;

[0259] in, Represents the loss value calculated based on the perceptual loss value corresponding to each resolution, represents the first resolution classification loss function corresponding to the first resolution-independent feature extraction network, represents the weight coefficient of the first resolution classification loss function, represents the second resolution classification loss function corresponding to the resolution-dependent feature extraction network, represents the weight coefficient of the second resolution classification loss function, represents the third-resolution classification loss function corresponding to the feature aggregation network, Represents the weight coefficient of the third resolution classification loss function.

[0260] Weight coefficient , , A preset value may be used, or a coefficient updated through self-learning during the iterative training process of the visual perception model may be used.

[0261] When using the visual perception model to perform target perception tasks, the trained visual perception model can be directly deployed in actual application scenarios to perform target perception tasks.

[0262] It should be noted that in the embodiments of the target perception method of the present invention, some of the steps or features may be ignored or not executed. The hardware or software functional modules divided for the convenience of description are not the only implementation form of the target perception method provided by the embodiment of the present invention.

[0263] The above describes in detail the various embodiments corresponding to the target perception method. On this basis, the present invention also discloses a target perception device, equipment, non-volatile storage medium and computer program product corresponding to the above method.

[0264] Fig.15 A schematic diagram of the structure of a target sensing device provided in an embodiment of the present invention.

[0265] like Fig.15 As shown, the target sensing device provided by the embodiment of the present invention includes:

[0266] The model training unit 1501 is used to train a visual perception model with a resolution-independent feature extraction network added using an image sample set including image samples of multiple resolutions; the resolution-independent feature extraction network is used to extract resolution-independent features from input image features and then output them; in iterative training, for each resolution, image samples of the resolution are input into the visual perception model and a perceptual loss value is calculated; the perceptual loss value corresponding to each resolution is used to update the parameters of the visual perception model and the parameters of the resolution-independent feature extraction network; after the iterative training end condition is met, the trained visual perception model is obtained;

[0267] The target perception unit 1502 is used to input the input image corresponding to the target perception task to be processed into the trained visual perception model to obtain the perception result of the perception target of the target perception task to be processed.

[0268] It should be noted that in each implementation of the target sensing device provided by the embodiment of the present invention, the division of units is only a logical functional division, and other division methods may be used. The connection method between different units may be electrical, mechanical or other connection methods. Separated units may be located in the same physical location or distributed on multiple network nodes. Each unit may be implemented in the form of hardware or in the form of a software functional unit. That is, part or all of the units provided in the embodiment of the present invention may be selected according to actual needs and a corresponding connection method or integration method may be adopted to achieve the purpose of the embodiment of the present invention.

[0269] Since the embodiments of the apparatus part correspond to the embodiments of the method part, please refer to the description of the embodiments of the method part for the embodiments of the apparatus part, which will not be repeated here.

[0270] Fig.16 A schematic diagram of the structure of a target sensing device provided in an embodiment of the present invention.

[0271] like Fig.16 As shown, the target perception device provided by an embodiment of the present invention includes: a memory 1610, used to store a computer program 1611; a processor 1620, used to execute the computer program 1611, and when the computer program 1611 is executed by the processor 1620, the steps of the target perception method provided in any one of the above embodiments are implemented.

[0272] Among them, the processor 1620 may include one or more processing cores, such as a 3-core processor, an 8-core processor, etc. The processor 1620 may be implemented in at least one hardware form of digital signal processing (DSP), field-programmable gate array (FPGA), and programmable logic array (PLA). The processor 1620 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a central processing unit (CPU); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 1620 may be integrated with a graphics processing unit (GPU), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 1620 may also include an artificial intelligence (AI) processor, which is used to process computing operations related to machine learning.

[0273] The memory 1610 may include one or more non-volatile storage media, which may be non-transitory. The memory 1610 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In this embodiment, the memory 1610 is at least used to store the following computer program 1611, wherein the computer program 1611, after being loaded and executed by the processor 1620, can implement the relevant steps in the target perception method disclosed in any of the aforementioned embodiments. In addition, the resources stored in the memory 1610 may also include an operating system 1612 and data 1613, etc., and the storage method may be short-term storage or permanent storage. Among them, the operating system 1612 may be Windows or other types of operating systems. Data 1613 may include, but is not limited to, the data involved in the above method.

[0274] In some embodiments, the target sensing device may also include a display screen 1630 , a power supply 1640 , a communication interface 1650 , an input / output interface 1660 , a sensor 1670 , and a communication bus 1680 .

[0275] Those skilled in the art will understand that Fig.16 The structure shown in the figure does not constitute a limitation on the target sensing device, and may include more or fewer components than shown in the figure.

[0276] The target perception device provided in an embodiment of the present invention includes a memory and a processor. When the processor executes the program stored in the memory, it can implement the steps of the target perception method provided in the above embodiment, and the effect is the same as above.

[0277] An embodiment of the present invention provides a non-volatile storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of the target perception method provided in any one of the above embodiments can be implemented.

[0278] The non-volatile storage medium may include: a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and other media that can store program codes.

[0279] For an introduction to the non-volatile storage medium provided in the embodiment of the present invention, please refer to the above-mentioned method embodiment, and the effect thereof is the same as the target sensing method provided in the embodiment of the present invention, and the present invention will not elaborate on it here.

[0280] An embodiment of the present invention provides a computer program product, including a computer program, which, when executed by a processor, implements the steps of the target perception method provided in any one of the above embodiments.

[0281] For an introduction to the computer program product provided by the embodiment of the present invention, please refer to the above-mentioned method embodiment, and the effect thereof is the same as the target perception method provided by the embodiment of the present invention, and the present invention will not elaborate on it here.

[0282] The above is a detailed introduction to a target perception method, device, equipment, medium and computer program product provided by the present invention. The various embodiments in the specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same and similar parts between the embodiments can refer to each other. For the devices, equipment, non-volatile storage media and computer program products disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple, and the relevant parts can refer to the method part description. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of the present invention, the present invention can also be improved and modified in several ways, and these improvements and modifications also fall within the scope of protection of the present invention.

[0283] It should also be noted that, in this specification, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the statement "comprises a ..." does not exclude the presence of other identical elements in the process, method, article or device including the element.

Claims

1. A target perception method, characterized in that: include: Using an image sample set including image samples of various resolutions to train a visual perception model to which a resolution-independent feature extraction network is added; The resolution-independent feature extraction network is used to extract resolution-independent features from input image features and then output them; In iterative training, for each resolution, an image sample of the resolution is input into the visual perception model and a perceptual loss value is calculated; the parameters of the visual perception model and the parameters of the resolution-independent feature extraction network are updated using the perceptual loss value corresponding to each resolution; After reaching the iterative training end condition, obtaining the trained visual perception model; Inputting the input image corresponding to the target perception task to be processed into the trained visual perception model to obtain a perception result of the perception target of the target perception task to be processed; Wherein, the resolution-independent feature extraction network comprises: A first resolution-independent feature extraction network, used to extract a first resolution-independent feature from an input image feature of the resolution-independent feature extraction network; A resolution-dependent feature extraction network for extracting resolution-dependent features from input image features of the resolution-independent feature extraction network; A feature aggregation network is used to aggregate the first resolution-independent feature and the second resolution-independent feature in the resolution-dependent feature, and output the resolution-independent feature.

2. The target perception method according to claim 1, characterized in that: Extracting a first resolution-independent feature from an input image feature of the resolution-independent feature extraction network comprises: A first deep convolution calculation, a batch normalization calculation, a second deep convolution calculation, and an activation function calculation are performed on the input image features of the resolution-independent feature extraction network to obtain the first resolution-independent features.

3. The target perception method according to claim 1, characterized in that: The resolution-independent feature extraction network also includes: A first resolution classification network, used for outputting a first resolution classification result according to the first resolution-independent feature; Updating the parameters of the visual perception model and the parameters of the resolution-independent feature extraction network using the perceptual loss values ​​corresponding to the resolutions includes: A model learning loss value corresponding to a current iterative training batch is calculated according to the perception loss value corresponding to each of the resolutions and the first resolution classification result corresponding to each of the resolutions, and the parameters of the visual perception model and the parameters of the resolution-independent feature extraction network are updated using the model learning loss value; Among them, the model learning loss value is positively correlated with the resolution classification confidence corresponding to the first resolution classification result.

4. The target perception method according to claim 1, characterized in that: Extracting resolution-dependent features from input image features of the resolution-independent feature extraction network includes: A third deep convolution calculation, a batch normalization calculation, a fourth deep convolution calculation, and an activation function calculation are performed on the input image features of the resolution-independent feature extraction network to obtain the resolution-dependent features.

5. The target perception method according to claim 1, characterized in that: The resolution-independent feature extraction network also includes: A second resolution classification network, used for outputting a second resolution classification result according to the resolution-related features; Updating the parameters of the visual perception model and the parameters of the resolution-independent feature extraction network using the perceptual loss values ​​corresponding to the resolutions includes: A model learning loss value corresponding to a current iterative training batch is calculated according to the perception loss value corresponding to each of the resolutions and the second resolution classification result corresponding to each of the resolutions, and the parameters of the visual perception model and the parameters of the resolution-independent feature extraction network are updated using the model learning loss value; Among them, the model learning loss value is negatively correlated with the resolution classification confidence corresponding to the second resolution classification result.

6. The target perception method according to claim 1, characterized in that: Aggregating the first resolution-independent feature and the second resolution-independent feature in the resolution-dependent feature, and outputting the resolution-independent feature, includes: The first resolution-independent feature is used as a query feature, and the resolution-dependent feature is used as a key feature and a value feature. Attention calculation is performed using the query feature, the key feature, and the value feature to output the resolution-independent feature.

7. The target perception method according to claim 6, characterized in that: Performing attention calculation using the query feature, the key feature, and the value feature to output the resolution-independent feature includes: After aggregating features from the resolution-dependent features according to the query features, the key features, and the value features using a cross attention layer, the features are processed sequentially through a first normalization layer, a self-attention layer, a second normalization layer, a feedforward neural network, and a third normalization layer, and the above steps are repeated for a preset number of times to output the resolution-independent features; Among them, the cross-attention layer, the self-attention layer and the feedforward neural network processing are all provided with a residual connection for adding output features and output features.

8. The target perception method according to claim 1, characterized in that: The resolution-independent feature extraction network also includes: A third resolution classification network, used for outputting a third resolution classification result according to the resolution-independent features; Updating the parameters of the visual perception model and the parameters of the resolution-independent feature extraction network using the perceptual loss values ​​corresponding to the resolutions includes: A model learning loss value corresponding to the current iterative training batch is calculated according to the perception loss value corresponding to each of the resolutions and the third resolution classification result corresponding to each of the resolutions, and the parameters of the visual perception model and the parameters of the resolution-independent feature extraction network are updated using the model learning loss value; Among them, the model learning loss value is positively correlated with the resolution classification confidence corresponding to the third resolution classification result.

9. The target perception method according to claim 1, characterized in that: The resolution-independent feature extraction network also includes: A first resolution classification network, used for outputting a first resolution classification result according to the first resolution-independent feature; A second resolution classification network, used for outputting a second resolution classification result according to the resolution-related features; A third resolution classification network, used for outputting a third resolution classification result according to the resolution-independent features; Updating the parameters of the visual perception model and the parameters of the resolution-independent feature extraction network using the perceptual loss values ​​corresponding to the resolutions includes: A model learning loss value corresponding to a current iterative training batch is calculated according to the perception loss value corresponding to each of the resolutions, the first resolution classification result corresponding to each of the resolutions, the second resolution classification result corresponding to each of the resolutions, and the third resolution classification result corresponding to each of the resolutions, and the parameters of the visual perception model and the parameters of the resolution-independent feature extraction network are updated using the model learning loss value; Among them, the model learning loss value is positively correlated with the resolution classification confidence corresponding to the first resolution classification result, the model learning loss value is negatively correlated with the resolution classification confidence corresponding to the second resolution classification result, and the model learning loss value is positively correlated with the resolution classification confidence corresponding to the third resolution classification result.

10. The target perception method according to claim 3 or 9, characterized in that: The value of the first resolution classification result is negatively correlated with the resolution classification confidence corresponding to the first resolution classification result; The model learning loss value is negatively correlated with the value of the first resolution classification result.

11. The target perception method according to claim 10, characterized in that: The first resolution classification network includes: A first gradient reversal layer, used for performing gradient reversal on the first resolution-independent feature; The first resolution classifier is used to input the output result of the first gradient reversal layer and then output the first resolution classification result.

12. The target perception method according to claim 8 or 9, characterized in that: The value of the third resolution classification result is negatively correlated with the resolution classification confidence corresponding to the third resolution classification result; The model learning loss value is negatively correlated with the value of the third resolution classification result.

13. The target perception method according to claim 12, characterized in that: The third resolution classification network includes: A second gradient reversal layer, used for performing gradient reversal on the resolution-independent feature; The third resolution classifier is used to input the output result of the second gradient reversal layer and then output the third resolution classification result.

14. The target perception method according to claim 1, characterized in that: The visual perception model also includes an orthogonal projection unit arranged at the input end of the resolution-independent feature extraction network.

15. The target perception method according to claim 14, characterized in that: Also includes: In the iterative training of the visual perception model, the parameters of the orthogonal linear layer of the orthogonal projection unit are updated using the perceptual loss values ​​corresponding to each of the resolutions; Determining whether the updated parameters of the orthogonal linear layer satisfy the orthogonality constraint; If not, the parameters of the orthogonal linear layer are updated again using the orthogonal matrix corresponding to the updated parameters of the orthogonal linear layer.

16. A target sensing device, characterized in that: include: A model training unit, for training a visual perception model to which a resolution-independent feature extraction network is added using an image sample set including image samples of multiple resolutions; The resolution-independent feature extraction network is used to extract resolution-independent features from input image features and then output them; In the iterative training, for each resolution, an image sample of the resolution is input into the visual perception model and a perceptual loss value is calculated; the parameters of the visual perception model and the parameters of the resolution-independent feature extraction network are updated using the perceptual loss values ​​corresponding to each resolution; after the iterative training end condition is reached, the trained visual perception model is obtained; A target perception unit, used for inputting an input image corresponding to the target perception task to be processed into the trained visual perception model to obtain a perception result of the perception target of the target perception task to be processed; Wherein, the resolution-independent feature extraction network comprises: A first resolution-independent feature extraction network, used to extract a first resolution-independent feature from an input image feature of the resolution-independent feature extraction network; A resolution-dependent feature extraction network for extracting resolution-dependent features from input image features of the resolution-independent feature extraction network; A feature aggregation network is used to aggregate the first resolution-independent feature and the second resolution-independent feature in the resolution-dependent feature, and output the resolution-independent feature.

17. A target sensing device, characterized in that: include: Memory for storing computer programs; A processor is used to execute the computer program, and the computer program, when executed by the processor, implements the steps of the target perception method as described in any one of claims 1 to 15.

18. A non-volatile storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the target perception method as described in any one of claims 1 to 15 are implemented.

19. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the target perception method as described in any one of claims 1 to 15 are implemented.

Images