A visual perception method, device, equipment, medium and computer program product

Abstract

The present invention relates to the field of computer vision technology, and specifically discloses a visual perception method, device, equipment, medium and computer program product. When training a visual perception model with multi-resolution image samples and learning image samples of each resolution with the visual perception model, the model parameter adjustment coefficient obtained by inputting image feature information corresponding to the resolution into a model parameter prediction network is used to adjust the parameters of the visual perception model, and the parameters of the visual perception model and the parameters of the model parameter prediction network are updated using the perception loss values ​​corresponding to each resolution, so as to reduce the difficulty of the visual perception model learning multi-resolution image samples and ensure the multi-resolution training effect; the resolution of the visual perception task to be processed is used to update the parameters of the visual perception model and then execute the visual perception task to be processed, so that the model can better adapt to the current input image resolution, and high-precision visual perception results can be guaranteed for input images with different resolutions.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a visual perception method, device, equipment, medium and computer program product. Background Art

[0002] Visual perception refers to the use of computer vision to identify information in collected images to obtain corresponding scene information. It is widely used in autonomous driving, intelligent robots, smart homes, smart terminals, as well as industrial production such as transportation, aerial remote sensing, and industrial product assembly.

[0003] When performing visual perception tasks, the visual perception model determines the accuracy of perception. In practical applications, it is sometimes necessary for the visual perception model to be able to process input images of different resolutions. For example, for application scenarios with limited computing resources, lower-resolution image samples are usually used to train the visual perception model, while for perception scenarios of small targets and long-distance targets, higher-resolution image samples are used to train the visual perception model. However, if the resolution of the input image of the visual perception model in the inference calculation stage is different from the resolution of the image samples in the training stage, the detection accuracy of the visual perception model will be significantly reduced. As a result, in practical applications, once the visual perception model is deployed, it can only process input images with a fixed resolution.

[0004] How to improve the resolution generalization capability of adapting to input images of different resolutions in visual perception tasks is a technical problem that technicians in this field need to solve. Summary of the invention

[0005] The purpose of the present invention is to provide a visual perception method, apparatus, device, medium and computer program product for improving the resolution generalization capability of adapting to input images of different resolutions in visual perception tasks of computer vision.

[0006] In order to solve the above technical problems, the present invention provides a visual perception method, comprising:

[0007] Training a visual perception model using an image sample set including image samples at multiple resolutions;

[0008] In the iterative training of the visual perception model, for each resolution, the image feature information corresponding to the resolution is input into the model parameter prediction network, and the model parameter adjustment coefficient is output. After the parameters of the visual perception model are adjusted using the model parameter adjustment coefficient, the image sample of the resolution is input into the adjusted visual perception model, and the perception loss value corresponding to the resolution is output;

[0009] Using the perceptual loss values ​​corresponding to each of the resolutions, updating the parameters of the visual perception model and the parameters of the model parameter prediction network, to obtain the visual perception model corresponding to the current iterative training batch;

[0010] After reaching the iterative training end condition, obtaining the trained visual perception model;

[0011] After using the model parameter prediction network to update the parameters of the visual perception model according to the resolution corresponding to the visual perception task to be processed, the input image of the visual perception task to be processed is input into the updated visual perception model to obtain a visual perception result of the perception target of the visual perception task to be processed.

[0012] On the one hand, the image feature information corresponding to the resolution is input into the model parameter prediction network, and the model parameter adjustment coefficient is output, including:

[0013] Determining the corresponding model parameter adjustment coefficient according to the parameter type of the visual perception model;

[0014] The image feature information corresponding to the resolution is input into the model parameter prediction network to generate the model parameter adjustment coefficients corresponding to each parameter type according to the image feature information corresponding to the resolution.

[0015] On the other hand, adjusting the parameters of the visual perception model using the model parameter adjustment coefficient includes:

[0016] Determining a model parameter adjustment function according to a calculation type of the parameters of the visual perception model;

[0017] Substitute the parameters of the visual perception model and the corresponding model parameter type into the corresponding model parameter adjustment function to obtain adjusted model parameters, so as to update the parameters of the visual perception model to the adjusted model parameters.

[0018] On the other hand, the image feature information corresponding to the resolution includes image features of image samples of the resolution.

[0019] On the other hand, the image feature information corresponding to the resolution includes the image feature size corresponding to the resolution.

[0020] On the other hand, the image feature information corresponding to the resolution includes image features of image samples of the resolution and image feature sizes corresponding to the resolution.

[0021] On the other hand, if the image feature information corresponding to the resolution is the image feature of the image sample of the resolution, the image feature information corresponding to the resolution is input into the model parameter prediction network, and the model parameter adjustment coefficient is output, including:

[0022] Performing global mean pooling processing and global maximum pooling processing on the image features corresponding to the resolution respectively, to obtain a global mean pooling feature vector corresponding to the resolution and a global maximum pooling feature vector corresponding to the resolution;

[0023] Calculating a mean of the image features corresponding to the resolution and a variance of the image features corresponding to the resolution;

[0024] Perform channel dimension splicing processing on the global mean pooling feature vector corresponding to the resolution, the mean of the image feature corresponding to the resolution, and the variance of the image feature corresponding to the resolution to obtain a global mean pooling feature splicing result;

[0025] Perform channel dimension splicing processing on the global maximum pooling feature vector corresponding to the resolution, the mean of the image feature corresponding to the resolution, and the variance of the image feature corresponding to the resolution to obtain a global maximum pooling feature splicing result;

[0026] The model parameter adjustment coefficient is obtained according to the global mean pooling feature splicing result and the global maximum pooling feature splicing result.

[0027] On the other hand, the model parameter adjustment coefficient is obtained according to the global mean pooling feature splicing result and the global maximum pooling feature splicing result, including:

[0028] Using a first multi-layer perceptron and a first activation function in sequence to process the global mean pooling feature concatenation result to obtain a mean feature vector;

[0029] Using a second multi-layer perceptron and a second activation function in sequence to process the global maximum pooling feature concatenation result to obtain a maximum feature vector;

[0030] Performing a feature splicing operation on the mean feature vector and the maximum feature vector to obtain a first feature splicing result;

[0031] The first feature splicing result is input into a first model parameter prediction function corresponding to the model parameter adjustment coefficient to obtain the model parameter adjustment coefficient.

[0032] On the other hand, if the image feature information corresponding to the resolution is the image feature size corresponding to the resolution, the image feature information corresponding to the resolution is input into the model parameter prediction network, and the model parameter adjustment coefficient is output, including:

[0033] Calculating a first normalized ratio of the image feature size corresponding to the resolution to the reference image feature size;

[0034] Calculating a second normalized ratio of the input image feature size of the network layer to be adjusted in the visual perception model to the reference image feature size;

[0035] Concatenating the first normalized ratio and the second normalized ratio to obtain a third normalized ratio;

[0036] The third normalized ratio is input into a second model parameter prediction function corresponding to the model parameter adjustment coefficient to obtain the model parameter adjustment coefficient.

[0037] On the other hand, the image feature information corresponding to the resolution is input into the model parameter prediction network, and the model parameter adjustment coefficient is output, including:

[0038] Inputting the image features corresponding to the resolution into a first model parameter prediction network, and outputting a first model parameter adjustment coefficient;

[0039] Inputting the image feature size corresponding to the resolution into a second model parameter prediction network, and outputting a second model parameter adjustment coefficient;

[0040] The model parameter adjustment coefficient is calculated according to the first model parameter adjustment coefficient and the second model parameter adjustment coefficient.

[0041] On the other hand, calculating the model parameter adjustment coefficient according to the first model parameter adjustment coefficient and the second model parameter adjustment coefficient includes:

[0042] Determining a model parameter combination function according to a calculation type of the parameters of the visual perception model;

[0043] Substitute the first model parameter adjustment coefficient and the second model parameter adjustment coefficient into the model parameter combination function to obtain the model parameter adjustment coefficient corresponding to the parameters of the visual perception model.

[0044] On the other hand, the weight coefficient of the first model parameter adjustment coefficient and the weight coefficient of the second model parameter adjustment coefficient in the model parameter combination function are both learnable parameters.

[0045] On the other hand, inputting the image feature information corresponding to the resolution into a model parameter prediction network, outputting a model parameter adjustment coefficient, and using the model parameter adjustment coefficient to adjust the parameters of the visual perception model, including:

[0046] According to the network layer type of the visual perception model, the visual perception model is processed in blocks;

[0047] Inputting the image feature information corresponding to the resolution into the model parameter prediction network corresponding to each network block of the visual perception model, and outputting the corresponding model parameter adjustment coefficient;

[0048] The model parameter adjustment coefficient is used to adjust the parameters of each network block of the visual perception model accordingly.

[0049] On the other hand, inputting the image feature information corresponding to the resolution into a model parameter prediction network, outputting a model parameter adjustment coefficient, and using the model parameter adjustment coefficient to adjust the parameters of the visual perception model, including:

[0050] Determining a target network layer for parameter adjustment in the visual perception model;

[0051] Inputting the image feature information corresponding to the resolution into the model parameter prediction network corresponding to the target network layer, and outputting the corresponding model parameter adjustment coefficient;

[0052] The model parameter adjustment coefficient is used to adjust the parameters of the target network layer accordingly.

[0053] On the other hand, after updating the parameters of the visual perception model according to the resolution corresponding to the visual perception task to be processed by using the model parameter prediction network, the input image of the visual perception task to be processed is input into the updated visual perception model to obtain a visual perception result of the perception target of the visual perception task to be processed, including:

[0054] After updating the parameters of the visual perception model according to the image feature size corresponding to the resolution using the first model parameter prediction network, the input image of the visual perception task to be processed is input into the updated visual perception model to obtain the visual perception result;

[0055] In the process of processing the input image using the updated visual perception model, before the image features of the resolution are input into the network layer of the updated visual perception model, the parameters of the network layer are adjusted according to the image features of the resolution using a second model parameter prediction network.

[0056] On the other hand, the parameters of the visual perception model and the parameters of the model parameter prediction network are updated using the perceptual loss values ​​corresponding to each of the resolutions to obtain the visual perception model corresponding to the current iterative training batch, including:

[0057] The regularization constraint items corresponding to the resolutions are calculated according to the model parameter adjustment coefficients corresponding to the resolutions;

[0058] According to the perceptual loss value corresponding to each of the resolutions and the regularization constraint item corresponding to each of the resolutions, the parameters of the visual perception model and the parameters of the model parameter prediction network are updated to obtain the visual perception model corresponding to the current iterative training batch.

[0059] In order to solve the above technical problems, the present invention further provides a visual perception device, comprising:

[0060] A model training unit is used to train a visual perception model using an image sample set including image samples of multiple resolutions; in the iterative training of the visual perception model, for each resolution, input the image feature information corresponding to the resolution into a model parameter prediction network, output a model parameter adjustment coefficient, and after adjusting the parameters of the visual perception model using the model parameter adjustment coefficient, input the image samples of the resolution into the adjusted visual perception model, and output a perceptual loss value corresponding to the resolution; update the parameters of the visual perception model and the parameters of the model parameter prediction network using the perceptual loss values ​​corresponding to each resolution, and obtain the visual perception model corresponding to the current iterative training batch; after the iterative training end condition is met, obtain the trained visual perception model;

[0061] The visual perception unit is used to use the model parameter prediction network to update the parameters of the visual perception model according to the resolution corresponding to the visual perception task to be processed, and then input the input image of the visual perception task to be processed into the updated visual perception model to obtain a visual perception result of the perception target of the visual perception task to be processed.

[0062] In order to solve the above technical problems, the present invention further provides a visual perception device, comprising:

[0063] Memory for storing computer programs;

[0064] A processor is used to execute the computer program, and when the computer program is executed by the processor, the steps of the visual perception method as described in any one of the above items are implemented.

[0065] In order to solve the above technical problems, the present invention also provides a non-volatile storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the visual perception method as described in any one of the above items are implemented.

[0066] In order to solve the above technical problems, the present invention also provides a computer program product, including a computer program, which, when executed by a processor, implements the steps of the visual perception method as described in any one of the above items.

[0067] The visual perception method provided by the present invention has the beneficial effect that in the process of training the visual perception model with multi-resolution image samples, when using the visual perception model to learn image samples of each resolution, the model parameter adjustment coefficient obtained by inputting the image feature information corresponding to the resolution into the model parameter prediction network is used to adjust the parameters of the visual perception model, and the perception loss value corresponding to each resolution is used to update the parameters of the visual perception model and the parameters of the model parameter prediction network, thereby reducing the difficulty of the visual perception model to learn multi-resolution image samples and ensuring the multi-resolution training effect; after using the trained visual perception model to update the parameters of the visual perception model according to the resolution corresponding to the visual perception task to be processed, the input image of the visual perception task to be processed is input into the updated visual perception model to obtain the visual perception result of the perception target of the visual perception task to be processed, and the model parameters can be adjusted to make it more adaptable to the current input image resolution, thereby ensuring high-precision visual perception results for input images of different resolutions.

[0068] The present invention also provides a visual perception device, equipment, medium and computer program product, which have the above-mentioned beneficial effects and are not described in detail here. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] In order to more clearly illustrate the embodiments of the present invention or the technical solutions of the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0070] Figure 1 A flowchart of a visual perception method provided by an embodiment of the present invention;

[0071] Figure 2 A schematic diagram of the structure of a visual perception system provided by an embodiment of the present invention;

[0072] Figure 3 A flowchart of multi-resolution loss optimization provided by an embodiment of the present invention;

[0073] Figure 4 An architecture diagram of a visual perception model adjusted by using a model parameter prediction network provided by an embodiment of the present invention;

[0074] Figure 5 A training flow chart of a visual perception model provided by an embodiment of the present invention;

[0075] Figure 6 A schematic diagram of the structure of a model parameter prediction network provided by an embodiment of the present invention;

[0076] Figure 7 A schematic diagram of the structure of another model parameter prediction network provided by an embodiment of the present invention;

[0077] Figure 8 A schematic diagram of the structure of a visual perception device provided by an embodiment of the present invention;

[0078] Fig. 9 A schematic diagram of the structure of a visual perception device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0079] The core of the present invention is to provide a visual perception method, apparatus, device, medium and computer program product for improving the resolution generalization capability of adapting to input images of different resolutions in visual perception tasks of computer vision.

[0080] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0081] Resolution is used to describe the clarity of an image, video or display device. It indicates the number of pixels that can be displayed per unit length. Resolution is usually expressed in the following forms:

[0082] Pixel resolution: This is the most intuitive way of expressing it, which is expressed directly in pixels. For example, 1920x1080 means there are 1920 pixels in the horizontal direction (image width) and 1080 pixels in the vertical direction (image height).

[0083] Screen resolution: Usually used to describe the resolution of a monitor, TV or other display device. For example, Full HD usually refers to a resolution of 1920x1080, while 4K resolution usually refers to 3840x2160.

[0084] Print resolution: In the printing field, resolution is usually expressed in dots per inch (DPI). For example, 300 DPI means there are 300 dots per inch.

[0085] Image scanning resolution: The resolution of the scanner is also expressed in DPI, which determines the clarity of the scanned image.

[0086] Video resolution: The resolution of a video is usually expressed in pixels, such as 720p (1280x720), 1080p (1920x1080), and 4K (3840x2160 or 4096x2160).

[0087] The visual perception method provided in the embodiment of the present invention can be used to process images captured by a visual sensor such as a camera, or to process video frames captured from a captured video. In the embodiment of the present invention, the resolution described can be pixel resolution.

[0088] The resolution generalization ability of a visual perception model refers to the ability of the visual perception model to process input images with resolutions that are inconsistent with the training resolution during the testing phase.

[0089] Traditional visual perception solutions usually use the same resolution in the model training and testing stages. This is because the resolution generalization ability of visual perception models is poor. Specifically, when the test resolution is inconsistent with the training resolution, the model accuracy will be seriously reduced. The greater the difference between the test resolution and the training resolution, the more serious the reduction in model accuracy, which limits the application of visual perception models in different scenarios. The fundamental reason for this phenomenon is that the distribution of image data of different resolutions is different, and current visual perception models have difficulty in handling such distribution differences.

[0090] However, in practical applications, the visual perception model after deployment may need to process input images with higher resolutions. For example, a visual perception model trained with a resolution of (256x704) may need to process input images with a resolution of (512x1408) or a resolution of (128x352) after deployment. To meet the application needs of processing images with higher resolutions, one current solution is to adjust the input image to the same size as the resolution of the image sample in the training phase during the application phase, that is, adjust the input image with a resolution of (512x1408) and the input image with a resolution of (128x352) to a resolution of (256x704).

[0091] Although this solution of unifying the resolution of input images is feasible, in some practical application scenarios, there is a high demand for different resolution settings. For some application scenarios with limited computing resources, the visual perception model is required to process low-resolution input images (such as 128x352 resolution in the above example) to reduce the computational burden and ensure real-time performance. If the input image resolution is increased to align it to the training resolution, the computational burden will increase and affect the real-time performance of the system. For some application scenarios with sufficient computing resources, it is hoped that a higher image resolution (such as 512x1408 resolution in the above example) will be used to improve the accuracy of three-dimensional visual perception. By reducing the high resolution of the image to align it to the training resolution, a lot of detail information will be lost, which may be detrimental to the detection of small and distant targets.

[0092] It can be seen that although this method of processing images of different resolutions to training resolution before inputting the input images into the visual perception model solves the problem of the difference in different resolutions, it causes the aligned and adjusted images to lose a lot of detail information, which is not conducive to practical applications. That is, in practical applications, in order to solve the problem, the visual perception model with the same training resolution can only be selected according to the resolution of the input image, which results in the inability to process images to be recognized with different resolutions, poor flexibility, and high deployment cost.

[0093] Therefore, in actual visual perception applications, there is still a great demand for the resolution generalization ability of the visual perception model itself. In order to improve the resolution generalization ability of the visual perception model, it is necessary to use image samples of different resolutions to train the visual perception model. In order to solve the problem that the feature distribution of image samples of different resolutions is quite different, the present invention provides a visual perception solution. In the iterative training process of training the visual perception model using multi-resolution image samples, a model parameter prediction network is used to calculate the model parameter prediction network according to the image feature information corresponding to the resolution, and the model parameter prediction network is used to adjust the parameters of the visual perception model, and then the visual perception model is used to process the image samples and calculate the perception loss value. The parameters of the model parameter prediction network are updated along with the iterative training of the visual perception model, thereby realizing a solution for extracting resolution-related information from the input image features in a data-driven manner, and adjusting the parameters of the visual perception model to make it more suitable for the current input image resolution. After using the trained visual perception model to update the parameters of the visual perception model according to the resolution corresponding to the visual perception task to be processed, the input image of the visual perception task to be processed is input into the updated visual perception model to obtain the visual perception result of the perception target of the visual perception task to be processed. The model parameters can be adjusted to make it more adaptable to the current input image resolution, thereby ensuring high-precision visual perception results for input images of different resolutions.

[0094] The visual perception method provided by the embodiment of the present invention is described below with reference to the accompanying drawings.

[0095] Figure 1 A flowchart of a visual perception method provided by an embodiment of the present invention; Figure 2 A schematic diagram of the structure of a visual perception system provided by an embodiment of the present invention; Figure 3 A flowchart of multi-resolution loss optimization is provided in an embodiment of the present invention.

[0096] like Figure 1 As shown, the visual perception method provided by the embodiment of the present invention includes:

[0097] S101: training a visual perception model using an image sample set including image samples of multiple resolutions;

[0098] S102: In the iterative training of the visual perception model, for each resolution, input the image feature information corresponding to the resolution into the model parameter prediction network, output the model parameter adjustment coefficient, adjust the parameters of the visual perception model using the model parameter adjustment coefficient, input the image sample of the resolution into the adjusted visual perception model, and output the perception loss value corresponding to the resolution;

[0099] S103: using the perceptual loss values ​​corresponding to each resolution to update the parameters of the visual perception model and the parameters of the model parameter prediction network, to obtain the visual perception model corresponding to the current iterative training batch;

[0100] S104: after reaching the iterative training end condition, obtaining a trained visual perception model;

[0101] S105: After updating the parameters of the visual perception model according to the resolution corresponding to the visual perception task to be processed using the model parameter prediction network, the input image of the visual perception task to be processed is input into the updated visual perception model to obtain a visual perception result of the perception target of the visual perception task to be processed.

[0102] The visual perception method provided by the embodiment of the present invention mainly includes two major links: training of the visual perception model and application of the visual perception model. In this regard, the visual perception method provided by the embodiment of the present invention can be applied to Figure 2 The visual perception system shown includes two parts: a model training system and a perception computing device.

[0103] The model training system may be composed of one or more computing devices for training a visual perception model based on an image sample set composed of image samples of multiple resolutions. Figure 2As shown, the model training system can be composed of multiple threads of multiple artificial intelligence devices to form a thread pool, and the image samples corresponding to each resolution are assigned to a thread pool, so as to achieve parallel accelerated learning of multiple resolution image sets. It should be noted that m, n, and k only represent the meaning of multiple, and do not represent a specific number or the same number of threads of each artificial intelligence processor, nor do they mean that each thread pool is composed of one thread in each artificial intelligence device. The model training system can be deployed in a cloud computing center. The device structure of the cloud computing center is similar Figure 2 The perceptual computing devices in the system include computing, storage, network and other infrastructure, operating system, database, middleware and other software environments and application software. The model training system receives image data sets of various resolutions, uses the stored pre-trained models, combined with manual annotation, to annotate visual perception targets (such as the category, location, size of the bounding box, target orientation information, etc.) and stores the annotated data.

[0104] The perception computing device can be a computing device deployed in a cloud computing center or an edge computing device deployed in an edge device. Its structure is as follows: Figure 2 As shown in the perception computing device shown. If the computing device of the cloud computing center is used, after the trained visual perception model is deployed, the edge device collects the image to be identified through the visual sensor such as the camera and uploads it to the cloud computing center, and uses the perception computing device that deploys the visual perception model to perform visual perception calculation on the image to be identified to obtain the visual perception result of the visual perception target. If the perception computing device uses the edge computing device deployed on the edge device, the visual perception model is trained by the model training system and sent to the edge computing device through the communication unit, and deployed in the memory of the edge computing device. After the edge computing device collects the image to be identified through the visual sensor such as the camera, it performs visual perception calculation through the local visual perception model to obtain the visual perception result of the visual perception target.

[0105] Applied to Figure 2 The multi-thread pool shown, in an embodiment of the present invention, the step of calculating the perceptual loss value of the visual perception model for image samples of each resolution may include: inputting the image samples corresponding to each resolution into different thread pools one by one, so as to obtain the perceptual loss value of the visual perception model for the image samples of each resolution by parallel calculation.

[0106] like Figure 3 As shown, in an embodiment of the present invention, when training a visual perception model, all images of each resolution can be stacked to obtain a set of input images of different resolutions, so that the training speed can be improved through batch processing.

[0107] In some optional implementations of the embodiments of the present invention, the parameters of the visual perception model and the parameters of the model parameter prediction network are updated using the perceptual loss values ​​corresponding to each resolution in S103, which may include: calculating the model learning loss value of the current iteration number according to the perceptual loss values ​​corresponding to each resolution, and updating the parameters of the visual perception model and the parameters of the model parameter prediction network according to the model learning loss value.

[0108] Among them, calculating the model learning loss value of the current iteration number according to the perceptual loss value corresponding to each resolution may include: taking the average of the loss values ​​calculated by the loss function corresponding to each resolution as the model learning loss value.

[0109] like Figure 3 As shown, for each resolution, all image samples are input into the visual perception model to obtain the corresponding visual perception results, and the loss value of the visual perception model for each resolution compared with the true value (which can be manually labeled) is calculated through the loss function. Finally, the model learning loss value of the visual perception model can be the average of the perceptual loss values ​​of the visual perception model for image samples of all resolutions, as shown in the following formula:

[0110] ;

[0111] in, represents the loss function of the visual perception model, represents the perceptual loss value corresponding to the i-th resolution, represents the loss function corresponding to the i-th resolution, and k+1 represents the number of resolutions used in visual perception model training.

[0112] It is very challenging to directly use input images of multiple resolutions to train a visual perception model because there are large differences in data distribution between input images of different resolutions, which increases the difficulty of optimizing the perception model. To solve this problem, an embodiment of the present invention provides a model parameter prediction network and an architecture for adjusting a visual perception model using the model parameter prediction network, so as to extract resolution-related information from input image features in a data-driven manner, and adjust the parameters of the visual perception model to make it more suitable for the current input image resolution.

[0113] Figure 4 An architecture diagram of an embodiment of the present invention for adjusting a visual perception model using a model parameter prediction network.

[0114] like Figure 4As shown, an embodiment of the present invention provides a model parameter adjustment module, which mainly includes two parts: a model parameter prediction network and a model parameter adjustment unit, wherein the model parameter prediction network is used to generate a model parameter adjustment coefficient according to the image feature information of the input image, and the model parameter adjustment unit is used to adjust the parameters of the visual perception model using the model parameter adjustment coefficient. The model parameter adjustment module provided by the embodiment of the present invention can act on the training stage of the visual perception model to reduce the problem of difficulty in optimizing the visual perception model caused by the large difference in the distribution of image features of different resolutions, thereby obtaining a visual perception model with resolution generalization capability. The model parameter adjustment module provided by the embodiment of the present invention can also act on the reasoning stage of the visual perception model to further optimize the visual perception effect of the visual perception model in processing input images of different resolutions.

[0115] It should be noted that for a visual perception model, Figure 4 The model parameter adjustment module in can be one or more. The network structure of the visual perception model is mainly an image feature extraction network. If the visual perception model contains multiple image feature extraction networks, and the parameter types of different image feature extraction networks are different, each image feature extraction network can use a set of model parameter adjustment coefficients.

[0116] In some optional implementations of the embodiments of the present invention, in S102, the image feature information corresponding to the resolution is input into the model parameter prediction network, the model parameter adjustment coefficient is output, and the parameters of the visual perception model are adjusted using the model parameter adjustment coefficient, which may include: according to the network layer type of the visual perception model, the visual perception model is divided into blocks; the image feature information corresponding to the resolution is input into the model parameter prediction network corresponding to each network block of the visual perception model, and the corresponding model parameter adjustment coefficient is output; the parameters of each network block of the visual perception model are adjusted using the model parameter adjustment coefficient. That is to say, the model parameter adjustment module can be deployed in units of network blocks including multiple network layers. Among them, the block division method of the visual perception model can be to classify adjacent network layers belonging to the same type of image feature extraction network and the same parameter type into one network block, or such a network block can be divided into multiple network blocks to achieve fine-grained adjustment.

[0117] In some other optional implementations of the embodiments of the present invention, in S102, the image feature information corresponding to the resolution is input into the model parameter prediction network, the model parameter adjustment coefficient is output, and the parameters of the visual perception model are adjusted using the model parameter adjustment coefficient. It may also include: determining the target network layer for parameter adjustment in the visual perception model; inputting the image feature information corresponding to the resolution into the model parameter prediction network corresponding to the target network layer, and outputting the corresponding model parameter adjustment coefficient; and adjusting the parameters of the target network layer using the model parameter adjustment coefficient. That is to say, the model parameter adjustment module can also be deployed in units of network layers, so as to further achieve fine-grained adjustment of the parameters of the visual perception model. Considering the cost of deploying the model parameter adjustment module, some network layers can be selected as target network layers in the visual perception model.

[0118] The visual perception method provided in the embodiment of the present invention can be applied to various visual perception models, for example, the visual perception model can adopt the ResNet-50 network. Of course, the visual perception model can also adopt a model other than the ResNet-50 network, and the structure of the deployed model parameter prediction network is determined according to the type of network layer to be adjusted.

[0119] In the embodiment of the present invention, inputting the image feature information corresponding to the resolution into the model parameter prediction network and outputting the model parameter adjustment coefficient in S102 may include: determining the corresponding model parameter adjustment coefficient according to the parameter type of the visual perception model; inputting the image feature information corresponding to the resolution into the model parameter prediction network, so as to generate the model parameter adjustment coefficient corresponding to each parameter type according to the image feature information corresponding to the resolution. That is, considering the types of some or all parameters in the visual perception model, the corresponding model parameter adjustment coefficient is designed.

[0120] In an embodiment of the present invention, adjusting the parameters of the visual perception model using the model parameter adjustment coefficient in S102 may include: determining a model parameter adjustment function according to the calculation type of the parameters of the visual perception model; substituting the parameters of the visual perception model and the corresponding model parameter type into the corresponding model parameter adjustment function to obtain the adjusted model parameters, so as to update the parameters of the visual perception model to the adjusted model parameters. That is to say, considering the calculation method corresponding to the parameters in the visual perception model, such as the parameters used for adding the input and the parameters used for multiplying the input, the calculation method corresponding to the parameters in the visual perception model needs to be considered when designing the model parameter adjustment function that uses the model parameter adjustment coefficient to adjust the parameters of the visual perception model.

[0121] Therefore, by utilizing the model parameter prediction network provided by the embodiment of the present invention, the parameters of the visual perception model can be adaptively adjusted when the visual perception model processes input images of different resolutions, so as to ensure the perception and recognition accuracy of the visual perception model for different image resolutions, thereby solving the problem of difficulty in model optimization caused by large data distribution differences between input images of different resolutions, and also solving the problem of weak resolution generalization ability of the visual perception model in application scenarios.

[0122] Figure 5 A training flow chart of a visual perception model provided by an embodiment of the present invention.

[0123] like Figure 5 As shown, for image samples corresponding to resolutions 1, 2, ..., m, they are respectively input into the visual perception model to obtain corresponding perceptual loss values ​​1, 2, ..., m. As described in Embodiment S102 of the present invention, in one iterative training of the visual perception model, for a resolution, an image sample of the resolution is first input into the visual perception model, and in the image feature adjustment network of the visual perception model, the image features corresponding to the resolution are adjusted, and then the adjusted image features are input into the subsequent network layer, and the perceptual loss value corresponding to the resolution is calculated using the perceptual result finally output by the visual perception model.

[0124] In S103, in one iterative training of the visual perception model, the image samples of each resolution are subjected to the operation of S102, the perception loss values ​​corresponding to all resolutions are summarized to calculate the model learning loss value of the current iterative training, and the original model parameters of the visual perception model are updated according to the model learning loss value while updating the parameters of the model parameter prediction network.

[0125] like Figure 2 As shown in the figure, in order to accelerate the training of the visual perception model, the self-supervised loss optimization tasks corresponding to m resolutions can be deployed in m thread pools for execution to achieve parallel accelerated training.

[0126] For S104, the iterative training end condition may be that the number of iterative training times of the visual perception model reaches a preset number of iterations, or the model learning loss value of the visual perception model is less than a preset loss value.

[0127] For S105, since the resolution to be processed in the actual application scenario is usually fixed, the parameters of the visual perception model can be updated according to the resolution corresponding to the visual perception task to be processed, and then the updated visual perception model can be used to perform the visual perception task to be processed. When the resolution corresponding to the visual perception task to be processed changes, the parameters of the visual perception model are updated again.

[0128] In order to further improve the perception recognition accuracy, each time a different image is input, the model parameter prediction network can be used to update the parameters of the visual perception model.

[0129] Then, in S105, after using the model parameter prediction network to update the parameters of the visual perception model according to the resolution corresponding to the visual perception task to be processed, the input image of the visual perception task to be processed is input into the updated visual perception model to obtain the visual perception result of the perception target of the visual perception task to be processed, which may include: after using the first model parameter prediction network to update the parameters of the visual perception model according to the image feature size corresponding to the resolution, the input image of the visual perception task to be processed is input into the updated visual perception model to obtain the visual perception result; in the process of using the updated visual perception model to process the input image, before inputting the image features of the resolution into the network layer of the updated visual perception model, using the second model parameter prediction network to adjust the parameters of the network layer according to the image features of the resolution.

[0130] The visual perception method provided by the embodiment of the present invention reduces the difficulty of the visual perception model learning multi-resolution image samples and ensures the multi-resolution training effect by using the trained visual perception model to update the parameters of the visual perception model and the parameters of the model parameter prediction network using the perceptual loss value corresponding to each resolution in the process of training the visual perception model using multi-resolution image samples. After using the trained visual perception model to update the parameters of the visual perception model according to the resolution corresponding to the visual perception task to be processed, the input image of the visual perception task to be processed is input into the updated visual perception model to obtain the visual perception result of the perception target of the visual perception task to be processed, and the model parameters can be adjusted to make it more adaptable to the current input image resolution, thereby ensuring high-precision visual perception results for input images of different resolutions.

[0131] Based on the above embodiment, the embodiment of the present invention further illustrates the steps of collecting the image sample set.

[0132] The image sample set used for training the visual perception model in the embodiment of the present invention includes image samples of multiple resolutions. When collecting the image sample set, a visual sensor such as a camera can be used to directly collect image samples of different resolutions through set resolution parameters to form an image sample set; when collecting the image sample set, image samples of one resolution can also be collected first, and image samples of multiple resolutions can be generated by scaling or enlarging to form an image sample set.

[0133] In some optional implementations of the embodiments of the present invention, the step of collecting the image sample set may include: obtaining an original image set; processing each original image in the original image set using different scaling factors to obtain scaled images of different resolutions corresponding to each original image; and using each original image and the scaled image corresponding to each original image as the image sample set.

[0134] In practical applications, each original image in the original image set is processed with different scaling factors to obtain scaled images of different resolutions corresponding to each original image, which may include: processing each original image with a reference scaling factor to obtain a reference scaled image corresponding to the original image; sampling multiple scaling factors from a preset distribution range, and processing each original image with each scaling factor to obtain scaled images of multiple resolutions corresponding to each original image; using the corresponding reference scaled image and each corresponding scaled image as an image set corresponding to the original image to obtain an image set corresponding to each original image. That is, the generated image sample set is composed of image sets corresponding to different original images, which include image samples of different resolutions, and different image samples of the same resolution may be obtained by processing different original images.

[0135] As described in the above embodiment of the present invention, the resolution used for processing by the visual perception model of the visual perception task is usually expressed in pixel resolution, that is, expressed as the product of the number of pixels in the horizontal direction and the number of pixels in the vertical direction. In the embodiment of the present invention, the scaling factor may include a scaling factor acting on the image height and / or a scaling factor on the image width. In order to obtain image samples of multiple resolutions with greater resolution differences, the scaling factor may include an image height scaling factor and an image width scaling factor.

[0136] To reduce the computational difficulty, the scaling factor can also be a scaling ratio for the image height and image width. For example, for an original image with a resolution of H×W , the base scaling factor can be used The processed resolution is After the image is distributed in Uniform distribution Sample k scaling factors , and process the image using each scaling factor , and finally obtain a collection of images with multiple resolutions ,in The resolution is . Where H represents the vertical size of the image, and W represents the horizontal size of the image.

[0137] When applied to the field of autonomous driving, cameras and other visual sensors are usually set up in different directions of the vehicle, such as left front (fl), front (f), right front (fr), left rear (bl), rear (b), right rear (br), etc., so that images of multiple perspectives of the road scene are collected each time. In the actual training process, in order to use the batch method to accelerate model training, the embodiment of the present invention samples a set of scaling factors at each iteration, and , the above-obtained scaling coefficient set is used for processing, that is, a multi-resolution image set is obtained for each viewing angle .

[0138] Since the resolution difference between the multi-resolution image samples used in training the visual perception model is small, it is not conducive to the training of the model's resolution generalization ability. In order to obtain as many resolutions with large differences as possible and improve the training effect of the model's resolution generalization ability, in other optional implementations of the embodiments of the present invention, the difference between the scaling factors can be set to be no less than the scaling factor difference threshold, that is, when determining the scaling factor, the difference between the scaling factor and other scaling factors in the collected scaling factor set is judged, and only when the difference is greater than the set scaling factor difference threshold is the scaling factor added to the scaling factor set, thereby ensuring the diversity between the scaling factors obtained by sampling and improving the training effect of the model's resolution generalization ability.

[0139] Furthermore, the scaling factors may be designed to all belong to a preset distribution range and the difference between any two scaling factors may not be less than a scaling factor difference threshold.

[0140] In some optional implementations of the embodiments of the present invention, the scaling factor difference threshold is calculated based on the interval length of the preset distribution range divided by twice the number of scaling factors. That is, the scaling factor difference threshold can be expressed by the following formula:

[0141] ;

[0142] Where M represents the scaling factor difference threshold, Indicates the lower limit of the preset distribution range. represents the upper limit of the preset distribution range, and k represents the total number of scaling factors to be collected from the preset distribution range.

[0143] In some other alternative embodiments of the embodiments of the present invention, for each original image in the original image set, different scaling factors are used for processing to obtain scaled images with different resolutions corresponding to each original image, which may also include: processing each original image with a reference scaling factor to obtain a reference scaled image corresponding to the original image; sampling a plurality of scaling factors from a preset distribution range until a preset number of scaling factors are collected; during the process of collecting scaling factors, for each newly collected scaling factor, calculating the difference between the new scaling factor and the existing scaling factors, and if the difference is not less than the scaling factor difference threshold, adding the new scaling factor to the set of existing scaling factors; if the difference is less than the scaling factor difference threshold, discarding the new scaling factor; processing each original image with each scaling factor respectively to obtain scaled images with multiple resolutions corresponding to each original image; using the corresponding reference scaled image and the corresponding scaled images as the image set corresponding to the original image to obtain the image set corresponding to each original image.

[0144] That is, the following algorithm can be designed to obtain the set of scaling factors:

[0145] Input: reference scaling factor , uniformly distributed in , set of scaling factors A, scaling factor difference threshold M;

[0146] Add the reference scaling factor to the set of scaling factors A ;

[0147] If the length of the set of scaling factors A < k + 1, then:

[0148] Randomly sample from the uniform distribution to obtain ;

[0149] Check the difference between and all the scaling factors in the current set of scaling factors A. If the difference is greater than the threshold scaling factor difference M, then add to the set of scaling factors A;

[0150] Output: set of scaling factors A.

[0151] In each embodiment of the present invention, the reference scaled image obtained by the above method can be used as the reference image in the image sample set to obtain the reference image features. Alternatively, an image sample with one of the resolutions can also be used as the reference image.

[0152] In order to further improve the ability of the visual perception model to process input images of different resolutions, image sample sets can be generated multiple times during the model training process. That is, the scaling factor can be re-collected to generate a new image sample set in each iterative training, or the scaling factor can be re-collected to generate a new image sample set after every several iterative trainings.

[0153] If the image sample set is regenerated through the above method in each iterative training, it may make it difficult for the visual perception model to converge.

[0154] To ensure model convergence, in some optional implementations of the embodiments of the present invention, a method of generating an image sample set once before training the visual perception model can be adopted, and the image sample set is no longer updated in subsequent iterative training. In this way, the same batch of resolutions are learned in each iterative training of the visual perception model, and the diversity of samples can be ensured by collecting different image samples corresponding to the resolutions. If computing resources and time permit, a method of generating an image sample set once before training the visual perception model can be adopted, but multiple rounds of iterative training are performed, and the image sample set is regenerated before the start of each round of iterative training, and the same image sample set is used in different iterative training steps in each round of iterative training.

[0155] In some other optional implementations of the embodiments of the present invention, the image sample set may be regenerated every multiple iterations of training, and the frequency of generating the image sample set may be determined according to the convergence of the visual perception model.

[0156] In some other optional implementations of the embodiments of the present invention, if the image sample set is regenerated in each iterative training or every few iterative trainings, the multiple resolutions corresponding to the resampling scaling factors can arrange the resolutions in a fixed order (such as from small to large or from large to small) according to the scaling factors, thereby ensuring the correspondence between the arrangement of the resolutions and the arrangement of the loss function, so that the model training can inherit the experience of the previous iterative training or the previous few iterative trainings.

[0157] In each embodiment of the present invention, the loss function corresponding to the resolution of the visual perception model in each iterative training number inherits the parameters of the loss function of the corresponding position in the previous iterative training, but the corresponding resolution can be the resolution in the previous iterative training or the updated resolution.

[0158] Based on the above embodiments, the embodiments of the present invention further illustrate the model parameter prediction network.

[0159] In the visual perception method provided by the embodiment of the present invention, the model parameter adjustment coefficient can be calculated from two perspectives: the image feature size corresponding to the resolution and the image feature itself of the image sample.

[0160] In some optional implementations of the present invention, the image feature information corresponding to the resolution may use the image features of the image sample of the resolution. The image features of the image sample are image features extracted from the image sample using the visual perception model or a partial image feature extraction layer in the visual perception model.

[0161] In some other optional implementations of the present invention, the image feature information corresponding to the resolution may also use the image feature size corresponding to the resolution. For example, the image vertical size and the image horizontal size described in the above embodiment may be used as the image feature size corresponding to the resolution.

[0162] In some further optional implementations of the embodiments of the present invention, the image feature information corresponding to the resolution may adopt image features of image samples of the resolution and image feature sizes corresponding to the resolution.

[0163] Figure 6 A schematic diagram of the structure of a model parameter prediction network provided by an embodiment of the present invention.

[0164] like Figure 6 As shown, if the image feature information corresponding to the resolution is the image feature of the image sample of the resolution, then in S102, the image feature information corresponding to the resolution is input into the model parameter prediction network, and the model parameter adjustment coefficient is output, which may include: performing global mean pooling processing and global maximum pooling processing on the image features corresponding to the resolution, respectively, to obtain a global mean pooling feature vector corresponding to the resolution and a global maximum pooling feature vector corresponding to the resolution; calculating the mean of the image features corresponding to the resolution and the variance of the image features corresponding to the resolution; performing channel dimension splicing processing on the global mean pooling feature vector corresponding to the resolution, the mean of the image features corresponding to the resolution, and the variance of the image features corresponding to the resolution to obtain a global mean pooling feature splicing result; performing channel dimension splicing processing on the global maximum pooling feature vector corresponding to the resolution, the mean of the image features corresponding to the resolution, and the variance of the image features corresponding to the resolution to obtain a global maximum pooling feature splicing result; and obtaining the model parameter adjustment coefficient according to the global mean pooling feature splicing result and the global maximum pooling feature splicing result.

[0165] Among them, obtaining the model parameter adjustment coefficient according to the global mean pooling feature splicing result and the global maximum pooling feature splicing result can include: using the first multi-layer perceptron and the first activation function in sequence to process the global mean pooling feature splicing result to obtain a mean feature vector; using the second multi-layer perceptron and the second activation function in sequence to process the global maximum pooling feature splicing result to obtain a maximum feature vector; performing feature splicing operation on the mean feature vector and the maximum feature vector to obtain a first feature splicing result; inputting the first feature splicing result into the first model parameter prediction function corresponding to the model parameter adjustment coefficient to obtain the model parameter adjustment coefficient.

[0166] Figure 7 A schematic diagram of the structure of another model parameter prediction network provided by an embodiment of the present invention.

[0167] like Figure 7 As shown, if the image feature information corresponding to the resolution is the image feature size corresponding to the resolution, then in S102, the image feature information corresponding to the resolution is input into the model parameter prediction network, and the model parameter adjustment coefficient is output, which may include: calculating a first normalized ratio of the image feature size corresponding to the resolution to the reference image feature size; calculating a second normalized ratio of the input image feature size of the network layer to be adjusted in the visual perception model to the reference image feature size; concatenating the first normalized ratio and the second normalized ratio to obtain a third normalized ratio; and inputting the third normalized ratio into the second model parameter prediction function corresponding to the model parameter adjustment coefficient to obtain the model parameter adjustment coefficient.

[0168] Among them, inputting the image feature information corresponding to the resolution into the model parameter prediction network and outputting the model parameter adjustment coefficient can include: inputting the image feature corresponding to the resolution into the first model parameter prediction network and outputting the first model parameter adjustment coefficient; inputting the image feature size corresponding to the resolution into the second model parameter prediction network and outputting the second model parameter adjustment coefficient; and calculating the model parameter adjustment coefficient according to the first model parameter adjustment coefficient and the second model parameter adjustment coefficient.

[0169] When the image features of the image samples with the resolution and the image feature size corresponding to the resolution are used as input parameters for generating the model parameter adjustment coefficient, the above-mentioned Figure 6 and Figure 7 The two model parameter prediction networks introduced generate a first model parameter adjustment coefficient and a second model parameter adjustment coefficient respectively, and a combination of the two is used to obtain the model parameter adjustment coefficient for the visual perception model.

[0170] Among them, calculating the model parameter adjustment coefficient based on the first model parameter adjustment coefficient and the second model parameter adjustment coefficient may include: determining the model parameter combination function according to the calculation type of the parameters of the visual perception model; substituting the first model parameter adjustment coefficient and the second model parameter adjustment coefficient into the model parameter combination function to obtain the model parameter adjustment coefficient corresponding to the parameters of the visual perception model.

[0171] In practical applications, the model parameter combination function may be a weighted sum of the first model parameter adjustment coefficient and the second model parameter adjustment coefficient. In some optional implementations of the embodiments of the present invention, the weight coefficient of the first model parameter adjustment coefficient and the weight coefficient of the second model parameter adjustment coefficient in the model parameter combination function may both be learnable parameters.

[0172] Based on the above embodiments, an embodiment of the present invention provides a method for adjusting model parameters of a convolutional network in a visual perception model.

[0173] Taking a convolutional network as an example, an embodiment of the present invention introduces the process of using a model parameter prediction network to generate a model parameter adjustment coefficient and adjust the parameters of a visual perception model as described in the above embodiment.

[0174] Assume that the rth network layer of the visual perception model is a convolutional network, and use represents the input features of the rth layer of the visual perception model, represents the output features of the rth layer of the visual perception model, represents the number of input channels of the rth layer of the visual perception model, represents the height of the input image features of the rth layer of the visual perception model, represents the width of the input image feature of the rth layer of the visual perception model, represents the convolution kernel parameters of the rth layer of the visual perception model, represents the bias parameter of the rth layer of the visual perception model, Represents the number of output channels of the rth layer of the visual perception model.

[0175] refer to Figure 6 The embodiment of the present invention provides a model parameter prediction network based on image features. , which takes the input features of the rth layer as input and outputs the model parameter adjustment coefficient for the convolution kernel parameters of the rth layer , and the model parameter adjustment coefficient for the bias parameter , using these two types of model parameter adjustment coefficients to adjust the parameters of the rth layer of the visual perception model, can be expressed as follows:

[0176] ;

[0177] The model parameter adjustment coefficients obtained above are used to adjust the convolution kernel parameters and bias parameters of the rth layer of the visual perception model. Since the convolution kernel parameters are used for multiplication operations in the convolutional network and the bias parameters are used for addition operations, the model parameter adjustment function can be shown as follows:

[0178] ;

[0179] ;

[0180] in, represents the convolution kernel parameters after adjustment of the rth layer of the visual perception model, Represents the adjusted bias parameter of the rth layer of the visual perception model.

[0181] The input features are processed using the adjusted parameters to obtain the output features, as shown in the following formula:

[0182] .

[0183] References Figure 6 The structure of the model parameter prediction network shown calculates the global mean pooling feature vector, global maximum pooling feature vector, mean, and variance of the image features of the image samples corresponding to the resolution.

[0184] Among them, the input features of the rth layer of the input visual perception model are calculated The mean and variance , which can be expressed as follows:

[0185] ;

[0186] ;

[0187] in, Represents input features In spatial position The value of the i-th channel at .

[0188] For the input features Perform global mean pooling and global maximum pooling Processing, and the above calculated mean and variance The features after global mean pooling and global maximum pooling are concatenated, and then two multi-layer perceptrons are used respectively. and After processing the concatenated features, the sigmoid activation function is used to obtain the mean feature vector and the maximum eigenvector , as shown below:

[0189] ;

[0190] ;

[0191] in, The symbol represents the feature concatenation operation.

[0192] For the convolution kernel parameters and bias parameters of the convolutional network, the first model parameter prediction function can be designed as shown below:

[0193] ;

[0194] ;

[0195] in, is a smaller constant, such as 0.05 or 0.1. In the embodiment of the present invention, the first multilayer perceptron and the second multilayer perceptron The input layer can be designed to include neurons, and the hidden layer contains neurons, and the output layer contains The structure of a neuron.

[0196] refer to Figure 7 The embodiment of the present invention provides another model parameter prediction network based on the image feature size of the resolution , the parameters of the visual perception model are adjusted by the image feature size corresponding to the resolution, so that it can better adapt to the current input image resolution.

[0197] The rth network layer for the visual perception model is a convolutional network, also using represents the input features of the rth layer of the visual perception model, represents the output features of the rth layer of the visual perception model, represents the number of input channels of the rth layer of the visual perception model, Represents the input image features of the rth layer of the visual perception model, represents the width of the input image feature of the rth layer of the visual perception model, represents the convolution kernel parameters of the rth layer of the visual perception model, represents the bias parameter of the rth layer of the visual perception model, Represents the number of output channels of the rth layer of the visual perception model.

[0198] The model parameter prediction network based on the image feature size of the resolution Based on the feature size of the reference image , the image feature size corresponding to the current resolution , the input image feature size of the rth layer As input, output the model parameter adjustment coefficient of the convolution kernel parameter of the rth layer , and the adjustment coefficient for the bias parameter , which can be expressed as follows:

[0199] .

[0200] The model parameter adjustment coefficients obtained above are used to adjust the convolution kernel parameters and bias parameters of the rth layer of the visual perception model. Since the convolution kernel parameters are used for multiplication operations in the convolutional network and the bias parameters are used for addition operations, the model parameter adjustment function can be shown as follows:

[0201] ;

[0202] ;

[0203] in, represents the convolution kernel parameters after adjustment of the rth layer of the visual perception model, Represents the adjusted bias parameter of the rth layer of the visual perception model.

[0204] Use the adjusted parameters to process the input features and obtain the output features, as shown in the following formula:

[0205] .

[0206] References Figure 7 The model parameter prediction network structure shown in the figure calculates the image feature size corresponding to the current resolution. And the input image feature size of the rth layer Relative reference image feature size The first normalized ratio and the second normalized ratio can be expressed as follows:

[0207] ;

[0208] ;

[0209] ;

[0210] ;

[0211] in, Indicates the normalized ratio of the vertical size of the image corresponding to the current resolution. Indicates the normalized ratio of the horizontal size of the image corresponding to the current resolution. Represents the normalized ratio of the vertical size of the input image feature of the rth layer, Represents the normalized ratio of the horizontal size of the input image feature of the rth layer, Represents a logarithmic function.

[0212] The above normalized ratios are spliced ​​together to obtain the spliced ​​normalized ratio, which is recorded as the third normalized ratio. :

[0213] ;

[0214] in, The symbol represents the concatenation operation.

[0215] The second model parameter prediction function provided by the embodiment of the present invention is used to process , for the convolution kernel parameters and bias parameters of the convolutional network, it is possible to use a multi-layer perceptron and Separate treatment After that, the sigmoid activation function is used to process and generate parameters, as shown in the following formula:

[0216] ;

[0217] ;

[0218] in, is a smaller constant, such as 0.05 or 0.1. In the embodiment of the present invention, the third multilayer perceptron And the fourth multi-layer perceptron The input layer can be designed to include neurons, and the hidden layer contains neurons, and the output layer contains The structure of a neuron.

[0219] If the model parameter adjustment coefficients are generated based on the image features and the image feature size corresponding to the resolution for the convolution network, the model parameter adjustment coefficients corresponding to the convolution kernel parameters can be combined using the following formula:

[0220] ;

[0221] in, It can be a learnable parameter and its initialization value can be 0.5.

[0222] The model parameter adjustment coefficients corresponding to the bias parameters can be combined using the following formula:

[0223] .

[0224] On the basis of the above embodiments, in order to further optimize the model parameter adjustment effect of the visual perception model, in the visual perception method provided in the embodiment of the present invention, the perceptual loss value corresponding to each resolution is used in S103 to update the parameters of the visual perception model and the parameters of the model parameter prediction network to obtain the visual perception model corresponding to the current iterative training batch, which may include: calculating the regularization constraint items corresponding to each resolution according to the model parameter adjustment coefficients corresponding to each resolution; updating the parameters of the visual perception model and the parameters of the model parameter prediction network according to the perceptual loss value corresponding to each resolution and the regularization constraint items corresponding to each resolution to obtain the visual perception model corresponding to the current iterative training batch.

[0225] In an embodiment of the present invention, on the basis of the loss calculated according to the perceptual loss value corresponding to each resolution, a regularization constraint term corresponding to each resolution is calculated according to the model parameter adjustment coefficient corresponding to each resolution to ensure the optimization effect.

[0226] The design principle of the regularization constraint term is that the model parameter adjustment coefficient corresponding to the resolution makes the parameter change of the visual perception model smaller, and the regularization constraint term corresponding to the resolution is closer to 0.

[0227] Taking the convolutional network introduced in the above embodiment as an example, the model parameter adjustment coefficient of the convolution kernel parameter based on image features of the rth layer of the visual perception model is , the corresponding regularization constraint It can be calculated using the following formula:

[0228] .

[0229] Model parameter adjustment coefficient of the convolution kernel parameter based on the image feature size of the rth layer of the visual perception model , the corresponding regularization constraint It can be calculated using the following formula:

[0230] .

[0231] Model parameter adjustment coefficient for the image feature-based bias parameter of the rth layer of the visual perception model , the corresponding regularization constraint It can be calculated using the following formula:

[0232] .

[0233] Model parameter adjustment coefficient for the bias parameter of the rth layer of the visual perception model based on the image feature size , the corresponding regularization constraint It can be calculated using the following formula:

[0234] .

[0235] The regularization term of the model parameter adjustment coefficient of the target network layer in the visual perception model can be averaged to obtain the regularization constraint term corresponding to each resolution in the visual perception model. For example, for the above convolutional network, the following formula can be used to calculate the regularization constraint term corresponding to the i-th resolution in the visual perception model: :

[0236] ;

[0237] Where L represents the total number of target network layers in the visual perception model.

[0238] The target network layer may be all network layers in the visual perception model or part of the network layers.

[0239] Based on the visual perception model parameter adjustment scheme based on resolution-based image feature information introduced in the above embodiment, the embodiment of the present invention continues to explain the training steps and reasoning steps of the visual perception model.

[0240] As described in the above embodiments of the present invention, the training task of the visual perception model can be executed in a cloud computing center.

[0241] If applied to the field of autonomous driving, the visual perception model can be trained on the collected and annotated road scene image data, and the multi-resolution meta-learning optimization method provided by the embodiment of the present invention is used. In each training batch, the image samples corresponding to m resolutions are respectively input into the visual perception model in the current iterative training, and the multi-resolution image correction network is used in the visual perception model to correct the image features. Finally, the perception result is output through the visual perception model and the perception loss value is calculated according to the true value. All the perception loss values ​​in the training batch are collected and substituted into the loss function of the visual perception model to obtain the model learning loss value, and then the model is optimized by the back propagation algorithm. Among them, the optimizer can use a gradient descent (GD) optimizer, an adaptive moment estimation (Adam) optimizer, an adaptive weight decay regularization (Adam Weight Decay Regularization, AdamW) optimizer, etc. The model parameter adjustment method of the visual perception model proposed by the present invention is used during training to improve the training efficiency. After the model training is completed, the visual perception model obtained by training is deployed to the edge computing platform of the unmanned vehicle, and the parameters of the visual perception model are updated according to the resolution required to be processed by the edge computing platform and the trained model parameter prediction network.

[0242] The visual perception method provided by the embodiment of the present invention can be applied to, but not limited to, the field of autonomous driving, and can also be applied to intelligent terminals such as intelligent robots, smart homes, and mobile phones, and can also be applied to visual perception models involved in industrial production fields such as transportation, aerial remote sensing, and industrial product assembly. For example, the visual perception model can be an autonomous driving perception model; for this, the image sample set can be constructed based on image samples collected by vehicle-mounted sensors from different perspectives, and the visual perception model can be trained on the collected and annotated road scene image data set when executing model training in the cloud data center.

[0243] In an embodiment of the present invention, updating the parameters of the visual perception model using the perceptual loss values ​​corresponding to each resolution may include: calculating the model learning loss value of the visual perception model using the sum of the perceptual loss values ​​corresponding to each resolution; and updating the parameters of the visual perception model using the model learning loss value. For example, a loss function of a visual perception model provided in the above embodiment of the present invention may be used. If the optimization effect of image feature adjustment is enhanced by adding a regularization constraint term in combination with the above-mentioned embodiment of the present invention, the loss function of the visual perception model is It can be expressed as:

[0244] ;

[0245] in, Represents the model learning loss value, represents the regularization constraint corresponding to the i-th resolution, It is a hyperparameter or learnable parameter used to balance the model learning loss value and the regularization constraint term calculated based on the model parameter adjustment coefficient, and can be initialized to 0.5.

[0246] It should be noted that in the embodiments of the visual perception method of the present invention, some of the steps or features may be ignored or not executed. The hardware or software functional modules divided for the convenience of description are not the only implementation form of the visual perception method provided by the embodiments of the present invention.

[0247] The above describes in detail various embodiments corresponding to the visual perception method. On this basis, the present invention also discloses a visual perception device, equipment, non-volatile storage medium and computer program product corresponding to the above method.

[0248] Figure 8 A schematic diagram of the structure of a visual perception device provided in an embodiment of the present invention.

[0249] like Figure 8 As shown, the visual perception device provided by the embodiment of the present invention includes:

[0250] The model training unit 801 is used to train the visual perception model using an image sample set including image samples of multiple resolutions; in the iterative training of the visual perception model, for each resolution, the image feature information corresponding to the resolution is input into the model parameter prediction network, the model parameter adjustment coefficient is output, and after the parameters of the visual perception model are adjusted using the model parameter adjustment coefficient, the image samples of the resolution are input into the adjusted visual perception model, and the perceptual loss value corresponding to the resolution is output; the parameters of the visual perception model and the parameters of the model parameter prediction network are updated using the perceptual loss values ​​corresponding to each resolution to obtain the visual perception model corresponding to the current iterative training batch; after the iterative training end condition is met, the trained visual perception model is obtained;

[0251] The visual perception unit 802 is used to use the model parameter prediction network to update the parameters of the visual perception model according to the resolution corresponding to the visual perception task to be processed, and then input the input image of the visual perception task to be processed into the updated visual perception model to obtain the visual perception result of the perception target of the visual perception task to be processed.

[0252] It should be noted that in each implementation of the visual perception device provided by the embodiment of the present invention, the division of units is only a logical functional division, and other division methods may be used. The connection method between different units may be electrical, mechanical or other connection methods. Separated units may be located in the same physical location or distributed on multiple network nodes. Each unit may be implemented in the form of hardware or in the form of a software functional unit. That is, part or all of the units provided in the embodiment of the present invention may be selected according to actual needs and a corresponding connection method or integration method may be adopted to achieve the purpose of the embodiment of the present invention.

[0253] Since the embodiments of the apparatus part correspond to the embodiments of the method part, please refer to the description of the embodiments of the method part for the embodiments of the apparatus part, which will not be repeated here.

[0254] Fig. 9 A schematic diagram of the structure of a visual perception device provided in an embodiment of the present invention.

[0255] like Fig. 9 As shown, the visual perception device provided by an embodiment of the present invention includes: a memory 910, used to store a computer program 911; a processor 920, used to execute the computer program 911, and when the computer program 911 is executed by the processor 920, the steps of the visual perception method provided in any of the above embodiments are implemented.

[0256] Among them, the processor 920 may include one or more processing cores, such as a 3-core processor, an 8-core processor, etc. The processor 920 may be implemented in at least one hardware form of digital signal processing (DSP), field-programmable gate array (FPGA), and programmable logic array (PLA). The processor 920 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a central processing unit (CPU); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 920 may be integrated with a graphics processing unit (GPU), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 920 may also include an artificial intelligence (AI) processor, which is used to process computing operations related to machine learning.

[0257] The memory 910 may include one or more non-volatile storage media, which may be non-transitory. The memory 910 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In this embodiment, the memory 910 is at least used to store the following computer program 911, wherein the computer program 911, after being loaded and executed by the processor 920, can implement the relevant steps in the visual perception method disclosed in any of the aforementioned embodiments. In addition, the resources stored in the memory 910 may also include an operating system 912 and data 913, etc., and the storage method may be temporary storage or permanent storage. Among them, the operating system 912 may be Windows or other types of operating systems. Data 913 may include, but is not limited to, the data involved in the above method.

[0258] In some embodiments, the visual perception device may also include a display screen 930 , a power supply 940 , a communication interface 950 , an input / output interface 960 , a sensor 970 , and a communication bus 980 .

[0259] Those skilled in the art will understand that Fig. 9 The structure shown in the figure does not constitute a limitation of the visual perception device and may include more or fewer components than shown in the figure.

[0260] The visual perception device provided in an embodiment of the present invention includes a memory and a processor. When the processor executes the program stored in the memory, it can implement the steps of the visual perception method provided in the above embodiment, and the effect is the same as above.

[0261] An embodiment of the present invention provides a non-volatile storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the visual perception method provided in any one of the above embodiments can be implemented.

[0262] The non-volatile storage medium may include: a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and other media that can store program codes.

[0263] For an introduction to the non-volatile storage medium provided in the embodiment of the present invention, please refer to the above method embodiment, and the effect thereof is the same as the visual perception method provided in the embodiment of the present invention, and the present invention will not elaborate on it here.

[0264] An embodiment of the present invention provides a computer program product, including a computer program, which, when executed by a processor, implements the steps of the visual perception method provided in any one of the above embodiments.

[0265] For an introduction to the computer program product provided by the embodiment of the present invention, please refer to the above-mentioned method embodiment, and the effect thereof is the same as the visual perception method provided by the embodiment of the present invention, and the present invention will not elaborate on it here.

[0266] The above is a detailed introduction to a visual perception method, device, equipment, medium and computer program product provided by the present invention. The various embodiments in the specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same and similar parts between the embodiments can be referenced to each other. For the devices, equipment, non-volatile storage media and computer program products disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple, and the relevant parts can be referred to the method part description. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of the present invention, the present invention can also be improved and modified in a number of ways, and these improvements and modifications also fall within the scope of protection of the present invention.

[0267] It should also be noted that, in this specification, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the statement "comprises a ..." does not exclude the presence of other identical elements in the process, method, article or device including the element.

Claims

1. A visual perception method, characterized in that: include: Training a visual perception model using an image sample set including image samples at multiple resolutions; In the iterative training of the visual perception model, for each resolution, the image feature information corresponding to the resolution is input into the model parameter prediction network, and the model parameter adjustment coefficient is output. After the parameters of the visual perception model are adjusted using the model parameter adjustment coefficient, the image sample of the resolution is input into the adjusted visual perception model, and the perception loss value corresponding to the resolution is output; Using the perceptual loss values ​​corresponding to each of the resolutions, updating the parameters of the visual perception model and the parameters of the model parameter prediction network, to obtain the visual perception model corresponding to the current iterative training batch; After reaching the iterative training end condition, obtaining the trained visual perception model; After updating the parameters of the visual perception model according to the resolution corresponding to the visual perception task to be processed by using the model parameter prediction network, inputting the input image of the visual perception task to be processed into the updated visual perception model to obtain a visual perception result of the perception target of the visual perception task to be processed; The image feature information corresponding to the resolution is input into the model parameter prediction network, and the model parameter adjustment coefficient is output, including: Determining the corresponding model parameter adjustment coefficient according to the parameter type of the visual perception model; Inputting the image feature information corresponding to the resolution into the model parameter prediction network to generate the model parameter adjustment coefficients corresponding to each parameter type according to the image feature information corresponding to the resolution; Adjusting the parameters of the visual perception model using the model parameter adjustment coefficient includes: Determining a model parameter adjustment function according to a calculation type of the parameters of the visual perception model; Substitute the parameters of the visual perception model and the corresponding model parameter type into the corresponding model parameter adjustment function to obtain adjusted model parameters, so as to update the parameters of the visual perception model to the adjusted model parameters.

2. The visual perception method according to claim 1, characterized in that: The image feature information corresponding to the resolution includes image features of image samples of the resolution.

3. The visual perception method according to claim 1, characterized in that: The image feature information corresponding to the resolution includes the image feature size corresponding to the resolution.

4. The visual perception method according to claim 1, characterized in that: The image feature information corresponding to the resolution includes image features of image samples of the resolution and image feature sizes corresponding to the resolution.

5. The visual perception method according to claim 2 or 4, characterized in that: If the image feature information corresponding to the resolution is the image feature of the image sample of the resolution, the image feature information corresponding to the resolution is input into the model parameter prediction network, and the model parameter adjustment coefficient is output, including: Performing global mean pooling processing and global maximum pooling processing on the image features corresponding to the resolution respectively, to obtain a global mean pooling feature vector corresponding to the resolution and a global maximum pooling feature vector corresponding to the resolution; Calculating a mean of the image features corresponding to the resolution and a variance of the image features corresponding to the resolution; Perform channel dimension splicing processing on the global mean pooling feature vector corresponding to the resolution, the mean of the image feature corresponding to the resolution, and the variance of the image feature corresponding to the resolution to obtain a global mean pooling feature splicing result; Perform channel dimension splicing processing on the global maximum pooling feature vector corresponding to the resolution, the mean of the image feature corresponding to the resolution, and the variance of the image feature corresponding to the resolution to obtain a global maximum pooling feature splicing result; The model parameter adjustment coefficient is obtained according to the global mean pooling feature splicing result and the global maximum pooling feature splicing result.

6. The visual perception method according to claim 5, characterized in that: The model parameter adjustment coefficient is obtained according to the global mean pooling feature splicing result and the global maximum pooling feature splicing result, including: Using a first multi-layer perceptron and a first activation function in sequence to process the global mean pooling feature concatenation result to obtain a mean feature vector; Using a second multi-layer perceptron and a second activation function in sequence to process the global maximum pooling feature concatenation result to obtain a maximum feature vector; Performing a feature splicing operation on the mean feature vector and the maximum feature vector to obtain a first feature splicing result; The first feature splicing result is input into a first model parameter prediction function corresponding to the model parameter adjustment coefficient to obtain the model parameter adjustment coefficient.

7. The visual perception method according to claim 3 or 4, characterized in that: If the image feature information corresponding to the resolution is the image feature size corresponding to the resolution, the image feature information corresponding to the resolution is input into a model parameter prediction network, and a model parameter adjustment coefficient is output, including: Calculating a first normalized ratio of the image feature size corresponding to the resolution to the reference image feature size; Calculating a second normalized ratio of the input image feature size of the network layer to be adjusted in the visual perception model to the reference image feature size; Concatenating the first normalized ratio and the second normalized ratio to obtain a third normalized ratio; The third normalized ratio is input into a second model parameter prediction function corresponding to the model parameter adjustment coefficient to obtain the model parameter adjustment coefficient.

8. The visual perception method according to claim 4, characterized in that: Inputting the image feature information corresponding to the resolution into the model parameter prediction network and outputting the model parameter adjustment coefficient includes: Inputting the image features corresponding to the resolution into a first model parameter prediction network, and outputting a first model parameter adjustment coefficient; Inputting the image feature size corresponding to the resolution into a second model parameter prediction network, and outputting a second model parameter adjustment coefficient; The model parameter adjustment coefficient is calculated according to the first model parameter adjustment coefficient and the second model parameter adjustment coefficient.

9. The visual perception method according to claim 8, characterized in that: The model parameter adjustment coefficient is calculated according to the first model parameter adjustment coefficient and the second model parameter adjustment coefficient, including: Determining a model parameter combination function according to a calculation type of the parameters of the visual perception model; Substitute the first model parameter adjustment coefficient and the second model parameter adjustment coefficient into the model parameter combination function to obtain the model parameter adjustment coefficient corresponding to the parameters of the visual perception model.

10. The visual perception method according to claim 9, characterized in that: The weight coefficient of the first model parameter adjustment coefficient and the weight coefficient of the second model parameter adjustment coefficient in the model parameter combination function are both learnable parameters.

11. The visual perception method according to claim 1, characterized in that: Inputting the image feature information corresponding to the resolution into a model parameter prediction network, outputting a model parameter adjustment coefficient, and using the model parameter adjustment coefficient to adjust the parameters of the visual perception model, including: According to the network layer type of the visual perception model, the visual perception model is processed in blocks; Inputting the image feature information corresponding to the resolution into the model parameter prediction network corresponding to each network block of the visual perception model, and outputting the corresponding model parameter adjustment coefficient; The model parameter adjustment coefficient is used to adjust the parameters of each network block of the visual perception model accordingly.

12. The visual perception method according to claim 1, characterized in that: Inputting the image feature information corresponding to the resolution into a model parameter prediction network, outputting a model parameter adjustment coefficient, and using the model parameter adjustment coefficient to adjust the parameters of the visual perception model, including: Determining a target network layer for parameter adjustment in the visual perception model; Inputting the image feature information corresponding to the resolution into the model parameter prediction network corresponding to the target network layer, and outputting the corresponding model parameter adjustment coefficient; The model parameter adjustment coefficient is used to adjust the parameters of the target network layer accordingly.

13. The visual perception method according to claim 1, characterized in that: After updating the parameters of the visual perception model according to the resolution corresponding to the visual perception task to be processed by using the model parameter prediction network, inputting the input image of the visual perception task to be processed into the updated visual perception model to obtain a visual perception result of the perception target of the visual perception task to be processed, including: After updating the parameters of the visual perception model according to the image feature size corresponding to the resolution using the first model parameter prediction network, the input image of the visual perception task to be processed is input into the updated visual perception model to obtain the visual perception result; In the process of processing the input image using the updated visual perception model, before the image features of the resolution are input into the network layer of the updated visual perception model, the parameters of the network layer are adjusted according to the image features of the resolution using a second model parameter prediction network.

14. The visual perception method according to claim 1, characterized in that: Using the perceptual loss values ​​corresponding to each of the resolutions to update the parameters of the visual perception model and the parameters of the model parameter prediction network to obtain the visual perception model corresponding to the current iterative training batch, including: The regularization constraint items corresponding to the resolutions are calculated according to the model parameter adjustment coefficients corresponding to the resolutions; According to the perceptual loss value corresponding to each of the resolutions and the regularization constraint item corresponding to each of the resolutions, the parameters of the visual perception model and the parameters of the model parameter prediction network are updated to obtain the visual perception model corresponding to the current iterative training batch.

15. A visual perception device, characterized in that: include: A model training unit is used to train a visual perception model using an image sample set including image samples of multiple resolutions; in the iterative training of the visual perception model, for each resolution, input the image feature information corresponding to the resolution into a model parameter prediction network, output a model parameter adjustment coefficient, adjust the parameters of the visual perception model using the model parameter adjustment coefficient, input the image sample of the resolution into the adjusted visual perception model, and output a perceptual loss value corresponding to the resolution; update the parameters of the visual perception model and the parameters of the model parameter prediction network using the perceptual loss value corresponding to each resolution, and obtain the visual perception model corresponding to the current iterative training batch; After reaching the iterative training end condition, obtaining the trained visual perception model; A visual perception unit, configured to update the parameters of the visual perception model according to the resolution corresponding to the visual perception task to be processed by using the model parameter prediction network, and then input the input image of the visual perception task to be processed into the updated visual perception model to obtain a visual perception result of the perception target of the visual perception task to be processed; The image feature information corresponding to the resolution is input into the model parameter prediction network, and the model parameter adjustment coefficient is output, including: Determining the corresponding model parameter adjustment coefficient according to the parameter type of the visual perception model; Inputting the image feature information corresponding to the resolution into the model parameter prediction network to generate the model parameter adjustment coefficients corresponding to each parameter type according to the image feature information corresponding to the resolution; Adjusting the parameters of the visual perception model using the model parameter adjustment coefficient includes: Determining a model parameter adjustment function according to a calculation type of the parameters of the visual perception model; Substitute the parameters of the visual perception model and the corresponding model parameter type into the corresponding model parameter adjustment function to obtain adjusted model parameters, so as to update the parameters of the visual perception model to the adjusted model parameters.

16. A visual perception device, characterized in that: include: Memory for storing computer programs; A processor is used to execute the computer program, and when the computer program is executed by the processor, the steps of the visual perception method as claimed in any one of claims 1 to 14 are implemented.

17. A non-volatile storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the visual perception method according to any one of claims 1 to 14 are implemented.

18. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the visual perception method according to any one of claims 1 to 14 are implemented.

Images