Image recognition method, device, equipment, medium and computer program product

Abstract

The present invention relates to the field of image processing technology, and specifically discloses an image recognition method, device, equipment, medium and computer program product. A visual perception model is trained by adopting multi-resolution image samples, and a multi-resolution gradient regularization term is calculated by similarity coefficients between resolutions in iterative training to adjust the relationship between gradient data of image features with different resolutions, so that the visual perception model understands the difference between multiple resolutions with large differences in gradient descent directions and understands the similarity between multiple resolutions with small differences in gradient descent directions. The model parameters of the visual perception model are updated by using a model learning loss value calculated according to a visual perception loss value and a multi-resolution gradient regularization term, so as to solve the problem that when the visual perception model learns the perception ability of images with multiple resolutions, different image features brought by different resolutions and interference between gradient data lead to difficulty in model optimization, thereby improving the resolution generalization ability of computer vision.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to an image recognition method, device, equipment, medium and computer program product. Background Art

[0002] Computer vision is an important branch of artificial intelligence technology. One of its main tasks is to obtain corresponding scene information by processing the collected images. It is widely used in autonomous driving, intelligent robots, smart homes, smart terminals, as well as industrial production such as transportation, aerial remote sensing, and industrial product assembly.

[0003] The visual perception model is one of the core technologies of computer vision. The perception ability of the visual perception model determines the visual understanding ability of artificial intelligence. Due to the differences in the needs of different application scenarios, different image samples need to be used when training the visual perception model. For example, for application scenarios with limited computing resources, lower-resolution image samples are usually used to train the visual perception model. For the perception scenarios of small targets and long-distance targets, higher-resolution image samples are used to train the visual perception model. In order to improve the resolution generalization ability of the visual perception model to process input images of different resolutions, it is necessary to use image samples of different resolutions to train the visual perception model. However, due to the large differences in the feature distribution of images of different resolutions, model optimization is difficult to succeed, resulting in the fact that only visual perception models that process specific resolutions can be deployed in actual applications to handle image recognition tasks.

[0004] How to improve the resolution generalization capability of computer vision to adapt to input images of different resolutions is a technical problem that technicians in this field need to solve. Summary of the invention

[0005] The object of the present invention is to provide an image recognition method, apparatus, device, medium and computer program product for improving the resolution generalization capability of computer vision to adapt to input images of different resolutions.

[0006] In order to solve the above technical problems, the present invention provides an image recognition method, comprising:

[0007] Training a visual perception model using an image sample set including image samples at multiple resolutions;

[0008] During the iterative training process, a visual perception loss value and a multi-resolution gradient regularization term of the visual perception model are calculated based on the perception loss value of the visual perception model for the image samples of each resolution, and a model learning loss value calculated based on the visual perception loss value and the multi-resolution gradient regularization term is used to update the model parameters of the visual perception model until the iterative training end condition is reached, thereby obtaining the trained visual perception model;

[0009] Inputting the input image of the visual perception task to be processed into the trained visual perception model to obtain a visual perception result of the perception target of the visual perception task to be processed;

[0010] Among them, the multi-resolution gradient regularization term is calculated by the gradient regularization term corresponding to each of the resolutions, and the gradient regularization term is the product of a function of the resolution similarity coefficient corresponding to the two resolutions and a function of the gradient angle between the gradients corresponding to the two resolutions, so that if the similarity between the two resolutions is higher, the gradient regularization term is used to enhance the similarity constraint between the two resolutions; if the similarity between the two resolutions is lower, the gradient regularization term is used to enhance the gradient orthogonality between the two resolutions.

[0011] On the one hand, the resolution similarity coefficient is calculated based on the image heights corresponding to the two resolutions and the image widths corresponding to the two resolutions.

[0012] On the other hand, the resolution similarity coefficient is calculated based on the ratio of image heights corresponding to the two resolutions and the ratio of image widths corresponding to the two resolutions.

[0013] On the other hand, the resolution similarity coefficient is a difference between a first preset constant and a first fractional term;

[0014] Among them, the numerator of the first fractional term is the maximum value of the difference between the image heights corresponding to the two resolutions and the difference between the image widths corresponding to the two resolutions, and the denominator of the first fractional term is the maximum value of the image heights corresponding to the two resolutions and the image widths corresponding to the two resolutions.

[0015] On the other hand, if the similarity between the two resolutions is higher, the gradient regularization term is used to enhance the similarity constraint between the two resolutions, including:

[0016] If the resolution similarity coefficient is greater than the resolution similarity threshold, the function of the resolution similarity coefficient is positively correlated with the resolution similarity coefficient, and the function of the gradient angle is positively correlated with the gradient angle;

[0017] If the similarity between the two resolutions is lower, the gradient regularization term is used to enhance the gradient orthogonality between the two resolutions, including:

[0018] If the resolution similarity coefficient is less than or equal to the resolution similarity threshold, and the gradient angles corresponding to the two resolutions are less than the gradient angle threshold, then the function of the resolution similarity coefficient is negatively correlated with the resolution similarity coefficient, and the function of the gradient angle is negatively correlated with the gradient angle;

[0019] If the resolution similarity coefficient is less than or equal to the resolution similarity threshold, and the gradient angles corresponding to the two resolutions are greater than or equal to the gradient angle threshold, then the function of the resolution similarity coefficient is negatively correlated with the resolution similarity coefficient, and the function of the gradient angle is positively correlated with the gradient angle.

[0020] On the other hand, if the resolution similarity coefficient is greater than the resolution similarity threshold, then the function of the resolution similarity coefficient is a directly proportional function of the resolution similarity coefficient;

[0021] If the resolution similarity coefficient is less than or equal to the resolution similarity threshold, the function of the resolution similarity coefficient is a direct proportional function of a second preset constant minus the resolution similarity coefficient.

[0022] On the other hand, if the resolution similarity coefficient is greater than the resolution similarity threshold, the function of the gradient angle is 1 minus the difference of the cosine function of the gradient angle;

[0023] If the resolution similarity coefficient is less than or equal to the resolution similarity threshold, and the gradient angles corresponding to the two resolutions are less than the gradient angle threshold, the function of the gradient angle is 1 plus half of the sum of the cosine function of the gradient angle;

[0024] If the resolution similarity coefficient is less than or equal to the resolution similarity threshold, and the gradient angles corresponding to the two resolutions are greater than or equal to the gradient angle threshold, then the function of the gradient angle is the negative value of the cosine function of the gradient angle.

[0025] On the other hand, the multi-resolution gradient regularization term is the sum of the gradient regularization terms corresponding to all pairwise combinations of the resolutions corresponding to the image sample set.

[0026] On the other hand, the step of collecting the image sample set includes:

[0027] Get the original image set;

[0028] Processing each original image in the original image set with a different scaling factor to obtain a scaled image with a different resolution corresponding to each original image;

[0029] Each of the original images and the zoomed images corresponding to the original images are used as the image sample set.

[0030] On the other hand, each original image in the original image set is processed using a different scaling factor to obtain a scaled image with a different resolution corresponding to each original image, including:

[0031] Processing each of the original images using a reference scaling factor to obtain a reference scaling image corresponding to the original image;

[0032] Sampling a plurality of the scaling factors from a preset distribution range, and processing each of the original images using each of the scaling factors, respectively, to obtain the scaled images of the plurality of resolutions corresponding to each of the original images;

[0033] The corresponding reference zoomed image and the corresponding zoomed images are used as an image set corresponding to the original image to obtain the image set corresponding to each original image.

[0034] On the other hand, the difference between the two scaling factors is not less than a scaling factor difference threshold.

[0035] On the other hand, the scaling factors all belong to a preset distribution range and the differences between any two of the scaling factors are not less than a scaling factor difference threshold.

[0036] On the other hand, the scaling factor difference threshold is calculated based on the interval length of the preset distribution range divided by twice the number of scaling factors.

[0037] On the other hand, each original image in the original image set is processed using a different scaling factor to obtain a scaled image with a different resolution corresponding to each original image, including:

[0038] Processing each of the original images using a reference scaling factor to obtain a reference scaling image corresponding to the original image;

[0039] Sampling a plurality of the scaling factors from a preset distribution range until a preset number of the scaling factors are collected;

[0040] In the process of collecting the zoom coefficients, each time a new zoom coefficient is collected, the difference between the new zoom coefficient and the existing zoom coefficient is calculated, and if the difference is not less than the zoom coefficient difference threshold, the new zoom coefficient is added to the set of the existing zoom coefficients; if the difference is less than the zoom coefficient difference threshold, the new zoom coefficient is discarded;

[0041] Processing each of the original images using each of the scaling factors respectively to obtain the scaled images of the plurality of resolutions corresponding to each of the original images;

[0042] The corresponding reference zoomed image and the corresponding zoomed images are used as an image set corresponding to the original image to obtain the image set corresponding to each original image.

[0043] On the other hand, the step of calculating the perceptual loss value of the image sample of each resolution by the visual perception model comprises:

[0044] The image samples corresponding to the resolutions are input into different thread pools one by one, so as to obtain the perception loss value of the visual perception model for the image samples of the resolutions in parallel.

[0045] On the other hand, the step of calculating the visual perception loss value includes:

[0046] Update the loss function corresponding to each of the resolutions according to the perceptual loss value of the image samples of each of the resolutions by the visual perception model;

[0047] The visual perception loss value of the current iteration number is calculated according to the loss function corresponding to each of the resolutions.

[0048] In order to solve the above technical problems, the present invention further provides an image recognition device, comprising:

[0049] A model training unit trains a visual perception model using an image sample set including image samples of multiple resolutions; in an iterative training process, a visual perception loss value and a multi-resolution gradient regularization term of the visual perception model are calculated based on the perception loss value of the visual perception model for the image samples of each resolution, and a model learning loss value calculated based on the visual perception loss value and the multi-resolution gradient regularization term is used to update the model parameters of the visual perception model until an iterative training end condition is reached, thereby obtaining the trained visual perception model;

[0050] A recognition unit, used for inputting an input image of a visual perception task to be processed into the trained visual perception model to obtain a visual perception result of a perception target of the visual perception task to be processed;

[0051] Among them, the multi-resolution gradient regularization term is calculated by the gradient regularization term corresponding to each other between the resolutions, and the gradient regularization term is the product of a function of the resolution similarity coefficient corresponding to the two resolutions and a function of the gradient angle between the gradients corresponding to the two resolutions, so that if the similarity between the two resolutions is higher, the gradient regularization term is used to enhance the similarity constraint between the two resolutions; if the similarity between the two resolutions is lower, the gradient regularization term is used to enhance the gradient orthogonality between the two resolutions.

[0052] In order to solve the above technical problems, the present invention further provides an image recognition device, comprising:

[0053] Memory for storing computer programs;

[0054] A processor is used to execute the computer program, and when the computer program is executed by the processor, the steps of any one of the image recognition methods described above are implemented.

[0055] In order to solve the above technical problems, the present invention also provides a non-volatile storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the image recognition method as described in any one of the above items are implemented.

[0056] In order to solve the above technical problems, the present invention also provides a computer program product, including a computer program, which implements the steps of any one of the above-mentioned image recognition methods when executed by a processor.

[0057] The image recognition method provided by the present invention has the beneficial effect that, when training a visual perception model, an image sample set including image samples of multiple resolutions is used for training, and in the iterative training process, a visual perception loss value of the visual perception model and a multi-resolution gradient regularization term are calculated according to the perception loss value of the visual perception model for image samples of each resolution, wherein the multi-resolution gradient regularization term is calculated by the gradient regularization term corresponding to each other between resolutions, and if the similarity between the two resolutions is higher, the gradient regularization term is used to enhance the similarity constraint between the two resolutions, and if the similarity between the two resolutions is lower, the gradient regularization term is used to enhance the gradient orthogonality between the two resolutions, so as to adjust the relationship between the gradient data of image features of different resolutions, strengthen the visual perception model to understand the difference between multiple resolutions with large differences in gradient descent directions and the similarity between multiple resolutions with small differences in gradient descent directions, and update the model parameters of the visual perception model using the model learning loss value calculated according to the visual perception loss value and the multi-resolution gradient regularization term, so as to solve the problem that when the visual perception model learns the perception ability of images with multiple resolutions, the image features brought by different resolutions are different, and the interference between gradient data leads to the difficulty of model optimization. By using the trained visual perception model to perform the visual perception task to be processed, the resolution of the input image can be set as needed, and high-precision perception results of the perceived target can be obtained, which greatly improves the applicability of the visual perception model of computer vision in different applications.

[0058] The present invention also provides an image recognition device, equipment, medium and computer program product, which have the above-mentioned beneficial effects and are not described in detail here. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] In order to more clearly illustrate the embodiments of the present invention or the technical solutions of the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0060] Figure 1 A flowchart of an image recognition method provided by an embodiment of the present invention;

[0061] Figure 2 A schematic diagram of the structure of an image recognition system provided by an embodiment of the present invention;

[0062] Figure 3 A training flow chart of a visual perception model provided by an embodiment of the present invention;

[0063] Figure 4 A schematic diagram of a calculation flow of a multi-resolution gradient regularization term provided in an embodiment of the present invention;

[0064] Figure 5 A schematic diagram of the structure of an image recognition device provided by an embodiment of the present invention;

[0065] Figure 6 A schematic diagram of the structure of an image recognition device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0066] The core of the present invention is to provide an image recognition method, device, equipment, medium and computer program product for improving the resolution generalization capability of computer vision to adapt to input images of different resolutions.

[0067] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0068] Resolution is used to describe the clarity of an image, video or display device. It indicates the number of pixels that can be displayed per unit length. Resolution is usually expressed in the following forms:

[0069] Pixel resolution: This is the most intuitive way of expressing it, which is expressed directly in pixels. For example, 1920x1080 means there are 1920 pixels in the horizontal direction (image width) and 1080 pixels in the vertical direction (image height).

[0070] Screen resolution: Usually used to describe the resolution of a monitor, TV or other display device. For example, Full HD usually refers to a resolution of 1920x1080, while 4K resolution usually refers to 3840x2160.

[0071] Print resolution: In the printing field, resolution is usually expressed in dots per inch (DPI). For example, 300 DPI means there are 300 dots per inch.

[0072] Image scanning resolution: The resolution of the scanner is also expressed in DPI, which determines the clarity of the scanned image.

[0073] Video resolution: The resolution of a video is usually expressed in pixels, such as 720p (1280x720), 1080p (1920x1080), and 4K (3840x2160 or 4096x2160).

[0074] The image recognition method provided in the embodiment of the present invention can be used to process images captured by a visual sensor such as a camera, or to process video frames captured from a captured video. In the embodiment of the present invention, the resolution described can be pixel resolution.

[0075] The resolution generalization ability of a visual perception model refers to the ability of the visual perception model to process input images with resolutions that are inconsistent with the training resolution during the testing phase.

[0076] Traditional computer vision solutions usually use the same resolution in the model training and testing stages. This is because the resolution generalization ability of visual perception models is poor. Specifically, when the test resolution is inconsistent with the training resolution, the model accuracy will be seriously reduced. The greater the difference between the test resolution and the training resolution, the more serious the reduction in model accuracy, which limits the application of visual perception models in different scenarios. The fundamental reason for this phenomenon is that the distribution of image data with different resolutions is different, and current visual perception models have difficulty in handling such distribution differences.

[0077] However, in practical applications, the visual perception model after deployment may need to process input images with higher resolutions. For example, a visual perception model trained with a resolution of (256x704) may need to process input images with a resolution of (512x1408) or a resolution of (128x352) after deployment. To meet the application needs of processing images with higher resolutions, one current solution is to adjust the input image to the same size as the resolution of the image sample in the training phase during the application phase, that is, adjust the input image with a resolution of (512x1408) and the input image with a resolution of (128x352) to a resolution of (256x704).

[0078] Although this solution of unifying the resolution of input images is feasible, in some practical application scenarios, there is a high demand for different resolution settings. For some application scenarios with limited computing resources, the visual perception model is required to process low-resolution input images (such as 128x352 resolution in the above example) to reduce the computational burden and ensure real-time performance. If the input image resolution is increased to align it to the training resolution, the computational burden will increase and affect the real-time performance of the system. For some application scenarios with sufficient computing resources, it is hoped that a higher image resolution (such as 512x1408 resolution in the above example) will be used to improve the accuracy of 3D target detection. By reducing the high resolution of the image to align it to the training resolution, a lot of detail information will be lost, which may be detrimental to the detection of small and distant targets.

[0079] It can be seen that in actual computer vision applications, there is still a great demand for the resolution generalization capability of the visual perception model itself. However, the resolution generalization capability of the current visual perception model is very poor and it is difficult to directly apply it to scenes with different resolutions.

[0080] In order to improve the resolution generalization ability of the visual perception model, it is necessary to use image samples of different resolutions to train the visual perception model. However, due to the large differences in the feature distribution of image samples of different resolutions, model optimization is difficult to succeed.

[0081] In order to meet the needs of computer vision applications for processing input images of different resolutions, an embodiment of the present invention provides an image recognition solution. When training a visual perception model, an image sample set including image samples of multiple resolutions is used for training. In the iterative training process, a visual perception loss value of the visual perception model and a multi-resolution gradient regularization term are calculated according to the perception loss value of the visual perception model for image samples of each resolution. The multi-resolution gradient regularization term is calculated by the gradient regularization term corresponding to each other between resolutions. If the similarity between the two resolutions is higher, the gradient regularization term is used to enhance the similarity constraint between the two resolutions. If the similarity between the two resolutions is lower, the gradient regularization term is used to enhance the gradient orthogonality between the two resolutions, so as to adjust the relationship between the gradient data of image features of different resolutions, strengthen the visual perception model to understand the difference between multiple resolutions with large differences in gradient descent directions and the similarity between multiple resolutions with small differences in gradient descent directions, and update the model parameters of the visual perception model using the model learning loss value calculated according to the visual perception loss value and the multi-resolution gradient regularization term, so as to solve the problem that when the visual perception model learns the perception ability of images with multiple resolutions, the image features brought by different resolutions are different and the interference between gradient data leads to the difficulty of model optimization. By using the trained visual perception model to perform the visual perception task to be processed, the resolution of the input image can be set as needed, and high-precision perception results of the perceived target can be obtained, which greatly improves the applicability of the visual perception model of computer vision in different applications.

[0082] The image recognition method provided by the embodiment of the present invention is described below with reference to the accompanying drawings.

[0083] Figure 1 A flowchart of an image recognition method provided by an embodiment of the present invention; Figure 2 A schematic diagram of the structure of an image recognition system provided by an embodiment of the present invention; Figure 3 A training flow chart of a visual perception model provided by an embodiment of the present invention.

[0084] like Figure 1 As shown, the image recognition method provided by the embodiment of the present invention includes:

[0085] S101: training a visual perception model using an image sample set including image samples of multiple resolutions;

[0086] S102: During the iterative training process, a visual perception loss value and a multi-resolution gradient regularization term of the visual perception model are calculated according to the perception loss value of the visual perception model for image samples of each resolution, and a model learning loss value calculated according to the visual perception loss value and the multi-resolution gradient regularization term is used to update the model parameters of the visual perception model until the iterative training end condition is reached, thereby obtaining a trained visual perception model;

[0087] S103: inputting the input image of the visual perception task to be processed into the trained visual perception model to obtain a visual perception result of the perception target of the visual perception task to be processed;

[0088] Among them, the multi-resolution gradient regularization term is calculated by the gradient regularization term corresponding to each other between the resolutions. The gradient regularization term is the product of a function of the resolution similarity coefficient corresponding to the two resolutions and a function of the gradient angle between the gradients corresponding to the two resolutions. If the similarity between the two resolutions is higher, the gradient regularization term is used to enhance the similarity constraint between the two resolutions. If the similarity between the two resolutions is lower, the gradient regularization term is used to enhance the gradient orthogonality between the two resolutions.

[0089] The image recognition method provided by the embodiment of the present invention mainly includes two major links: training of the visual perception model and application of the visual perception model. In this regard, the image recognition method provided by the embodiment of the present invention can be applied to Figure 2 The image detection system shown in FIG. 1 includes two parts: a model training system and a perception computing device.

[0090] The model training system may be composed of one or more computing devices for training a visual perception model based on an image sample set composed of image samples of multiple resolutions. Figure 2 As shown, the model training system can be composed of multiple threads of multiple artificial intelligence devices to form a thread pool, and the image samples corresponding to each resolution are assigned to a thread pool, so as to achieve parallel accelerated learning of multiple resolution image sets. It should be noted that m, n, and k only represent the meaning of multiple, and do not represent a specific number or the same number of threads of each artificial intelligence processor, nor do they mean that each thread pool is composed of one thread in each artificial intelligence device. The model training system can be deployed in a cloud computing center. The device structure of the cloud computing center is similar Figure 2The perceptual computing devices in the system include computing, storage, network and other infrastructure, operating system, database, middleware and other software environments and application software. The model training system receives image data sets of various resolutions, uses the stored pre-trained models, combined with manual annotation, to annotate visual perception targets (such as the category, location, size of the bounding box, target orientation information, etc.) and stores the annotated data.

[0091] The perception computing device can be a computing device deployed in a cloud computing center or an edge computing device deployed in an edge device. Its structure is as follows: Figure 2 As shown in the perception computing device shown. If the computing device of the cloud computing center is used, after the trained visual perception model is deployed, the edge device collects the image to be detected through a visual sensor such as a camera and uploads it to the cloud computing center, and uses the perception computing device that deploys the visual perception model to perform visual perception calculation on the image to be detected to obtain the visual perception result of the visual perception target. If the perception computing device uses an edge computing device deployed on the edge device, the visual perception model is trained by the model training system and sent to the edge computing device through the communication unit, and deployed in the memory of the edge computing device. After the edge computing device collects the image to be detected through a visual sensor such as a camera, it performs visual perception calculation through the local visual perception model to obtain the visual perception result of the visual perception target.

[0092] Applied to Figure 2 The multi-thread pool shown, in an embodiment of the present invention, the step of calculating the perceptual loss value of the visual perception model for image samples of each resolution may include: inputting the image samples corresponding to each resolution into different thread pools one by one, so as to obtain the perceptual loss value of the visual perception model for the image samples of each resolution by parallel calculation.

[0093] like Figure 3 As shown, in an embodiment of the present invention, when training a visual perception model, all images of each resolution can be stacked to obtain a set of input images of different resolutions, so that the training speed can be improved through batch processing.

[0094] Since there are large differences in data distribution between input images of different resolutions, the optimization difficulty of the visual perception model increases. Specifically, the data distribution of input images of different resolutions is different, and there are large differences in the distribution of image features extracted by the middle layer of the visual perception model, resulting in differences in the final prediction results, and thus differences in the calculated loss function values, which ultimately leads to large differences in the gradient data of the visual perception model parameters corresponding to different resolutions in the back-propagation stage. There is interference between the gradient data, which increases the difficulty of model optimization. In order to solve the problem of difficult model optimization, in an embodiment of the present invention, in addition to calculating the perceived loss value of the visual perception model for image samples of different resolutions, the multi-resolution gradient regularization term is calculated to adjust the gradient directions corresponding to different resolutions to avoid mutual interference between gradients of different resolutions.

[0095] Then, as in S102 of an embodiment of the present invention, the model learning loss value of the visual perception model in one iterative training is calculated based on the visual perception loss value and the multi-resolution gradient regularization term. Among them, the visual perception loss value can be calculated using the perceptual loss value of the visual perception model for image samples of each resolution, such as the sum or mean of the perceptual loss values ​​of the visual perception model for image samples of each resolution. For S102, the iterative training end condition may be that the number of iterative training times of the visual perception model reaches a preset number of iterations, or the model learning loss value of the visual perception model is less than a preset loss value.

[0096] As described above, the multi-resolution gradient regularization term proposed in the embodiment of the present invention is calculated by the gradient regularization term corresponding to each other between the resolutions, and the gradient regularization term is the product of the function of the resolution similarity coefficient corresponding to the two resolutions and the function of the gradient angle between the gradients corresponding to the two resolutions, so that if the similarity between the two resolutions is higher, the gradient regularization term is used to enhance the similarity constraint between the two resolutions, and if the similarity between the two resolutions is lower, the gradient regularization term is used to enhance the gradient orthogonality between the two resolutions. That is to say, if the similarity between the two resolutions is higher, the gradient regularization term is used to enhance the similarity between the gradient directions corresponding to the two resolutions, thereby enhancing the visual perception model to understand the two resolutions as more similar resolutions. If the similarity between the two resolutions is lower, the gradient regularization term is used to enhance the orthogonality between the gradient directions corresponding to the two resolutions, that is, the gradient angle between the gradients of the two resolutions is adjusted to 90°, thereby enhancing the visual perception model to understand the two resolutions as more dissimilar resolutions.

[0097] The image recognition method provided by the embodiment of the present invention is applied. When training a visual perception model, an image sample set including image samples of multiple resolutions is used for training. In the iterative training process, a visual perception loss value of the visual perception model and a multi-resolution gradient regularization term are calculated according to the perception loss value of the visual perception model for image samples of each resolution. The multi-resolution gradient regularization term is calculated by the gradient regularization term corresponding to each other between resolutions. If the similarity between the two resolutions is higher, the gradient regularization term is used to enhance the similarity constraint between the two resolutions. If the similarity between the two resolutions is lower, the gradient regularization term is used to enhance the gradient orthogonality between the two resolutions, so as to adjust the relationship between the gradient data of image features of different resolutions, strengthen the visual perception model to understand the difference between multiple resolutions with large differences in gradient descent directions and the similarity between multiple resolutions with small differences in gradient descent directions, and update the model parameters of the visual perception model using the model learning loss value calculated according to the visual perception loss value and the multi-resolution gradient regularization term, so as to solve the problem that when the visual perception model learns the perception ability of images with multiple resolutions, the image features brought by different resolutions are different and the interference between gradient data makes the model optimization difficult. By using the trained visual perception model to perform the visual perception task to be processed, the resolution of the input image can be set as needed, and high-precision perception results of the perceived target can be obtained, which greatly improves the applicability of the visual perception model of computer vision in different applications.

[0098] Based on the above embodiment, the embodiment of the present invention further illustrates the steps of collecting the image sample set.

[0099] The image sample set used for training the visual perception model in the embodiment of the present invention includes image samples of multiple resolutions. When collecting the image sample set, a visual sensor such as a camera can be used to directly collect image samples of different resolutions through set resolution parameters to form an image sample set; when collecting the image sample set, image samples of one resolution can also be collected first, and image samples of multiple resolutions can be generated by scaling or enlarging to form an image sample set.

[0100] In some optional implementations of the embodiments of the present invention, the step of collecting the image sample set may include: obtaining an original image set; processing each original image in the original image set using different scaling factors to obtain scaled images of different resolutions corresponding to each original image; and using each original image and the scaled image corresponding to each original image as the image sample set.

[0101] In practical applications, each original image in the original image set is processed with different scaling factors to obtain scaled images of different resolutions corresponding to each original image, which may include: processing each original image with a reference scaling factor to obtain a reference scaled image corresponding to the original image; sampling multiple scaling factors from a preset distribution range, and processing each original image with each scaling factor to obtain scaled images of multiple resolutions corresponding to each original image; using the corresponding reference scaled image and each corresponding scaled image as an image set corresponding to the original image to obtain an image set corresponding to each original image. That is, the generated image sample set is composed of image sets corresponding to different original images, which include image samples of different resolutions, and different image samples of the same resolution may be obtained by processing different original images.

[0102] As described in the above embodiments of the present invention, the resolution used for processing by the visual perception model of computer vision tasks is usually expressed in pixel resolution, that is, expressed as the product of the number of pixels in the horizontal direction and the number of pixels in the vertical direction. In the embodiments of the present invention, the scaling factor may include a scaling factor acting on the image height and / or a scaling factor on the image width. In order to obtain image samples of multiple resolutions with greater resolution differences, the scaling factor may include an image height scaling factor and an image width scaling factor.

[0103] To reduce the computational difficulty, the scaling factor can also be a scaling ratio for the image height and image width. For example, for an original image with a resolution of H×W , the base scaling factor can be used The processed resolution is After the image is distributed in Uniform distribution Sample k scaling factors , and process the image using each scaling factor , and finally obtain a collection of images with multiple resolutions ,in The resolution is . Where H represents the vertical size of the image, and W represents the horizontal size of the image.

[0104] When applied to the field of autonomous driving, cameras and other visual sensors are usually set up in different directions of the vehicle, such as left front (fl), front (f), right front (fr), left rear (bl), rear (b), right rear (br), etc., so that images of multiple perspectives of the road scene are collected each time. In the actual training process, in order to use the batch method to accelerate model training, the embodiment of the present invention samples a set of scaling factors at each iteration, and , the above-obtained scaling coefficient set is used for processing, that is, a multi-resolution image set is obtained for each viewing angle .

[0105] Since the resolution difference between the multi-resolution image samples used in training the visual perception model is small, it is not conducive to the training of the model's resolution generalization ability. In order to obtain as many resolutions with large differences as possible and improve the training effect of the model's resolution generalization ability, in other optional implementations of the embodiments of the present invention, the difference between the scaling factors can be set to be no less than the scaling factor difference threshold, that is, when determining the scaling factor, the difference between the scaling factor and other scaling factors in the collected scaling factor set is judged, and only when the difference is greater than the set scaling factor difference threshold is the scaling factor added to the scaling factor set, thereby ensuring the diversity between the scaling factors obtained by sampling and improving the training effect of the model's resolution generalization ability.

[0106] Furthermore, the scaling factors may be designed to all belong to a preset distribution range and the difference between any two scaling factors may not be less than a scaling factor difference threshold.

[0107] In some optional implementations of the embodiments of the present invention, the scaling factor difference threshold is calculated based on the interval length of the preset distribution range divided by twice the number of scaling factors. That is, the scaling factor difference threshold can be expressed by the following formula:

[0108] ;

[0109] Where M represents the scaling factor difference threshold, Indicates the lower limit of the preset distribution range. represents the upper limit of the preset distribution range, and k represents the total number of scaling factors to be collected from the preset distribution range.

[0110] In some other alternative embodiments of the embodiments of the present invention, for each original image in the original image set, different scaling factors are used for processing to obtain scaled images with different resolutions corresponding to each original image. It may also include: processing each original image with a reference scaling factor to obtain a reference scaled image corresponding to the original image; sampling multiple scaling factors from a preset distribution range until a preset number of scaling factors are collected; during the process of collecting scaling factors, for each newly collected scaling factor, calculating the difference between the new scaling factor and the existing scaling factors. If the difference is not less than the scaling factor difference threshold, then add the new scaling factor to the set of existing scaling factors; if the difference is less than the scaling factor difference threshold, then discard the new scaling factor; processing each original image with each scaling factor respectively to obtain scaled images with multiple resolutions corresponding to each original image; using the corresponding reference scaled image and the corresponding scaled images as the image set corresponding to the original image to obtain the image set corresponding to each original image.

[0111] That is, the following algorithm can be designed to obtain the set of scaling factors:

[0112] Input: reference scaling factor , distributed in uniform distribution , set A of scaling factors, scaling factor difference threshold M;

[0113] Add the reference scaling factor to the set A of scaling factors;

[0114] If the length of the set A of scaling factors < k + 1, then:

[0115] Randomly sample from the uniform distribution to obtain ;

[0116] Check the difference between and all the scaling factors in the current set A of scaling factors. If the difference is greater than the threshold scaling factor difference M, then add to the set A of scaling factors;

[0117] Output: set A of scaling factors.

[0118] To further improve the ability of the visual perception model to process input images with different resolutions, image sample sets can be generated multiple times during the model training process. That is to say, new image sample sets can be generated by re-collecting scaling factors in each iteration training, or new image sample sets can be generated by re-collecting scaling factors after every few iteration trainings.

[0119] However, in the embodiment of the present invention, since the model learning loss function of the visual perception model is obtained by designing a combination of loss functions for each resolution to enable the visual perception model to acquire the ability to process multi-resolution input images, if the image sample set is regenerated by the above method for each iterative training, it may make it difficult for the model to converge.

[0120] To ensure model convergence, in some optional implementations of the embodiments of the present invention, a method of generating an image sample set once before training the visual perception model can be adopted, and the image sample set is no longer updated in subsequent iterative training. In this way, the same batch of resolutions are learned in each iterative training of the visual perception model, and the diversity of samples can be ensured by collecting different image samples corresponding to the resolutions. If computing resources and time permit, a method of generating an image sample set once before training the visual perception model can be adopted, but multiple rounds of iterative training are performed, and the image sample set is regenerated before the start of each round of iterative training, and the same image sample set is used in different iterative training steps in each round of iterative training.

[0121] In some other optional implementations of the embodiments of the present invention, the image sample set may be regenerated every multiple iterations of training, and the frequency of generating the image sample set may be determined according to the convergence of the visual perception model.

[0122] In some other optional implementations of the embodiments of the present invention, if the image sample set is regenerated in each iterative training or every few iterative trainings, the multiple resolutions corresponding to the resampling scaling factors can arrange the resolutions in a fixed order (such as from small to large or from large to small) according to the scaling factors, thereby ensuring the correspondence between the arrangement of the resolutions and the arrangement of the loss function, so that the model training can inherit the experience of the previous iterative training or the previous few iterative trainings.

[0123] In each embodiment of the present invention, the loss function corresponding to the resolution of the visual perception model in each iterative training number inherits the parameters of the loss function of the corresponding position in the previous iterative training, but the corresponding resolution can be the resolution in the previous iterative training or the updated resolution.

[0124] In an embodiment of the present invention, if a scheme of generating image sample sets multiple times in iterative training of a visual perception model is adopted, each time an image sample set is regenerated, it is necessary to re-execute the step of calculating the similarity coefficient between each pair of resolutions, and regenerate the multi-resolution gradient regularization term accordingly.

[0125] Based on the above embodiment, the embodiment of the present invention further illustrates a method for calculating the resolution similarity coefficient.

[0126] In the embodiment of the present invention, the resolution can be expressed by pixel resolution, that is, the product of the number of pixels in the horizontal direction of the image and the number of pixels in the vertical direction of the image. Then the resolution similarity coefficient can be calculated based on the image height corresponding to the two resolutions and the image width corresponding to the two resolutions. , The similarity coefficient between Can be used Indicates that Resolution The pixel height, Resolution The pixel width, Resolution The pixel height, Resolution The pixel width.

[0127] In some optional implementations of the present invention, the resolution similarity coefficient can be calculated based on the image height ratio corresponding to the two resolutions and the image width ratio corresponding to the two resolutions. .

[0128] Alternatively, the resolution similarity coefficient can also be calculated based on the image height difference corresponding to the two resolutions and the image width difference corresponding to the two resolutions. .

[0129] In some other optional implementations of the embodiments of the present invention, the resolution similarity coefficient may also be the difference of the first preset constant minus the first fractional term; wherein the numerator of the first fractional term is the maximum value of the difference of the image height corresponding to the two resolutions and the difference of the image width corresponding to the two resolutions, and the denominator of the first fractional term is the maximum value of the image height corresponding to the two resolutions and the image width corresponding to the two resolutions. Based on this, the resolution similarity coefficient can be expressed by the following formula:

[0130] ;

[0131] in, Indicates finding the maximum value.

[0132] The calculation method of the similarity coefficient introduced above makes the similarity coefficient higher when the similarity between two resolutions is higher. In other embodiments, the calculation method of the similarity coefficient can also be designed so that the similarity coefficient is lower when the similarity between two resolutions is higher.

[0133] Based on the above embodiment, the embodiment of the present invention further illustrates a method for calculating a multi-resolution gradient regularization term.

[0134] In the above embodiment of the present invention, the design principle of the multi-resolution gradient regularization term is that if the similarity between two resolutions is higher, the gradient regularization term is used to enhance the similarity constraint between the two resolutions; if the similarity between the two resolutions is lower, the gradient regularization term is used to enhance the gradient orthogonality between the two resolutions, thereby strengthening the visual perception model to understand the differences between multiple resolutions with large differences in gradient descent directions and to understand the similarities between multiple resolutions with small differences in gradient descent directions.

[0135] In an embodiment of the present invention, the multi-resolution gradient regularization term can be represented by the sum or mean of the gradient regularization terms corresponding to multiple groups of resolutions. The gradient regularization term corresponding to one group of resolutions is calculated based on the similarity coefficient between the two resolutions and the gradient direction difference of the gradient data corresponding to the two resolutions.

[0136] In applications, the difference in gradient direction between two resolutions can be expressed by the gradient angle between the gradient vectors corresponding to the two resolutions, and the gradient angle ranges from 0° to 180°. The size of the gradient angle corresponding to the two resolutions is related to the similarity between the two resolutions, that is, the higher the similarity between the two resolutions, the smaller the gradient angle between their gradients, that is, the closer the gradient descent direction is. In order to solve the interference between gradients of different resolutions, the constraints of the gradient regularization items corresponding to the two resolutions with higher similarity or smaller gradient angles can be strengthened according to the size of the gradient angle or the similarity coefficient between the resolutions, and the constraints of the gradient regularization items corresponding to the two resolutions with lower similarity or larger gradient angles can be strengthened at the same time.

[0137] A resolution similarity threshold may be designed to divide the similarity coefficients corresponding to two resolutions into two intervals. Two resolutions whose similarity coefficients are less than or equal to the resolution similarity threshold are considered to have low similarity, while two resolutions whose pixel coefficients are greater than the resolution similarity threshold are considered to have high similarity.

[0138] A gradient angle threshold may also be designed to divide the gradient angle into intervals, so that the gradients of the two resolutions with smaller gradient angles are adjusted toward the direction of the gradient angle of 0, and the gradients of the two resolutions with larger gradient angles are adjusted toward the direction of the gradient angle of 90°. Then, corresponding to the resolution similarity threshold, the gradient angle threshold is a value between 45° and 90°.

[0139] In some optional implementations of the embodiments of the present invention, if the similarity between the two resolutions is higher, the gradient regularization term is used to enhance the similarity constraint between the two resolutions, which may include: if the resolution similarity coefficient is greater than the resolution similarity threshold, then the function of the resolution similarity coefficient is positively correlated with the resolution similarity coefficient, and the function of the gradient angle is positively correlated with the gradient angle.

[0140] If the similarity between the two resolutions is lower, the gradient regularization term is used to enhance the gradient orthogonality between the two resolutions, which may include: if the resolution similarity coefficient is less than or equal to the resolution similarity threshold, and the gradient angle corresponding to the two resolutions is less than the gradient angle threshold, then the function of the resolution similarity coefficient is negatively correlated with the resolution similarity coefficient, and the function of the gradient angle is negatively correlated with the gradient angle; if the resolution similarity coefficient is less than or equal to the resolution similarity threshold, and the gradient angle corresponding to the two resolutions is greater than or equal to the gradient angle threshold, then the function of the resolution similarity coefficient is negatively correlated with the resolution similarity coefficient, and the function of the gradient angle is positively correlated with the gradient angle.

[0141] In the embodiment of the present invention, the gradient angle function can be represented by a sine function or a cosine function of the gradient angle. Then, taking the resolution similarity threshold as the boundary, according to the change trend of the sine function or the cosine function, the effect of adjusting the gradients of the two resolutions with smaller gradient angles toward the direction of the gradient angle of 0 and adjusting the gradients of the two resolutions with larger gradient angles toward the direction of the gradient angle of 90° is achieved, and the gradient angle threshold is 90°.

[0142] Based on this, the gradient regularization term can be expressed as follows:

[0143] ;

[0144] in, Indicates resolution The gradient vector of Indicates resolution The gradient vector of Indicates resolution With resolution The corresponding gradient angle, Indicates resolution With resolution The corresponding gradient regularization term is, , , represents the function of the gradient angle, where , It is positively correlated with the gradient angle. Negatively correlated with the gradient angle, , , Represents the function of the similarity coefficient, where It is positively correlated with the similarity coefficient. , It is negatively correlated with the similarity coefficient. Indicates the resolution similarity threshold.

[0145] In order to better adjust the relationship between feature gradients of images with different resolutions, the strength of the gradient similarity constraint can be adjusted according to the similarity coefficient. If the resolution similarity coefficient is greater than the resolution similarity threshold, the function of the resolution similarity coefficient is a proportional function of the resolution similarity coefficient; if the resolution similarity coefficient is less than or equal to the resolution similarity threshold, the function of the resolution similarity coefficient is a proportional function of the second preset constant minus the resolution similarity coefficient.

[0146] At the same time, the design principle of the gradient angle function can be: if the resolution similarity coefficient is greater than the resolution similarity threshold, the gradient angle function is the difference between 1 and the cosine function of the gradient angle; if the resolution similarity coefficient is less than or equal to the resolution similarity threshold, and the gradient angles corresponding to the two resolutions are less than the gradient angle threshold, the gradient angle function is 1 plus half of the sum of the cosine function of the gradient angle; if the resolution similarity coefficient is less than or equal to the resolution similarity threshold, and the gradient angles corresponding to the two resolutions are greater than or equal to the gradient angle threshold, the gradient angle function is the negative value of the cosine function of the gradient angle.

[0147] In some other optional implementations of the embodiments of the present invention, the gradient regularization term can be expressed by the following formula:

[0148] ;

[0149] in, Indicates resolution With resolution The cosine function of the gradient angle between .

[0150] Alternatively, the function of the gradient angle may also be a sinusoidal function. For example, the gradient regularization term may be expressed as follows:

[0151] ;

[0152] in, Indicates resolution With resolution The sine function of the gradient angle between .

[0153] Figure 4 A schematic diagram of a calculation flow of a multi-resolution gradient regularization term provided in an embodiment of the present invention. Figure 4 As shown, the input resolution The corresponding image feature gradient , and resolution The corresponding image feature gradient , calculate the resolution With resolution and determine whether the similarity coefficient is greater than the resolution similarity threshold; if so, the gradient similarity increases the constraint, which is equivalent to reducing the resolution With resolution The corresponding gradient direction difference; if not, the gradient orthogonality is enhanced, which is equivalent to the resolution With resolution The corresponding gradient direction is adjusted to 90°.

[0154] In some optional implementations of the embodiments of the present invention, the multi-resolution gradient regularization term is the sum of the gradient regularization terms corresponding to all pairwise combinations of resolutions corresponding to the image sample set. That is, the multi-resolution gradient regularization term can be expressed by the following formula:

[0155] ;

[0156] in, Represents the multi-resolution gradient regularization term.

[0157] In some other optional implementations of the embodiments of the present invention, the multi-resolution gradient regularization term may be calculated by using only the gradient regularization term corresponding to some resolution combinations. For example, when the similarity coefficient is greater than the high similarity threshold (the high similarity threshold is greater than the resolution similarity threshold) or the gradient angles corresponding to the two resolutions are less than the low gradient angle threshold (the low gradient angle threshold is less than the gradient angle threshold), the gradient regularization term corresponding to the resolution combination may not be added to the calculation of the multi-resolution gradient regularization term.

[0158] Based on the above embodiment, the model learning loss function of the visual perception model can be expressed by the following formula:

[0159] ;

[0160] in, Learning loss functions for models of visual perception, represents the visual perception loss value, Represents the multi-resolution gradient regularization term.

[0161] Visual perception loss value It can be the sum or mean of the perceptual loss values ​​of the visual perception model for image samples of each resolution.

[0162] In order to further reduce the interference between image samples of different resolutions, in the image recognition method provided in an embodiment of the present invention, the step of calculating the visual perception loss value may include: updating the loss function corresponding to each resolution according to the perceptual loss value of the image samples of each resolution based on the visual perception model; and calculating the visual perception loss value of the current number of iterations based on the loss function corresponding to each resolution.

[0163] That is to say, the interference between image samples of different resolutions can be further reduced by updating the loss function corresponding to each resolution according to the perceptual loss value of the image samples of each resolution according to the visual perception model.

[0164] In the above embodiment of the present invention, the step of calculating the visual perception loss value in S102 may include: taking the average of the loss values ​​of the loss functions of each resolution as the visual perception loss value.

[0165] refer to Figure 3 As shown, for each resolution, all image samples are input into the visual perception model to obtain the corresponding visual perception results, and the loss value of the visual perception model for each resolution compared with the true value (which can be manually labeled) is calculated through the loss function. Finally, the model learning loss value of the visual perception model can be the average of the perceptual loss values ​​of the visual perception model for image samples of all resolutions, as shown in the following formula:

[0166] ;

[0167] in, represents the visual perception loss value, represents the perceptual loss value corresponding to the r-th resolution, represents the loss function corresponding to the rth resolution, and k+1 represents the number of resolutions used in visual perception model training.

[0168] Among them, the loss function corresponding to the resolution can be expressed as a function of the self-learning loss weight corresponding to the resolution and the perceptual loss value of the image sample of the resolution by the visual perception model. Updating the loss function corresponding to each resolution according to the perceptual loss value of the image sample of each resolution by the visual perception model can include: updating the self-learning loss weight in the loss function corresponding to each resolution according to the perceptual loss value of the image sample of each resolution by the visual perception model. That is to say, by setting the self-learning loss weight for the image samples of each resolution, in the iterative training of the visual perception model, not only the model parameters of the visual perception model are updated through loss optimization, but also the self-learning loss weight corresponding to each resolution is updated, thereby automatically adjusting the loss functions corresponding to different resolutions to avoid mutual interference between the loss functions corresponding to different resolutions.

[0169] The loss function corresponding to the resolution can be expressed by the product of the self-learning loss weight corresponding to the resolution and the perceptual loss value corresponding to the resolution. Other representations can also be used. For example, the loss function corresponding to the resolution can be expressed by the following formula:

[0170] ;

[0171] in, represents the visual perception loss value, represents the perceptual loss value corresponding to the r-th resolution, represents the self-learning loss weight corresponding to the rth resolution, and k+1 represents the number of resolutions used in visual perception model training.

[0172] By adding logarithmic terms, the training process of the visual perception model can be made more stable, which can have the effect of punishing overfitting.

[0173] Since the difficulty of learning image samples of different resolutions for the visual perception model is different, the embodiment of the present invention also proposes a learning difficulty coefficient to further balance the learning effect of the visual perception model at different resolutions. In practical applications, the higher the resolution of the input image, the easier it is to optimize the visual perception model, that is, the lower the learning difficulty. Accordingly, the loss function corresponding to the resolution can also be calculated based on the self-learning loss weight corresponding to the resolution, the learning difficulty coefficient of the visual perception model for image samples of the resolution, and the perceived loss value of the visual perception model for image samples of the resolution; wherein, the higher the resolution, the lower the learning difficulty of the visual perception model for image samples of the resolution.

[0174] That is to say, according to the relationship between the learning difficulty of the visual perception model and the resolution of the input image, the learning difficulty coefficient is added to the loss function of different resolutions to balance the learning difficulty of the visual perception model for image samples with different resolutions, so as to further balance the loss value at each resolution and improve the training effect of the visual perception model on multi-resolution images.

[0175] In some other optional implementations of the embodiments of the present invention, the loss function of the visual perception model can also be expressed by the following formula:

[0176] ;

[0177] in, represents the learning difficulty coefficient corresponding to the rth resolution, Represents the self-learning loss weight corresponding to the r-th resolution.

[0178] The learning difficulty coefficient corresponding to the resolution can be calculated based on the relationship between the spatial size of the image sample of the resolution and the spatial size of the image sample of other resolutions. Then the learning difficulty coefficient can be calculated by the following formula:

[0179] ;

[0180] in, , , A collection representing the number of pixels of height for all resolutions, A collection of width pixels for all resolutions. Indicates the number of height pixels corresponding to the rth resolution, Indicates the maximum number of height pixels in the image sample set, Indicates the number of pixels of width corresponding to the r-th resolution, Indicates the maximum number of pixels of width in the image sample set. Indicates the larger value of the maximum number of height pixels in the image sample set and the maximum number of width pixels in the image sample set.

[0181] It can be seen that when adding the learning difficulty coefficient After that, for higher resolution input images, the learning difficulty of the visual perception model is smaller and the value of the loss function is smaller, so the learning difficulty coefficient The value of approaches 1, then The adjustment of the perceptual loss value is small. For lower resolution input images, the learning difficulty of the visual perception model is greater, and the value of the loss function is larger, so the learning difficulty coefficient The value of is in the range (0,1) and decreases as the resolution decreases. The adjustment of the perceptual loss value is large to avoid the excessively high loss function value at low resolution from dominating and interfering with the training effect of the visual perception model on high-resolution images. It is a learnable parameter that is automatically adjusted during the training process through an adaptive learning mechanism to further balance the loss values ​​at each resolution and improve the training effect of the visual perception model on multi-resolution images. Therefore, in the image recognition method provided by the embodiment of the present invention, by balancing the learning difficulty of the visual perception model for image samples of different resolutions to improve the training effect of the visual perception model on multi-resolution images, a more outstanding resolution generalization capability can be obtained in densely predicted perception tasks such as three-dimensional object detection.

[0182] In another optional implementation of the embodiment of the present invention, the learning difficulty of the visual perception model on images of different resolutions is different. Generally, the learning difficulty on high-resolution images is lower because high-resolution images contain rich detail information, which is conducive to dense prediction tasks. The learning difficulty on low-resolution images is higher because low-resolution images lose a lot of detail information. Therefore, the learning speed of the visual perception model on images of different resolutions is different. The visual perception model can converge faster on high-resolution images, but converge slowly on low-resolution images.

[0183] In order to balance the learning progress of the visual perception model for images with different resolutions, an embodiment of the present invention provides a solution for optimizing the learning effect of the visual perception model for multi-resolution images from the perspective of balancing the learning rate.

[0184] In an embodiment of the present invention, the loss function corresponding to the resolution can be calculated based on the learning rate of the visual perception model for image samples of the resolution and the perceptual loss value of the visual perception model for image samples of the resolution; wherein the learning rate corresponding to the resolution is the rate of change of the perceptual loss value of image samples of the resolution in adjacent iterative training times of the visual perception model.

[0185] In some optional implementations of the embodiments of the present invention, the learning rate of resolution may be the ratio of the perceptual loss value of the visual perception model for the image sample of resolution in the current iterative training to the perceptual loss value of the visual perception model for the image sample of resolution in the previous iterative training. That is, the learning rate may be expressed by the following formula:

[0186] ;

[0187] in, represents the learning rate of the rth resolution in the tth iterative training of the visual perception model, represents the perceptual loss value of the rth resolution in the tth iterative training of the visual perception model, Represents the loss value of the r-th resolution in the t-1-th iterative training of the visual perception model.

[0188] In some other optional implementations of the embodiments of the present invention, the learning rate of resolution may also be the difference between the perceptual loss value of the visual perception model for image samples of resolution in the current iterative training and the perceptual loss value of the visual perception model for image samples of resolution in the previous iterative training.

[0189] For the convenience of calculation, the learning rates of all resolutions can be normalized. The normalized learning rate can be shown as follows:

[0190] ;

[0191] in, It represents the normalized learning rate of the r-th resolution in the t-th iterative training of the visual perception model. The larger the value, the slower the learning rate.

[0192] After considering the learning rate, the loss function of the visual perception model can be designed as follows:

[0193] ;

[0194] in, represents the visual perception loss value, Represents the perceptual loss value corresponding to the r-th resolution.

[0195] Considering that there is a certain relationship between the learning difficulty and learning rate of the visual perception model for image samples of different resolutions, in other optional implementations of the embodiments of the present invention, the loss function corresponding to the resolution can also be calculated based on the learning difficulty coefficient of the visual perception model for image samples of resolution, the learning rate of the visual perception model for image samples of resolution, and the perceptual loss value of the visual perception model for image samples of resolution; wherein, the higher the resolution, the lower the learning difficulty of the visual perception model for image samples of resolution; the learning rate corresponding to the resolution is the rate of change of the perceptual loss value of the image samples of resolution in the adjacent iterative training times of the visual perception model. Based on this, the loss function of the visual perception model can be designed as shown in the following formula:

[0196] ;

[0197] in, represents the visual perception loss value, represents the perceptual loss value corresponding to the r-th resolution, represents the learning difficulty coefficient corresponding to the rth resolution, represents the normalized learning rate of the r-th resolution in the t-th iterative training of the visual perception model.

[0198] Learning difficulty It is calculated based on the resolution based on prior knowledge, and the learning rate It is calculated in real time during the training of the visual perception model. The combination of the two can better balance the training process of the visual perception model at different resolutions, thereby improving the multi-resolution generalization ability of the visual perception model.

[0199] In some other optional implementations of the embodiments of the present invention, the design idea of ​​updating the loss function corresponding to each resolution is achieved by designing self-learning loss weights, learning difficulty coefficients, and learning rates. Based on this, the form of the loss function of other visual perception models can be obtained by combining one or more types of parameters.

[0200] The loss function of the visual perception model can also be expressed as follows:

[0201] ;

[0202] in, represents the visual perception loss value, represents the perceptual loss value corresponding to the r-th resolution, represents the normalized learning rate of the rth resolution in the tth iterative training of the visual perception model, represents the learning difficulty coefficient corresponding to the rth resolution, represents the self-learning loss weight corresponding to the r-th resolution, is the weight of the fourth fraction, is the weight of the fifth fraction, It is a learnable parameter that can be initialized to 0.5 and updated as the visual perception model is trained. It is used to weigh the two loss weights of the multi-resolution loss weight adaptation method and the loss weight adjustment method based on learning speed.

[0203] Therefore, by combining multiple types of loss weights, the visual perception model can perform adaptive loss weight adjustment during iterative training, loss weight adjustment based on learning difficulty, and loss weight adjustment based on learning rate.

[0204] During the execution of reasoning tasks using the visual perception model, the visual perception model trained by the training method provided by the embodiment of the present invention can be deployed in a perception computing device after removing the multi-resolution gradient regularization method. For example, the trained visual perception model is deployed in the edge computing platform of an autonomous driving vehicle to obtain the visual perception results of the perception target of the visual perception task to be processed. Since the visual perception model obtains the processing capability of multi-resolution input images during the training phase, it can have good adaptability to input images of different resolutions in different application scenarios to ensure image detection accuracy.

[0205] It should be noted that in the embodiments of the image recognition method of the present invention, some of the steps or features may be ignored or not executed. The hardware or software functional modules divided for the convenience of description are not the only implementation form of the image recognition method provided by the embodiment of the present invention.

[0206] The above describes in detail various embodiments corresponding to the image recognition method. On this basis, the present invention also discloses an image recognition device, equipment, non-volatile storage medium and computer program product corresponding to the above method.

[0207] Figure 5 A schematic diagram of the structure of an image recognition device provided by an embodiment of the present invention.

[0208] like Figure 5 As shown, the image recognition device provided by the embodiment of the present invention includes:

[0209] The model training unit 501 trains the visual perception model using an image sample set including image samples of multiple resolutions; in the iterative training process, the visual perception loss value and the multi-resolution gradient regularization term of the visual perception model are calculated according to the perceptual loss value of the visual perception model for the image samples of each resolution, and the model parameters of the visual perception model are updated using the model learning loss value calculated according to the visual perception loss value and the multi-resolution gradient regularization term, until the iterative training end condition is reached, thereby obtaining the trained visual perception model;

[0210] The recognition unit 502 is used to input the input image of the visual perception task to be processed into the trained visual perception model to obtain the visual perception result of the perception target of the visual perception task to be processed;

[0211] Among them, the multi-resolution gradient regularization term is calculated by the gradient regularization term corresponding to each other between the resolutions. The gradient regularization term is the product of a function of the resolution similarity coefficient corresponding to the two resolutions and a function of the gradient angle between the gradients corresponding to the two resolutions. If the similarity between the two resolutions is higher, the gradient regularization term is used to enhance the similarity constraint between the two resolutions. If the similarity between the two resolutions is lower, the gradient regularization term is used to enhance the gradient orthogonality between the two resolutions.

[0212] It should be noted that in each implementation of the image recognition device provided by the embodiment of the present invention, the division of units is only a logical functional division, and other division methods can be used. The connection method between different units can be electrical, mechanical or other connection methods. The separated units can be located in the same physical location or distributed on multiple network nodes. Each unit can be implemented in the form of hardware or in the form of a software functional unit. That is, part or all of the units provided in the embodiment of the present invention can be selected according to actual needs and the corresponding connection method or integration method can be adopted to achieve the purpose of the embodiment of the present invention.

[0213] Since the embodiments of the apparatus part correspond to the embodiments of the method part, please refer to the description of the embodiments of the method part for the embodiments of the apparatus part, which will not be repeated here.

[0214] Figure 6 A schematic diagram of the structure of an image recognition device provided by an embodiment of the present invention.

[0215] like Figure 6 As shown, the image recognition device provided by the embodiment of the present invention includes: a memory 610, used to store a computer program 611; a processor 620, used to execute the computer program 611, and when the computer program 611 is executed by the processor 620, the steps of the image recognition method provided in any of the above embodiments are implemented.

[0216] Among them, the processor 620 may include one or more processing cores, such as a 3-core processor, an 8-core processor, etc. The processor 620 may be implemented in at least one hardware form of digital signal processing (DSP), field-programmable gate array (FPGA), and programmable logic array (PLA). The processor 620 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a central processing unit (CPU); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 620 may be integrated with a graphics processing unit (GPU), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 620 may also include an artificial intelligence (AI) processor, which is used to process computing operations related to machine learning.

[0217] The memory 610 may include one or more non-volatile storage media, which may be non-transitory. The memory 610 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In this embodiment, the memory 610 is at least used to store the following computer program 611, wherein the computer program 611, after being loaded and executed by the processor 620, can implement the relevant steps in the image recognition method disclosed in any of the aforementioned embodiments. In addition, the resources stored in the memory 610 may also include an operating system 612 and data 613, etc., and the storage method may be temporary storage or permanent storage. Among them, the operating system 612 may be Windows or other types of operating systems. Data 613 may include, but is not limited to, the data involved in the above method.

[0218] In some embodiments, the image recognition device may further include a display screen 630 , a power supply 640 , a communication interface 650 , an input / output interface 660 , a sensor 670 , and a communication bus 680 .

[0219] Those skilled in the art will understand that Figure 6 The structure shown in the figure does not constitute a limitation on the image recognition device, and may include more or fewer components than those shown in the figure.

[0220] The image recognition device provided by the embodiment of the present invention includes a memory and a processor. When the processor executes the program stored in the memory, it can implement the steps of the image recognition method provided by the above embodiment, and the effect is the same as above.

[0221] An embodiment of the present invention provides a non-volatile storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the image recognition method provided in any one of the above embodiments can be implemented.

[0222] The non-volatile storage medium may include: a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and other media that can store program codes.

[0223] For the introduction of the non-volatile storage medium provided in the embodiment of the present invention, please refer to the above method embodiment, and the effect thereof is the same as the image recognition method provided in the embodiment of the present invention, and the present invention will not elaborate on it here.

[0224] An embodiment of the present invention provides a computer program product, including a computer program, which, when executed by a processor, implements the steps of the image recognition method provided in any one of the above embodiments.

[0225] For an introduction to the computer program product provided by the embodiment of the present invention, please refer to the above method embodiment, and the effect thereof is the same as the image recognition method provided by the embodiment of the present invention, and the present invention will not elaborate on it here.

[0226] The above is a detailed introduction to an image recognition method, device, equipment, medium and computer program product provided by the present invention. The various embodiments in the specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same and similar parts between the embodiments can be referenced to each other. For the devices, equipment, non-volatile storage media and computer program products disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple, and the relevant parts can be referred to the method part description. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of the present invention, the present invention can also be improved and modified in a number of ways, and these improvements and modifications also fall within the scope of protection of the present invention.

[0227] It should also be noted that, in this specification, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the statement "comprises a ..." does not exclude the presence of other identical elements in the process, method, article or device including the element.

Claims

1. An image recognition method, characterized in that: include: Training a visual perception model using an image sample set including image samples at multiple resolutions; During the iterative training process, a visual perception loss value and a multi-resolution gradient regularization term of the visual perception model are calculated based on the perception loss value of the visual perception model for the image samples of each resolution, and a model learning loss value calculated based on the visual perception loss value and the multi-resolution gradient regularization term is used to update the model parameters of the visual perception model until the iterative training end condition is reached, thereby obtaining the trained visual perception model; Inputting the input image of the visual perception task to be processed into the trained visual perception model to obtain a visual perception result of the perception target of the visual perception task to be processed; Among them, the multi-resolution gradient regularization term is calculated by the gradient regularization term corresponding to each of the resolutions, and the gradient regularization term is the product of a function of the resolution similarity coefficient corresponding to the two resolutions and a function of the gradient angle between the gradients corresponding to the two resolutions, so that if the similarity between the two resolutions is higher, the gradient regularization term is used to enhance the similarity constraint between the two resolutions; if the similarity between the two resolutions is lower, the gradient regularization term is used to enhance the gradient orthogonality between the two resolutions.

2. The image recognition method according to claim 1, characterized in that: The resolution similarity coefficient is calculated based on the image heights corresponding to the two resolutions and the image widths corresponding to the two resolutions.

3. The image recognition method according to claim 2, characterized in that: The resolution similarity coefficient is calculated based on a ratio of image heights corresponding to the two resolutions and a ratio of image widths corresponding to the two resolutions.

4. The image recognition method according to claim 2, characterized in that: The resolution similarity coefficient is a difference between a first preset constant and a first fractional term; Among them, the numerator of the first fractional term is the maximum value of the difference between the image heights corresponding to the two resolutions and the difference between the image widths corresponding to the two resolutions, and the denominator of the first fractional term is the maximum value of the image heights corresponding to the two resolutions and the image widths corresponding to the two resolutions.

5. The image recognition method according to claim 1, characterized in that: If the similarity between the two resolutions is higher, the gradient regularization term is used to enhance the similarity constraint between the two resolutions, including: If the resolution similarity coefficient is greater than the resolution similarity threshold, the function of the resolution similarity coefficient is positively correlated with the resolution similarity coefficient, and the function of the gradient angle is positively correlated with the gradient angle; If the similarity between the two resolutions is lower, the gradient regularization term is used to enhance the gradient orthogonality between the two resolutions, including: If the resolution similarity coefficient is less than or equal to the resolution similarity threshold, and the gradient angles corresponding to the two resolutions are less than the gradient angle threshold, then the function of the resolution similarity coefficient is negatively correlated with the resolution similarity coefficient, and the function of the gradient angle is negatively correlated with the gradient angle; If the resolution similarity coefficient is less than or equal to the resolution similarity threshold, and the gradient angles corresponding to the two resolutions are greater than or equal to the gradient angle threshold, then the function of the resolution similarity coefficient is negatively correlated with the resolution similarity coefficient, and the function of the gradient angle is positively correlated with the gradient angle.

6. The image recognition method according to claim 5, characterized in that: If the resolution similarity coefficient is greater than the resolution similarity threshold, then the function of the resolution similarity coefficient is a directly proportional function of the resolution similarity coefficient; If the resolution similarity coefficient is less than or equal to the resolution similarity threshold, the function of the resolution similarity coefficient is a direct proportional function of a second preset constant minus the resolution similarity coefficient.

7. The image recognition method according to claim 5, characterized in that: If the resolution similarity coefficient is greater than the resolution similarity threshold, the function of the gradient angle is 1 minus the difference of the cosine function of the gradient angle; If the resolution similarity coefficient is less than or equal to the resolution similarity threshold, and the gradient angles corresponding to the two resolutions are less than the gradient angle threshold, the function of the gradient angle is 1 plus half of the sum of the cosine function of the gradient angle; If the resolution similarity coefficient is less than or equal to the resolution similarity threshold, and the gradient angles corresponding to the two resolutions are greater than or equal to the gradient angle threshold, then the function of the gradient angle is the negative value of the cosine function of the gradient angle.

8. The image recognition method according to claim 1, characterized in that: The multi-resolution gradient regularization term is the sum of the gradient regularization terms corresponding to all pairwise combinations of the resolutions corresponding to the image sample set.

9. The image recognition method according to claim 1, characterized in that: The step of collecting the image sample set includes: Get the original image set; Processing each original image in the original image set with a different scaling factor to obtain a scaled image with a different resolution corresponding to each original image; Each of the original images and the zoomed images corresponding to the original images are used as the image sample set.

10. The image recognition method according to claim 9, characterized in that: Processing each original image in the original image set with a different scaling factor to obtain a scaled image with a different resolution corresponding to each original image, including: Processing each of the original images using a reference scaling factor to obtain a reference scaling image corresponding to the original image; Sampling a plurality of the scaling factors from a preset distribution range, and processing each of the original images using each of the scaling factors respectively, to obtain the scaled images of the plurality of resolutions corresponding to each of the original images; The corresponding reference zoomed image and the corresponding zoomed images are used as an image set corresponding to the original image to obtain the image set corresponding to each original image.

11. The image recognition method according to claim 9, characterized in that: The difference between the two scaling factors is not less than a scaling factor difference threshold.

12. The image recognition method according to claim 9, characterized in that: The scaling factors all belong to a preset distribution range and the difference between any two of the scaling factors is not less than a scaling factor difference threshold.

13. The image recognition method according to claim 12, characterized in that: The scaling factor difference threshold is calculated based on the interval length of the preset distribution range divided by twice the number of scaling factors.

14. The image recognition method according to claim 9, characterized in that: Processing each original image in the original image set with a different scaling factor to obtain a scaled image with a different resolution corresponding to each original image, including: Processing each of the original images using a reference scaling factor to obtain a reference scaling image corresponding to the original image; Sampling a plurality of the scaling factors from a preset distribution range until a preset number of the scaling factors are collected; In the process of collecting the zoom coefficients, each time a new zoom coefficient is collected, the difference between the new zoom coefficient and the existing zoom coefficient is calculated, and if the difference is not less than the zoom coefficient difference threshold, the new zoom coefficient is added to the set of the existing zoom coefficients; if the difference is less than the zoom coefficient difference threshold, the new zoom coefficient is discarded; Processing each of the original images using each of the scaling factors respectively to obtain the scaled images of the plurality of resolutions corresponding to each of the original images; The corresponding reference zoomed image and the corresponding zoomed images are used as an image set corresponding to the original image to obtain the image set corresponding to each original image.

15. The image recognition method according to claim 1, characterized in that: The step of calculating the perceptual loss value of the image sample of each resolution by the visual perception model comprises: The image samples corresponding to the resolutions are input into different thread pools one by one, so as to obtain the perception loss value of the visual perception model for the image samples of the resolutions in parallel.

16. The image recognition method according to claim 1, characterized in that: The step of calculating the visual perception loss value comprises: Update the loss function corresponding to each of the resolutions according to the perceptual loss value of the image samples of each of the resolutions by the visual perception model; The visual perception loss value of the current iteration number is calculated according to the loss function corresponding to each of the resolutions.

17. An image recognition device, characterized in that: include: A model training unit, which trains a visual perception model using an image sample set including image samples of multiple resolutions; During the iterative training process, a visual perception loss value and a multi-resolution gradient regularization term of the visual perception model are calculated based on the perception loss value of the visual perception model for the image samples of each resolution, and a model learning loss value calculated based on the visual perception loss value and the multi-resolution gradient regularization term is used to update the model parameters of the visual perception model until the iterative training end condition is reached, thereby obtaining the trained visual perception model; A recognition unit, used for inputting an input image of a visual perception task to be processed into the trained visual perception model to obtain a visual perception result of a perception target of the visual perception task to be processed; Among them, the multi-resolution gradient regularization term is calculated by the gradient regularization term corresponding to each other between the resolutions, and the gradient regularization term is the product of a function of the resolution similarity coefficient corresponding to the two resolutions and a function of the gradient angle between the gradients corresponding to the two resolutions, so that if the similarity between the two resolutions is higher, the gradient regularization term is used to enhance the similarity constraint between the two resolutions; if the similarity between the two resolutions is lower, the gradient regularization term is used to enhance the gradient orthogonality between the two resolutions.

18. An image recognition device, characterized in that: include: Memory for storing computer programs; A processor is used to execute the computer program, and when the computer program is executed by the processor, the steps of the image recognition method according to any one of claims 1 to 16 are implemented.

19. A non-volatile storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the image recognition method according to any one of claims 1 to 16 are implemented.

20. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the image recognition method according to any one of claims 1 to 16 are implemented.

Images