Image detection method, device, equipment, medium and computer program product

Abstract

The present invention relates to the field of image processing technology, and specifically discloses an image detection method, device, equipment, medium and computer program product. When training a visual perception model, an image sample set including image samples of multiple resolutions is used for training. In the iterative training process, the loss function corresponding to each resolution is updated according to the perceived loss value of the visual perception model for the image samples of each resolution, thereby reducing the interference between image samples of different resolutions. The model learning loss value of the current iteration number is calculated according to the loss function corresponding to each resolution, and the complementary advantages of the loss functions of each resolution are brought into play, so that the visual perception model learns the perception ability of input images of different resolutions, and improves the resolution generalization ability of the visual perception model. The trained visual perception model is used to perform visual perception tasks, ensure the perception accuracy of input images of different resolutions, and improve the applicability of computer visual perception in different applications.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to an image detection method, device, equipment, medium and computer program product. Background Art

[0002] Computer vision is an important branch of artificial intelligence technology. One of its main tasks is to obtain corresponding scene information by processing the collected images. It is widely used in autonomous driving, intelligent robots, smart homes, smart terminals, as well as industrial production such as transportation, aerial remote sensing, and industrial product assembly.

[0003] The visual perception model is one of the core technologies of computer vision. The perception ability of the visual perception model determines the visual understanding ability of artificial intelligence. Due to the differences in requirements of different application scenarios, different image samples need to be used when training visual perception models for different application scenarios. For example, for application scenarios with limited computing resources, low-resolution image samples are usually used to train visual perception models, while for perception scenarios of small targets and long-distance targets, high-resolution image samples are used to train visual perception models. This results in that in practical applications, once the visual perception model is deployed, it can only process input images with a fixed resolution. It is impossible to reduce the computing burden and ensure real-time performance by reducing the resolution of the input image, nor is it possible to increase the resolution of the input image to provide more detailed information, which limits the practical application of the visual perception model.

[0004] How to improve the resolution generalization capability of computer vision to adapt to input images of different resolutions is a technical problem that technicians in this field need to solve. Summary of the invention

[0005] The object of the present invention is to provide an image detection method, apparatus, device, medium and computer program product for improving the resolution generalization capability of computer vision to adapt to input images of different resolutions.

[0006] In order to solve the above technical problems, the present invention provides an image detection method, comprising:

[0007] training a visual perception model using an image sample set including image samples at multiple resolutions;

[0008] During the iterative training process, the loss function corresponding to each resolution is updated according to the perceived loss value of the visual perception model for the image samples of each resolution, and the model learning loss value of the current iteration number is calculated according to the loss function corresponding to each resolution, and the model parameters of the visual perception model are updated using the model learning loss value until the iterative training end condition is reached, thereby obtaining the trained visual perception model;

[0009] The input image of the visual perception task to be processed is input into the trained visual perception model to obtain a visual perception result of the perception target of the visual perception task to be processed.

[0010] On the one hand, updating the loss function corresponding to each resolution according to the perceptual loss value of the image sample of each resolution by the visual perception model includes:

[0011] The self-learning loss weight in the loss function corresponding to each resolution is updated according to the perceptual loss value of the image sample of each resolution by the visual perception model.

[0012] On the other hand, the loss function corresponding to the resolution consists of a first fractional term and a first logarithmic term, the numerator of the first fractional term is the perceptual loss value of the visual perception model for the image sample of the resolution, the denominator of the first fractional term is the self-learning loss weight corresponding to the resolution, and the real number of the first logarithmic term is a function of the self-learning loss weight corresponding to the resolution.

[0013] On the other hand, the loss function corresponding to the resolution is calculated according to the self-learning loss weight corresponding to the resolution, the learning difficulty coefficient of the visual perception model for the image sample of the resolution, and the perceptual loss value of the visual perception model for the image sample of the resolution;

[0014] The higher the resolution, the easier it is for the visual perception model to learn image samples of the resolution.

[0015] On the other hand, the learning difficulty coefficient corresponding to the resolution is calculated based on the relationship between the spatial size of the image sample of the resolution and the spatial sizes of other image samples of the resolution.

[0016] On the other hand, the loss function corresponding to the resolution is composed of a second fractional term and a first logarithmic term, the numerator of the second fractional term is the product of the perceptual loss value of the visual perception model for the image sample of the resolution and the learning difficulty coefficient corresponding to the resolution, and the denominator of the second fractional term is the self-learning loss weight corresponding to the resolution; the real number of the first logarithmic term is a function of the self-learning loss weight corresponding to the resolution.

[0017] On the other hand, the loss function corresponding to the resolution is calculated according to the learning rate of the visual perception model for the image samples of the resolution and the perceptual loss value of the visual perception model for the image samples of the resolution;

[0018] The learning rate corresponding to the resolution is the rate of change of the perceptual loss value of the image samples of the resolution in adjacent iterative training times of the visual perception model.

[0019] On the other hand, the learning rate of the resolution is the ratio of the perceptual loss value of the visual perception model for image samples of the resolution in the current iterative training to the perceptual loss value of the visual perception model for image samples of the resolution in the previous iterative training.

[0020] On the other hand, the loss function corresponding to the resolution is calculated according to the learning difficulty coefficient of the visual perception model for the image samples of the resolution, the learning rate of the visual perception model for the image samples of the resolution, and the perceptual loss value of the visual perception model for the image samples of the resolution;

[0021] Among them, the higher the resolution, the easier it is for the visual perception model to learn image samples of the resolution; the learning rate corresponding to the resolution is the rate of change of the perceptual loss value of the image samples of the resolution in adjacent iterative training times of the visual perception model.

[0022] On the other hand, the loss function corresponding to the resolution is calculated according to the self-learning loss weight corresponding to the resolution, the learning rate of the visual perception model for the image samples of the resolution, and the perceptual loss value of the visual perception model for the image samples of the resolution;

[0023] The learning rate corresponding to the resolution is the rate of change of the perceptual loss value of the image samples of the resolution in adjacent iterative training times of the visual perception model.

[0024] On the other hand, the loss function corresponding to the resolution is calculated according to the self-learning loss weight corresponding to the resolution, the learning difficulty coefficient of the visual perception model for the image samples of the resolution, the learning rate of the visual perception model for the image samples of the resolution, and the perceptual loss value of the visual perception model for the image samples of the resolution;

[0025] Among them, the higher the resolution, the easier it is for the visual perception model to learn image samples of the resolution; the learning rate corresponding to the resolution is the rate of change of the perceptual loss value of the image samples of the resolution in adjacent iterative training times of the visual perception model.

[0026] On the other hand, the loss function corresponding to the resolution is a weighted sum of the fourth fractional term and the fifth fractional term;

[0027] The numerator of the fourth fractional term is the product of the perceptual loss value of the visual perception model for the image sample of the resolution and the learning difficulty coefficient corresponding to the resolution, and the denominator of the fourth fractional term is the self-learning loss weight corresponding to the resolution; the fifth fractional term is the average of the learning difficulty coefficient corresponding to the resolution and the normalized learning rate corresponding to the resolution;

[0028] The weight of the fourth fractional item and the weight of the fifth fractional item are both coefficients that are adaptively updated during the iterative training of the visual perception model.

[0029] On the other hand, updating the loss function corresponding to each resolution according to the perceptual loss value of the image sample of each resolution by the visual perception model includes:

[0030] In the current iterative training, judging whether the visual perception model satisfies a risk judgment condition of falling into a local optimum according to a learning rate of the visual perception model for image samples corresponding to each of the resolutions;

[0031] If the risk judgment condition of falling into a local optimum is met, a random weight is set for the loss function of each resolution in the current iterative training;

[0032] If the risk judgment condition of falling into a local optimum is not met, then in the current iterative training, the self-learning loss weight in the loss function corresponding to each resolution is updated according to the perceptual loss value of the image sample of each resolution by the visual perception model;

[0033] The learning rate corresponding to the resolution is the rate of change of the perceptual loss value of the image samples of the resolution in adjacent iterative training times of the visual perception model.

[0034] On the other hand, in the current iterative training, judging whether the visual perception model satisfies the risk judgment condition of falling into a local optimum according to the learning rate of the visual perception model for the image samples corresponding to each resolution includes:

[0035] In each iterative training of the visual perception model, if the number of the resolutions whose learning rate is less than the learning rate threshold exceeds the resolution number threshold, the number of abnormal iterative training times is increased by one;

[0036] If the abnormal number of iterative trainings corresponding to the historical iterative training reaches the abnormal number threshold, it is determined that the visual perception model meets the risk judgment condition of falling into the local optimum; otherwise, the visual perception model does not meet the risk judgment condition of falling into the local optimum.

[0037] On the other hand, in the current iterative training, random weights are set for the loss functions of each resolution, including:

[0038] Randomly sampling the same number of random sampling values ​​as the number of types of the resolution from a preset probability distribution;

[0039] The normalized random sampling value is used as the random weight of the resolution.

[0040] On the other hand, it also includes:

[0041] Determining the learning rates of different network layers of the visual perception model according to the change rates of model parameters of different network layers of the visual perception model in adjacent iterative training times;

[0042] Standardizing the learning rates of the network layers according to the learning rates of the different network layers to obtain standardized processing parameters;

[0043] When updating the model parameters of the visual perception model, the model parameters of the corresponding network layer are updated according to the standardized processing parameters to reduce the learning rate differences between different network layers.

[0044] On the other hand, the model learning loss value of the current iteration number is calculated according to the loss function corresponding to each of the resolutions, including:

[0045] The average of the loss values ​​of the loss functions of the resolutions is taken as the model learning loss value.

[0046] On the other hand, the visual perception model is an autonomous driving perception model;

[0047] The image sample set is constructed based on image samples collected by a vehicle-mounted sensor at different viewing angles.

[0048] In order to solve the above technical problems, the present invention further provides an image detection device, comprising:

[0049] A model training unit is used to train a visual perception model using an image sample set including image samples of multiple resolutions; in an iterative training process, the loss function corresponding to each resolution is updated according to the perceptual loss value of the visual perception model for the image samples of each resolution, and the model learning loss value of the current iteration number is calculated according to the loss function corresponding to each resolution, and the model parameters of the visual perception model are updated using the model learning loss value until the iterative training end condition is reached, thereby obtaining the trained visual perception model;

[0050] The perception processing unit is used to input the input image of the visual perception task to be processed into the trained visual perception model to obtain the visual perception result of the perception target of the visual perception task to be processed.

[0051] In order to solve the above technical problems, the present invention further provides an image detection device, comprising:

[0052] Memory for storing computer programs;

[0053] A processor is used to execute the computer program, and when the computer program is executed by the processor, the steps of the image detection method as described in any one of the above items are implemented.

[0054] In order to solve the above technical problems, the present invention also provides a non-volatile storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the image detection method as described in any one of the above items are implemented.

[0055] In order to solve the above technical problem, the present invention also provides a computer program product, including a computer program, and when the computer program is executed by a processor, the steps of the image detection method as described in any one of the above items are implemented.

[0056] The image detection method provided by the present invention has the beneficial effect of using an image sample set including image samples of multiple resolutions for training when training a visual perception model, and updating the loss function corresponding to each resolution according to the perceived loss value of the visual perception model for image samples of each resolution during the iterative training process, thereby reducing the interference between image samples of different resolutions, and calculating the model learning loss value of the current iteration number according to the loss function corresponding to each resolution, giving play to the complementary advantages of the loss functions of each resolution, so that the visual perception model learns the perception ability of input images of different resolutions, and improves the resolution generalization ability of the visual perception model. By using the trained visual perception model to perform the visual perception task to be processed, the resolution of the input image can be set as needed, and high-precision perception results of the perceived target can be obtained, which greatly improves the applicability of the visual perception model of computer vision in different applications.

[0057] The present invention also provides an image detection device, equipment, medium and computer program product, which have the above-mentioned beneficial effects and are not described in detail here. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] In order to more clearly illustrate the embodiments of the present invention or the technical solutions of the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0059] Figure 1 A flowchart of an image detection method provided by an embodiment of the present invention;

[0060] Figure 2 A schematic diagram of the structure of an image detection system provided by an embodiment of the present invention;

[0061] Figure 3 A training flow chart of a visual perception model provided by an embodiment of the present invention;

[0062] Figure 4 A schematic diagram of the structure of an image detection device provided by an embodiment of the present invention;

[0063] Figure 5 A schematic diagram of the structure of an image detection device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0064] The core of the present invention is to provide an image detection method, device, equipment, medium and computer program product for improving the resolution generalization capability of computer vision to adapt to input images of different resolutions.

[0065] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0066] Resolution is used to describe the clarity of an image, video or display device. It indicates the number of pixels that can be displayed per unit length. Resolution is usually expressed in the following forms:

[0067] Pixel resolution: This is the most intuitive way of expressing it, which is expressed directly in pixels. For example, 1920x1080 means there are 1920 pixels in the horizontal direction (image width) and 1080 pixels in the vertical direction (image height).

[0068] Screen resolution: Usually used to describe the resolution of a monitor, TV or other display device. For example, Full HD usually refers to a resolution of 1920x1080, while 4K resolution usually refers to 3840x2160.

[0069] Print resolution: In the printing field, resolution is usually expressed in dots per inch (DPI). For example, 300 DPI means there are 300 dots per inch.

[0070] Image scanning resolution: The resolution of the scanner is also expressed in DPI, which determines the clarity of the scanned image.

[0071] Video resolution: The resolution of a video is usually expressed in pixels, such as 720p (1280x720), 1080p (1920x1080), and 4K (3840x2160 or 4096x2160).

[0072] The image recognition method provided in the embodiment of the present invention can be used to process images captured by a visual sensor such as a camera, or to process video frames captured from a captured video. In the embodiment of the present invention, the resolution described can be pixel resolution.

[0073] The resolution generalization ability of a visual perception model refers to the ability of the visual perception model to process input images with resolutions that are inconsistent with the training resolution during the testing phase.

[0074] Traditional computer vision solutions usually use the same resolution in the model training and testing stages. This is because the resolution generalization ability of visual perception models is poor. Specifically, when the test resolution is inconsistent with the training resolution, the model accuracy will be seriously reduced. The greater the difference between the test resolution and the training resolution, the more serious the reduction in model accuracy, which limits the application of visual perception models in different scenarios. The fundamental reason for this phenomenon is that the distribution of image data with different resolutions is different, and current visual perception models have difficulty in handling such distribution differences.

[0075] However, in practical applications, the visual perception model after deployment may need to process input images with higher resolutions. For example, a visual perception model trained with a resolution of (256x704) may need to process input images with a resolution of (512x1408) or a resolution of (128x352) after deployment. To meet the application needs of processing images with higher resolutions, one current solution is to adjust the input image to the same size as the resolution of the image sample in the training phase during the application phase, that is, adjust the input image with a resolution of (512x1408) and the input image with a resolution of (128x352) to a resolution of (256x704).

[0076] Although this solution of unifying the resolution of input images is feasible, in some practical application scenarios, there is a high demand for different resolution settings. For some application scenarios with limited computing resources, the visual perception model is required to process low-resolution input images (such as 128x352 resolution in the above example) to reduce the computational burden and ensure real-time performance. If the input image resolution is increased to align it to the training resolution, the computational burden will increase and affect the real-time performance of the system. For some application scenarios with sufficient computing resources, it is hoped that a higher image resolution (such as 512x1408 resolution in the above example) will be used to improve the accuracy of 3D target detection. By reducing the high resolution of the image to align it to the training resolution, a lot of detail information will be lost, which may be detrimental to the detection of small and distant targets.

[0077] It can be seen that in actual computer vision applications, there is still a great demand for the resolution generalization capability of the visual perception model itself. However, the resolution generalization capability of the current visual perception model is very poor and it is difficult to directly apply it to scenes with different resolutions.

[0078] In order to improve the resolution generalization ability of the visual perception model, it is necessary to use image samples of different resolutions to train the visual perception model. However, due to the large differences in the feature distribution of image samples of different resolutions, model optimization is difficult to succeed.

[0079] In order to meet the needs of computer vision applications to process input images of different resolutions, an embodiment of the present invention provides an image detection solution. Through an adaptive learning mechanism, the loss functions corresponding to different resolutions are automatically adjusted during the training process of the visual perception model to avoid interference between different loss functions. The visual perception model is used to combine the loss functions of different resolutions to obtain the model learning loss value based on the iterative training and update of the model, and the complementary advantages of the loss functions of each resolution are brought into play, so that the visual perception model can learn the perception ability of input images of different resolutions, and improve the resolution generalization ability of the visual perception model. By using the trained visual perception model to perform the visual perception task to be processed, the resolution of the input image can be set as needed, and high-precision perception results of the perceived target can be obtained, which greatly improves the applicability of the visual perception model of computer vision in different applications.

[0080] The image detection method provided by the embodiment of the present invention is described below with reference to the accompanying drawings.

[0081] Figure 1 A flowchart of an image detection method provided by an embodiment of the present invention; Figure 2 A schematic diagram of the structure of an image detection system provided by an embodiment of the present invention; Figure 3A training flow chart of a visual perception model provided by an embodiment of the present invention.

[0082] like Figure 1 As shown, the image detection method provided by the embodiment of the present invention includes:

[0083] S101: training a visual perception model using an image sample set including image samples of multiple resolutions;

[0084] S102: During the iterative training process, the loss function corresponding to each resolution is updated according to the perceptual loss value of the visual perception model for the image samples of each resolution, and the model learning loss value of the current iteration number is calculated according to the loss function corresponding to each resolution, and the model parameters of the visual perception model are updated using the model learning loss value until the iterative training end condition is reached, thereby obtaining the trained visual perception model;

[0085] S103: Inputting the input image of the visual perception task to be processed into the trained visual perception model to obtain a visual perception result of the perception target of the visual perception task to be processed.

[0086] The image detection method provided by the embodiment of the present invention mainly includes two major links: training of the visual perception model and application of the visual perception model. In this regard, the image detection method provided by the embodiment of the present invention can be applied to Figure 2 The image detection system shown in FIG. 1 includes two parts: a model training system and a perception computing device.

[0087] The model training system may be composed of one or more computing devices for training a visual perception model based on an image sample set composed of image samples of multiple resolutions. Figure 2 As shown, the model training system can be composed of multiple threads of multiple artificial intelligence devices to form a thread pool, and the image samples corresponding to each resolution are assigned to a thread pool, so as to achieve parallel accelerated learning of multiple resolution image sets. It should be noted that m, n, and k only represent the meaning of multiple, and do not represent a specific number or the same number of threads of each artificial intelligence processor, nor do they mean that each thread pool is composed of one thread in each artificial intelligence device. The model training system can be deployed in a cloud computing center. The device structure of the cloud computing center is similar Figure 2 The perceptual computing devices in the system include computing, storage, network and other infrastructure, operating system, database, middleware and other software environments and application software. The model training system receives image data sets of various resolutions, uses the stored pre-trained models, combined with manual annotation, to annotate visual perception targets (such as the category, location, size of the bounding box, target orientation information, etc.) and stores the annotated data.

[0088] The perception computing device can be a computing device deployed in a cloud computing center or an edge computing device deployed in an edge device. Its structure is as follows: Figure 2 As shown in the perception computing device shown. If the computing device of the cloud computing center is used, after the trained visual perception model is deployed, the edge device collects the image to be detected through a visual sensor such as a camera and uploads it to the cloud computing center, and uses the perception computing device that deploys the visual perception model to perform visual perception calculation on the image to be detected to obtain the visual perception result of the visual perception target. If the perception computing device uses an edge computing device deployed on the edge device, the visual perception model is trained by the model training system and sent to the edge computing device through the communication unit, and deployed in the memory of the edge computing device. After the edge computing device collects the image to be detected through a visual sensor such as a camera, it performs visual perception calculation through the local visual perception model to obtain the visual perception result of the visual perception target.

[0089] Applied to Figure 2 The multi-thread pool shown, in an embodiment of the present invention, the step of calculating the perceptual loss value of the visual perception model for image samples of each resolution may include: inputting the image samples corresponding to each resolution into different thread pools one by one, so as to obtain the perceptual loss value of the visual perception model for the image samples of each resolution by parallel calculation.

[0090] like Figure 3 As shown, in an embodiment of the present invention, when training a visual perception model, all images of each resolution can be stacked to obtain a set of input images of different resolutions, so that the training speed can be improved through batch processing.

[0091] In some optional implementations of the embodiments of the present invention, calculating the model learning loss value of the current iteration number according to the loss function corresponding to each resolution in S102 may include: taking the average of the loss values ​​of the loss function of each resolution as the model learning loss value.

[0092] like Figure 3 As shown, for each resolution, all image samples are input into the visual perception model to obtain the corresponding visual perception results, and the loss value of the visual perception model for each resolution compared with the true value (which can be manually labeled) is calculated through the loss function. Finally, the model learning loss value of the visual perception model can be the average of the perceptual loss values ​​of the visual perception model for image samples of all resolutions, as shown in the following formula:

[0093] ;

[0094] in, represents the loss function of the visual perception model, represents the perceptual loss value corresponding to the r-th resolution, represents the loss function corresponding to the rth resolution, and k+1 represents the number of resolutions used in visual perception model training.

[0095] It is very challenging to directly use input images of multiple resolutions to train visual perception models. This is because there are large differences in data distribution between input images of different resolutions, and there are large differences in the distribution of image features extracted by the middle layer of the visual perception model, resulting in differences in the final prediction results. As a result, there are differences in the calculated loss function values, and there is interference between different resolutions, which increases the difficulty of model optimization. In this regard, an embodiment of the present invention provides a multi-resolution loss weight adaptive adjustment method, which automatically adjusts the loss functions corresponding to different resolutions during the training process through an adaptive learning mechanism, thereby avoiding interference between different loss functions.

[0096] In some optional implementations of the embodiments of the present invention, updating the loss function corresponding to each resolution according to the perceptual loss value of the image samples of each resolution based on the visual perception model in S102 may include: updating the self-learning loss weight in the loss function corresponding to each resolution according to the perceptual loss value of the image samples of each resolution based on the visual perception model.

[0097] That is to say, by setting self-learning loss weights for image samples of each resolution, in the iterative training of the visual perception model, not only the model parameters of the visual perception model are updated through loss optimization, but also the self-learning loss weights corresponding to each resolution are updated, thereby automatically adjusting the loss functions corresponding to different resolutions to avoid mutual interference between loss functions corresponding to different resolutions.

[0098] In practical applications, the loss function corresponding to the resolution may be a function of the self-learning loss weight corresponding to the resolution and the perceptual loss value of the visual perception model for image samples of each resolution.

[0099] For S102, the iterative training end condition may be that the number of iterative training times of the visual perception model reaches a preset number of iterations, or the model learning loss value of the visual perception model is less than a preset loss value.

[0100] The image detection method provided by the embodiment of the present invention is applied. When training the visual perception model, an image sample set including image samples of multiple resolutions is used for training. In the iterative training process, the loss function corresponding to each resolution is updated according to the perceived loss value of the visual perception model for the image samples of each resolution, thereby reducing the interference between image samples of different resolutions. The model learning loss value of the current iteration number is calculated according to the loss function corresponding to each resolution, and the complementary advantages of the loss functions of each resolution are brought into play, so that the visual perception model learns the perception ability of input images of different resolutions, and improves the resolution generalization ability of the visual perception model. By using the trained visual perception model to perform the visual perception task to be processed, the resolution of the input image can be set as needed, and high-precision perception results of the perceived target can be obtained, which greatly improves the applicability of the visual perception model of computer vision in different applications.

[0101] Based on the above embodiment, the embodiment of the present invention further illustrates the steps of collecting the image sample set.

[0102] The image sample set used for training the visual perception model in the embodiment of the present invention includes image samples of multiple resolutions. When collecting the image sample set, a visual sensor such as a camera can be used to directly collect image samples of different resolutions through set resolution parameters to form an image sample set; when collecting the image sample set, image samples of one resolution can also be collected first, and image samples of multiple resolutions can be generated by scaling or enlarging to form an image sample set.

[0103] In some optional implementations of the embodiments of the present invention, the step of collecting the image sample set may include: obtaining an original image set; processing each original image in the original image set using different scaling factors to obtain scaled images of different resolutions corresponding to each original image; and using each original image and the scaled image corresponding to each original image as the image sample set.

[0104] In practical applications, each original image in the original image set is processed with different scaling factors to obtain scaled images of different resolutions corresponding to each original image, which may include: processing each original image with a reference scaling factor to obtain a reference scaled image corresponding to the original image; sampling multiple scaling factors from a preset distribution range, and processing each original image with each scaling factor to obtain scaled images of multiple resolutions corresponding to each original image; using the corresponding reference scaled image and each corresponding scaled image as an image set corresponding to the original image to obtain an image set corresponding to each original image. That is, the generated image sample set is composed of image sets corresponding to different original images, which include image samples of different resolutions, and different image samples of the same resolution may be obtained by processing different original images.

[0105] As described in the above embodiments of the present invention, the resolution used for processing by the visual perception model of computer vision tasks is usually expressed in pixel resolution, that is, expressed as the product of the number of pixels in the horizontal direction and the number of pixels in the vertical direction. In the embodiments of the present invention, the scaling factor may include a scaling factor acting on the image height and / or a scaling factor on the image width. In order to obtain image samples of multiple resolutions with greater resolution differences, the scaling factor may include an image height scaling factor and an image width scaling factor.

[0106] To reduce the computational difficulty, the scaling factor can also be a scaling ratio for the image height and image width. For example, for an original image with a resolution of H×W , the base scaling factor can be used The processed resolution is After the image is distributed in Uniform distribution Sample k scaling factors , and process the image using each scaling factor , and finally obtain a collection of images with multiple resolutions ,in The resolution is . Where H represents the vertical size of the image, and W represents the horizontal size of the image.

[0107] When applied to the field of autonomous driving, cameras and other visual sensors are usually set up in different directions of the vehicle, such as left front (fl), front (f), right front (fr), left rear (bl), rear (b), right rear (br), etc., so that images of multiple perspectives of the road scene are collected each time. In the actual training process, in order to use the batch method to accelerate model training, the embodiment of the present invention samples a set of scaling factors at each iteration, and , the above-obtained scaling coefficient set is used for processing, that is, a multi-resolution image set is obtained for each viewing angle .

[0108] Since the resolution difference between the multi-resolution image samples used in training the visual perception model is small, it is not conducive to the training of the model's resolution generalization ability. In order to obtain as many resolutions with large differences as possible and improve the training effect of the model's resolution generalization ability, in other optional implementations of the embodiments of the present invention, the difference between the scaling factors can be set to be no less than the scaling factor difference threshold, that is, when determining the scaling factor, the difference between the scaling factor and other scaling factors in the collected scaling factor set is judged, and only when the difference is greater than the set scaling factor difference threshold is the scaling factor added to the scaling factor set, thereby ensuring the diversity between the scaling factors obtained by sampling and improving the training effect of the model's resolution generalization ability.

[0109] Furthermore, the scaling factors may be designed to all belong to a preset distribution range and the difference between any two scaling factors may not be less than a scaling factor difference threshold.

[0110] In some optional implementations of the embodiments of the present invention, the scaling factor difference threshold is calculated based on the interval length of the preset distribution range divided by twice the number of scaling factors. That is, the scaling factor difference threshold can be expressed by the following formula:

[0111] ;

[0112] Where M represents the scaling factor difference threshold, Indicates the lower limit of the preset distribution range. represents the upper limit of the preset distribution range, and k represents the total number of scaling factors to be collected from the preset distribution range.

[0113] In some other optional implementations of the embodiments of the present invention, each original image in the original image set is processed using a different scaling factor to obtain a scaled image of different resolutions corresponding to each original image, which may also include: using a benchmark scaling factor to process each original image to obtain a benchmark scaled image corresponding to the original image; sampling multiple scaling factors from a preset distribution range until a preset number of scaling factors are collected; in the process of collecting scaling factors, each time a new scaling factor is collected, the difference between the new scaling factor and the existing scaling factor is calculated, and if the difference is not less than the scaling factor difference threshold, the new scaling factor is added to the set of existing scaling factors; if the difference is less than the scaling factor difference threshold, the new scaling factor is discarded; using each scaling factor to process each original image respectively to obtain scaled images of multiple resolutions corresponding to each original image; using the corresponding benchmark scaling image and the corresponding scaled images as the image set corresponding to the original image to obtain the image set corresponding to each original image.

[0114] That is, the following algorithm can be designed to obtain the set of scaling factors:

[0115] Input: reference scaling factor , distributed in uniform distribution , scaling factor set A, scaling factor difference threshold M;

[0116] Add the reference scaling factor to the scaling factor set A ;

[0117] If the length of the scaling factor combination A < k + 1, then:

[0118] Randomly sample from the uniform distribution to obtain ;

[0119] Check the difference between and all the scaling factors in the current scaling factor set A. If the difference is greater than the threshold scaling factor difference M, then add to the scaling factor set A;

[0120] Output: scaling factor set A.

[0121] To further improve the ability of the visual perception model to process input images of different resolutions, an image sample set can be generated multiple times during the model training process. That is to say, the scaling factors can be re-collected to generate a new image sample set in each iteration training, or the scaling factors can be re-collected to generate a new image sample set after every few iteration trainings.

[0122] However, in the embodiments of the present invention, since the model learning loss function of the visual perception model is obtained by designing the loss function combination of each resolution, if a new image sample set is regenerated by the above method in each iteration training, it may cause the model to be difficult to converge.

[0123] To ensure model convergence, in some optional implementation manners of the embodiments of the present invention, an image sample set can be generated一次性 before the training of the visual perception model, and the image sample set is not updated during the subsequent iteration training. Then, in each iteration training of the visual perception model, the same batch of resolutions are learned, and the diversity of the samples can be ensured by collecting different image samples corresponding to the resolutions. If computing resources and time permit, an image sample set can be generated一次性 before the training of the visual perception model, but multiple rounds of iteration training are performed, and a new image sample set is regenerated before the start of each round of iteration training, and the same image sample set is used in different iteration training steps during each round of iteration training.

[0124] In some other optional implementations of the embodiments of the present invention, the image sample set may be regenerated every multiple iterations of training, and the frequency of generating the image sample set may be determined according to the convergence of the visual perception model.

[0125] In some other optional implementations of the embodiments of the present invention, if the image sample set is regenerated in each iterative training or every few iterative trainings, the multiple resolutions corresponding to the resampling scaling factors can arrange the resolutions in a fixed order (such as from small to large or from large to small) according to the scaling factors, thereby ensuring the correspondence between the arrangement of the resolutions and the arrangement of the loss function, so that the model training can inherit the experience of the previous iterative training or the previous few iterative trainings.

[0126] In each embodiment of the present invention, the loss function corresponding to the resolution of the visual perception model in each iterative training number inherits the parameters of the loss function of the corresponding position in the previous iterative training, but the corresponding resolution can be the resolution in the previous iterative training or the updated resolution.

[0127] On the basis of the above embodiments, in order to adapt to the problem that multi-resolution models are difficult to optimize, the embodiments of the present invention continue to explain the method for updating the self-learning loss weights.

[0128] As introduced in the above embodiments, the loss function corresponding to the resolution can be a function of the self-learning loss weight corresponding to the resolution and the perceived loss value of the visual perception model for image samples of each resolution. In some optional implementations of the embodiments of the present invention, the loss function corresponding to the resolution can be composed of a first fractional term and a first logarithmic term, the numerator of the first fractional term is the perceived loss value of the visual perception model for image samples of the resolution, the denominator of the first fractional term is the self-learning loss weight corresponding to the resolution, and the real number of the first logarithmic term is a function of the self-learning loss weight corresponding to the resolution.

[0129] That is to say, the loss function of the visual perception model can be expressed as follows:

[0130] ;

[0131] in, represents the loss function of the visual perception model, represents the perceptual loss value corresponding to the r-th resolution, represents the self-learning loss weight corresponding to the rth resolution, and k+1 represents the number of resolutions used in visual perception model training.

[0132] By adding the first logarithmic term, the training process of the visual perception model can be made more stable, which can have the effect of punishing overfitting.

[0133] Since the difficulty of learning image samples of different resolutions for the visual perception model is different, the embodiment of the present invention also proposes a learning difficulty coefficient to further balance the learning effect of the visual perception model at different resolutions. In practical applications, the higher the resolution of the input image, the easier it is to optimize the visual perception model, that is, the lower the learning difficulty. Accordingly, the loss function corresponding to the resolution can also be calculated based on the self-learning loss weight corresponding to the resolution, the learning difficulty coefficient of the visual perception model for image samples of the resolution, and the perceived loss value of the visual perception model for image samples of the resolution; wherein, the higher the resolution, the lower the learning difficulty of the visual perception model for image samples of the resolution.

[0134] That is to say, according to the relationship between the learning difficulty of the visual perception model and the resolution of the input image, the learning difficulty coefficient is added to the loss function of different resolutions to balance the learning difficulty of the visual perception model for image samples with different resolutions, so as to further balance the loss value at each resolution and improve the training effect of the visual perception model on multi-resolution images.

[0135] In some other optional implementations of the embodiments of the present invention, the loss function corresponding to the resolution can also be composed of a second fractional term and a first logarithmic term, the numerator of the second fractional term is the product of the perceptual loss value of the visual perception model for the image sample of the resolution and the learning difficulty coefficient corresponding to the resolution, and the denominator of the second fractional term is the self-learning loss weight corresponding to the resolution; the real number of the first logarithmic term is a function of the self-learning loss weight corresponding to the resolution.

[0136] That is to say, the loss function of the visual perception model can also be expressed as follows:

[0137] ;

[0138] in, represents the learning difficulty coefficient corresponding to the rth resolution, Represents the self-learning loss weight corresponding to the r-th resolution.

[0139] The learning difficulty coefficient corresponding to the resolution can be calculated based on the relationship between the spatial size of the image sample of the resolution and the spatial size of the image sample of other resolutions. Then the learning difficulty coefficient can be calculated by the following formula:

[0140] ;

[0141] in, , , A collection representing the number of pixels of height for all resolutions, A collection of width pixels for all resolutions. Indicates the number of height pixels corresponding to the rth resolution, Indicates the maximum number of height pixels in the image sample set, Indicates the number of pixels of width corresponding to the rth resolution, Indicates the maximum number of pixels of width in the image sample set. Indicates the larger value of the maximum number of height pixels in the image sample set and the maximum number of width pixels in the image sample set.

[0142] It can be seen that when adding the learning difficulty coefficient After that, for higher resolution input images, the learning difficulty of the visual perception model is smaller and the value of the loss function is smaller, so the learning difficulty coefficient The value of approaches 1, then The adjustment of the perceptual loss value is small. For lower resolution input images, the learning difficulty of the visual perception model is greater, and the value of the loss function is larger, so the learning difficulty coefficient The value of is in the range (0,1) and decreases as the resolution decreases. The adjustment of the perceptual loss value is large to avoid the excessively high loss function value at low resolution from dominating and interfering with the training effect of the visual perception model on high-resolution images. It is a learnable parameter that is automatically adjusted during the training process through an adaptive learning mechanism to further balance the loss values ​​at each resolution and improve the training effect of the visual perception model on multi-resolution images. Therefore, in the image detection method provided by the embodiment of the present invention, by balancing the learning difficulty of the visual perception model for image samples of different resolutions to improve the training effect of the visual perception model on multi-resolution images, a more outstanding resolution generalization capability can be obtained in densely predicted perception tasks such as three-dimensional object detection.

[0143] As described in the above embodiments, the learning difficulty of the visual perception model on images of different resolutions is different. Generally, the learning difficulty on high-resolution images is lower because high-resolution images contain rich detail information, which is conducive to dense prediction tasks. The learning difficulty on low-resolution images is higher because low-resolution images lose a lot of detail information. Therefore, the learning speed of the visual perception model on images of different resolutions is different. The visual perception model can converge faster on high-resolution images, but converge slowly on low-resolution images.

[0144] In order to balance the learning progress of the visual perception model for images with different resolutions, an embodiment of the present invention provides a solution for optimizing the learning effect of the visual perception model for multi-resolution images from the perspective of balancing the learning rate.

[0145] In an embodiment of the present invention, the loss function corresponding to the resolution can be calculated based on the learning rate of the visual perception model for image samples of the resolution and the perceptual loss value of the visual perception model for image samples of the resolution; wherein the learning rate corresponding to the resolution is the rate of change of the perceptual loss value of image samples of the resolution in adjacent iterative training times of the visual perception model.

[0146] In some optional implementations of the embodiments of the present invention, the learning rate of resolution may be the ratio of the perceptual loss value of the visual perception model for the image sample of resolution in the current iterative training to the perceptual loss value of the visual perception model for the image sample of resolution in the previous iterative training. That is, the learning rate may be expressed by the following formula:

[0147] ;

[0148] in, represents the learning rate of the rth resolution in the tth iterative training of the visual perception model, represents the loss value of the r-th resolution in the t-th iterative training of the visual perception model, Represents the loss value of the r-th resolution in the t-1-th iterative training of the visual perception model.

[0149] In some other optional implementations of the embodiments of the present invention, the learning rate of resolution may also be the difference between the perceptual loss value of the visual perception model for image samples of resolution in the current iterative training and the perceptual loss value of the visual perception model for image samples of resolution in the previous iterative training.

[0150] For the convenience of calculation, the learning rates of all resolutions can be normalized. The normalized learning rate corresponding to the resolution can include a third fractional term, the numerator of which is an exponential term with a natural constant as the base and the learning rate corresponding to the resolution as the exponent, and the denominator of which is the sum of the exponential terms corresponding to each resolution. That is, the normalized learning rate can be shown as follows:

[0151] ;

[0152] in, It represents the normalized learning rate of the r-th resolution in the t-th iterative training of the visual perception model. The larger the value, the slower the learning rate.

[0153] After considering the learning rate, the loss function of the visual perception model can be designed as follows:

[0154] ;

[0155] in, represents the loss function of the visual perception model, Represents the perceptual loss value corresponding to the r-th resolution.

[0156] Considering that there is a certain relationship between the learning difficulty and learning rate of the visual perception model for image samples of different resolutions, in other optional implementations of the embodiments of the present invention, the loss function corresponding to the resolution can also be calculated based on the learning difficulty coefficient of the visual perception model for image samples of resolution, the learning rate of the visual perception model for image samples of resolution, and the perceptual loss value of the visual perception model for image samples of resolution; wherein, the higher the resolution, the lower the learning difficulty of the visual perception model for image samples of resolution; the learning rate corresponding to the resolution is the rate of change of the perceptual loss value of the image samples of resolution in the adjacent iterative training times of the visual perception model. Based on this, the loss function of the visual perception model can be designed as shown in the following formula:

[0157] ;

[0158] in, represents the loss function of the visual perception model, represents the perceptual loss value corresponding to the r-th resolution, represents the learning difficulty coefficient corresponding to the rth resolution, represents the normalized learning rate of the r-th resolution in the t-th iterative training of the visual perception model.

[0159] Learning difficulty It is calculated based on the resolution based on prior knowledge, and the learning rate It is calculated in real time during the training of the visual perception model. The combination of the two can better balance the training process of the visual perception model at different resolutions, thereby improving the multi-resolution generalization ability of the visual perception model.

[0160] In the above embodiment, the design idea of ​​updating the loss function corresponding to each resolution by designing self-learning loss weights, learning difficulty coefficients, and learning rates is introduced. Based on this, the form of the loss function of other visual perception models can be obtained by combining one or more types of parameters.

[0161] In some optional implementations of the embodiments of the present invention, the loss function corresponding to the resolution can be calculated based on the self-learning loss weight corresponding to the resolution, the learning rate of the visual perception model for the image samples of the resolution, and the perceptual loss value of the visual perception model for the image samples of the resolution; wherein the learning rate corresponding to the resolution is the rate of change of the perceptual loss value of the image samples of the resolution in the adjacent iterative training times of the visual perception model. Based on this, the loss function of the visual perception model can be designed as shown in the following formula:

[0162] ;

[0163] in, represents the loss function of the visual perception model, represents the perceptual loss value corresponding to the r-th resolution, represents the normalized learning rate of the rth resolution in the tth iterative training of the visual perception model, represents the self-learning loss weight corresponding to the r-th resolution, It is a learnable parameter that can be initialized to 0.5 and updated as the visual perception model is trained. It is used to weigh the two loss weights of the multi-resolution loss weight adaptation method and the loss weight adjustment method based on learning speed.

[0164] In some other optional implementations of the embodiments of the present invention, the loss function corresponding to the resolution can also be calculated based on the self-learning loss weight corresponding to the resolution, the learning difficulty coefficient of the visual perception model for image samples of the resolution, the learning rate of the visual perception model for image samples of the resolution, and the perceptual loss value of the visual perception model for image samples of the resolution; wherein, the higher the resolution, the smaller the learning difficulty of the visual perception model for image samples of the resolution; the learning rate corresponding to the resolution is the rate of change of the perceptual loss value of image samples of the resolution in adjacent iterative training times of the visual perception model.

[0165] Based on this, the loss function corresponding to the resolution can be designed as the weighted sum of the fourth and fifth fractional terms; the numerator of the fourth fractional term is the product of the perceived loss value of the image sample of the resolution by the visual perception model and the learning difficulty coefficient corresponding to the resolution, and the denominator of the fourth fractional term is the self-learning loss weight corresponding to the resolution; the fifth fractional term is the average of the learning difficulty coefficient corresponding to the resolution and the normalized learning rate corresponding to the resolution; the weight of the fourth fractional term and the weight of the fifth fractional term are both coefficients that are adaptively updated in the iterative training of the visual perception model. That is to say, the loss function of the visual perception model can also be expressed by the following formula:

[0166] ;

[0167] in, represents the loss function of the visual perception model, represents the perceptual loss value corresponding to the r-th resolution, represents the normalized learning rate of the rth resolution in the tth iterative training of the visual perception model, represents the learning difficulty coefficient corresponding to the rth resolution, represents the self-learning loss weight corresponding to the r-th resolution, is the weight of the fourth fraction, is the weight of the fifth fraction, It is a learnable parameter that can be initialized to 0.5 and updated as the visual perception model is trained. It is used to weigh the two loss weights of the multi-resolution loss weight adaptation method and the loss weight adjustment method based on learning speed.

[0168] Therefore, by combining multiple types of loss weights, the visual perception model can perform adaptive loss weight adjustment during iterative training, loss weight adjustment based on learning difficulty, and loss weight adjustment based on learning rate.

[0169] On the basis of the above embodiments, in order to prevent the visual perception model from falling into a local optimum during the training process, in the image detection method provided in the embodiment of the present invention, in S102, the loss function corresponding to each resolution is updated according to the perceived loss value of the visual perception model for image samples of each resolution, which may include: in the current iterative training, judging whether the visual perception model meets the risk judgment condition of falling into a local optimum according to the learning rate of the visual perception model for image samples corresponding to each resolution; if the risk judgment condition of falling into a local optimum is met, setting random weights for the loss function of each resolution in the current iterative training; if the risk judgment condition of falling into a local optimum is not met, updating the self-learning loss weights in the loss function corresponding to each resolution according to the perceived loss value of the image samples of each resolution by the visual perception model in the current iterative training; wherein the learning rate corresponding to the resolution is the rate of change of the perceived loss value of the image samples of the resolution in adjacent iterative training times of the visual perception model.

[0170] In a specific implementation, in the current iterative training, whether the visual perception model meets the risk judgment condition of falling into the local optimum is judged according to the learning rate of the visual perception model for the image samples corresponding to each resolution, which may include: in each iterative training of the visual perception model, if the number of resolutions whose learning rate is less than the learning rate threshold exceeds the resolution number threshold, then the number of abnormal iterative training is increased by one; if the number of abnormal iterative training corresponding to the historical iterative training reaches the abnormal number threshold, then it is determined that the visual perception model meets the risk judgment condition of falling into the local optimum, otherwise the visual perception model does not meet the risk judgment condition of falling into the local optimum.

[0171] Specifically, a learning speed set S can be set to collect the learning rates corresponding to all resolutions in each iterative training; a counter C can be set to count the number of resolutions with slow learning rates in each iterative training; a learning rate threshold λ can be set to determine whether the learning rate of each resolution is slow; a resolution number threshold μ can be set to determine whether the number of resolutions with slow learning rates in each iterative training is large enough; a queue Q with a maximum length of L can be set to collect information on whether the learning rate is slow in historical iterative training; an activation threshold , used to determine whether to perform random weight adjustment.

[0172] At the beginning of each iterative training of the visual perception model, the learning rate set S is first cleared, the counter C is set to 0, and then the elements in the queue Q are summed to obtain the number of iterations with slow learning rates in the historical iterative training. If this number is greater than the activation threshold , then random weight adjustment is performed, otherwise, the normal training process is performed, that is, the loss function of any resolution provided in the above embodiment can be adopted. If random weight adjustment is performed, random weights can be generated for each resolution in one or more subsequent iterative trainings to weight the perceived loss values ​​of each resolution and average them, to obtain the model learning loss value of the iterative training, and to update the model parameters using the back propagation algorithm.

[0173] In an embodiment of the present invention, setting random weights for the loss functions of each resolution in the current iterative training may include: randomly sampling the same number of random sampling values ​​as the number of resolution types from a preset probability distribution; and using the normalized random sampling values ​​as random weights for the resolution.

[0174] The preset probability distribution may adopt a standard Gaussian distribution or other types of probability distribution. Randomly sample k+1 values ​​from the preset probability distribution , followed by normalization, the normalization calculation method of the rth random weight can be shown as follows:

[0175] ;

[0176] The perceptual loss value of each resolution is weighted using a normalized random weight as shown below:

[0177] ;

[0178] in, represents the random weight corresponding to the rth resolution, represents the rth value sampled from a standard Gaussian distribution, represents the standard Gaussian distribution, represents the loss function of the visual perception model, Represents the perceptual loss value corresponding to the r-th resolution.

[0179] In the normal training process of the visual perception model, the perceptual loss value corresponding to each resolution is calculated , learning rate , and add it to the learning rate set S, then traverse the learning rate set S, normalize each learning rate, and obtain the normalized learning rate ,if >λ, then the counter C is incremented by 1. After traversing the learning rate set S, if the counter C>μ, then add the value 1 to the queue Q; otherwise, add the value 0 to the queue Q. Get the model learning loss value of the current training step:

[0180] .

[0181] Use back propagation to update the model parameters. Any loss function of the resolution provided in the above embodiments may be adopted.

[0182] By adopting the random weight updating method provided in the embodiment of the present invention, the iterative training process of the visual perception model can be designed to adopt the following algorithm:

[0183] Input: learning rate set S, counter C, learning rate threshold λ, resolution number threshold μ, random weight adjustment times N, queue Q=[] with maximum length L, activation threshold ;

[0184] Traverse all training samples until the training is completed:

[0185] Clear the learning rate set S and set the counter C to 0;

[0186] Sum the elements in queue Q, if the sum > activation threshold , then random weight adjustment is performed,

[0187] Perform N training iterations:

[0188] Iterate over all resolutions:

[0189] Get the loss function value of the current resolution for the current training iteration ;

[0190] Generate random weight values ​​for all resolutions , weight the corresponding resolution loss function value to obtain the loss value of the current training step: , back-propagation updates model parameters;

[0191] Clear the queue Q and exit the random weight adjustment;

[0192] Otherwise, perform the normal training steps:

[0193] Iterate over all resolutions:

[0194] Get the loss function value of the current resolution for the current training iteration , calculate the learning rate , and add it to the learning rate set S;

[0195] Iterate over all resolutions:

[0196] Get the learning rate of the current resolution from the learning rate set S , calculate the normalized learning rate ,if >λ, then counter C increases by 1;

[0197] If counter C>μ, add value 1 to queue Q; otherwise, add value 0 to queue Q;

[0198] Get the loss value of the current training step: , back-propagation updates the model parameters.

[0199] On the basis of the above embodiments, considering that different network layers of the visual perception model have different learning effects on multi-resolution training, the image detection method provided by the embodiment of the present invention may also include: determining the learning rates of different network layers of the visual perception model according to the change rates of model parameters of different network layers of the visual perception model in adjacent iterative training times; standardizing the learning rates of the network layers according to the learning rates of the different network layers to obtain standardized processing parameters; when updating the model parameters of the visual perception model, updating the model parameters of the corresponding network layers according to the standardized processing parameters to narrow the learning rate differences between different network layers.

[0200] That is to say, by monitoring the learning rates of different network layers of the visual perception model, the update rate of the corresponding network layer parameters is adaptively adjusted based on the learning rate.

[0201] In the specific implementation, for the perception model containing L layers, represents the parameters of the i-th layer in the t-th training step, represents the parameters of the i-th layer at step tl, then the parameter learning speed can be expressed as follows:

[0202] ;

[0203] The parameter learning rate of each network layer is standardized as shown in the following formula:

[0204] ;

[0205] in, is a smaller constant, such as 0.05 or 0.1.

[0206] The parameters of the i-th layer in the t-th training step , update the parameters according to the following formula:

[0207] ;

[0208] in, represents the learning rate of visual perception model training, represents the loss function of the visual perception model, represents the gradient of the parameters of the i-th layer at the t-1th training step.

[0209] In an embodiment of the present invention, the network layer parameter learning rate adjustment method proposed in the embodiment of the present invention can be applied to all network layers or part of the network layers of the visual perception model. For example, if the resources of the model training system are sufficient, the learning rates of all network layers can be monitored and adjusted. If the resources of the model training system are insufficient, only the learning rates of some network layers can be monitored and adjusted. For example, based on the learning rates of each network layer in the historical iterative training, the network layer with a large difference from the mean learning rate can be selected as the key monitoring object of the current iterative training.

[0210] The multi-resolution loss weight adaptive adjustment method, the learning rate-based loss weight adjustment method, the random loss weight adjustment method, the combination method of multiple loss weights, and the network layer parameter learning rate adjustment method provided in the above embodiments of the present invention are used to train the perception model, and the total loss function is set as follows:

[0211] ;

[0212] in, represents the loss function of the visual perception model, represents the perceptual loss value corresponding to the r-th resolution, represents the learning difficulty coefficient corresponding to the rth resolution, represents the self-learning loss weight corresponding to the r-th resolution, represents the normalized learning rate of the rth resolution in the tth iterative training of the visual perception model, is a learnable parameter, Represents the random weight corresponding to the r-th resolution.

[0213] In each iteration of the visual perception model training, the loss function is calculated The value of is used to optimize the model through the back propagation algorithm, wherein the optimizer may adopt a gradient descent (GD) optimizer, an adaptive moment estimation (Adam) optimizer, an adaptive weight decay regularization (Adam WeightDecay Regularization, AdamW) optimizer, etc. During training, the multi-resolution parallel acceleration method proposed in the above embodiment of the present invention may be adopted to improve the training efficiency. After the model training is completed, the multi-resolution loss weight adaptive adjustment method, the loss weight adjustment method based on the learning speed, the random loss weight adjustment method, the combination method of multiple loss weights, and the network layer parameter learning speed adjustment method are removed, and the perception model obtained by training is deployed to the edge computing platform of the unmanned vehicle to perform visual perception tasks.

[0214] The image detection method provided by the embodiment of the present invention can be applied but not limited to the field of autonomous driving. It can also be applied to smart terminals such as smart robots, smart homes, and mobile phones. It can also be applied to visual perception models involved in industrial production such as transportation, aerial remote sensing, and industrial product assembly.

[0215] For example, the visual perception model can be an autonomous driving perception model; in this regard, an image sample set can be constructed based on image samples collected by vehicle-mounted sensors from different perspectives, and the visual perception model can be trained on a collected and annotated road scene image dataset when executing model training in a cloud data center.

[0216] When performing reasoning tasks using a visual perception model, the visual perception model trained using the training method provided by an embodiment of the present invention can be deployed on a perception computing device, and the input image of the visual perception task to be processed can be directly input into the trained visual perception model. For example, the trained visual perception model can be deployed in an edge computing platform of an autonomous driving vehicle to obtain a visual perception result of the perception target of the visual perception task to be processed. Since the visual perception model acquires the processing capability of multi-resolution input images during the training phase, it can have good adaptability to input images of different resolutions in different application scenarios to ensure image detection accuracy.

[0217] It should be noted that in the embodiments of the image detection method of the present invention, some of the steps or features may be ignored or not executed. The hardware or software functional modules divided for the convenience of description are not the only implementation form of the image detection method provided by the embodiment of the present invention.

[0218] The above describes in detail various embodiments corresponding to the image detection method. On this basis, the present invention also discloses an image detection device, equipment, non-volatile storage medium and computer program product corresponding to the above method.

[0219] Figure 4 A schematic diagram of the structure of an image detection device provided by an embodiment of the present invention.

[0220] like Figure 4 As shown, the image detection device provided by the embodiment of the present invention includes:

[0221] The model training unit 401 is used to train the visual perception model using an image sample set including image samples of multiple resolutions; in the iterative training process, the loss function corresponding to each resolution is updated according to the perceptual loss value of the visual perception model for the image samples of each resolution, and the model learning loss value of the current iteration number is calculated according to the loss function corresponding to each resolution, and the model parameters of the visual perception model are updated using the model learning loss value until the iterative training end condition is reached, thereby obtaining the trained visual perception model;

[0222] The detection unit 402 is used to input the input image of the visual perception task to be processed into the trained visual perception model to obtain the visual perception result of the perception target of the visual perception task to be processed.

[0223] The image detection device provided by the embodiment of the present invention may further include:

[0224] A monitoring unit, used to determine the learning rates of different network layers of the visual perception model according to the change rates of model parameters of different network layers of the visual perception model in adjacent iteration training times;

[0225] A standardization processing unit, used for standardizing the learning rate of the network layer according to the learning rates of different network layers to obtain a standardized processing parameter;

[0226] The network layer updating unit is used to update the model parameters of the corresponding network layer according to the standardized processing parameters when updating the model parameters of the visual perception model, so as to narrow the learning rate differences between different network layers.

[0227] It should be noted that in each implementation of the image detection device provided by the embodiment of the present invention, the division of the units is only a logical functional division, and other division methods can be adopted. The connection method between different units can adopt electrical, mechanical or other connection methods. The separated units can be located in the same physical location or distributed on multiple network nodes. Each unit can be implemented in the form of hardware or in the form of a software functional unit. That is, part or all of the units provided by the embodiment of the present invention can be selected according to actual needs and the corresponding connection method or integration method can be adopted to achieve the purpose of the embodiment of the present invention.

[0228] Since the embodiments of the apparatus part correspond to the embodiments of the method part, please refer to the description of the embodiments of the method part for the embodiments of the apparatus part, which will not be repeated here.

[0229] Figure 5 A schematic diagram of the structure of an image detection device provided by an embodiment of the present invention.

[0230] like Figure 5 As shown, the image detection device provided by the embodiment of the present invention includes: a memory 510, used to store a computer program 511; a processor 520, used to execute the computer program 511, and when the computer program 511 is executed by the processor 520, the steps of the image detection method provided in any of the above embodiments are implemented.

[0231] Among them, the processor 520 may include one or more processing cores, such as a 3-core processor, an 8-core processor, etc. The processor 520 may be implemented in at least one hardware form of digital signal processing (DSP), field-programmable gate array (FPGA), and programmable logic array (PLA). The processor 520 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a central processing unit (CPU); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 520 may be integrated with a graphics processing unit (GPU), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 520 may also include an artificial intelligence (AI) processor, which is used to process computing operations related to machine learning.

[0232] The memory 510 may include one or more non-volatile storage media, which may be non-transitory. The memory 510 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In this embodiment, the memory 510 is at least used to store the following computer program 511, wherein the computer program 511 can implement the relevant steps in the image detection method disclosed in any of the aforementioned embodiments after being loaded and executed by the processor 520. In addition, the resources stored in the memory 510 may also include an operating system 512 and data 513, etc., and the storage method may be temporary storage or permanent storage. Among them, the operating system 512 may be Windows or other types of operating systems. Data 513 may include, but is not limited to, the data involved in the above method.

[0233] In some embodiments, the image detection device may further include a display screen 530 , a power supply 540 , a communication interface 550 , an input / output interface 560 , a sensor 570 , and a communication bus 580 .

[0234] Those skilled in the art will understand that Figure 5 The structure shown in the figure does not constitute a limitation on the image detection device, and may include more or fewer components than those shown in the figure.

[0235] The image detection device provided in the embodiment of the present invention includes a memory and a processor. When the processor executes the program stored in the memory, it can implement the steps of the image detection method provided in the above embodiment, and the effect is the same as above.

[0236] An embodiment of the present invention provides a non-volatile storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the image detection method provided in any one of the above embodiments can be implemented.

[0237] The non-volatile storage medium may include: a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and other media that can store program codes.

[0238] For the introduction of the non-volatile storage medium provided in the embodiment of the present invention, please refer to the above method embodiment, and the effect thereof is the same as the image detection method provided in the embodiment of the present invention, and the present invention will not elaborate on it here.

[0239] An embodiment of the present invention provides a computer program product, including a computer program, which, when executed by a processor, implements the steps of the image detection method provided in any one of the above embodiments.

[0240] For an introduction to the computer program product provided by the embodiment of the present invention, please refer to the above method embodiment, and the effect thereof is the same as the image detection method provided by the embodiment of the present invention, and the present invention will not elaborate on it here.

[0241] The above is a detailed introduction to an image detection method, device, equipment, medium and computer program product provided by the present invention. The various embodiments in the specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same and similar parts between the embodiments can be referred to each other. For the device, equipment, non-volatile storage medium and computer program product disclosed in the embodiment, since they correspond to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part description. It should be pointed out that for ordinary technicians in this technical field, without departing from the principle of the present invention, the present invention can also be improved and modified in a number of ways, and these improvements and modifications also fall within the scope of protection of the present invention.

[0242] It should also be noted that, in this specification, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the statement "comprises a ..." does not exclude the presence of other identical elements in the process, method, article or device including the element.

Claims

1. An image detection method, characterized in that: include: Training a visual perception model using an image sample set including image samples at multiple resolutions; During the iterative training process, the loss function corresponding to each resolution is updated according to the perceived loss value of the visual perception model for the image samples of each resolution, and the model learning loss value of the current iteration number is calculated according to the loss function corresponding to each resolution, and the model parameters of the visual perception model are updated using the model learning loss value until the iterative training end condition is reached, thereby obtaining the trained visual perception model; Inputting the input image of the visual perception task to be processed into the trained visual perception model to obtain a visual perception result of the perception target of the visual perception task to be processed; Among them, the loss function corresponding to the resolution includes at least one parameter of the self-learning loss weight corresponding to the resolution, the learning difficulty coefficient corresponding to the resolution, and the learning rate of the visual perception model for image samples of the resolution; the higher the resolution, the lower the learning difficulty of the visual perception model for image samples of the resolution; the learning rate corresponding to the resolution is the rate of change of the perceptual loss value of the image samples of the resolution in adjacent iterative training times of the visual perception model.

2. The image detection method according to claim 1, characterized in that: Updating the loss function corresponding to each resolution according to the perceptual loss value of the image sample of each resolution by the visual perception model includes: The self-learning loss weight in the loss function corresponding to each resolution is updated according to the perceptual loss value of the image sample of each resolution by the visual perception model.

3. The image detection method according to claim 2, characterized in that: The loss function corresponding to the resolution consists of a first fractional term and a first logarithmic term, the numerator of the first fractional term is the perceptual loss value of the visual perception model for the image sample of the resolution, the denominator of the first fractional term is the self-learning loss weight corresponding to the resolution, and the real number of the first logarithmic term is a function of the self-learning loss weight corresponding to the resolution.

4. The image detection method according to claim 1, characterized in that: The loss function corresponding to the resolution is calculated according to the self-learning loss weight corresponding to the resolution, the learning difficulty coefficient of the visual perception model for the image sample of the resolution, and the perceptual loss value of the visual perception model for the image sample of the resolution; The higher the resolution, the easier it is for the visual perception model to learn image samples of the resolution.

5. The image detection method according to claim 4, characterized in that: The learning difficulty coefficient corresponding to the resolution is calculated according to the relationship between the spatial size of the image sample of the resolution and the spatial sizes of other image samples of the resolution.

6. The image detection method according to claim 4, characterized in that: The loss function corresponding to the resolution is composed of a second fractional term and a first logarithmic term, the numerator of the second fractional term is the product of the perceptual loss value of the visual perception model for the image sample of the resolution and the learning difficulty coefficient corresponding to the resolution, and the denominator of the second fractional term is the self-learning loss weight corresponding to the resolution; the real number of the first logarithmic term is a function of the self-learning loss weight corresponding to the resolution.

7. The image detection method according to claim 1, characterized in that: The loss function corresponding to the resolution is calculated based on the learning rate of the visual perception model for the image samples of the resolution and the perceptual loss value of the visual perception model for the image samples of the resolution; The learning rate corresponding to the resolution is the rate of change of the perceptual loss value of the image samples of the resolution in adjacent iterative training times of the visual perception model.

8. The image detection method according to claim 7, characterized in that: The learning rate of the resolution is a ratio of a perceptual loss value of the visual perception model for image samples of the resolution in the current iterative training to a perceptual loss value of the visual perception model for image samples of the resolution in the previous iterative training.

9. The image detection method according to claim 1, characterized in that: The loss function corresponding to the resolution is calculated based on the learning difficulty coefficient of the visual perception model for the image samples of the resolution, the learning rate of the visual perception model for the image samples of the resolution, and the perceptual loss value of the visual perception model for the image samples of the resolution; Among them, the higher the resolution, the easier it is for the visual perception model to learn image samples of the resolution; the learning rate corresponding to the resolution is the rate of change of the perceptual loss value of the image samples of the resolution in adjacent iterative training times of the visual perception model.

10. The image detection method according to claim 1, characterized in that: The loss function corresponding to the resolution is calculated according to the self-learning loss weight corresponding to the resolution, the learning rate of the visual perception model for the image samples of the resolution, and the perceptual loss value of the visual perception model for the image samples of the resolution; The learning rate corresponding to the resolution is the rate of change of the perceptual loss value of the image samples of the resolution in adjacent iterative training times of the visual perception model.

11. The image detection method according to claim 1, characterized in that: The loss function corresponding to the resolution is calculated according to the self-learning loss weight corresponding to the resolution, the learning difficulty coefficient of the visual perception model for the image samples of the resolution, the learning rate of the visual perception model for the image samples of the resolution, and the perceptual loss value of the visual perception model for the image samples of the resolution; Among them, the higher the resolution, the easier it is for the visual perception model to learn image samples of the resolution; the learning rate corresponding to the resolution is the rate of change of the perceptual loss value of the image samples of the resolution in adjacent iterative training times of the visual perception model.

12. The image detection method according to claim 11, characterized in that: The loss function corresponding to the resolution is a weighted sum of the fourth fractional term and the fifth fractional term; The numerator of the fourth fractional term is the product of the perceptual loss value of the visual perception model for the image sample of the resolution and the learning difficulty coefficient corresponding to the resolution, and the denominator of the fourth fractional term is the self-learning loss weight corresponding to the resolution; the fifth fractional term is the average of the learning difficulty coefficient corresponding to the resolution and the normalized learning rate corresponding to the resolution; The weight of the fourth fractional item and the weight of the fifth fractional item are both coefficients that are adaptively updated during the iterative training of the visual perception model.

13. The image detection method according to claim 1, characterized in that: Updating the loss function corresponding to each resolution according to the perceptual loss value of the image sample of each resolution by the visual perception model includes: In the current iterative training, judging whether the visual perception model satisfies a risk judgment condition of falling into a local optimum according to a learning rate of the visual perception model for image samples corresponding to each of the resolutions; If the risk judgment condition of falling into a local optimum is met, a random weight is set for the loss function of each resolution in the current iterative training; If the risk judgment condition of falling into a local optimum is not met, then in the current iterative training, the self-learning loss weight in the loss function corresponding to each resolution is updated according to the perceptual loss value of the image sample of each resolution by the visual perception model; The learning rate corresponding to the resolution is the rate of change of the perceptual loss value of the image samples of the resolution in adjacent iterative training times of the visual perception model.

14. The image detection method according to claim 13, characterized in that: In the current iterative training, judging whether the visual perception model satisfies the risk judgment condition of falling into a local optimum according to the learning rate of the visual perception model for the image samples corresponding to each resolution includes: In each iterative training of the visual perception model, if the number of the resolutions whose learning rate is less than the learning rate threshold exceeds the resolution number threshold, the number of abnormal iterative training times is increased by one; If the abnormal number of iterative trainings corresponding to the historical iterative training reaches the abnormal number threshold, it is determined that the visual perception model meets the risk judgment condition of falling into the local optimum; otherwise, the visual perception model does not meet the risk judgment condition of falling into the local optimum.

15. The image detection method according to claim 13, characterized in that: In the current iterative training, random weights are set for the loss functions of each resolution, including: Randomly sampling the same number of random sampling values ​​as the number of types of the resolution from a preset probability distribution; The normalized random sampling value is used as the random weight of the resolution.

16. The image detection method according to claim 1, characterized in that: Also includes: Determining the learning rates of different network layers of the visual perception model according to the change rates of model parameters of different network layers of the visual perception model in adjacent iterative training times; Standardizing the learning rates of the network layers according to the learning rates of the different network layers to obtain standardized processing parameters; When updating the model parameters of the visual perception model, the model parameters of the corresponding network layer are updated according to the standardized processing parameters to reduce the learning rate differences between different network layers.

17. The image detection method according to claim 1, characterized in that: The model learning loss value of the current iteration number is calculated according to the loss function corresponding to each resolution, including: The average of the loss values ​​of the loss functions of the resolutions is taken as the model learning loss value.

18. The image detection method according to claim 1, characterized in that: The visual perception model is an autonomous driving perception model; The image sample set is constructed based on image samples collected by a vehicle-mounted sensor at different viewing angles.

19. An image detection device, characterized in that: include: A model training unit, for training a visual perception model using an image sample set including image samples of multiple resolutions; During the iterative training process, the loss function corresponding to each resolution is updated according to the perceived loss value of the visual perception model for the image samples of each resolution, and the model learning loss value of the current iteration number is calculated according to the loss function corresponding to each resolution, and the model parameters of the visual perception model are updated using the model learning loss value until the iterative training end condition is reached, thereby obtaining the trained visual perception model; A perception processing unit, used for inputting an input image of a visual perception task to be processed into the trained visual perception model to obtain a visual perception result of a perception target of the visual perception task to be processed; Among them, the loss function corresponding to the resolution includes at least one parameter of the self-learning loss weight corresponding to the resolution, the learning difficulty coefficient corresponding to the resolution, and the learning rate of the visual perception model for image samples of the resolution; the higher the resolution, the lower the learning difficulty of the visual perception model for image samples of the resolution; the learning rate corresponding to the resolution is the rate of change of the perceptual loss value of the image samples of the resolution in adjacent iterative training times of the visual perception model.

20. An image detection device, characterized in that: include: Memory for storing computer programs; A processor is used to execute the computer program, and when the computer program is executed by the processor, the steps of the image detection method according to any one of claims 1 to 18 are implemented.

21. A non-volatile storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the image detection method according to any one of claims 1 to 18 are implemented.

22. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the image detection method according to any one of claims 1 to 18 are implemented.

Images