A target detection method, device, equipment, medium and computer program product

Abstract

The present invention relates to the field of image processing technology, and specifically discloses a target detection method, device, equipment, medium and computer program product. A multi-resolution image correction network is added to a visual perception model. In the process of iterating the visual perception model using multi-resolution image samples, the multi-resolution image correction network is used to determine the image feature position encoding corresponding to the resolution according to the relationship between the image feature size corresponding to the resolution and the feature size of a reference image. According to the image feature position encoding, self-attention calculation is performed on the input image feature and the corrected image feature is output, so that the image features of different resolutions are corrected to be consistent with the reference image features, the interference between image samples of different resolutions is reduced when training the visual perception model, and when the trained visual perception model is used to perform the target detection task, the accurate target detection results can be guaranteed for the input images of different resolutions that need to be processed.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a target detection method, device, equipment, medium and computer program product. Background Art

[0002] Object detection is a common application of computer vision. Its main task is to obtain corresponding scene information by processing the collected images. It is widely used in autonomous driving, intelligent robots, smart homes, smart terminals, as well as industrial production such as transportation, aerial remote sensing, and industrial product assembly.

[0003] When performing target detection tasks, the visual perception model determines the accuracy of detection. In practical applications, it is sometimes necessary for the visual perception model to be able to process input images of different resolutions. For example, for application scenarios with limited computing resources, lower-resolution image samples are usually used to train the visual perception model, while for perception scenarios of small targets and long-distance targets, higher-resolution image samples are used to train the visual perception model. However, if the resolution of the input image of the visual perception model in the inference calculation stage is different from the resolution of the image samples in the training stage, the detection accuracy of the visual perception model will be significantly reduced. As a result, in practical applications, once the visual perception model is deployed, it can only process input images with a fixed resolution.

[0004] How to improve the resolution generalization capability of adapting to input images of different resolutions in target detection tasks is a technical problem that technicians in this field need to solve. Summary of the invention

[0005] The purpose of the present invention is to provide a target detection method, device, equipment, medium and computer program product for improving the resolution generalization capability of adapting to input images of different resolutions in target detection tasks.

[0006] In order to solve the above technical problems, the present invention provides a target detection method, comprising:

[0007] Training a visual perception model to which a multi-resolution image correction network is added using an image sample set including image samples of multiple resolutions;

[0008] In iterative training, for each resolution, an image sample of the resolution is input into the visual perception model and a perception loss value is calculated, and in the multi-resolution image correction network, an image feature position encoding corresponding to the resolution is determined according to a relationship between an image feature size corresponding to the resolution and a reference image feature size in the image sample set, and self-attention calculation is performed on image features input into the multi-resolution image correction network according to the image feature position encoding corresponding to the resolution, and corrected image features are output;

[0009] Using the perceptual loss values ​​corresponding to each of the resolutions, the parameters of the visual perception model and the parameters of the multi-resolution image correction network are updated to obtain the visual perception model corresponding to the current iterative training batch;

[0010] After reaching the iterative training end condition, obtaining the trained visual perception model;

[0011] The input image corresponding to the target detection task is input into the trained visual perception model to obtain the target detection result of the target detection task.

[0012] On the one hand, determining the image feature position encoding corresponding to the resolution according to the relationship between the image feature size corresponding to the resolution and the reference image feature size in the image sample set includes:

[0013] Using a first feature mapping network, mapping the image feature size corresponding to the resolution and the reference image feature size is performed to obtain a first feature mapping result;

[0014] Generate a first size transformation coefficient according to the first feature mapping result using a first size transformation prediction network;

[0015] The first size transformation coefficient is used to perform size transformation processing on the current image feature position code to obtain the image feature position code corresponding to the resolution.

[0016] On the other hand, generating a first size transformation coefficient according to the first feature mapping result using a first size transformation prediction network includes:

[0017] A first scaling factor prediction network is used to generate a first scaling factor according to the first feature mapping result, and the first scaling factor is used as the first size transformation coefficient.

[0018] On the other hand, generating a first size transformation coefficient according to the first feature mapping result using a first size transformation prediction network includes:

[0019] Generate a first scaling factor according to the first feature mapping result using a first scaling factor prediction network;

[0020] Generate a first translation coefficient according to the first feature mapping result using a first translation coefficient prediction network;

[0021] performing normalization processing on the first scaling factor and the first translation factor respectively;

[0022] The first normalized scaling factor and the first normalized translation factor are used as the first size conversion factor.

[0023] On the other hand, updating the parameters of the multi-resolution image correction network using the perceptual loss values ​​corresponding to the resolutions includes:

[0024] The perceptual loss values ​​corresponding to the resolutions are used to update parameters of the first feature mapping network and parameters of the first size transformation prediction network.

[0025] On the other hand, in the multi-resolution image correction network, the image feature position code corresponding to the resolution is determined according to the relationship between the image feature size corresponding to the resolution and the reference image feature size in the image sample set, self-attention calculation is performed on the image features input to the multi-resolution image correction network according to the image feature position code corresponding to the resolution, and the corrected image features are output, including:

[0026] In the multi-resolution image correction network, self-attention calculation is performed on image features input into the multi-resolution image correction network using the image feature position encoding corresponding to the resolution and the embedding vector corresponding to the resolution, and the corrected image features are output;

[0027] The step of determining the embedding vector corresponding to the resolution includes:

[0028] Perform mapping processing according to the image feature corresponding to the resolution, the image feature size corresponding to the resolution, and the reference image feature size to obtain a second feature mapping result;

[0029] Generate a second size transformation coefficient according to the second feature mapping result using a second size transformation prediction network;

[0030] The current embedding vector is resized using the second resize coefficient to obtain the embedding vector corresponding to the resolution.

[0031] On the other hand, mapping processing is performed according to the image feature corresponding to the resolution, the image feature size corresponding to the resolution, and the reference image feature size to obtain a second feature mapping result, including:

[0032] Performing global mean pooling processing and global maximum pooling processing on the image features corresponding to the resolution respectively, to obtain a global mean pooling feature vector corresponding to the resolution and a global maximum pooling feature vector corresponding to the resolution;

[0033] Calculating a mean value of the image features corresponding to the resolution and a standard deviation of the image features corresponding to the resolution;

[0034] Perform channel dimension stitching processing on the global mean pooling feature vector corresponding to the resolution, the global maximum pooling feature vector corresponding to the resolution, the mean of the image feature corresponding to the resolution, the standard deviation of the image feature corresponding to the resolution, the image feature size corresponding to the resolution, and the reference image feature size to obtain a channel dimension stitching result;

[0035] The channel dimension splicing result is input into a second feature mapping network, and the second feature mapping result is output.

[0036] On the other hand, generating a second size transformation coefficient according to the second feature mapping result using a second size transformation prediction network includes:

[0037] A second scaling factor prediction network is used to generate a second scaling factor according to the second feature mapping result, and the second scaling factor is used as the second size transformation coefficient.

[0038] On the other hand, generating a second size transformation coefficient according to the second feature mapping result using a second size transformation prediction network includes:

[0039] Generate a second scaling factor according to the second feature mapping result using a second scaling factor prediction network;

[0040] Generate a second translation coefficient according to the second feature mapping result using a second translation coefficient prediction network;

[0041] performing normalization processing on the second scaling factor and the second translation factor respectively;

[0042] The second normalized scaling coefficient and the second normalized translation coefficient are used as the second size conversion coefficient.

[0043] On the other hand, updating the parameters of the multi-resolution image correction network using the perceptual loss values ​​corresponding to the resolutions includes:

[0044] The parameters of the second feature mapping network are updated using the perceptual loss values ​​corresponding to each of the resolutions.

[0045] On the other hand, performing self-attention calculation on the image features input into the multi-resolution image correction network using the image feature position encoding corresponding to the resolution and the embedding vector corresponding to the resolution, including:

[0046] The query vector position code in the image feature position code corresponding to the resolution is concatenated with the embedding vector corresponding to the resolution to obtain a multi-resolution query vector;

[0047] The multi-resolution query vector is input into the self-attention layer for self-attention calculation.

[0048] On the other hand, the parameters of the visual perception model and the parameters of the multi-resolution image correction network are updated using the perceptual loss values ​​corresponding to each of the resolutions to obtain the visual perception model corresponding to the current iterative training batch, including:

[0049] The regularization constraint item corresponding to each of the resolutions is calculated according to the size transformation coefficient of each of the resolutions compared with the characteristic size of the reference image;

[0050] The parameters of the visual perception model and the parameters of the multi-resolution image correction network are updated according to the perceptual loss value corresponding to each resolution and the regularization constraint item corresponding to each resolution to obtain the visual perception model corresponding to the current iterative training batch.

[0051] On the other hand, the size transformation coefficient corresponding to the resolution makes the transformation of the image feature size corresponding to the resolution smaller, and the regularization constraint item corresponding to the resolution approaches 0.

[0052] On the other hand, it also includes:

[0053] The weight coefficient of the regularization constraint term is updated during the iterative training of the visual perception model.

[0054] On the other hand, the image feature position encoding includes query vector position encoding and value vector position encoding.

[0055] On the other hand, the image feature position encoding includes query vector position encoding, key vector position encoding and value vector position encoding.

[0056] In order to solve the above technical problems, the present invention further provides a target detection device, comprising:

[0057] A model training unit is used to train a visual perception model with a multi-resolution image correction network using an image sample set including image samples of multiple resolutions; in iterative training, for each resolution, input the image sample of the resolution into the visual perception model and calculate the perceptual loss value, and in the multi-resolution image correction network, determine the image feature position encoding corresponding to the resolution according to the relationship between the image feature size corresponding to the resolution and the reference image feature size in the image sample set, perform self-attention calculation on the image features input to the multi-resolution image correction network according to the image feature position encoding corresponding to the resolution, and output the corrected image features; update the parameters of the visual perception model and the parameters of the multi-resolution image correction network using the perceptual loss value corresponding to each resolution, and obtain the visual perception model corresponding to the current iterative training batch; after the iterative training end condition is met, obtain the trained visual perception model;

[0058] The target detection unit is used to input the input image corresponding to the target detection task into the trained visual perception model to obtain the target detection result of the target detection task.

[0059] In order to solve the above technical problems, the present invention further provides a target detection device, comprising:

[0060] Memory for storing computer programs;

[0061] A processor is used to execute the computer program, and when the computer program is executed by the processor, the steps of the target detection method as described in any one of the above items are implemented.

[0062] In order to solve the above technical problems, the present invention also provides a non-volatile storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the target detection method as described in any one of the above items are implemented.

[0063] In order to solve the above technical problems, the present invention also provides a computer program product, including a computer program, which implements the steps of any one of the target detection methods described above when executed by a processor.

[0064] The target detection method provided by the present invention has the beneficial effect of adding a multi-resolution image correction network to a visual perception model. In the process of iteratively training the visual perception model using multi-resolution image samples, the multi-resolution image correction network is used to determine the image feature position encoding corresponding to the resolution based on the relationship between the image feature size corresponding to the resolution and the reference image feature size in the image sample set. According to the image feature position encoding, self-attention calculation is performed on the input image features and the corrected image features are output, thereby correcting the image features of different resolutions to be consistent with the reference image features. When training the visual perception model, the interference between image samples of different resolutions can be reduced, so that when the trained visual perception model is used to perform target detection tasks, accurate target detection results can be guaranteed for input images of different resolutions that need to be processed.

[0065] The present invention also provides a target detection device, equipment, medium and computer program product, which have the above-mentioned beneficial effects and are not described in detail here. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] In order to more clearly illustrate the embodiments of the present invention or the technical solutions of the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0067] Figure 1 A flow chart of a target detection method provided by an embodiment of the present invention;

[0068] Figure 2 A schematic diagram of the structure of a target detection system provided by an embodiment of the present invention;

[0069] Figure 3 A flowchart of multi-resolution loss optimization provided by an embodiment of the present invention;

[0070] Figure 4 A schematic diagram of the structure of a visual perception model with a multi-resolution image correction network added provided by an embodiment of the present invention;

[0071] Figure 5 A training flow chart of a visual perception model provided by an embodiment of the present invention;

[0072] Figure 6 A schematic diagram of the structure of a multi-resolution image correction network provided by an embodiment of the present invention;

[0073] Figure 7 An optimization flow chart of query vector position encoding provided by an embodiment of the present invention;

[0074] Figure 8 An optimization flow chart of value vector position encoding provided by an embodiment of the present invention;

[0075] Fig. 9 A schematic diagram of the structure of another multi-resolution image correction network provided by an embodiment of the present invention;

[0076] Fig.10 An optimization flow chart of an embedding vector provided by an embodiment of the present invention;

[0077] Fig.11 A schematic diagram of the structure of a target detection device provided by an embodiment of the present invention;

[0078] Fig.12 A schematic diagram of the structure of a target detection device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0079] The core of the present invention is to provide a target detection method, device, equipment, medium and computer program product for improving the resolution generalization capability of adapting to input images of different resolutions in target detection tasks.

[0080] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0081] Resolution is used to describe the clarity of an image, video or display device. It indicates the number of pixels that can be displayed per unit length. Resolution is usually expressed in the following forms:

[0082] Pixel resolution: This is the most intuitive way of expressing it, which is expressed directly in pixels. For example, 1920x1080 means there are 1920 pixels in the horizontal direction (image width) and 1080 pixels in the vertical direction (image height).

[0083] Screen resolution: Usually used to describe the resolution of a monitor, TV or other display device. For example, Full HD usually refers to a resolution of 1920x1080, while 4K resolution usually refers to 3840x2160.

[0084] Print resolution: In the printing field, resolution is usually expressed in dots per inch (DPI). For example, 300 DPI means there are 300 dots per inch.

[0085] Image scanning resolution: The resolution of the scanner is also expressed in DPI, which determines the clarity of the scanned image.

[0086] Video resolution: The resolution of a video is usually expressed in pixels, such as 720p (1280x720), 1080p (1920x1080), and 4K (3840x2160 or 4096x2160).

[0087] The object detection method provided in the embodiment of the present invention can be used to process images captured by a visual sensor such as a camera, or to process video frames captured from a captured video. In the embodiment of the present invention, the resolution described can be pixel resolution.

[0088] The resolution generalization ability of a visual perception model refers to the ability of the visual perception model to process input images with resolutions that are inconsistent with the training resolution during the testing phase.

[0089] Traditional object detection solutions usually use the same resolution in the model training and testing stages. This is because the resolution generalization ability of visual perception models is poor. Specifically, when the test resolution is inconsistent with the training resolution, the model accuracy will be seriously reduced. The greater the difference between the test resolution and the training resolution, the more serious the reduction in model accuracy, which limits the application of visual perception models in different scenarios. The fundamental reason for this phenomenon is that the distribution of image data with different resolutions is different, and current visual perception models have difficulty in handling such distribution differences.

[0090] However, in practical applications, the visual perception model after deployment may need to process input images with higher resolutions. For example, a visual perception model trained with a resolution of (256x704) may need to process input images with a resolution of (512x1408) or a resolution of (128x352) after deployment. To meet the application needs of processing images with higher resolutions, one current solution is to adjust the input image to the same size as the resolution of the image sample in the training phase during the application phase, that is, adjust the input image with a resolution of (512x1408) and the input image with a resolution of (128x352) to a resolution of (256x704).

[0091] Although this solution of unifying the resolution of input images is feasible, in some practical application scenarios, there is a high demand for different resolution settings. For some application scenarios with limited computing resources, the visual perception model is required to process low-resolution input images (such as 128x352 resolution in the above example) to reduce the computational burden and ensure real-time performance. If the input image resolution is increased to align it to the training resolution, the computational burden will increase and affect the real-time performance of the system. For some application scenarios with sufficient computing resources, it is hoped that a higher image resolution (such as 512x1408 resolution in the above example) will be used to improve the accuracy of 3D target detection. By reducing the high resolution of the image to align it to the training resolution, a lot of detail information will be lost, which may be detrimental to the detection of small and distant targets.

[0092] It can be seen that in actual object detection applications, there is still a great demand for the resolution generalization capability of the visual perception model itself. However, the resolution generalization capability of the current visual perception model is very poor and it is difficult to directly apply it to scenes with different resolutions.

[0093] In order to improve the resolution generalization ability of the visual perception model, it is necessary to use image samples of different resolutions to train the visual perception model. However, due to the large differences in the feature distribution of image samples of different resolutions, model optimization is difficult to succeed.

[0094] In order to meet the needs of target detection tasks for processing input images of different resolutions, an embodiment of the present invention provides a target detection solution, which adds a multi-resolution image correction network to a visual perception model. In the process of iteratively training the visual perception model using multi-resolution image samples, the multi-resolution image correction network is used to determine the image feature position encoding corresponding to the resolution according to the relationship between the image feature size corresponding to the resolution and the reference image feature size in the image sample set. According to the image feature position encoding, self-attention calculation is performed on the input image features and the corrected image features are output, thereby correcting the image features of different resolutions to be consistent with the reference image features. When training the visual perception model, the interference between image samples of different resolutions can be reduced. Therefore, when the trained visual perception model is used to perform the target detection task, accurate target detection results can be guaranteed for input images of different resolutions that need to be processed.

[0095] The target detection method provided by the embodiment of the present invention is described below with reference to the accompanying drawings.

[0096] Figure 1 A flow chart of a target detection method provided by an embodiment of the present invention; Figure 2 A schematic diagram of the structure of a target detection system provided by an embodiment of the present invention; Figure 3 A flowchart of multi-resolution loss optimization is provided in an embodiment of the present invention.

[0097] like Figure 1 As shown, the target detection method provided by the embodiment of the present invention includes:

[0098] S101: training a visual perception model to which a multi-resolution image correction network is added using an image sample set including image samples of multiple resolutions;

[0099] S102: In iterative training, for each resolution, an image sample of the resolution is input into a visual perception model and a perception loss value is calculated, and in a multi-resolution image correction network, an image feature position encoding corresponding to the resolution is determined according to a relationship between an image feature size corresponding to the resolution and a reference image feature size in an image sample set, and self-attention calculation is performed on image features input into the multi-resolution image correction network according to the image feature position encoding corresponding to the resolution, and corrected image features are output;

[0100] S103: using the perceptual loss values ​​corresponding to each resolution to update the parameters of the visual perception model and the parameters of the multi-resolution image correction network, to obtain the visual perception model corresponding to the current iterative training batch;

[0101] S104: after reaching the iterative training end condition, obtaining a trained visual perception model;

[0102] S105: Inputting the input image corresponding to the target detection task into the trained visual perception model to obtain the target detection result of the target detection task.

[0103] The target detection method provided by the embodiment of the present invention mainly includes two major links: training of the visual perception model and application of the visual perception model. In this regard, the target detection method provided by the embodiment of the present invention can be applied to Figure 2 The target detection system shown in FIG. 1 includes two parts: a model training system and a perception computing device.

[0104] The model training system may be composed of one or more computing devices for training a visual perception model based on an image sample set composed of image samples of multiple resolutions. Figure 2 As shown, the model training system can be composed of multiple threads of multiple artificial intelligence devices to form a thread pool, and the image samples corresponding to each resolution are assigned to a thread pool, so as to achieve parallel accelerated learning of multiple resolution image sets. It should be noted that m, n, and k only represent the meaning of multiple, and do not represent a specific number or the same number of threads of each artificial intelligence processor, nor do they mean that each thread pool is composed of one thread in each artificial intelligence device. The model training system can be deployed in a cloud computing center. The device structure of the cloud computing center is similar Figure 2The perceptual computing devices in the system include computing, storage, network and other infrastructure, operating system, database, middleware and other software environments and application software. The model training system receives image data sets of various resolutions, uses the stored pre-trained models, combined with manual annotation, to annotate visual perception targets (such as the category, location, size of the bounding box, target orientation information, etc.) and stores the annotated data.

[0105] The perception computing device can be a computing device deployed in a cloud computing center or an edge computing device deployed in an edge device. Its structure is as follows: Figure 2 As shown in the perception computing device shown. If the computing device of the cloud computing center is used, after the trained visual perception model is deployed, the edge device collects the image to be detected through a visual sensor such as a camera and uploads it to the cloud computing center, and uses the perception computing device that deploys the visual perception model to perform visual perception calculation on the image to be detected to obtain the target detection result of the visual perception target. If the perception computing device uses an edge computing device deployed on the edge device, the visual perception model is trained by the model training system and sent to the edge computing device through the communication unit, and deployed in the memory of the edge computing device. After the edge computing device collects the image to be detected through a visual sensor such as a camera, it performs visual perception calculation through the local visual perception model to obtain the target detection result of the visual perception target.

[0106] Applied to Figure 2 The multi-thread pool shown, in an embodiment of the present invention, the step of calculating the perceptual loss value of the visual perception model for image samples of each resolution may include: inputting the image samples corresponding to each resolution into different thread pools one by one, so as to obtain the perceptual loss value of the visual perception model for the image samples of each resolution by parallel calculation.

[0107] like Figure 3 As shown, in an embodiment of the present invention, when training a visual perception model, all images of each resolution can be stacked to obtain a set of input images of different resolutions, so that the training speed can be improved through batch processing.

[0108] In some optional implementations of the embodiments of the present invention, updating the parameters of the visual perception model and the parameters of the multi-resolution image correction network using the perceptual loss values ​​corresponding to each resolution in S103 may include: calculating the model learning loss value of the current iteration number according to the perceptual loss values ​​corresponding to each resolution, and updating the parameters of the visual perception model and the parameters of the multi-resolution image correction network according to the model learning loss value.

[0109] Among them, calculating the model learning loss value of the current iteration number according to the perceptual loss value corresponding to each resolution may include: taking the average of the loss values ​​calculated by the loss function corresponding to each resolution as the model learning loss value.

[0110] like Figure 3 As shown, for each resolution, all image samples are input into the visual perception model to obtain the corresponding visual perception results, and the loss value of the visual perception model for each resolution compared with the true value (which can be manually labeled) is calculated through the loss function. Finally, the model learning loss value of the visual perception model can be the average of the perceptual loss values ​​of the visual perception model for image samples of all resolutions, as shown in the following formula:

[0111] ;

[0112] in, represents the loss function of the visual perception model, represents the perceptual loss value corresponding to the r-th resolution, represents the loss function corresponding to the rth resolution, and k+1 represents the number of resolutions used in visual perception model training.

[0113] It is very challenging to directly use input images of multiple resolutions to train a visual perception model because there are large data distribution differences between input images of different resolutions, which makes it more difficult to optimize the perception model. To solve this problem, an embodiment of the present invention provides a visual perception model with a multi-resolution image correction network added. The multi-resolution image correction network is used to correct image features of different resolutions to be consistent with the feature size of the reference image, and optimization is achieved through self-attention calculation, thereby improving the generalization ability of the visual perception model for multiple resolutions.

[0114] Figure 4 A schematic diagram of the structure of a visual perception model with a multi-resolution image correction network added provided by an embodiment of the present invention.

[0115] like Figure 4 As shown, an embodiment of the present invention proposes a multi-resolution image correction network for correcting input image features, and in the iterative training of the visual perception model, the perceptual loss values ​​of image samples of each resolution of the visual perception model are used to simultaneously update the parameters of the multi-resolution image correction network, so that image features of different resolutions can be corrected to be consistent with the feature size of the reference image through the multi-resolution image correction network, thereby solving the interference problem between different resolutions when the visual perception model learns image samples of multiple resolutions.

[0116] In an embodiment of the present invention, one or more multi-resolution image correction networks may be added to a traditional visual perception model. A visual perception model is usually composed of multiple image feature extraction layers. The input of the multi-resolution image correction network is image features, and the output is corrected image features, so the multi-resolution image correction network may be added at any position after the first image feature extraction layer of the visual perception model.

[0117] For example, if the visual perception model adopts the ResNet-50 network, the multi-resolution image correction network provided by the embodiment of the present invention can be added after the image feature extraction network therein.

[0118] It should be noted that the visual perception model can also use models other than the ResNet-50 network, and the location of adding the multi-resolution image correction network can be set as needed, and does not necessarily have to be added to the Figure 4 Position shown.

[0119] It can be understood that the model parameters of the multi-resolution image correction network are related to the position in the visual perception model in which they are located. Then, determining the image feature position encoding corresponding to the resolution according to the relationship between the image feature size corresponding to the resolution and the reference image feature size in the image sample set in S102 may include: determining the image feature position encoding corresponding to the resolution according to the relationship between the image feature size corresponding to the resolution and the reference image feature size and the image feature size corresponding to the multi-resolution image correction network.

[0120] Image feature position encoding is responsible for providing the specific position information of each feature vector in the image features, which is very important for understanding the image structure and context. Image feature position encoding is not only related to the relationship between the image feature size corresponding to the resolution and the reference image feature size, but also related to the position of the multi-resolution image correction network in the visual perception model. When there are multiple multi-resolution image correction networks in the visual perception model, it is also possible to achieve targeted correction of the image features of the input image of the resolution at different network layers of the visual perception model, solving the problem of difficulty in model optimization caused by large data distribution differences between input images of different resolutions.

[0121] Figure 5 A training flow chart of a visual perception model provided by an embodiment of the present invention.

[0122] like Figure 5As shown, for image samples corresponding to resolutions 1, 2, ..., m, they are respectively input into the visual perception model to obtain corresponding perceptual loss values ​​1, 2, ..., m. As described in Embodiment S102 of the present invention, in one iterative training of the visual perception model, for a resolution, an image sample of the resolution is first input into the visual perception model, and in the multi-resolution image correction network of the visual perception model, the image features corresponding to the resolution are corrected, and then the corrected image features are input into the subsequent network layer, and the perceptual loss value corresponding to the resolution is calculated using the perceptual result finally output by the visual perception model.

[0123] In S103, in one iterative training of the visual perception model, the image samples of each resolution are subjected to the operation of S102, the perception loss values ​​corresponding to all resolutions are summarized to calculate the model learning loss value of the current iterative training, and the original model parameters of the visual perception model are updated according to the model learning loss value while updating the model parameters of the multi-resolution image correction network in the visual perception model.

[0124] like Figure 2 As shown in the figure, in order to accelerate the training of the visual perception model, the self-supervised loss optimization tasks corresponding to m resolutions can be deployed in m thread pools for execution to achieve parallel accelerated training.

[0125] For S104, the iterative training end condition may be that the number of iterative training times of the visual perception model reaches a preset number of iterations, or the model learning loss value of the visual perception model is less than a preset loss value.

[0126] For S105, after the trained visual perception model is deployed in an actual application scenario, it can be directly used to perform the target detection task. When performing the target detection task, in the process of using the visual perception model to perceive the input image corresponding to the target detection task, regardless of the resolution of the input image, the multi-resolution image correction network in the visual perception model can be used to correct the input image features to ensure the perception accuracy of the visual perception model.

[0127] The target detection method provided by the embodiment of the present invention adds a multi-resolution image correction network to the visual perception model. In the process of iteratively training the visual perception model using multi-resolution image samples, the multi-resolution image correction network is used to determine the image feature position code corresponding to the resolution according to the relationship between the image feature size corresponding to the resolution and the reference image feature size in the image sample set. According to the image feature position code, self-attention calculation is performed on the input image feature and the corrected image feature is output, thereby correcting the image features of different resolutions to be consistent with the reference image features. When training the visual perception model, the interference between image samples of different resolutions can be reduced, so that when the trained visual perception model is used to perform the target detection task, the accurate target detection results can be guaranteed for the input images of different resolutions that need to be processed.

[0128] Based on the above embodiment, the embodiment of the present invention further illustrates the steps of collecting the image sample set.

[0129] The image sample set used for training the visual perception model in the embodiment of the present invention includes image samples of multiple resolutions. When collecting the image sample set, a visual sensor such as a camera can be used to directly collect image samples of different resolutions through set resolution parameters to form an image sample set; when collecting the image sample set, image samples of one resolution can also be collected first, and image samples of multiple resolutions can be generated by scaling or enlarging to form an image sample set.

[0130] In some optional implementations of the embodiments of the present invention, the step of collecting the image sample set may include: obtaining an original image set; processing each original image in the original image set using different scaling factors to obtain scaled images of different resolutions corresponding to each original image; and using each original image and the scaled image corresponding to each original image as the image sample set.

[0131] In practical applications, each original image in the original image set is processed with different scaling factors to obtain scaled images of different resolutions corresponding to each original image, which may include: processing each original image with a reference scaling factor to obtain a reference scaled image corresponding to the original image; sampling multiple scaling factors from a preset distribution range, and processing each original image with each scaling factor to obtain scaled images of multiple resolutions corresponding to each original image; using the corresponding reference scaled image and each corresponding scaled image as an image set corresponding to the original image to obtain an image set corresponding to each original image. That is, the generated image sample set is composed of image sets corresponding to different original images, which include image samples of different resolutions, and different image samples of the same resolution may be obtained by processing different original images.

[0132] As described in the above embodiment of the present invention, the resolution used for processing by the visual perception model of the target detection task is usually expressed in pixel resolution, that is, expressed as the product of the number of pixels in the horizontal direction and the number of pixels in the vertical direction. In the embodiment of the present invention, the scaling factor may include a scaling factor acting on the image height and / or a scaling factor on the image width. In order to obtain image samples of multiple resolutions with greater resolution differences, the scaling factor may include an image height scaling factor and an image width scaling factor.

[0133] To reduce the computational difficulty, the scaling factor can also be a scaling ratio for the image height and image width. For example, for an original image with a resolution of H×W , the base scaling factor can be used The processed resolution is After the image is distributed in Uniform distribution Sample k scaling factors , and process the image using each scaling factor , and finally obtain a collection of images with multiple resolutions ,in The resolution is . Where H represents the vertical size of the image, and W represents the horizontal size of the image.

[0134] When applied to the field of autonomous driving, cameras and other visual sensors are usually set up in different directions of the vehicle, such as left front (fl), front (f), right front (fr), left rear (bl), rear (b), right rear (br), etc., so that images of multiple perspectives of the road scene are collected each time. In the actual training process, in order to use the batch method to accelerate model training, the embodiment of the present invention samples a set of scaling factors at each iteration, and , the above-obtained scaling coefficient set is used for processing, that is, a multi-resolution image set is obtained for each viewing angle .

[0135] Since the resolution difference between the multi-resolution image samples used in training the visual perception model is small, it is not conducive to the training of the model's resolution generalization ability. In order to obtain as many resolutions with large differences as possible and improve the training effect of the model's resolution generalization ability, in other optional implementations of the embodiments of the present invention, the difference between the scaling factors can be set to be no less than the scaling factor difference threshold, that is, when determining the scaling factor, the difference between the scaling factor and other scaling factors in the collected scaling factor set is judged, and only when the difference is greater than the set scaling factor difference threshold is the scaling factor added to the scaling factor set, thereby ensuring the diversity between the scaling factors obtained by sampling and improving the training effect of the model's resolution generalization ability.

[0136] Furthermore, the scaling factors may be designed to all belong to a preset distribution range and the difference between any two scaling factors may not be less than a scaling factor difference threshold.

[0137] In some optional implementations of the embodiments of the present invention, the scaling factor difference threshold is calculated based on the interval length of the preset distribution range divided by twice the number of scaling factors. That is, the scaling factor difference threshold can be expressed by the following formula:

[0138] ;

[0139] Where M represents the scaling factor difference threshold, Indicates the lower limit of the preset distribution range. represents the upper limit of the preset distribution range, and k represents the total number of scaling factors to be collected from the preset distribution range.

[0140] In some other optional implementations of the embodiments of the present invention, each original image in the original image set is processed using a different scaling factor to obtain a scaled image of different resolutions corresponding to each original image, which may also include: using a benchmark scaling factor to process each original image to obtain a benchmark scaled image corresponding to the original image; sampling multiple scaling factors from a preset distribution range until a preset number of scaling factors are collected; in the process of collecting scaling factors, each time a new scaling factor is collected, the difference between the new scaling factor and the existing scaling factor is calculated, and if the difference is not less than the scaling factor difference threshold, the new scaling factor is added to the set of existing scaling factors; if the difference is less than the scaling factor difference threshold, the new scaling factor is discarded; using each scaling factor to process each original image respectively to obtain scaled images of multiple resolutions corresponding to each original image; using the corresponding benchmark scaling image and the corresponding scaled images as the image set corresponding to the original image to obtain the image set corresponding to each original image.

[0141] That is, the following algorithm can be designed to obtain the set of scaling factors:

[0142] Input: reference scaling factor , distributed in uniform distribution , scaling factor set A, scaling factor difference threshold M;

[0143] Add the reference scaling factor to the scaling factor set A ;

[0144] If the length of the scaling factor set A < k + 1, then:

[0145] Randomly sample from the uniform distribution to obtain ;

[0146] Check the difference between and all the scaling factors in the current scaling factor set A. If the difference is greater than the threshold scaling factor difference M, then add to the scaling factor set A;

[0147] Output: scaling factor set A.

[0148] To further improve the ability of the visual perception model to process input images of different resolutions, an image sample set can be generated multiple times during the model training process. That is to say, the scaling factors can be re - collected to generate a new image sample set in each iteration training, or the scaling factors can be re - collected to generate a new image sample set after every few iteration trainings.

[0149] Since if the image sample set is regenerated by the above method in each iteration training, it may cause the visual perception model to be difficult to converge.

[0150] To ensure model convergence, in some optional implementation manners of the embodiments of the present invention, an image sample set can be generated once before the training of the visual perception model. In the subsequent iteration trainings, the image sample set is not updated. Then, in each iteration training of the visual perception model, the same batch of resolutions are learned, and the diversity of the samples can be ensured by collecting different image samples corresponding to the resolutions. When computing resources and time permit, an image sample set can be generated once before the training of the visual perception model, but multiple rounds of iteration training are performed. Before the start of each round of iteration training, a new image sample set is regenerated, and the same image sample set is used in different iteration training steps in each round of iteration training.

[0151] In some other optional implementation manners of the embodiments of the present invention, the image sample set can also be regenerated every few iteration trainings, and the frequency of generating the image sample set can be determined according to the convergence situation of the visual perception model.

[0152] In some other optional implementations of the embodiments of the present invention, if the image sample set is regenerated in each iterative training or every few iterative trainings, the multiple resolutions corresponding to the resampling scaling factors can arrange the resolutions in a fixed order (such as from small to large or from large to small) according to the scaling factors, thereby ensuring the correspondence between the arrangement of the resolutions and the arrangement of the loss function, so that the model training can inherit the experience of the previous iterative training or the previous few iterative trainings.

[0153] In each embodiment of the present invention, the loss function corresponding to the resolution of the visual perception model in each iterative training number inherits the parameters of the loss function of the corresponding position in the previous iterative training, but the corresponding resolution can be the resolution in the previous iterative training or the updated resolution.

[0154] On the basis of the above embodiments, in order to adapt to the problem that the multi-resolution model is difficult to optimize, the embodiment of the present invention continues to explain the structure of the multi-resolution image correction network.

[0155] Figure 6 A schematic diagram of the structure of a multi-resolution image correction network provided by an embodiment of the present invention.

[0156] As described in the above embodiment of the present invention, the multi-resolution image correction network provided by the embodiment of the present invention is used to perform self-attention calculation on the image features input to the multi-resolution image correction network according to the image feature position encoding corresponding to the resolution, and output the corrected image features. Figure 5 As shown, an embodiment of the present invention provides a network structure of a multi-resolution image correction network, including an attention module and an image feature position encoding multi-resolution optimization unit, wherein the image feature position encoding multi-resolution optimization unit is used to input the attention module by generating a query vector, a key value vector, etc., so as to perform self-attention learning on the image correction method. The attention module can be implemented by the self-attention layer of the Transformer model.

[0157] In some optional implementations of the embodiments of the present invention, Figure 6 As shown, image feature position encoding may include query vector position encoding and value vector position encoding.

[0158] In some other optional implementations of the embodiments of the present invention, the image feature position encoding includes query vector position encoding, key vector position encoding and value vector position encoding.

[0159] In the target detection method provided by the embodiment of the present invention, the reference image feature size is the image feature size of the reference image sample. The reference image sample is an image sample corresponding to a resolution in the image sample set. The reference image sample can adopt the reference scaled image introduced in the above embodiment.

[0160] In the target detection method provided by the embodiment of the present invention, the image feature position coding is responsible for providing the specific position information of each feature vector in the image feature, which is very important for understanding the image structure and context. The image feature space sizes corresponding to input images of different resolutions are different, and the existing position coding method has poor generalization ability for the image feature space size. Therefore, the embodiment of the present invention proposes an image feature position coding optimization unit for adjusting the image feature position coding using resolution information to make it better adapt to image features of different resolutions.

[0161] In order to enable the multi-resolution image correction network to process image features of different resolutions, that is, the multi-resolution image correction network can adapt to the resolution of the input image to determine the required image feature position encoding, and correct the image features of the input image to the same size as the reference image feature, the embodiment of the present invention provides a reference query vector, the spatial size of which is aligned with the reference image feature size, and can unify image features of different resolutions to the reference image feature size, thereby achieving consistency in spatial size of image features of different resolutions. For example, the resolution of the reference image is , for an image feature extraction network with a downsampling factor of s, the size of the baseline image feature space is , then the benchmark query vector , It is randomly initialized at the beginning of the perception model training. During the training process of the perception model, Gradually accumulate information corresponding to image features of different resolutions, thereby achieving generalization capabilities for different resolutions. Among them, the image feature extraction network with a downsampling factor of s means that the image features of the input image have been downsampled s times before entering the multi-resolution image correction network.

[0162] In some optional implementations of the embodiments of the present invention, the image feature position coding optimization unit may be composed of a first feature mapping network, a first size transformation prediction network, and a first size transformation mapping unit. Determining the image feature position coding corresponding to the resolution according to the relationship between the image feature size corresponding to the resolution and the reference image feature size in the image sample set may include: using the first feature mapping network to map the image feature size corresponding to the resolution and the reference image feature size to obtain a first feature mapping result; using the first size transformation prediction network to generate a first size transformation coefficient according to the first feature mapping result; using the first size transformation coefficient to perform size transformation processing on the current image feature position coding to obtain the image feature position coding corresponding to the resolution.

[0163] Among them, using the first size transformation prediction network to generate the first size transformation coefficient according to the first feature mapping result may include: using the first scaling coefficient prediction network to generate the first scaling coefficient according to the first feature mapping result, and using the first scaling coefficient as the first size transformation coefficient.

[0164] Alternatively, using a first size transformation prediction network to generate a first size transformation coefficient according to a first feature mapping result may also include: using a first scaling coefficient prediction network to generate a first scaling coefficient according to a first feature mapping result; using a first translation coefficient prediction network to generate a first translation coefficient according to a first feature mapping result; standardizing the first scaling coefficient and the first translation coefficient respectively; and using the standardized first scaling coefficient and the standardized first translation coefficient as the first size transformation coefficient.

[0165] In some optional implementations of the embodiments of the present invention, the types of image feature position coding optimization units may include query vector position coding multi-resolution optimization units, key vector position coding multi-resolution optimization units, and value vector position coding multi-resolution optimization units. Among them, the query vector position coding multi-resolution optimization unit is used to optimize the query vector position coding to obtain the query vector position coding corresponding to the current resolution. The key vector position coding multi-resolution optimization unit is used to optimize the key vector position coding to obtain the key vector position coding corresponding to the current resolution. The value vector position coding multi-resolution optimization unit is used to optimize the value vector position coding to obtain the value vector position coding corresponding to the current resolution.

[0166] Figure 7 An optimization flow chart of query vector position encoding provided by an embodiment of the present invention.

[0167] like Figure 7 As shown, the query vector position coding multi-resolution optimization unit can be composed of a query vector feature mapping network, a query vector size transformation prediction network and a query vector size transformation mapping unit, and is used to adjust the query vector position coding using resolution information so that it can better adapt to image features of different resolutions. Among them, the types of query vector size transformation prediction networks can include query vector scaling coefficient prediction networks and query vector translation coefficient prediction networks. The query vector size transformation prediction network can use a multi-layer perceptron.

[0168] The feature size of the reference image is , the image feature size corresponding to the current resolution (i-th resolution) is , s means that the image features have been downsampled s times before entering the multi-resolution image rectification network.

[0169] Then in the query vector position encoding multi-resolution optimization unit, the query vector feature mapping network ( ) Processing image feature size , , , , and then use the query vector scaling factor to predict the network ( ), query vector translation coefficient prediction network ( ) and then processed by the sigmoid activation function to obtain the query vector scaling factor and query vector translation coefficient , and then the query vector scaling factor can be calculated by the following formula and query vector translation coefficient To standardize:

[0170] ;

[0171] ;

[0172] Get the normalized query vector scaling factor and the normalized query vector translation coefficient .

[0173] Scaling factor using the normalized query vector and the normalized query vector translation coefficient Adjusting the query vector position encoding , get the optimized query vector position encoding , as shown below:

[0174] ;

[0175] Among them, the query vector position encoding represents the query vector position encoding before optimization, and represents the query vector position encoding after the last optimization saved in the iterative training of the visual perception model.

[0176] The query vector position encoding can adopt a fixed position encoding or a learnable position encoding.

[0177] In some optional implementations of the embodiments of the present invention, before the iterative training of the visual perception model, a baseline query vector position code can be obtained by random initialization. The baseline query vector position code can also be generated by a sine function position coding method or other position coding methods. In each iterative training of the visual perception model, the query vector position code is continuously updated by image samples of different resolutions so that the query vector position code obtains multi-resolution information. Then the last optimized query vector position code introduced above can be the query vector position code optimized in the last iterative training, or it can be the query vector position code optimized by using the resolution of the previous input visual perception model in the current iterative training.

[0178] Figure 8 An optimization flow chart for value vector position encoding provided by an embodiment of the present invention.

[0179] like Figure 8 As shown, the value vector position coding multi-resolution optimization unit can be composed of a value vector feature mapping network, a value vector size transformation prediction network and a value vector size transformation mapping unit, which is used to adjust the value vector position coding using resolution information to better adapt to image features of different resolutions. Among them, the type of the value vector size transformation prediction network can include a value vector scaling coefficient prediction network and a value vector translation coefficient prediction network. The value vector size transformation prediction network can use a multi-layer perceptron.

[0180] The feature size of the reference image is , the image feature size corresponding to the current resolution (i-th resolution) is , s means that the image features have been downsampled s times before entering the multi-resolution image rectification network.

[0181] Then in the value vector position encoding multi-resolution optimization unit, the value vector feature mapping network is first used ( ) Processing image feature size , , , , and then use the value vector to scale the coefficient prediction network ( )、Value vector translation coefficient prediction network( ) and then processed by the sigmoid activation function to obtain the value vector scaling factor Sum vector translation coefficient , and then the value vector can be scaled by the following formula Sum vector translation coefficient To standardize:

[0182] ;

[0183] ;

[0184] Get the normalized value vector scaling factor and the normalized value vector translation coefficient .

[0185] Scale the coefficients with the normalized value vector and the normalized value vector translation coefficient Adjustment value vector position encoding , get the optimized value vector position encoding , as shown below:

[0186] ;

[0187] Among them, the value vector position encoding represents the position encoding of the value vector before optimization, and represents the position encoding of the value vector after the last optimization saved in the iterative training of the visual perception model.

[0188] The value vector position encoding can adopt fixed position encoding or learnable position encoding.

[0189] In some optional implementations of the embodiments of the present invention, before the iterative training of the visual perception model, a baseline value vector position coding can be obtained by random initialization. The baseline value vector position coding can also be generated by a sine function position coding method or other position coding methods. In each iterative training of the visual perception model, the value vector position coding is continuously updated by image samples of different resolutions so that the value vector position coding obtains multi-resolution information. Then the last optimized value vector position coding introduced above can be the value vector position coding optimized in the last iterative training, or it can be the value vector position coding optimized by the resolution of the previous input visual perception model in the current iterative training.

[0190] If the image feature position encoding also includes the key vector position encoding, then the optimized network structure is the same as Figure 8 Similarly, the optimization step of the principle reference value vector position encoding is sufficient.

[0191] In the target detection method provided by an embodiment of the present invention, in order to further enhance the ability of the multi-resolution image correction network to correct image features, the attention module in the multi-resolution image correction network can adopt a deformable cross-attention module to overcome the problems existing in the attention of the Transformer model.

[0192] Then the image feature position coding may only include the query vector position coding and the value vector position coding, that is, the query vector position coding and the value vector position coding are only optimized according to the current resolution. Then in some other optional implementations of the embodiments of the present invention, such as Figure 6 As shown, the image feature position encoding optimization unit may include a query vector position encoding multi-resolution optimization unit and a value vector position encoding multi-resolution optimization unit.

[0193] Then in the target detection method provided by the embodiment of the present invention, in S103, updating the parameters of the multi-resolution image correction network using the perceptual loss values ​​corresponding to each resolution may include: updating the parameters of the first feature mapping network and the parameters of the first size transformation prediction network using the perceptual loss values ​​corresponding to each resolution. For example, if the image feature position encoding may include query vector position encoding and value vector position encoding, then the perceptual loss values ​​corresponding to each resolution are used to update the parameters of the query vector feature mapping network, the parameters of the query vector size transformation prediction network, the parameters of the value vector feature mapping network, and the parameters of the value vector size transformation prediction network.

[0194] Fig. 9 A schematic diagram of the structure of another multi-resolution image correction network provided by an embodiment of the present invention.

[0195] On the basis of the above embodiments, in the target detection method provided by the embodiments of the present invention, in S102, in the multi-resolution image correction network, the image feature position code corresponding to the resolution is determined according to the relationship between the image feature size corresponding to the resolution and the reference image feature size in the image sample set, and self-attention calculation is performed on the image features input to the multi-resolution image correction network according to the image feature position code corresponding to the resolution, and the corrected image features are output, which may include: in the multi-resolution image correction network, self-attention calculation is performed on the image features input to the multi-resolution image correction network using the image feature position code corresponding to the resolution and the embedding vector corresponding to the resolution, and the corrected image features are output.

[0196] Among them, the step of determining the embedding vector corresponding to the resolution may include: performing mapping processing according to the image features corresponding to the resolution, the image feature size corresponding to the resolution and the reference image feature size to obtain a second feature mapping result; using a second size transformation prediction network to generate a second size transformation coefficient according to the second feature mapping result; using the second size transformation coefficient to perform size transformation processing on the current embedding vector to obtain the embedding vector corresponding to the resolution.

[0197] In order to further improve the optimization effect of multi-resolution image correction, the present invention provides another structure of multi-resolution image correction network, such as Fig. 9As shown, in addition to the attention module and the image feature position encoding multi-resolution optimization unit, the multi-resolution image correction network also includes an embedding vector multi-resolution optimization unit to add the image features of the current resolution to the correction learning of the multi-resolution image features, providing the query vector with richer information related to the current resolution, and improving the multi-resolution generalization ability of the multi-resolution feature correction method.

[0198] In some optional implementations of the embodiments of the present invention, using the image feature position encoding corresponding to the resolution and the embedding vector corresponding to the resolution to perform self-attention calculation on the image features input to the multi-resolution image correction network may include: concatenating the query vector position encoding in the image feature position encoding corresponding to the resolution with the embedding vector corresponding to the resolution to obtain a multi-resolution query vector; and inputting the multi-resolution query vector into the self-attention layer to perform self-attention calculation. Fig. 9 As shown, the embedding vector optimized according to the image features of the current resolution is concatenated with the optimized query vector position encoding and the reference query vector to obtain a multi-resolution query vector, and the multi-resolution query vector is input into the attention module.

[0199] Fig.10 An optimization flow chart of an embedding vector provided by an embodiment of the present invention.

[0200] like Fig.10 As shown, the embedded vector multi-resolution optimization unit can be composed of a resolution image feature processing network, a second feature mapping network, a second size transformation prediction network and a second size transformation mapping unit. Among them, the resolution image feature processing network can include a global mean pooling layer, a global maximum pooling layer, a mean calculation layer and a standard deviation calculation layer. The global mean pooling layer and the global maximum pooling layer are respectively used to perform global mean pooling calculations in the channel dimension and global maximum pooling calculations in the channel dimension on the image features of the current resolution. The mean calculation layer and the standard deviation calculation layer are respectively used to calculate the mean μ and the standard deviation σ of the image features of the current resolution.

[0201] Mapping processing is performed according to the image features corresponding to the resolution, the image feature size corresponding to the resolution, and the reference image feature size to obtain a second feature mapping result, which may include: performing global mean pooling processing and global maximum pooling processing on the image features corresponding to the resolution, respectively, to obtain a global mean pooling feature vector corresponding to the resolution and a global maximum pooling feature vector corresponding to the resolution; calculating the mean of the image features corresponding to the resolution and the standard deviation of the image features corresponding to the resolution; performing channel dimension splicing processing on the global mean pooling feature vector corresponding to the resolution, the global maximum pooling feature vector corresponding to the resolution, the mean of the image features corresponding to the resolution, the standard deviation of the image features corresponding to the resolution, the image feature size corresponding to the resolution, and the reference image feature size to obtain a channel dimension splicing result; inputting the channel dimension splicing result into the second feature mapping network, and outputting the second feature mapping result.

[0202] Among them, using the second size transformation prediction network to generate the second size transformation coefficient according to the second feature mapping result may include: using the second scaling coefficient prediction network to generate the second scaling coefficient according to the second feature mapping result, and using the second scaling coefficient as the second size transformation coefficient.

[0203] Alternatively, using a second size transformation prediction network to generate a second size transformation coefficient based on a second feature mapping result may also include: using a second scaling coefficient prediction network to generate a second scaling coefficient based on a second feature mapping result; using a second translation coefficient prediction network to generate a second translation coefficient based on a second feature mapping result; standardizing the second scaling coefficient and the second translation coefficient, respectively; and using the standardized second scaling coefficient and the standardized second translation coefficient as the second size transformation coefficient.

[0204] Then the reference image feature size is , the image feature size corresponding to the current resolution (i-th resolution) is , s means that the image features have been downsampled s times before entering the multi-resolution image rectification network.

[0205] like Fig.10 As shown, in the embedding vector multi-resolution optimization unit, the image features are firstly processed by global mean pooling ( ), global maximum pooling processing ( ), obtain the global mean pooling feature vector , global maximum pooling feature vector , calculate the mean μ and standard deviation σ of the image features, and then convert the benchmark image feature size , , image feature size , , global mean pooling feature vector , global maximum pooling feature vector , mean μ, standard deviation σ splicing, using the second feature mapping network ( ) and then respectively by the second scaling coefficient prediction network ( ), the second translation coefficient prediction network ( ) is processed, and after being processed by the sigmoid activation function, the second scaling factor is obtained , the second translation coefficient Finally, the second scaling factor and the second translation factor are normalized as shown in the following formula:

[0206] ;

[0207] ;

[0208] Get the standardized second scaling factor , the standardized second translation coefficient .

[0209] Using the normalized second scaling factor , the standardized second translation coefficient Adjusting the baseline embedding vector , and get the optimized embedding vector , as shown below:

[0210] .

[0211] Baseline Embedding Vectors It is randomly initialized at the beginning of the visual perception model training. During the training process of the visual perception model, the embedding vector gradually accumulates information corresponding to the image features of different resolutions, and is optimized by the image features and resolution information to obtain the optimized embedding vector , achieving generalization capabilities for different resolutions.

[0212] In an embodiment of the present invention, updating the parameters of the multi-resolution image correction network using the perceptual loss values ​​corresponding to each resolution may also include: updating the parameters of the second feature mapping network and the parameters of the second size transformation prediction network using the perceptual loss values ​​corresponding to each resolution.

[0213] In combination with the optimization process of image feature position encoding introduced in the above embodiment of the present invention, in the target detection method provided in the embodiment of the present invention, updating the parameters of the multi-resolution image correction network using the perceptual loss values ​​corresponding to each resolution in S103 may include: updating the parameters of the first feature mapping network, the parameters of the first size transformation prediction network, the parameters of the second feature mapping network, and the parameters of the second size transformation prediction network using the perceptual loss values ​​corresponding to each resolution.

[0214] On the basis of the above embodiments, in order to further improve the optimization effect of multi-resolution image correction, in the target detection method provided in the embodiment of the present invention, the perceptual loss value corresponding to each resolution is used in S103 to update the parameters of the visual perception model and the parameters of the multi-resolution image correction network to obtain the visual perception model corresponding to the current iterative training batch, including: calculating the regularization constraint term corresponding to each resolution according to the size transformation coefficient of each resolution compared with the feature size of the reference image; updating the parameters of the visual perception model and the parameters of the multi-resolution image correction network according to the perceptual loss value corresponding to each resolution and the regularization constraint term corresponding to each resolution to obtain the visual perception model corresponding to the current iterative training batch.

[0215] In an embodiment of the present invention, on the basis of the loss calculated according to the perceptual loss value corresponding to each resolution, a regularization constraint term corresponding to each resolution is calculated according to the size transformation coefficient of each resolution compared with the characteristic size of the reference image to ensure the optimization effect.

[0216] The design principle of the regularization constraint term is to minimize the adjustment of the input image features to avoid the visual perception model from being difficult to converge and the image correction effect from being poor when the size transformation coefficient is too large. The smaller the size transformation coefficient corresponding to the resolution makes the size transformation of the image feature corresponding to the resolution smaller, the closer the regularization constraint term corresponding to the resolution is to 0.

[0217] Taking the optimization of query vector position encoding, value vector position encoding and embedding vector in multi-resolution image rectification network as an example, the query vector scaling factor of query vector position encoding is and query vector translation coefficient , can be constrained according to the following regularization formula:

[0218] ;

[0219] ;

[0220] in, Represents the query vector scaling factor The corresponding regularization constraint is Represents the translation coefficient of the query vector The corresponding regularization constraint.

[0221] The value vector scaling factor for the value vector position encoding Sum vector translation coefficient , can be constrained according to the following regularization formula:

[0222] ;

[0223] ;

[0224] in, Represents the value vector scaling factor The corresponding regularization constraint is Represents the value vector translation coefficient The corresponding regularization constraint.

[0225] The second scaling factor for the embedding vector and the second translation coefficient , can be constrained according to the following regularization formula:

[0226] ;

[0227] ;

[0228] in, Indicates the second scaling factor The corresponding regularization constraint is The second translation coefficient The corresponding regularization constraint.

[0229] Then the loss function of the visual perception model is It can be expressed as follows:

[0230] ;

[0231] in, is the loss value calculated according to the perceptual loss value corresponding to each resolution.

[0232] In some optional implementations of the embodiments of the present invention, , , , , , is the weight coefficient of each regularization constraint term, which can be used as a hyperparameter to balance the loss function With regularization constraints , , , , , ,For example, , , , , , They can all be 0.5.

[0233] It should be noted that in one iteration training of the visual perception model, the image features corresponding to each resolution have a regularization constraint term of the size transformation coefficient. In the visual perception model, each regularization constraint term can be composed of the sum of the regularization constraint terms corresponding to each resolution.

[0234] In some other alternative embodiments of the embodiments of the present invention, 、 、 、 、 、 Self-learning parameters can also be adopted. Then, the object detection method provided by the embodiments of the present invention can further include: updating the weight coefficient of the regularization constraint term during the iterative training of the visual perception model.

[0235] Based on the multi-resolution image feature correction method solution introduced in the above embodiments, the embodiments of the present invention continue to describe the training steps and inference steps of the visual perception model.

[0236] As introduced in the above embodiments of the present invention, the training task of the visual perception model can be executed in the cloud computing center.

[0237] For example, when applied to the field of autonomous driving, the visual perception model can be trained on the collected and labeled road scene image data. Using the multi-resolution meta-learning optimization method provided by the embodiments of the present invention, for the image samples corresponding to m resolutions in each training batch, they are respectively input into the visual perception model in the current iterative training. After the image features are corrected by the multi-resolution image correction network in the visual perception model, finally, the perception result is output by the visual perception model and the perception loss value is calculated according to the ground truth. All the perception loss values in the training batch are collected and substituted into the loss function of the visual perception model to obtain the model learning loss value, and then the model is optimized through the backpropagation algorithm. Among them, the optimizer can adopt optimizers such as Gradient Descent (GD) optimizer, Adaptive Moment estimation (Adam) optimizer, and Adam Weight Decay Regularization (AdamW) optimizer. The multi-resolution parallel acceleration method proposed by the present invention is used during training to improve the training efficiency. After the model training is completed, the trained perception model is deployed to the edge computing platform of the driverless vehicle.

[0238] The target detection method provided by the embodiment of the present invention can be applied to, but not limited to, the field of autonomous driving, and can also be applied to intelligent terminals such as intelligent robots, smart homes, and mobile phones, and can also be applied to visual perception models involved in industrial production fields such as transportation, aerial remote sensing, and industrial product assembly. For example, the visual perception model can be an autonomous driving perception model; for this, the image sample set can be constructed based on image samples collected by vehicle-mounted sensors from different perspectives, and the visual perception model can be trained on the collected and annotated road scene image data set when executing model training in the cloud data center.

[0239] In an embodiment of the present invention, updating the parameters of the visual perception model using the perceptual loss values ​​corresponding to each resolution may include: calculating the model learning loss value of the visual perception model using the sum of the perceptual loss values ​​corresponding to each resolution; and updating the parameters of the visual perception model using the model learning loss value. For example, a loss function of a visual perception model provided in the above embodiment of the present invention may be used. If the optimization effect of image feature correction is enhanced by adding a regularization constraint term in combination with the above-mentioned embodiment of the present invention, the loss function of the visual perception model can be expressed as:

[0240] .

[0241] When using the visual perception model to perform target detection tasks, the trained visual perception model can be directly deployed in actual application scenarios to perform target detection tasks.

[0242] It should be noted that in the embodiments of the target detection method of the present invention, some of the steps or features may be ignored or not executed. The hardware or software functional modules divided for the convenience of description are not the only implementation form of the target detection method provided by the embodiment of the present invention.

[0243] The above describes in detail various embodiments corresponding to the target detection method. On this basis, the present invention also discloses a target detection device, equipment, non-volatile storage medium and computer program product corresponding to the above method.

[0244] Fig.11 A schematic diagram of the structure of a target detection device provided by an embodiment of the present invention.

[0245] like Fig.11 As shown, the target detection device provided by the embodiment of the present invention includes:

[0246] The model training unit 1101 is used to train a visual perception model with a multi-resolution image correction network using an image sample set including image samples of multiple resolutions; in iterative training, for each resolution, the image sample of the resolution is input into the visual perception model and a perceptual loss value is calculated, and in the multi-resolution image correction network, the image feature position encoding corresponding to the resolution is determined according to the relationship between the image feature size corresponding to the resolution and the reference image feature size in the image sample set, and self-attention calculation is performed on the image features input into the multi-resolution image correction network according to the image feature position encoding corresponding to the resolution, and the corrected image features are output; the perceptual loss value corresponding to each resolution is used to update the parameters of the visual perception model and the parameters of the multi-resolution image correction network to obtain the visual perception model corresponding to the current iterative training batch; after the iterative training end condition is met, the trained visual perception model is obtained;

[0247] The target detection unit 1102 is used to input the input image corresponding to the target detection task into the trained visual perception model to obtain the target detection result of the target detection task.

[0248] It should be noted that in each implementation of the target detection device provided by the embodiment of the present invention, the division of units is only a logical functional division, and other division methods can be adopted. The connection method between different units can be electrical, mechanical or other connection methods. The separated units can be located in the same physical location or distributed on multiple network nodes. Each unit can be implemented in the form of hardware or in the form of a software functional unit. That is, part or all of the units provided by the embodiment of the present invention can be selected according to actual needs and the corresponding connection method or integration method can be adopted to achieve the purpose of the embodiment of the present invention.

[0249] Since the embodiments of the apparatus part correspond to the embodiments of the method part, please refer to the description of the embodiments of the method part for the embodiments of the apparatus part, which will not be repeated here.

[0250] Fig.12 A schematic diagram of the structure of a target detection device provided by an embodiment of the present invention.

[0251] like Fig.12 As shown, the target detection device provided by an embodiment of the present invention includes: a memory 1210, used to store a computer program 1211; a processor 1220, used to execute the computer program 1211, and when the computer program 1211 is executed by the processor 1220, the steps of the target detection method provided in any of the above embodiments are implemented.

[0252] Among them, the processor 1220 may include one or more processing cores, such as a 3-core processor, an 8-core processor, etc. The processor 1220 may be implemented in at least one hardware form of digital signal processing (DSP), field-programmable gate array (FPGA), and programmable logic array (PLA). The processor 1220 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a central processing unit (CPU); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 1220 may be integrated with a graphics processing unit (GPU), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 1220 may also include an artificial intelligence (AI) processor, which is used to process computing operations related to machine learning.

[0253] The memory 1210 may include one or more non-volatile storage media, which may be non-transitory. The memory 1210 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In this embodiment, the memory 1210 is at least used to store the following computer program 1211, wherein the computer program 1211 can implement the relevant steps in the target detection method disclosed in any of the aforementioned embodiments after being loaded and executed by the processor 1220. In addition, the resources stored in the memory 1210 may also include an operating system 1212 and data 1213, etc., and the storage method may be temporary storage or permanent storage. Among them, the operating system 1212 may be Windows or other types of operating systems. Data 1213 may include, but is not limited to, the data involved in the above method.

[0254] In some embodiments, the target detection device may further include a display screen 1230 , a power supply 1240 , a communication interface 1250 , an input / output interface 1260 , a sensor 1270 , and a communication bus 1280 .

[0255] Those skilled in the art will understand that Fig.12 The structure shown in the figure does not constitute a limitation on the target detection device, and may include more or fewer components than those shown in the figure.

[0256] The target detection device provided by the embodiment of the present invention includes a memory and a processor. When the processor executes the program stored in the memory, it can implement the steps of the target detection method provided in the above embodiment, and the effect is the same as above.

[0257] An embodiment of the present invention provides a non-volatile storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the target detection method provided in any one of the above embodiments can be implemented.

[0258] The non-volatile storage medium may include: a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and other media that can store program codes.

[0259] For the introduction of the non-volatile storage medium provided in the embodiment of the present invention, please refer to the above method embodiment, and the effect thereof is the same as the target detection method provided in the embodiment of the present invention, and the present invention will not elaborate on it here.

[0260] An embodiment of the present invention provides a computer program product, including a computer program, which, when executed by a processor, implements the steps of the target detection method provided in any one of the above embodiments.

[0261] For an introduction to the computer program product provided by the embodiment of the present invention, please refer to the above method embodiment, and the effect thereof is the same as the target detection method provided by the embodiment of the present invention, and the present invention will not elaborate on it here.

[0262] The above is a detailed introduction to a target detection method, device, equipment, medium and computer program product provided by the present invention. The various embodiments in the specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same and similar parts between the embodiments can be referred to each other. For the devices, equipment, non-volatile storage media and computer program products disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple, and the relevant parts can be referred to the method part description. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of the present invention, the present invention can also be improved and modified in a number of ways, and these improvements and modifications also fall within the scope of protection of the present invention.

[0263] It should also be noted that, in this specification, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the statement "comprises a ..." does not exclude the presence of other identical elements in the process, method, article or device including the element.

Claims

1. A target detection method, characterized in that: include: Training a visual perception model to which a multi-resolution image correction network is added using an image sample set including image samples of multiple resolutions; In iterative training, for each resolution, an image sample of the resolution is input into the visual perception model and a perception loss value is calculated, and in the multi-resolution image correction network, an image feature position encoding corresponding to the resolution is determined according to a relationship between an image feature size corresponding to the resolution and a reference image feature size in the image sample set, and self-attention calculation is performed on image features input into the multi-resolution image correction network according to the image feature position encoding corresponding to the resolution, and corrected image features are output; Using the perceptual loss values ​​corresponding to each of the resolutions, the parameters of the visual perception model and the parameters of the multi-resolution image correction network are updated to obtain the visual perception model corresponding to the current iterative training batch; After reaching the iterative training end condition, obtaining the trained visual perception model; The input image corresponding to the target detection task is input into the trained visual perception model to obtain the target detection result of the target detection task.

2. The target detection method according to claim 1, characterized in that: Determining the image feature position encoding corresponding to the resolution according to the relationship between the image feature size corresponding to the resolution and the reference image feature size in the image sample set includes: Using a first feature mapping network, mapping the image feature size corresponding to the resolution and the reference image feature size is performed to obtain a first feature mapping result; Generate a first size transformation coefficient according to the first feature mapping result using a first size transformation prediction network; The first size transformation coefficient is used to perform size transformation processing on the current image feature position code to obtain the image feature position code corresponding to the resolution.

3. The target detection method according to claim 2, characterized in that: Generating a first size transformation coefficient according to the first feature mapping result using a first size transformation prediction network includes: A first scaling factor prediction network is used to generate a first scaling factor according to the first feature mapping result, and the first scaling factor is used as the first size transformation coefficient.

4. The target detection method according to claim 2, characterized in that: Generating a first size transformation coefficient according to the first feature mapping result using a first size transformation prediction network includes: Generate a first scaling factor according to the first feature mapping result using a first scaling factor prediction network; Generate a first translation coefficient according to the first feature mapping result using a first translation coefficient prediction network; performing normalization processing on the first scaling factor and the first translation factor respectively; The first normalized scaling factor and the first normalized translation factor are used as the first size conversion factor.

5. The target detection method according to claim 2, characterized in that: Updating the parameters of the multi-resolution image correction network using the perceptual loss values ​​corresponding to each of the resolutions includes: The perceptual loss values ​​corresponding to the resolutions are used to update parameters of the first feature mapping network and parameters of the first size transformation prediction network.

6. The target detection method according to claim 1, characterized in that: In the multi-resolution image correction network, the image feature position code corresponding to the resolution is determined according to the relationship between the image feature size corresponding to the resolution and the reference image feature size in the image sample set, self-attention calculation is performed on the image features input to the multi-resolution image correction network according to the image feature position code corresponding to the resolution, and the corrected image features are output, including: In the multi-resolution image correction network, self-attention calculation is performed on image features input into the multi-resolution image correction network using the image feature position encoding corresponding to the resolution and the embedding vector corresponding to the resolution, and the corrected image features are output; The step of determining the embedding vector corresponding to the resolution includes: Perform mapping processing according to the image feature corresponding to the resolution, the image feature size corresponding to the resolution, and the reference image feature size to obtain a second feature mapping result; Generate a second size transformation coefficient according to the second feature mapping result using a second size transformation prediction network; The current embedding vector is resized using the second resize coefficient to obtain the embedding vector corresponding to the resolution.

7. The target detection method according to claim 6, characterized in that: Mapping is performed according to the image feature corresponding to the resolution, the image feature size corresponding to the resolution, and the reference image feature size to obtain a second feature mapping result, including: Performing global mean pooling processing and global maximum pooling processing on the image features corresponding to the resolution respectively, to obtain a global mean pooling feature vector corresponding to the resolution and a global maximum pooling feature vector corresponding to the resolution; Calculating a mean value of the image features corresponding to the resolution and a standard deviation of the image features corresponding to the resolution; Perform channel dimension stitching processing on the global mean pooling feature vector corresponding to the resolution, the global maximum pooling feature vector corresponding to the resolution, the mean of the image feature corresponding to the resolution, the standard deviation of the image feature corresponding to the resolution, the image feature size corresponding to the resolution, and the reference image feature size to obtain a channel dimension stitching result; The channel dimension splicing result is input into a second feature mapping network, and the second feature mapping result is output.

8. The target detection method according to claim 6, characterized in that: Generating a second size transformation coefficient according to the second feature mapping result using a second size transformation prediction network includes: A second scaling factor prediction network is used to generate a second scaling factor according to the second feature mapping result, and the second scaling factor is used as the second size transformation coefficient.

9. The target detection method according to claim 6, characterized in that: Generating a second size transformation coefficient according to the second feature mapping result using a second size transformation prediction network includes: Generate a second scaling factor according to the second feature mapping result using a second scaling factor prediction network; Generate a second translation coefficient according to the second feature mapping result using a second translation coefficient prediction network; performing normalization processing on the second scaling factor and the second translation factor respectively; The second normalized scaling coefficient and the second normalized translation coefficient are used as the second size conversion coefficient.

10. The target detection method according to claim 7, characterized in that: Updating the parameters of the multi-resolution image correction network using the perceptual loss values ​​corresponding to each of the resolutions includes: The perceptual loss values ​​corresponding to the resolutions are used to update parameters of the second feature mapping network and parameters of the second size transformation prediction network.

11. The target detection method according to claim 6, characterized in that: Performing self-attention calculation on the image features input into the multi-resolution image correction network using the image feature position encoding corresponding to the resolution and the embedding vector corresponding to the resolution, including: The query vector position code in the image feature position code corresponding to the resolution is concatenated with the embedding vector corresponding to the resolution to obtain a multi-resolution query vector; The multi-resolution query vector is input into the self-attention layer for self-attention calculation.

12. The target detection method according to claim 1, characterized in that: The method of updating the parameters of the visual perception model and the parameters of the multi-resolution image correction network by using the perceptual loss values ​​corresponding to each of the resolutions to obtain the visual perception model corresponding to the current iterative training batch includes: The regularization constraint item corresponding to each of the resolutions is calculated according to the size transformation coefficient of each of the resolutions compared with the characteristic size of the reference image; The parameters of the visual perception model and the parameters of the multi-resolution image correction network are updated according to the perceptual loss value corresponding to each resolution and the regularization constraint item corresponding to each resolution to obtain the visual perception model corresponding to the current iterative training batch.

13. The target detection method according to claim 12, characterized in that: The size transformation coefficient corresponding to the resolution makes the transformation of the image feature size corresponding to the resolution smaller, and the regularization constraint item corresponding to the resolution approaches 0.

14. The target detection method according to claim 12, characterized in that: Also includes: The weight coefficient of the regularization constraint term is updated during the iterative training of the visual perception model.

15. The target detection method according to claim 1, characterized in that: The image feature position coding includes query vector position coding and value vector position coding.

16. The target detection method according to claim 1, characterized in that: The image feature position coding includes query vector position coding, key vector position coding and value vector position coding.

17. A target detection device, characterized in that: include: A model training unit, for training a visual perception model to which a multi-resolution image correction network is added using an image sample set including image samples of multiple resolutions; In iterative training, for each resolution, an image sample of the resolution is input into the visual perception model and a perceptual loss value is calculated, and in the multi-resolution image correction network, an image feature position encoding corresponding to the resolution is determined according to a relationship between an image feature size corresponding to the resolution and a reference image feature size in the image sample set, and self-attention calculation is performed on image features input into the multi-resolution image correction network according to the image feature position encoding corresponding to the resolution, and corrected image features are output; the perceptual loss value corresponding to each resolution is used to update the parameters of the visual perception model and the parameters of the multi-resolution image correction network, so as to obtain the visual perception model corresponding to the current iterative training batch; After reaching the iterative training end condition, obtaining the trained visual perception model; The target detection unit is used to input the input image corresponding to the target detection task into the trained visual perception model to obtain the target detection result of the target detection task.

18. A target detection device, characterized in that: include: Memory for storing computer programs; A processor is used to execute the computer program, and when the computer program is executed by the processor, the steps of the target detection method according to any one of claims 1 to 16 are implemented.

19. A non-volatile storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the target detection method according to any one of claims 1 to 16 are implemented.

20. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the target detection method according to any one of claims 1 to 16 are implemented.

Images