Target model generation method, image processing method and device

Abstract

The present disclosure provides a target model generation method, an image processing method, and an apparatus, relating to technical fields such as artificial intelligence, computer vision, deep learning, large models, autonomous driving, and intelligent transportation. The specific scheme is as follows: masking a sample image based on a mask map to obtain a masked sample image; inputting the masked sample image into a preset model to obtain features of the masked region and features of the unmasked region; obtaining mask feature information based on the features of the masked region; obtaining reconstructed features of the masked region based on the mask feature information and features of the unmasked region; and pre-training the preset model based on the reconstruction loss corresponding to the reconstructed features of the masked region and the distillation loss corresponding to the features of the unmasked region to obtain a trained target model.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, and in particular to computer vision, deep learning, large models, autonomous driving, intelligent transportation and other technical fields. Background Art

[0002] The main idea behind generative image-based self-supervised pre-training methods is to mask the feature map during feature extraction and then reconstruct the masked region. However, in these training methods, accelerating model convergence and, therefore, improving model training efficiency, becomes a challenge that needs to be addressed. Summary of the Invention

[0003] The present disclosure provides a target model generation method, an image processing method, and an apparatus.

[0004] According to a first aspect of the present disclosure, a method for generating a target model is provided, comprising:

[0005] Performing mask processing on the sample image based on the mask map to obtain a masked sample image, wherein the masked sample image includes a masked area image and an unmasked area image;

[0006] Inputting the masked sample image into a preset model to obtain features of the masked area and features of the unmasked area output by the preset model, wherein the features of the masked area correspond to the masked area image, and the features of the unmasked area correspond to the unmasked area image;

[0007] Obtaining mask feature information according to the features of the mask area;

[0008] Obtaining a reconstructed feature of the masked area based on the mask feature information and the features of the unmasked area;

[0009] Based on the reconstruction loss corresponding to the reconstruction features of the masked area and the distillation loss corresponding to the features of the unmasked area, the preset model is pre-trained to obtain a trained target model.

[0010] According to a second aspect of the present disclosure, a method for generating a downstream task target model is provided, comprising:

[0011] Obtain training datasets for downstream tasks;

[0012] Generate a preset model for the downstream task based on the target model and relevant parameters corresponding to the downstream task, wherein the target model is generated according to the method described in the first aspect;

[0013] Based on the training data set of the downstream task, relevant parameters of the downstream task in the downstream task preset model are adjusted to obtain a downstream task target model, wherein the downstream task target model is used to execute the downstream task.

[0014] According to a third aspect of the present disclosure, there is provided an image processing method, comprising:

[0015] Inputting the image to be processed into a downstream task target model, wherein the downstream task target model is generated according to the method described in the second aspect;

[0016] Image processing is performed according to the downstream task target model to obtain an image processing result.

[0017] According to a fourth aspect of the present disclosure, there is provided a target model generating device, comprising:

[0018] a masking module, configured to perform masking processing on the sample image based on the mask map to obtain a masked sample image, wherein the masked sample image includes a masked area image and an unmasked area image;

[0019] a preset model processing module, configured to input the masked sample image into a preset model to obtain features of the masked area and features of the unmasked area output by the preset model, wherein the features of the masked area correspond to the masked area image, and the features of the unmasked area correspond to the unmasked area image;

[0020] A mask feature information acquisition module, configured to obtain mask feature information based on the features of the mask area;

[0021] A training module is used to obtain the reconstruction features of the masked area based on the mask feature information and the features of the unmasked area; and pre-train the preset model based on the reconstruction loss corresponding to the reconstruction features of the masked area and the distillation loss corresponding to the features of the unmasked area to obtain a trained target model.

[0022] According to a fifth aspect of the present disclosure, there is provided a downstream task target model generation device, comprising:

[0023] Downstream task training data acquisition module, used to obtain training data sets for downstream tasks;

[0024] A downstream task preset model generation module, configured to generate a downstream task preset model based on the target model and relevant parameters corresponding to the downstream task, wherein the target model is generated according to the method described in the first aspect;

[0025] A downstream task target model generation module is used to adjust the relevant parameters of the downstream task in the downstream task preset model based on the training data set of the downstream task to obtain a downstream task target model, wherein the downstream task target model is used to execute the downstream task.

[0026] According to a sixth aspect of the present disclosure, there is provided an image processing apparatus, comprising:

[0027] An input module, configured to input an image to be processed into a downstream task target model, wherein the downstream task target model is generated according to the method described in the second aspect;

[0028] The image processing module is used to perform image processing according to the downstream task target model to obtain an image processing result.

[0029] According to a seventh aspect of the present disclosure, there is provided an electronic device, including:

[0030] at least one processor; and

[0031] a memory communicatively connected to the at least one processor; wherein,

[0032] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the method described in any one of the first, second, and third aspects.

[0033] According to an eighth aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute the method described in any one of the first, second, and third aspects.

[0034] According to a ninth aspect of the present disclosure, a computer program product is provided, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements the method described in any one of the first, second and third aspects.

[0035] According to a tenth aspect of the present disclosure, a vehicle is provided, comprising the electronic device described in the seventh aspect.

[0036] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will become readily understood through the following description.

[0037] The technical solution provided by the embodiment of the present disclosure uses a preset model to extract features from a masked sample image to obtain features of the masked area and features of the unmasked area. Mask feature information is obtained based on the extracted features of the masked area. Feature reconstruction is then performed based on the mask feature information to obtain reconstructed features of the masked area. The preset model is then pre-trained based on the reconstruction loss corresponding to the reconstructed features of the masked area and the distillation loss corresponding to the features of the unmasked area to obtain initialization parameters of the trained target model. The technical solution provided by the present disclosure can provide a better model for training downstream tasks, accelerate the convergence of the model, and improve the efficiency of model training. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] The accompanying drawings are used to better understand the present invention and do not constitute a limitation of the present invention.

[0039] Figure 1 is a flowchart of a method for generating a target model according to an embodiment of the present disclosure;

[0040] Figure 2 is a schematic flow chart of a target model generation method provided according to an embodiment of the present disclosure;

[0041] Figure 3 This is a flowchart of a method for generating a downstream task target model according to an embodiment of the present disclosure;

[0042] Figure 4 is a flowchart of an image processing method provided according to an embodiment of the present disclosure;

[0043] Figure 5 is a structural diagram of a model generation device provided according to an embodiment of the present disclosure;

[0044] Figure 6 is a structural diagram of another model generation device provided according to an embodiment of the present disclosure;

[0045] Figure 7 is a structural diagram of an image processing device provided according to an embodiment of the present disclosure;

[0046] Figure 8 It is a block diagram of an electronic device provided by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0047] The following description of exemplary embodiments of the present disclosure is made in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding. These details should be considered as merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0048] The first embodiment of the present disclosure provides a method for generating a target model, such as Figure 1 As shown, including:

[0049] S101, performing mask processing on a sample image based on a mask image to obtain a masked sample image, wherein the masked sample image includes a masked area image and an unmasked area image;

[0050] S102, inputting the masked sample image into a preset model to obtain features of the masked area and features of the unmasked area output by the preset model, wherein the features of the masked area correspond to the masked area image, and the features of the unmasked area correspond to the unmasked area image;

[0051] S103, obtaining mask feature information according to the features of the mask area;

[0052] S104, obtaining a reconstructed feature of the masked area based on the mask feature information and the features of the unmasked area;

[0053] S105 , pre-training the preset model based on the reconstruction loss corresponding to the reconstruction features of the masked area and the distillation loss corresponding to the features of the unmasked area to obtain a trained target model.

[0054] The target model generation method described above can be applied to a target model generation device. The target model generation device can be located on an electronic device. The electronic device includes, but is not limited to, fixed devices and / or mobile devices. For example, fixed devices include, but are not limited to, servers, which can be cloud servers or ordinary servers. For example, mobile devices include, but are not limited to, mobile phones, laptops, etc.

[0055] The technical solution provided by the present disclosure uses a preset model to extract features from a masked sample image to obtain features of the masked area and features of the unmasked area. Mask feature information is obtained based on the extracted features of the masked area. Feature reconstruction is then performed based on the mask feature information to obtain reconstructed features of the masked area. The preset model is then pre-trained based on the reconstruction loss corresponding to the reconstructed features of the masked area and the distillation loss corresponding to the features of the unmasked area to obtain initialization parameters of the trained target model. The technical solution provided by the present disclosure can provide better initialization parameters for the training of downstream tasks, thereby reducing the loss of model training, accelerating the convergence of the model, and improving the efficiency of model training.

[0056] In some possible implementations, the method further includes: generating an initial mask image based on a maximum downsampling multiple used in the preset model; and upsampling the initial mask image to the same size as the sample image to obtain the mask image.

[0057] The generating of the initial mask map based on the maximum downsampling multiple used in the preset model may include: downsampling an image of a first size based on the maximum downsampling multiple used in the preset model to obtain an initial image of a second size, wherein the first size is the size of the sample image; setting the values ​​of some positions in the initial image of the second size to a first value based on a preset mask ratio to obtain a masked area in the initial image of the second size; setting the values ​​of other positions in the initial image except the masked area to a second value to obtain an unmasked area in the initial image; and obtaining an initial mask map based on the masked area and the unmasked area.

[0058] The first value is less than the second value, and the first value is different from the second value. Exemplarily, the first value may be equal to 0, or the first value may be close to 0; for example, the first value may be one of the following: 0, 0.1, 0.0001, etc. The second value may be equal to 0, or the second value may be close to 1; for example, the second value may be one of the following: 1, 0.99, 1.0001, etc. Preferably, the first value is 0, and the second value is 1.

[0059] For example, if the size of the sample image is HxW (H is the height of the sample image, and W is the width of the sample image), and the maximum downsampling factor used in the preset model is 16 times, then the above processing process may be: based on the maximum downsampling factor used in the preset model, downsample the image of the first size (the image of size HxW) to obtain an initial image of the second size (the image of size H1 xW1, where H1 is the height of the initial image and W1 is the width of the initial image); according to a preset mask ratio, randomly define the values ​​of some positions in the initial image as a first value to obtain a masked area in the initial image; set the values ​​of other positions in the initial image except the masked area to a second value to obtain an unmasked area in the initial image; and obtain an initial mask map based on the masked area and the unmasked area. The value of the element (smallest unit) in the initial mask map is the first value or the second value, and the size of the initial mask map is H1xW1.

[0060] Upsampling the initial mask image to the same size as the sample image to obtain the mask image may include: obtaining an upsampling multiple based on the maximum downsampling multiple used in the preset model; and upsampling the initial mask image based on the upsampling multiple to obtain the mask image. For example, continuing with the above example, if the maximum downsampling multiple used in the preset model is 16 times, then the upsampling multiple is 16 times, and the initial mask image of H1xW1 is expanded upward by 16 times, so that each element (minimum unit) in the initial mask image is repeated 16x16 times, thereby obtaining a mask image of the same size HxW as the sample image.

[0061] Through the above technical solution, an initial mask map is generated based on the maximum downsampling multiple used in the preset model, and then the initial mask map is upsampled to obtain a mask map of the same size as the sample image. In this way, a block-level mask is defined, which alleviates the problem of information leakage, so that the target model trained based on the mask map can learn higher-dimensional and more abstract features.

[0062] The performing masking on the sample image based on the mask map to obtain the masked sample image may include: performing an element-wise (minimum unit) multiplication operation on the sample image and the mask map to obtain the masked sample image.

[0063] The sample image may be an RGB image, which may be acquired by a single camera or multiple cameras.

[0064] The masked sample image includes a masked area image and an unmasked area image.

[0065] Specifically, the mask region image may be determined by determining a mask position in the masked sample image based on the position of the first value of the mask image, and determining the mask region image from the masked sample image based on the mask position. Determining the mask region image from the masked sample image based on the mask position may mean obtaining a pixel value at the mask position from the masked sample image based on the mask position, and forming the mask region image with the mask position and its corresponding pixel value.

[0066] The unmasked area image may be determined by: determining a non-masked position in the masked sample image based on the position of the second value of the mask image, and determining the unmasked area image from the masked sample image based on the non-masked position. Determining the unmasked area image from the masked sample image based on the non-masked position may mean obtaining a pixel value at the non-masked position from the masked sample image based on the non-masked position, and forming the unmasked area image with the non-masked position and its corresponding pixel value.

[0067] In some possible implementations, the preset model includes: one or more convolutional layers and one or more transformation layers in cascade.

[0068] The one or more convolutional layers may be implemented by a convolutional neural network (CNN), and the one or more convolutional layers may be used to extract local features of the masked sample image. The one or more transformation layers may be implemented by a transformer, and the one or more transformation layers may be used to extract global features of the masked sample image.

[0069] The step of inputting the masked sample image into a preset model to obtain features of the masked area and features of the unmasked area output by the preset model may include: inputting the masked sample image into a preset model, processing the masked sample image through the one or more convolutional layers in the preset model to obtain local features of the masked sample image, inputting the local features of the masked sample image into the one or more transformation layers in the preset model to obtain global features of the masked sample image output by the one or more transformation layers; and fusing the local features of the masked sample image and the global features of the masked sample image in the preset model to obtain fused features, wherein the fused features include features of the masked area and features of the unmasked area.

[0070] For example, the fused feature includes N=H1xW1 regions (H1xW1xC), where C is the number of channels. The feature of the masked region is N1, and the feature of the unmasked region is N2, where N=N1+N2. The feature of the masked region can be represented as a feature matrix of N1xC dimensions, and the feature of the unmasked region can be represented as a matrix of N2xC dimensions.

[0071] The preset model may be a backbone network of MobileVit (mobile visual transformer) or TopFormer (TokenPyramid Transformer).

[0072] Through the above technical solution, the preset model includes a convolution layer and a transformation layer, which are the basic structures commonly used in related models in the field of image processing. This application pre-trains the preset model containing the convolution layer and the transformation layer, so that the target model obtained by training based on the preset model can better adapt to various possible image processing-related downstream tasks.

[0073] Obtaining the mask feature information according to the feature of the mask area may include: using the feature of the mask area as the mask feature information.

[0074] Specifically, before using the features of the mask region as the mask feature information, the method may further include: pre-constructing initial mask feature information, and setting the values ​​in the initial mask feature information to all 0s. Accordingly, using the features of the mask region as the mask feature information may include: assigning the features of the mask region to the initial mask feature information to obtain adjusted initial mask feature information; and using the adjusted initial mask feature information as the mask feature information.

[0075] The method of pre-constructing the initial mask feature information may be to construct the initial mask feature information according to the position of the first value in the mask map.

[0076] The mask feature information can also be alternatively referred to as a mask token in the field of image processing. In related technologies, the value of a mask token is usually set to 0 and the corresponding subsequent processing is performed. However, the present application assigns the features of the mask area obtained after processing the preset model to the mask feature information. Therefore, the solution provided by the present application can make reconstruction easier and reduce reconstruction losses. Moreover, by obtaining the reconstruction features of the mask area based on this method and pre-training the preset model, the convergence speed of the preset model can be accelerated.

[0077] In some possible implementations, obtaining the reconstructed features of the masked area based on the mask feature information and the features of the unmasked area includes: inputting the mask feature information and the features of the unmasked area into a decoder, and obtaining the reconstructed features of the masked area output by the decoder.

[0078] The decoder can be designed according to the structure of the preset model, and is not limited here. For example, when the preset model includes one or more convolutional layers and one or more transform layers, the decoder can include one or more transform layers, that is, the decoder adopts a transformer structure.

[0079] When the decoder adopts a transformer structure, inputting the mask feature information and the features of the unmasked area into the decoder to obtain the reconstructed features of the masked area output by the decoder can include: using the mask feature information as a query vector (for example, expressed as query), the features of the unmasked area as a key vector (for example, expressed as Key), and a value vector (for example, expressed as Value), inputting the query vector, the key vector, and the value vector into the decoder, and performing cross-attention calculation on the query vector, the key vector, and the value vector in the decoder to obtain the reconstructed features of the masked area output by the decoder.

[0080] Through the above technical solution, a decoder is used to obtain the reconstruction features of the mask area, which can more conveniently and accurately realize the feature reconstruction of the mask area and obtain accurate reconstruction features, which is then conducive to obtaining accurate reconstruction loss in the subsequent process, thereby obtaining a more accurate target model.

[0081] In some possible embodiments, the preset model is pre-trained based on the reconstruction loss corresponding to the reconstructed features of the masked area and the distillation loss corresponding to the features of the unmasked area to obtain a trained target model, including: inputting the sample image into a feature extraction model to obtain the features of the sample image output by the feature extraction model; determining the reference features of the masked area and the reference features of the unmasked area in the features of the sample image based on the corresponding positions of the masked area image and the unmasked area image; obtaining the reconstruction loss based on the reconstruction features of the masked area and the reference features of the masked area; obtaining the distillation loss based on the features of the unmasked area and the reference features of the unmasked area; and pre-training the preset model based on the reconstruction loss and the distillation loss to obtain a trained target model.

[0082] The feature extraction model is used to extract features from the sample image. There are multiple feature extraction models, which are not limited here. Optionally, the feature extraction model can be the image branch (image encoder branch) in the CLIP (Contrastive Language-Image Pre-Training) model. The CLIP model can better achieve consistency in the semantic space of image features and text features.

[0083] The size of the features of the sample image output by the feature extraction model may be different from the size of the sample image. For example, the sample image is an RGB image with a size of 224x224, and the downsampling multiple of the feature extraction model is 16 times, then the features of the sample image output by the feature extraction model may have a dimension of 14x14x768. The reason for 14x14x768 is that the maximum downsampling multiple corresponding to the feature extraction model is 16 times, that is, the features of the sample image have 224 / 16x224 / 16=14×14 elements (minimum units), and each element (minimum unit) has a dimension of 16×16×3=768, then the features of the sample image have a dimension of 14×14×768.

[0084] Determining the reference features of the masked area and the reference features of the unmasked area in the features of the sample image based on the corresponding positions of the masked area image and the unmasked area image may include: determining the reference features of the masked area in the features of the sample image based on the corresponding position of the masked area image; and determining the reference features of the unmasked area in the features of the sample image based on the corresponding position of the unmasked area image.

[0085] Determining the reference features of the mask area in the features of the sample image based on the position corresponding to the mask area image may include: determining, according to the position corresponding to the mask area image, the features in the features of the sample image that correspond to the position of the mask image; and determining the features in the features of the sample image that correspond to the position of the mask image as the reference features of the mask area.

[0086] Determining the reference features of the unmasked area in the features of the sample image based on the position corresponding to the unmasked area image may include: determining the features of the features of the sample image corresponding to the position of the unmasked image according to the position of the unmasked area image; and determining the features of the features of the sample image corresponding to the position of the unmasked image as the reference features of the unmasked area. Optionally, obtaining the reconstruction loss based on the reconstructed features of the masked area and the reference features of the masked area may include: calculating the distance between the reconstructed features of the masked area and the reference features of the masked area to obtain the reconstruction loss.

[0087] Here, in the process of calculating the distance between the reconstructed features of the masked area and the reference features of the masked area to obtain the reconstruction loss, a first loss function can be used for calculation. The distance can refer to the difference between the reconstructed features of the masked area and the reference features of the masked area, or the mean square error, or the cosine distance, etc.; accordingly, the first loss function can be one of the mean square error (MSE) loss function, the cosine distance loss function, etc., without limitation. In a preferred example, the first loss function is the cosine distance loss function.

[0088] Optionally, when the reconstructed features of the mask area are different from the reference features of the mask area in form, obtaining the reconstruction loss based on the reconstructed features of the mask area and the reference features of the mask area may include: mapping the reconstructed features of the mask area to obtain mapping features of the mask area in the same form as the reference features of the mask area; calculating the distance between the reference features of the mask area and the mapping features of the mask area to obtain the reconstruction loss.

[0089] Here, in the process of calculating the distance between the reference features of the mask area and the mapping features of the mask area to obtain the reconstruction loss, the first loss function can also be used for calculation. The relevant description of the first loss function and the relevant description of the distance are the same as those in the previous embodiment and will not be repeated.

[0090] The step of mapping the reconstructed features of the masked region to obtain mapped features of the masked region in the same form as the reference features of the masked region may include: inputting the reconstructed features of the masked region into a head network to obtain features output by the head network; and using the features output by the head network as the mapped features of the masked region. The head network may include a fully connected layer (FC) and a normalization layer (LN).

[0091] Optionally, obtaining the distillation loss based on the feature of the unmasked area and the reference feature of the unmasked area may include: calculating a distance between the feature of the unmasked area and the reference feature of the unmasked area to obtain the distillation loss.

[0092] Here, in the process of calculating the distance between the features of the unmasked area and the reference features of the unmasked area to obtain the distillation loss, a second loss function can be used for calculation. The distance can refer to the difference between the features of the unmasked area and the reference features of the unmasked area, or the mean square error, or the cosine distance, etc.; accordingly, the second loss function can be one of the mean square error loss function, the cosine distance loss function, etc. The second loss function can be the same as or different from the first loss function, and there is no limitation here. In a preferred example, the second loss function is the cosine distance loss function.

[0093] Optionally, when the features of the unmasked area are different from the forms of the reference features of the unmasked area, obtaining the distillation loss based on the features of the unmasked area and the reference features of the unmasked area may include: mapping the features of the unmasked area to obtain mapping features of the unmasked area in the same form as the reference features of the unmasked area; calculating the distance between the reference features of the unmasked area and the mapping features of the unmasked area to obtain the distillation loss.

[0094] Here, in the process of calculating the distance between the reference feature of the unmasked area and the mapping feature of the unmasked area to obtain the distillation loss, the second loss function can also be used for calculation. The relevant description of the second loss function and the relevant description of the distance are the same as those in the previous embodiment and will not be repeated.

[0095] The mapping process of the features of the unmasked area to obtain the mapping features of the unmasked area in the same form as the reference features of the unmasked area may include: inputting the features of the unmasked area into the above-mentioned head network to obtain the features output by the head network; and using the features output by the head network as the mapping features of the unmasked area.

[0096] It should be noted that in the solution provided in this embodiment, the head network will process both the reconstructed features of the masked area and the features of the unmasked area. Therefore, the head network may process the reconstructed features of the masked area and the features of the unmasked area at the same time, or the head network may process the reconstructed features of the masked area and the features of the unmasked area in sequence (or successively, or separately). For example, the reconstructed features of the masked area can be input into the head network while the features of the unmasked area are input into the above-mentioned head network to obtain the first feature and the second feature outputted simultaneously by the head network, wherein the first feature corresponds to the mapping feature of the masked area, and the second feature corresponds to the mapping feature of the unmasked area. For another example, the reconstructed features of the masked area and the features of the unmasked area can be input into the head network respectively to obtain the first feature and the second feature outputted respectively by the head network.

[0097] The pre-training of the preset model based on the reconstruction loss and the distillation loss to obtain the trained target model may include: constructing a target loss based on the reconstruction loss and the distillation loss; and adjusting the parameters of the preset model based on the target loss to obtain the target model.

[0098] The target loss constructed based on the reconstruction loss and the distillation loss may be obtained by performing a weighted summation of the reconstruction loss and the distillation loss. In the weighted summation, the weights corresponding to the reconstruction loss and the distillation loss may be preset based on actual conditions and may be the same or different.

[0099] When the weights corresponding to the reconstruction loss and the distillation loss are the same, the reconstruction loss and the distillation loss are weighted and summed to obtain the target loss, which can be expressed as:

[0100] L=min(||f mask -f target_mask ||2+||f unmask -f target_unmask ||2)

[0101] Among them, L represents the target loss; min represents the minimum value; “||·||2” represents the norm; f mask represents the reconstructed features of the mask area; f target_mas k represents the reference feature of the mask area; ||f mask -f target_mask ||2 represents the reconstruction loss; f unmask represents the features of the unmasked area; the ftarget_unmask represents the reference feature of the unmasked area; ||f unmask -f target_unmask ||2 represents distillation loss.

[0102] Adjusting the parameters of the preset model based on the target loss to obtain the target model may include: adjusting the parameters of the preset model and the decoder based on the target loss; and obtaining the target model when the preset model meets the convergence condition, wherein the parameters of the target model are equal to the parameters of the preset model when the convergence condition is met.

[0103] Among them, the convergence condition can be set according to actual conditions, for example, it can include at least one of the following: the calculation result of the target loss no longer changes, the number of updates of the preset model reaches a preset number, the calculation result of the target loss is less than a specified value, etc. The second convergence condition is not exhaustively listed and limited here.

[0104] Through the above technical solution, a feature extraction model is used to extract the features of the sample image, and the features of the sample image corresponding to the mask image position are used as reconstruction targets, and the features of the sample image corresponding to the unmasked image position are used as distillation targets, instead of directly using the original pixels as the reconstruction targets and distillation targets. This improves the quality of self-supervised pre-training learning of the preset model, enables the preset model to extract more effective features, and improves the performance of the preset model by using distillation loss.

[0105] Combine Figure 2 For example, the process of the above target model generation method is:

[0106] S201, performing mask processing on the sample image based on the mask image to obtain a masked sample image;

[0107] S202, inputting the masked sample image into a preset model to obtain the features of the masked area output by the preset model ( Figure 2 Not drawn in ) and features of the unmasked area (i.e. Figure 2 Zv);

[0108] S203, obtaining mask feature information according to the features of the mask area;

[0109] S204: the mask feature information and the features of the unmasked area (i.e. Figure 2 Zv) is input into the decoder to obtain the reconstruction features of the mask area output by the decoder (i.e. Figure 2 (Zm);

[0110] S205, the features of the unmasked area (i.e. Figure 2 Zv) and the reconstructed features of the mask area (i.e. Figure 2 The input head network of Zm is mapped to obtain the mapping features of the unmasked area (i.e. Figure 2 Yv) and the mapping features of the mask area (i.e. Figure 2 (m)

[0111] S206, inputting the sample image into a feature extraction model to obtain the features of the sample image output by the feature extraction model; determining the reference features of the masked area in the features of the sample image based on the position corresponding to the masked area image in the masked sample image (i.e., Figure 2 Based on the position corresponding to the unmasked area image in the masked sample image, determine the reference feature of the unmasked area in the feature of the sample image (ie Figure 2 Medium TV);

[0112] S207, calculating the reference features of the mask area (i.e. Figure 2 Tm) and the mapping characteristics of the mask region (i.e. Figure 2 The distance between Ym) in the mask is used to obtain the reconstruction loss; the reference features of the unmasked area (i.e. Figure 2 Tv) and the mapping features of the unmasked area (i.e. Figure 2 The distance between the two is obtained by calculating the distillation loss; and the preset model is pre-trained based on the reconstruction loss and the distillation loss to obtain a trained target model.

[0113] Before step S207 and after step S202, the method may further include: performing dimensionality transformation on the first target feature and the second target feature so that the dimensions of the mapped features of the unmasked area and the mapped features of the masked area are the same as the dimensions of the features of the sample image.

[0114] The timing of the dimensional transformation may be before step S207 and after step S202. Different first target features and different second target features may be determined according to different timings of the dimensional transformation.

[0115] The timing of the dimensional transformation may be after step S202 and step S203, the first target feature may be the feature of the unmasked area obtained in step S202, and the second target feature may be the mask feature information obtained in step S203. The dimensional transformation of the first target feature and the second target feature so that the dimensions of the mapped features of the unmasked area and the mapped features of the masked area are the same as the dimensions of the features of the sample image may include: resizing the features of the unmasked area and the mask feature information and processing them using a linear layer so that the dimensions of the mapped features of the unmasked area and the mapped features of the masked area are the same as the dimensions of the features of the sample image.

[0116] The timing of the dimensional transformation may be after step S202 and step S204, the first target feature may be the feature of the unmasked area obtained in step S202, and the second target feature may be the reconstructed feature of the masked area obtained in step S204. The dimensional transformation of the first target feature and the second target feature so that the dimensions of the mapped features of the unmasked area and the mapped features of the masked area are the same as the dimensions of the features of the sample image may include: resizing the features of the unmasked area and the reconstructed features of the masked area and processing them with a linear layer so that the dimensions of the mapped features of the unmasked area and the mapped features of the masked area are the same as the dimensions of the features of the sample image.

[0117] The timing of the dimensional transformation may be after step S205, the first target feature may be the mapping feature of the unmasked area obtained in step S205, and the second target feature may be the mapping feature of the masked area obtained in step S205. The dimensional transformation of the first target feature and the second target feature so that the dimensions of the mapping features of the unmasked area and the mapping features of the masked area are the same as the dimensions of the features of the sample image may include: resizing the mapping features of the unmasked area and the mapping features of the masked area and processing them with a linear layer so that the dimensions of the mapping features of the unmasked area and the mapping features of the masked area are the same as the dimensions of the features of the sample image.

[0118] In the above solution, the sample images may be images related to technical fields such as autonomous driving and intelligent transportation. For example, the sample images may be images of the environment surrounding the autonomous driving vehicle captured by an image acquisition device installed on the autonomous driving vehicle. The environment images may include at least one of the following objects: buildings, road traffic facilities, motor vehicles, non-motor vehicles, pedestrians, etc. Among them, road traffic facilities may include at least one of the following: traffic signs, road markings, pedestrian overpasses, pedestrian tunnels, guardrails, bollards, dividing facilities, anti-glare screens, lighting equipment, bus stops, etc.

[0119] Finally, the effect of the target model generation method provided by this solution is explained by combining relevant technologies:

[0120] In related technologies, the main idea behind generative image self-supervised pre-training methods is to mask the feature map during feature extraction and then reconstruct the masked region using the mask tokens and the features of the unmasked region. However, in these related technologies, the mask tokens are typically initialized to 0, which makes the reconstruction process difficult, thus affecting the model's convergence speed and further leading to low model training efficiency.

[0121] The technical solution provided by the present disclosure uses a preset model to extract features of the masked sample image to obtain features of the masked area and features of the unmasked area, obtains mask feature information based on the extracted features of the masked area, reconstructs the features of the masked area based on the mask feature information, and then trains the preset model based on the reconstructed features; in this way, compared with the model training method that uses mask feature information after 0 initialization, the solution provided by the present disclosure can use the features of the masked area to obtain mask feature information, thereby reducing reconstruction loss, accelerating the convergence speed of the preset model, and improving the training efficiency of the model.

[0122] In the above scheme, in the process of obtaining the masked sample image, the mask map used is a block-level mask, which alleviates the problem of information leakage and enables the target model trained based on the mask map to learn higher-dimensional and more abstract features.

[0123] In the above scheme, the preset model includes a convolution layer and a transformation layer. By pre-training the preset model including the convolution layer and the transformation layer, the target model obtained by training based on the preset model can be better adapted to various possible image processing-related downstream tasks.

[0124] In the above scheme, a decoder is used to obtain the reconstruction features of the mask area, which can more conveniently and accurately realize the feature reconstruction of the mask area and obtain accurate reconstruction features, which is then conducive to obtaining accurate reconstruction loss in the subsequent process, thereby obtaining a more accurate target model.

[0125] In addition, the above scheme uses a feature extraction model to extract the features of the sample image, and uses the features of the sample image corresponding to the mask image position as the reconstruction target, and uses the features of the sample image corresponding to the unmasked image position as the distillation target, instead of directly using the original pixels as the reconstruction target and the distillation target. This improves the quality of the self-supervised pre-training learning of the preset model, so that the preset model can extract more effective features, and by using distillation loss, the performance of the preset model is improved.

[0126] In addition, the above scheme only performs masking on the sample image before inputting the preset model. Compared with the prior art that performs masking operations in multiple layers of the model, it alleviates the problem of statistical bias and improves the generalization ability of the target model, making it applicable to various downstream tasks. The statistical bias refers to the difference or inconsistency between the statistical distribution of the pixel values ​​of the training data input during model training and the statistical distribution of the pixel values ​​of the actual data to be processed input when the model is applied. In the prior art, masking operations are performed in multiple layers of the model (for example, making the pixel value of the masked area 0, and the masked area accounts for 70% of the total area), which will cause the pixel value distribution of the training data input when training the multiple layers to be more biased towards 0, while the pixel value distribution of the actual data to be processed input when applying the multiple layers may not be more biased towards 0, which in turn leads to the existence of statistical bias in each layer, making the model unsuitable for actual application scenarios. The technical solution provided by the present disclosure only performs mask processing on the sample image before inputting the preset model, which helps to ensure that the pixel value distribution of the training data input when the preset model is trained is closer to the pixel value distribution of the actual data to be processed input when the model is applied, thereby alleviating the statistical bias problem and improving the generalization ability of the target model.

[0127] The second embodiment of the present disclosure provides a method for generating a downstream task target model, such as Figure 3 As shown, including:

[0128] S301, obtaining a training dataset for downstream tasks;

[0129] S302, generating a preset model for the downstream task based on the target model and relevant parameters corresponding to the downstream task, wherein the target model is generated by the above method;

[0130] S303: Based on the training data set of the downstream task, adjust the relevant parameters of the downstream task in the downstream task preset model to obtain a downstream task target model, wherein the downstream task target model is used to execute the downstream task.

[0131] The aforementioned downstream task target model generation method can be applied to a downstream task target model generation device. The downstream task target model generation device can be located on an electronic device. The electronic device includes, but is not limited to, fixed devices and / or mobile devices. For example, fixed devices include, but are not limited to, servers, which can be cloud servers or ordinary servers. For example, mobile devices include, but are not limited to, mobile phones, laptop computers, etc. The electronic device can store the aforementioned target model.

[0132] The technical solution provided by the present invention is based on the principle of generative image self-supervised pre-training. After the target model is obtained by the above-mentioned method, a downstream task preset model is generated based on the target model and the relevant parameters corresponding to the downstream task. Based on the training data set of the downstream task, the relevant parameters of the downstream task in the downstream task preset model are adjusted to obtain a downstream task target model for executing the downstream task. In the downstream task scenario, only a small amount of training data is used to fine-tune the relevant parameters of the downstream task to obtain a downstream task target model suitable for the downstream task, thereby achieving better model performance benefits.

[0133] The obtaining of the training data set for the downstream task may include: obtaining the original image of the downstream task; annotating the original image according to the requirements of the downstream task to obtain an annotated image; and using the annotated image as the training data set for the downstream task.

[0134] In some possible implementations, the downstream task includes one of the following: image classification, image detection, image segmentation, and monocular 3D object detection.

[0135] Among them, image classification is to distinguish different categories of images based on the semantic information of the image. It is an important basic problem in computer vision and the basis for other high-level visual tasks such as object detection, image segmentation, object tracking, behavior analysis, and face recognition.

[0136] Image detection can refer to image target detection, that is, finding objects of interest in an image and determining their categories and locations.

[0137] Image segmentation is the technology and process of dividing an image into several specific regions with unique properties and identifying targets of interest. It is a key step from image processing to image analysis. Image segmentation includes: Image semantic segmentation. Image semantic segmentation is an important branch of the field of artificial intelligence computer vision. It combines image classification, target detection, and image segmentation technologies, and mainly classifies images at the pixel level. The result of semantic segmentation is to transform the image into color blocks with certain semantic information. Semantic segmentation technology can identify the semantic category of each color block and label each pixel with its corresponding label, thereby realizing the semantic reasoning process from the bottom layer to the top layer, and ultimately obtaining a segmented image with semantic annotation information for each pixel.

[0138] Monocular 3D object detection is a branch of 3D object detection. 3D object detection includes multi-camera 3D object detection and monocular 3D object detection.

[0139] Through the above technical solution, when the downstream task is one of image classification, image detection, image segmentation, and monocular 3D object detection, a downstream task target model for performing the above downstream task is generated based on the training data set of the above downstream task, the relevant parameters corresponding to the above downstream task, and the above target model, so that the target model can be applied to a variety of downstream tasks, thereby making the target model more widely used.

[0140] Optionally, generating a downstream task preset model based on the target model and relevant parameters corresponding to the downstream task includes: adding relevant parameters corresponding to the downstream task to some or all layers of the target model to obtain the downstream task preset model.

[0141] In some possible implementations, generating a downstream task preset model based on the target model and the relevant parameters corresponding to the downstream task includes: adding the original model of the downstream task to the target model to obtain a downstream task preset model, wherein the original model of the downstream task is constructed by the relevant parameters corresponding to the downstream task.

[0142] The original model of the downstream task can vary depending on the downstream task. For example, when the downstream task is image classification, the original model of the downstream task can be a classification head. When the downstream task is image detection, the original model of the downstream task can be a detection head. When the downstream task is image segmentation, the original model of the downstream task can be a segmentation head.

[0143] Adding the original model of the downstream task to the target model to obtain the preset model of the downstream task may include: constructing the original model of the downstream task, and concatenating the target model with the original model of the downstream task to obtain the preset model of the downstream task. For example, when the original model of the downstream task may be a classification head, the target model is used as a backbone network, and the output of the target model is concatenated with the input of the classification head to obtain the preset model of the downstream task.

[0144] Through the above technical solution, the target model is spliced ​​with the original model of the downstream task to obtain the downstream task preset model, which can improve the generation efficiency of the downstream task preset model and make the downstream task preset model more in line with the needs of the downstream task.

[0145] In some possible implementations, the processing of adjusting relevant parameters of the downstream task in the downstream task preset model based on the training data set of the downstream task to obtain the downstream task target model is related to the type of the downstream task.

[0146] In one possible example, the downstream task is image classification. Image classification has a wide range of applications in many fields, such as face recognition and intelligent video analysis in security, traffic scene recognition and obstacle classification in transportation, content-based image retrieval and automatic album classification in the internet, and image recognition in medicine.

[0147] The content of the training dataset for the downstream task is also related to the downstream task. When the downstream task is image classification, the training dataset for the downstream task is a training dataset for image classification. The training dataset for image classification may include multiple images. When the image classification is used to classify obstacles (pedestrians, other vehicles, buildings, road signs, etc.), some or all of the multiple images include annotation information for the obstacles and obstacle types.

[0148] Based on the training data set of the downstream task, the relevant parameters of the downstream task in the downstream task preset model are adjusted to obtain the processing of the downstream task target model, which may refer to: inputting multiple images in the image classification training data set into the downstream task preset model separately or simultaneously to obtain the image classification processing result of each image output by the downstream task preset model; based on the image classification processing result of each image and the annotation information of the obstacle type of the image, determining the classification-related loss function, and adjusting the relevant parameters of the downstream task in the downstream task preset model based on the classification-related loss function to obtain the downstream task target model.

[0149] In another possible example, the downstream task is image detection. Image detection has a wide range of applications in many fields, including autonomous driving. When used in autonomous driving, image detection can be used to detect objects on the road and their locations, ensuring safe driving. Objects on the road can include vehicles, people, green belts, and other objects.

[0150] The content of the training data set of the downstream task is also related to the downstream task. When the downstream task is image detection, the training data set of the downstream task is a training data set for image detection. The training data set for image detection may include multiple images. When the image detection is used to detect targets (vehicles, people, green belts, etc.) on the road and the locations of the targets, some or all of the multiple images include annotation information of the targets and the locations of the targets.

[0151] Based on the training data set of the downstream task, the relevant parameters of the downstream task in the downstream task preset model are adjusted to obtain the processing of the downstream task target model, which may refer to: inputting multiple images in the image detection training data set into the downstream task preset model separately or simultaneously to obtain the image detection processing result of each image output by the downstream task preset model; based on the image detection processing result of each image and the annotation information of the target position in the image, determining the detection-related loss function, and adjusting the relevant parameters of the downstream task in the downstream task preset model based on the detection-related loss function to obtain the downstream task target model.

[0152] In another possible example, the downstream task is image segmentation. Image segmentation has widespread applications in many fields, including autonomous driving. When used in autonomous driving, images captured by onboard cameras or lidar are fed into a neural network. A trained algorithm automatically segments and categorizes the images, enabling the vehicle to avoid obstacles such as pedestrians and vehicles.

[0153] The content of the training dataset for the downstream task is also related to the downstream task. When the downstream task is image segmentation, the training dataset for the downstream task can be a training dataset for image segmentation. The training dataset for image segmentation can include multiple images. When the image segmentation is used to segment a traffic environment image, some or all of the multiple images include color blocks in the traffic environment image and annotation information about the color block type. The traffic environment image can be an image of the surrounding environment captured by a vehicle while it is driving.

[0154] Based on the training data set of the downstream task, the relevant parameters of the downstream task in the downstream task preset model are adjusted to obtain the processing of the downstream task target model, which may refer to: inputting multiple images in the image segmentation training data set into the downstream task preset model separately or simultaneously to obtain the image segmentation processing result of each image output by the downstream task preset model; based on the image segmentation processing result of each image and the annotation information of the color block type in the image, determining the segmentation-related loss function, and adjusting the relevant parameters of the downstream task in the downstream task preset model based on the segmentation-related loss function to obtain the downstream task target model.

[0155] In another possible example, the downstream task is monocular 3D object detection. Monocular 3D object detection has a wide range of applications in many fields, for example, autonomous driving. When monocular 3D object detection is used in autonomous driving, it can use LiDAR point cloud data (monocular images) as input to perform 3D object detection.

[0156] The content of the training dataset for the downstream task is also related to the downstream task. When the downstream task is monocular 3D object detection, the training dataset for the downstream task can be a training dataset for monocular 3D object detection. The training dataset for monocular 3D object detection can include multiple images. When the monocular 3D object detection is used to detect targets and the three-dimensional positions of targets in the environment surrounding the autonomous driving vehicle, some or all of the multiple images include annotation information of the targets and the three-dimensional positions of the targets.

[0157] Based on the training data set of the downstream task, adjusting the relevant parameters of the downstream task in the downstream task preset model to obtain the processing of the downstream task target model may refer to: inputting multiple images in the training data set of monocular 3D object detection into the downstream task preset model separately or simultaneously to obtain the monocular 3D object detection result of each image output by the downstream task preset model; based on the monocular 3D object detection result of each image and the annotation information of the three-dimensional position of the target in the image, determining the 3D object detection related loss function, and adjusting the relevant parameters of the downstream task in the downstream task preset model based on the 3D object detection related loss function to obtain the downstream task target model.

[0158] The third embodiment of the present disclosure provides an image processing method, such as Figure 4 As shown, including:

[0159] S401, inputting the image to be processed into a downstream task target model, wherein the downstream task target model is generated according to the above method;

[0160] S402: Perform image processing according to the downstream task target model to obtain an image processing result.

[0161] The above-described image processing method can be applied to an image processing device. The image processing device can be located on an electronic device. The electronic device includes, but is not limited to, fixed devices and / or mobile devices. For example, fixed devices include, but are not limited to, servers, which can be cloud servers or ordinary servers. For example, mobile devices include, but are not limited to, mobile phones, laptops, etc. The electronic device stores the above-described downstream task target model.

[0162] Through the above technical solution, by processing the image using the downstream task target model obtained by the above method, a more accurate image processing result can be obtained.

[0163] In the above solution, the image to be processed may be an image related to technical fields such as autonomous driving and intelligent transportation. For example, the image to be processed may be an image of the environment surrounding the autonomous driving vehicle, captured by an image acquisition device installed on the autonomous driving vehicle. The environment image may include at least one of the following objects: buildings, road traffic facilities, motor vehicles, non-motor vehicles, pedestrians, etc. Among them, road traffic facilities may include at least one of the following: traffic signs, road markings, pedestrian overpasses, pedestrian tunnels, guardrails, bollards, dividing facilities, anti-glare screens, lighting equipment, bus stops, etc.

[0164] In some possible implementations, the image processing result includes at least one of the following: an image classification processing result, an image detection processing result, an image segmentation processing result, and a monocular 3D object detection processing result.

[0165] As can be seen from the foregoing, the downstream task includes one of the following: image classification, image detection, image segmentation, or monocular 3D object detection. Accordingly, the downstream task target model performs image processing, and the resulting image processing results also include one of the following: image classification processing results, image detection processing results, image segmentation processing results, or monocular 3D object detection processing results. This is not detailed here.

[0166] Through the above technical solution, the downstream task target model is used to perform image processing on the image to be processed, and one of the image classification processing results, image detection processing results, image segmentation processing results, and monocular 3D object detection processing results can be obtained, so that the downstream task target model can be applied to a variety of downstream tasks, thereby improving the generalization ability of the downstream task target model.

[0167] It should be noted that the process of inputting the image to be processed into the downstream task target model, performing image processing according to the downstream task target model, and obtaining the image processing result is related to the downstream task target model. In addition, the downstream task target model is related to the downstream task.

[0168] In one possible example, when the downstream task is image classification, the downstream task target model may be an image classification target model; inputting the image to be processed into the downstream task target model, performing image processing according to the downstream task target model, and obtaining an image processing result may refer to: inputting the image to be processed into the image classification target model, performing image classification according to the image classification target model, and obtaining an image classification processing result. When the image classification target model is used to classify obstacles (pedestrians, other vehicles, buildings, road signs, etc.), the image classification processing result may include information about the obstacles in the image and the obstacle type.

[0169] In another possible example, when the downstream task is image detection, the downstream task target model may be an image detection target model; inputting the image to be processed into the downstream task target model, performing image processing according to the downstream task target model, and obtaining an image processing result may refer to: inputting the image to be processed into the image detection target model, performing image detection according to the image detection target model, and obtaining an image detection processing result. When the image detection target model is used to detect targets (vehicles, people, green belts, etc.) on the road and the location of the targets, the image detection processing result may include information about the targets in the image and the location of the targets.

[0170] In another possible example, when the downstream task is image segmentation, the downstream task target model may be an image segmentation target model; inputting the image to be processed into the downstream task target model, performing image processing according to the downstream task target model, and obtaining an image processing result may refer to: inputting the image to be processed into the image segmentation target model, performing image segmentation according to the image segmentation target model, and obtaining an image segmentation processing result. When the image segmentation target model is used to segment a traffic environment image, the image segmentation processing result may include information about color blocks and color block types in the image.

[0171] In another possible example, when the downstream task is monocular 3D object detection, the downstream task target model can be a monocular 3D object detection target model; inputting the image to be processed into the downstream task target model, performing image processing according to the downstream task target model, and obtaining an image processing result can refer to: inputting the image to be processed into the monocular 3D object detection target model, performing monocular 3D object detection according to the monocular 3D object detection target model, and obtaining a monocular 3D object detection processing result. When the monocular 3D object detection target model is used to detect targets and the three-dimensional position of targets in the environment surrounding the autonomous driving vehicle, the monocular 3D object detection processing result can include the three-dimensional position information of the targets and the targets in the image.

[0172] The fourth aspect of the present disclosure provides a target model generation device, such as Figure 5 As shown, including:

[0173] A masking module 501 is configured to perform masking processing on the sample image based on the mask map to obtain a masked sample image, wherein the masked sample image includes a masked area image and an unmasked area image;

[0174] A preset model processing module 502 is configured to input the masked sample image into a preset model to obtain features of the masked area and features of the unmasked area output by the preset model, wherein the features of the masked area correspond to the masked area image, and the features of the unmasked area correspond to the unmasked area image;

[0175] The mask feature information acquisition module 503 is used to obtain mask feature information according to the features of the mask area;

[0176] The training module 504 is used to obtain the reconstruction features of the masked area based on the mask feature information and the features of the unmasked area; and pre-train the preset model based on the reconstruction loss corresponding to the reconstruction features of the masked area and the distillation loss corresponding to the features of the unmasked area to obtain a trained target model.

[0177] In some possible implementations, the mask module 501 is configured to generate an initial mask image based on the maximum downsampling multiple used in the preset model; and upsample the initial mask image to the same size as the sample image to obtain the mask image.

[0178] In some possible implementations, the training module 504 is configured to input the mask feature information and the features of the unmasked region into a decoder to obtain the reconstructed features of the masked region output by the decoder.

[0179] In some possible embodiments, the training module 504 is used to input the sample image into a feature extraction model to obtain the features of the sample image output by the feature extraction model; determine the reference features of the masked area and the reference features of the unmasked area in the features of the sample image based on the corresponding positions of the masked area image and the unmasked area image; obtain the reconstruction loss based on the reconstructed features of the masked area and the reference features of the masked area; obtain the distillation loss based on the features of the unmasked area and the reference features of the unmasked area; and pre-train the preset model based on the reconstruction loss and the distillation loss to obtain a trained target model.

[0180] In some possible implementations, the preset model includes: one or more convolutional layers and one or more transformation layers in cascade.

[0181] The fifth embodiment of the present disclosure provides a downstream task target model generation device, such as Figure 6 As shown, including:

[0182] Downstream task training data acquisition module 601, used to obtain the training data set of downstream tasks;

[0183] A downstream task preset model generation module 602 is used to generate a downstream task preset model based on the target model and relevant parameters corresponding to the downstream task, wherein the target model is generated according to the above method;

[0184] The downstream task target model generation module 603 is used to adjust the relevant parameters of the downstream task in the downstream task preset model based on the training data set of the downstream task to obtain the downstream task target model, wherein the downstream task target model is used to execute the downstream task.

[0185] In some possible implementations, the downstream task preset model generation module 602 is used to add the original model of the downstream task to the target model to obtain a downstream task preset model, wherein the original model of the downstream task is constructed by relevant parameters corresponding to the downstream task.

[0186] In some possible implementations, the downstream task includes one of the following: image classification, image detection, image segmentation, and monocular 3D object detection.

[0187] A sixth aspect of the present disclosure provides an image processing device, such as Figure 7 As shown, including:

[0188] An input module 701 is used to input the image to be processed into a downstream task target model, wherein the downstream task target model is generated according to the above method;

[0189] The image processing module 702 is used to perform image processing according to the downstream task target model to obtain an image processing result.

[0190] In some possible implementations, the image processing result includes at least one of the following: an image classification processing result, an image detection processing result, an image segmentation processing result, and a monocular 3D object detection processing result.

[0191] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0192] Figure 8A schematic block diagram of an example electronic device 800 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are provided as examples only and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0193] like Figure 8 As shown, the electronic device 800 includes a computing unit 801, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 802 or a computer program loaded from a storage unit 808 into a random access memory (RAM) 803. In the RAM 803, various programs and data required for the operation of the electronic device 800 can also be stored. The computing unit 801, the ROM 802, and the RAM 803 are connected to each other via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0194] Multiple components in the electronic device 800 are connected to the I / O interface 805, including an input unit 806, such as a keyboard, a mouse, etc.; an output unit 807, such as various types of displays, speakers, etc.; a storage unit 808, such as a magnetic disk, an optical disk, etc.; and a communication unit 809, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 809 allows the electronic device 800 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0195] The computing unit 801 can be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units that run machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 801 performs the various methods and processes described above. For example, in some embodiments, the various methods described above can be implemented as a computer software program that is tangibly contained in a machine-readable medium, such as a storage unit 808. In some embodiments, part or all of the computer program can be loaded and / or installed on the electronic device 800 via the ROM 802 and / or the communication unit 809. When the computer program is loaded into the RAM 803 and executed by the computing unit 801, one or more steps of the various methods described above can be performed. Alternatively, in other embodiments, the computing unit 801 can be configured to perform the various methods described above by any other appropriate means (e.g., by means of firmware).

[0196] According to yet another embodiment of the present disclosure, a vehicle is provided. The vehicle includes the electronic device 800 according to the above embodiment.

[0197] Various embodiments of the systems and techniques described herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-chip systems (SOCs), programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system that includes at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0198] The program code for implementing the method of the present disclosure can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device so that when the program code is executed by the processor or controller, the functions / operations specified in the flow chart and / or block diagram are implemented. The program code can be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0199] In the context of the present disclosure, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in conjunction with an instruction execution system, device or equipment. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium can include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0200] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0201] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0202] A computer system may include a client and a server. The client and server are generally remote from each other and typically interact through a communication network. The client-server relationship arises through computer programs running on the respective computers and having a client-server relationship with each other. The server may be a cloud server, a server in a distributed system, or a server integrated with a blockchain.

[0203] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved. This is not limited herein.

[0204] The above specific embodiments do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure shall be included within the scope of protection of this disclosure.

Claims

1. A target model generation method, comprising: Performing mask processing on the sample image based on the mask map to obtain a masked sample image, wherein the masked sample image includes a masked area image and an unmasked area image; Inputting the masked sample image into a preset model to obtain features of the masked area and features of the unmasked area output by the preset model, wherein the features of the masked area correspond to the masked area image, and the features of the unmasked area correspond to the unmasked area image; Obtaining mask feature information according to the features of the mask area; wherein the mask feature information is obtained according to the feature assignment of the mask area output by the preset model; Obtaining a reconstructed feature of the masked area based on the mask feature information and the features of the unmasked area; Pre-training the preset model based on the reconstruction loss corresponding to the reconstructed features corresponding to the masked area and the distillation loss corresponding to the features of the unmasked area to obtain a trained target model; The mask image and the sample image have the same size, and the mask processing includes performing an element-wise multiplication operation on the sample image and the mask image; The pre-training of the preset model based on the reconstruction loss corresponding to the reconstruction features corresponding to the masked area and the distillation loss corresponding to the features of the unmasked area to obtain the trained target model includes: Inputting the sample image into a feature extraction model to obtain features of the sample image output by the feature extraction model; Determining, based on corresponding positions of the masked area image and the unmasked area image, reference features of the masked area and reference features of the unmasked area in the features of the sample image; Obtaining a reconstruction loss based on the reconstruction features of the masked area and the reference features of the masked area; Obtaining a distillation loss based on the features of the unmasked region and a reference feature of the unmasked region; The preset model is pre-trained based on the reconstruction loss and the distillation loss to obtain a trained target model.

2. The method according to claim 1, further comprising: Generate an initial mask map based on the maximum downsampling multiple used in the preset model; The initial mask image is upsampled to the same size as the sample image to obtain the mask image.

3. The method according to claim 1, wherein The obtaining, based on the mask feature information and the features of the unmasked area, a reconstructed feature of the masked area includes: The mask feature information and the features of the unmasked area are input into a decoder to obtain the reconstructed features of the masked area output by the decoder.

4. The method according to any one of claims 1 to 3, wherein The preset model includes: one or more convolutional layers and one or more transformation layers in cascade.

5. A method for generating a downstream task target model, comprising: Obtain training datasets for downstream tasks; generating a preset model for the downstream task based on the target model and relevant parameters corresponding to the downstream task, wherein the target model is generated by the method according to any one of claims 1 to 4; Based on the training data set of the downstream task, relevant parameters of the downstream task in the downstream task preset model are adjusted to obtain a downstream task target model, wherein the downstream task target model is used to execute the downstream task.

6. The method according to claim 5, wherein: The generating of a preset model for a downstream task based on the target model and relevant parameters corresponding to the downstream task includes: The original model of the downstream task is added to the target model to obtain a preset model of the downstream task, wherein the original model of the downstream task is constructed by relevant parameters corresponding to the downstream task.

7. The method according to claim 5 or 6, wherein the downstream task comprises one of the following: image classification, image detection, image segmentation, and monocular 3D object detection.

8. An image processing method, comprising: Inputting the image to be processed into a downstream task target model, wherein the downstream task target model is generated according to the method according to any one of claims 5 to 7; Image processing is performed according to the downstream task target model to obtain an image processing result.

9. The method according to claim 8, wherein The image processing result includes at least one of the following: an image classification processing result, an image detection processing result, an image segmentation processing result, and a monocular 3D object detection processing result.

10. A target model generation device, comprising: a masking module, configured to perform masking processing on the sample image based on the mask map to obtain a masked sample image, wherein the masked sample image includes a masked area image and an unmasked area image; a preset model processing module, configured to input the masked sample image into a preset model to obtain features of the masked area and features of the unmasked area output by the preset model, wherein the features of the masked area correspond to the masked area image, and the features of the unmasked area correspond to the unmasked area image; A mask feature information acquisition module, configured to obtain mask feature information based on the features of the mask area; wherein the mask feature information is obtained by assigning features of the mask area output by the preset model; A training module, configured to obtain a reconstruction feature of the masked area based on the mask feature information and the features of the unmasked area; and pre-train the preset model based on a reconstruction loss corresponding to the reconstruction feature of the masked area and a distillation loss corresponding to the features of the unmasked area to obtain a trained target model; The mask image and the sample image have the same size, and the mask processing includes performing an element-wise multiplication operation on the sample image and the mask image; Among them, the training module is used to input the sample image into the feature extraction model to obtain the features of the sample image output by the feature extraction model; based on the corresponding positions of the mask area image and the unmasked area image, determine the reference features of the mask area and the reference features of the unmasked area in the features of the sample image; based on the reconstructed features of the mask area and the reference features of the mask area, obtain the reconstruction loss; based on the features of the unmasked area and the reference features of the unmasked area, obtain the distillation loss; based on the reconstruction loss and the distillation loss, pre-train the preset model to obtain the trained target model.

11. The device according to claim 10, wherein The mask module is configured to generate an initial mask image based on the maximum downsampling multiple used in the preset model; and upsample the initial mask image to the same size as the sample image to obtain the mask image.

12. The device according to claim 10, wherein The training module is used to input the mask feature information and the features of the unmasked area into a decoder to obtain the reconstructed features of the masked area output by the decoder.

13. The device according to any one of claims 10 to 12, wherein: The preset model includes: one or more convolutional layers and one or more transformation layers in cascade.

14. A downstream task target model generation device, comprising: Downstream task training data acquisition module, used to obtain training data sets for downstream tasks; a downstream task preset model generation module, configured to generate a downstream task preset model based on the target model and relevant parameters corresponding to the downstream task, wherein the target model is generated according to the method according to any one of claims 1 to 4; A downstream task target model generation module is used to adjust the relevant parameters of the downstream task in the downstream task preset model based on the training data set of the downstream task to obtain a downstream task target model, wherein the downstream task target model is used to execute the downstream task.

15. The device according to claim 14, wherein The downstream task preset model generation module is used to add the original model of the downstream task to the target model to obtain the downstream task preset model, wherein the original model of the downstream task is constructed by relevant parameters corresponding to the downstream task.

16. The apparatus according to claim 14 or 15, wherein the downstream task comprises one of the following: image classification, image detection, image segmentation, and monocular 3D object detection.

17. An image processing apparatus, comprising: An input module, configured to input an image to be processed into a downstream task target model, wherein the downstream task target model is generated according to the method according to any one of claims 5 to 7; The image processing module is used to perform image processing according to the downstream task target model to obtain an image processing result.

18. The device according to claim 17, wherein The image processing result includes at least one of the following: an image classification processing result, an image detection processing result, an image segmentation processing result, and a monocular 3D object detection processing result.

19. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 9.

20. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 9.

21. A computer program product comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements the method according to any one of claims 1 to 9.

22. A vehicle comprising the electronic device according to claim 19.

Images