Method for training an image feature extraction model, image recognition method and medium

Abstract

The present disclosure relates to a method for training an image feature extraction model, an image recognition method and a medium, and relates to the technical field of image recognition. In the method, an initial image feature extraction model obtains a first reconstructed image based on a first image encoding feature, a first mask image block and a second image encoding feature. The initial image feature extraction model obtains a second reconstructed image based on the second image encoding feature, a second mask image block and the first image encoding feature. Then, a total loss function is constructed based on the first reconstructed image, the second reconstructed image, a first image and a second image, and the parameters of the initial image feature extraction model are adjusted based on the total loss function to obtain a trained image feature extraction model, so that the image feature extraction model can be trained by using multi-modal images in the training stage of the image feature extraction model, and the feature extraction of the trained image feature extraction model is more accurate.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of image recognition, and in particular, to a method for training an image feature extraction model, an image recognition method, an apparatus, a medium and an electronic device. BACKGROUND

[0002] Image recognition is a new image processing technology, which aims to identify the semantic category of each pixel point in an image according to the image content. In the related art, in order to improve the accuracy of image recognition, the model is generally pre-trained. Although the related pre-training method trains the model using multi-modal images, the related multi-modal training method is difficult to effectively train the model, resulting in low model accuracy. SUMMARY

[0003] This summary is provided to introduce a selection of concepts, which are further described below in the detailed description. This summary is not intended to identify key or essential features of the claimed subject matter, nor is it used to limit the scope of the claimed subject matter's scope.

[0004] In a first aspect, the present disclosure provides a method for training an image feature extraction model, comprising:

[0005] obtaining an image pair, the image pair comprising a first image and a second image obtained by photographing a same scene in different photographing modes;

[0006] performing block processing on the first image, and performing mask processing on part of the image blocks to obtain a first masked image block and a first unmasked image block;

[0007] performing block processing on the second image, and performing mask processing on part of the image blocks to obtain a second masked image block and a second unmasked image block;

[0008] inputting the first masked image block, the first unmasked image block, the second masked image block and the second unmasked image block into an initial image feature extraction model to obtain a first reconstructed image and a second reconstructed image output by the initial image feature extraction model, wherein the first reconstructed image is obtained based on first image encoding features of the first unmasked image block, the first masked image block and second image encoding features of the second unmasked image block, and the second reconstructed image is obtained based on second image encoding features of the second unmasked image block, the second masked image block and first image encoding features of the first unmasked image block;

[0009] construct a total loss function based on the first reconstructed image, the second reconstructed image, the first image and the second image;

[0010] adjust parameters of the initial image feature extraction model based on the total loss function to obtain a trained image feature extraction model.

[0011] In a second aspect, the present disclosure provides an image recognition method, comprising:

[0012] obtaining a target image;

[0013] inputting the target image into the image feature extraction model obtained according to the method of the first aspect to obtain image coding features corresponding to the target image;

[0014] obtaining an image recognition result of the target image according to the image coding features.

[0015] In a third aspect, the present disclosure provides a device for training an image feature extraction model, comprising:

[0016] a first obtaining module configured to obtain an image pair, the image pair comprising a first image and a second image obtained by photographing a same scene in different photographing modes;

[0017] a first processing module configured to perform block processing on the first image, and perform mask processing on part of the image blocks to obtain first masked image blocks and first unmasked image blocks;

[0018] a second processing module configured to perform block processing on the second image, and perform mask processing on part of the image blocks to obtain second masked image blocks and second unmasked image blocks;

[0019] a reconstruction module configured to input the first masked image blocks, the first unmasked image blocks, the second masked image blocks and the second unmasked image blocks into an initial image feature extraction model to obtain first reconstructed image and second reconstructed image output by the initial image feature extraction model, wherein the first reconstructed image is obtained based on first image coding features of the first unmasked image blocks, the first masked image blocks and second image coding features of the second unmasked image blocks, and the second reconstructed image is obtained based on second image coding features of the second unmasked image blocks, the second masked image blocks and first image coding features of the first unmasked image blocks;

[0020] a construction module configured to construct a total loss function based on the first reconstructed image, the second reconstructed image, the first image and the second image;

[0021] an adjusting module configured to adjust parameters of the initial image feature extraction model based on the total loss function, to obtain a trained image feature extraction model.

[0022] In a fourth aspect, the present disclosure provides an image recognition device, comprising:

[0023] a second obtaining module configured to obtain a target image;

[0024] an extracting module configured to input the target image into the image feature extraction model obtained according to the device of the third aspect, to obtain image coding features corresponding to the target image;

[0025] a recognition module configured to obtain an image recognition result of the target image according to the image coding features.

[0026] In a fifth aspect, the present disclosure provides a computer readable medium having a computer program stored thereon, which, when executed by a processing device, implements the steps of the method of the first aspect, or implements the steps of the method of the second aspect.

[0027] In a sixth aspect, the present disclosure provides an electronic device, comprising:

[0028] a storage device having a computer program stored thereon;

[0029] a processing device configured to execute the computer program in the storage device, to implement the steps of the method of the first aspect, or implement the steps of the method of the second aspect.

[0030] According to the above technical solution, the initial image feature extraction model obtains a first reconstructed image based on the first image coding features of the first unmasked image block, the first masked image block, and the second image coding features of the second unmasked image block. The initial image feature extraction model obtains a second reconstructed image based on the second image coding features of the second unmasked image block, the second masked image block, and the first image coding features of the first unmasked image block. Then, a total loss function is constructed based on the first reconstructed image, the second reconstructed image, the first image, and the second image, and parameters of the initial image feature extraction model are adjusted based on the total loss function, to obtain a trained image feature extraction model. Thus, in the training stage of the image feature extraction model, the image feature extraction model can be trained by using multi-modal images, so that the feature extraction of the trained image feature extraction model is more accurate.

[0031] Other features and advantages of the present disclosure will be described in detail in the following detailed description. BRIEF DESCRIPTION OF DRAWINGS

[0032] The above-described and other features, advantages, and aspects of embodiments of the present disclosure will become more apparent as various embodiments of the present disclosure are described in conjunction with the following detailed description, which, provided in connection with the following drawings. Like reference numerals used in the various figures indicate the same or similar elements. It should be understood that the drawings are not to scale unless specifically noted. In the drawings:

[0033] Figure 1 is a flowchart of a method for training an image feature extraction model according to some embodiments.

[0034] Figure 2 is a structural diagram of an initial image feature extraction model according to some embodiments.

[0035] Figure 3 is a flowchart of an image recognition method according to some embodiments.

[0036] Figure 4 is a structural diagram of an image recognition model according to some embodiments.

[0037] Figure 5 is a structural diagram of an apparatus for training an image feature extraction model according to some embodiments.

[0038] Figure 6 is a structural diagram of an image recognition apparatus according to some embodiments.

[0039] Figure 7 is a structural diagram of an electronic device according to some embodiments. DETAILED DESCRIPTION

[0040] Embodiments of the present disclosure will be described more fully hereinafter with reference to the accompanying drawings. While several embodiments of the present disclosure are shown in the drawings, it is understood that the present disclosure can be embodied in various forms and should not be construed as being limited to the embodiments set forth herein; rather, these embodiments are provided so that this disclosure will be thorough and complete, and fully convey the scope of the present disclosure to those skilled in the art. It should be understood that the drawings and embodiments are only for illustrative purposes and are not intended to limit the scope of the present disclosure.

[0041] It should be understood that each of the steps in the method embodiments of the present disclosure can be performed in a different order and / or in parallel. In addition, the method embodiments can include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.

[0042] As used herein, the term "includes" and its variants are to be read to be analogous to "comprises," or "comprising." The term "based on" is to be read as "based, at least in part, on." The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments." Related terms have corresponding meanings.

[0043] It should be noted that the terms "first", "second", etc. mentioned in the disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0044] It should be noted that the modification of "one" or "multiple" mentioned in the disclosure is illustrative and not limiting, and those skilled in the art should understand that unless the context clearly indicates otherwise, it should be understood as "one or more".

[0045] The names of the messages or information exchanged between the devices in the embodiments of the disclosure are only for illustrative purposes, and are not used to limit the scope of the messages or information.

[0046] It can be understood that before using the technical solutions disclosed in the embodiments of the disclosure, the type, use range, use scenario, etc. of the personal information involved in the disclosure should be informed to the user and the authorization of the user should be obtained through appropriate means according to relevant laws and regulations.

[0047] For example, in response to receiving the active request of the user, the user is sent prompt information to explicitly prompt the user that the operation requested to be performed will require obtaining and using the personal information of the user. Thus, the user can voluntarily choose whether to provide personal information to the software or hardware such as electronic devices, application programs, servers or storage media that perform the operation of the technical solutions of the disclosure according to the prompt information.

[0048] As an optional but not limiting implementation, in response to receiving the active request of the user, the way of sending prompt information to the user may, for example, be the way of a pop-up window, and the prompt information may be presented in the form of text in the pop-up window. In addition, the pop-up window may also carry selection controls for the user to select "agree" or "disagree" to provide personal information to the electronic device.

[0049] It can be understood that the above notification and user authorization process is only illustrative and does not limit the implementation of the disclosure, and other ways that meet the relevant laws and regulations can also be applied to the implementation of the disclosure.

[0050] Meanwhile, it can be understood that the data (including but not limited to the data itself, the acquisition or use of the data) involved in the technical solution should comply with the requirements of the corresponding laws, regulations and relevant provisions.

[0051] Figure 1 is a flow chart of a method for training an image feature extraction model according to some embodiments. As shown in Figure 1 The present disclosure provides a method for training an image feature extraction model, which can be executed by an electronic device, specifically by an apparatus for training an image feature extraction model. The apparatus can be implemented in software and / or hardware, and configured in an electronic device. As shown in Figure 1 The method can include the following steps.

[0052] In step 110, an image pair is obtained, which includes a first image and a second image obtained by photographing a same scene in different photographing manners.

[0053] Here, the image pair includes a first image and a second image obtained by photographing a same scene in different photographing manners. For example, the first image can be an RGB (three primary colors) image obtained by photographing the scene with a visible camera, and the second image can be point cloud data obtained by scanning the scene with a laser radar. For another example, the first image can be an infrared image obtained by photographing the scene with a thermal imaging camera, and the second image can be an RGB image obtained by photographing the scene with a visible camera. For still another example, the first image can be an RGB image obtained by photographing the scene with a visible camera, and the second image can be a depth image obtained by photographing the scene with a depth camera.

[0054] In step 120, the first image is processed in blocks, and part of the image blocks in the plurality of image blocks after the processing in blocks is processed with a mask to obtain a first masked image block and a first unmasked image block.

[0055] Here, the processing in blocks of the first image means that the first image is divided into a plurality of image blocks. After obtaining the plurality of image blocks, part of the image blocks in the plurality of image blocks can be randomly processed with a mask, so as to obtain a first masked image block after the processing with a mask and a first unmasked image block without the processing with a mask.

[0056] It is worth noting that the processing with a mask means that the image features in the image block are shielded, so that the first masked image block is essentially an invisible image block, and the first unmasked image block is a visible image block.

[0057] Exemplarily, in a case that the first image is point cloud data, a plurality of center points can be sampled in the point cloud data by a farthest point sampling algorithm, and for each center point, a point cloud image block corresponding to the center point is obtained by taking the center point as a core of a K-Nearest Neighbor (KNN) algorithm, thereby obtaining a plurality of point cloud image blocks.

[0058] In step 130, the second image is subjected to a blocking process, and part of the plurality of image blocks after blocking is subjected to a mask process, thereby obtaining a second mask image block and a second unmasked image block.

[0059] Here, the blocking process of the first and second images refers to dividing the second image into a plurality of image blocks. After obtaining the plurality of image blocks, part of the plurality of image blocks can be randomly subjected to a mask process, thereby obtaining a second mask image block after the mask process and a second unmasked image block which is not subjected to the mask process.

[0060] It is worth noting that the mask process refers to shielding the image features in the image block, and the second mask image block is essentially an invisible image block, while the second unmasked image block is a visible image block.

[0061] Exemplarily, in a case that the second image is an RGB image, the RGB image can be equally divided into a plurality of image blocks.

[0062] In step 140, the first mask image block, the first unmasked image block, the second mask image block, and the second unmasked image block are input into an initial image feature extraction model, thereby obtaining a first reconstructed image and a second reconstructed image output by the initial image feature extraction model, wherein the first reconstructed image is obtained based on a first image encoding feature of the first unmasked image block, the first mask image block, and a second image encoding feature of the second unmasked image block, and the second reconstructed image is obtained based on a second image encoding feature of the second unmasked image block, the second mask image block, and a first image encoding feature of the first unmasked image block.

[0063] Here, the first image encoding feature is obtained by the initial image feature extraction model based on encoding of the first unmasked image block by an encoder, and the second image encoding feature is obtained by the initial image feature extraction model based on encoding of the second unmasked image block by the encoder. It should be understood that the first image encoding feature and the second image encoding feature will be described in detail in subsequent embodiments.

[0064] After obtaining the first mask image block, the first unmasked image block, the second mask image block, and the second unmasked image block, the first mask image block, the first unmasked image block, the second mask image block, and the second unmasked image block are input into the initial image feature extraction model.

[0065] The initial image feature extraction model obtains the first reconstructed image based on the first image coding features of the first unmasked image block, the first masked image block, and the second image coding features of the second unmasked image block. Since the first image and the second image belong to the same scene, the initial image feature extraction model extracts useful information from the second unmasked image block, reconstructs the first masked image block based on the useful information extracted from the first unmasked image block and the useful information extracted from the second unmasked image block, and obtains the first reconstructed image.

[0066] The initial image feature extraction model obtains the second reconstructed image based on the second image coding features of the second unmasked image block, the second masked image block, and the first image coding features of the first unmasked image block. Since the first image and the second image belong to the same scene, the initial image feature extraction model extracts useful information from the first unmasked image block, reconstructs the second masked image block based on the useful information extracted from the second unmasked image block and the useful information extracted from the first unmasked image block, and obtains the second reconstructed image.

[0067] It should be noted that, since the random masking is performed on part of the image blocks in the first image and the second image, the initial image feature extraction model reconstructs the masked image blocks in the first image and the second image to obtain the first reconstructed image and the second reconstructed image.

[0068] It should be understood that, in the embodiments of the present disclosure, the initial image feature extraction model extracts useful information from the other image when reconstructing the first image or the second image, so that the complementarity between the multi-modal features can be captured from the first image and the second image when reconstructing the image, and a more accurate reconstructed image can be obtained.

[0069] In step 150, a total loss function is constructed based on the first reconstructed image, the second reconstructed image, the first image, and the second image.

[0070] Here, the second loss value can be determined based on the similarity between the first reconstructed image and the first image in combination with the first contrast loss function. The third loss value can be determined based on the similarity between the second reconstructed image and the second image in combination with the second contrast loss function. Then, the total loss function is constructed based on the second loss value and the third loss value.

[0071] It should be noted that, in the embodiments of the present disclosure, the first contrast loss function and the second contrast loss function are not limited, and can be any implementable loss function, such as a mean square error loss function.

[0072] In step 160, based on the total loss function, the parameters of the initial image feature extraction model are adjusted to obtain a trained image feature extraction model.

[0073] Here, the corresponding total loss value can be calculated by the total loss function, and the parameters of the image feature extraction model are adjusted based on the total loss value, so that the adjusted initial image feature extraction model can meet the preset training condition, and the trained image feature extraction model is obtained.

[0074] It is worth noting that in the embodiments of the present disclosure, the initial image feature extraction model can be continuously trained using a given training sample set, so that the total loss value calculated based on the total loss function can reach the optimal solution, i.e. the training of the initial image feature extraction model is completed. Among them, the training sample set includes a plurality of image pairs.

[0075] Based on the above embodiments, the initial image feature extraction model obtains the first reconstructed image based on the first image encoding feature of the first unmasked image block, the first masked image block and the second image encoding feature of the second unmasked image block. The initial image feature extraction model obtains the second reconstructed image based on the second image encoding feature of the second unmasked image block, the second masked image block and the first image encoding feature of the first unmasked image block. Then based on the first reconstructed image, the second reconstructed image, the first image and the second image, a total loss function is constructed, and based on the total loss function, the parameters of the initial image feature extraction model are adjusted to obtain a trained image feature extraction model, so that in the training stage of the image feature extraction model, the image feature extraction model can be trained using multiple modal image pairs, so that the feature extraction of the trained image feature extraction model is more accurate.

[0076] Figure 2 is a structural schematic diagram of an initial image feature extraction model according to some embodiments. As Figure 2 shown, in some implementable embodiments, the initial image feature extraction model includes a first feature representation layer, a first encoder and a first decoder connected in sequence, a second feature representation layer, a second encoder and a second decoder connected in sequence.

[0077] The first feature representation layer is configured to perform feature representation processing on the first unmasked image block to obtain a first feature representation corresponding to the first unmasked image block; the first encoder is configured to encode the first feature representation to obtain a first image encoding feature; the second feature representation layer is configured to perform feature representation processing on the second unmasked image block to obtain a second feature representation corresponding to the second unmasked image block; the second encoder is configured to encode the second feature representation to obtain a second image encoding feature; the first decoder is configured to obtain a first reconstructed image according to the first image encoding feature, the first masked image block, and the second image encoding feature; and the second decoder is configured to obtain a second reconstructed image according to the second image encoding feature, the second masked image block, and the first image encoding feature.

[0078] The first feature representation layer can be configured to map the first unmasked image block to a high-dimensional space through a linear representation layer to obtain the first feature representation. The second feature representation layer can be configured to map the second unmasked image block to the high-dimensional space through the linear representation layer to obtain the second feature representation.

[0079] The first feature representation layer and the second feature representation layer can be multi-layer perceptrons.

[0080] The first encoder can be regarded as an unsupervised feature extraction network, and the first encoder can encode the first feature representation into a low-dimensional representation. The second encoder can be regarded as an unsupervised feature extraction network, and the second encoder can encode the second feature representation into a low-dimensional representation. It should be noted that the first encoder and the second encoder are used to encode the visible image blocks.

[0081] The first decoder receives the first image encoding feature output by the first encoder and the second image encoding feature output by the second encoder, and obtains the first reconstructed image according to the first image encoding feature, the first masked image block, and the second image encoding feature. The first decoder actually uses the first image encoding feature representing the visible image in the first image and the second image encoding feature representing the visible image in the second image, and uses the first masked image block as a learnable parameter (supervision signal) of the first decoder to decode the learnable parameter to predict the masked image block in the first image and obtain the first reconstructed image. Through the second image encoding feature, the first decoder can extract useful information from the second image to assist in reconstructing the masked image block in the first image based on the useful information extracted from the second image.

[0082] The second decoder receives the second image encoding features output by the second encoder, the first image encoding features output by the first encoder, and obtains the second reconstructed image according to the first image encoding features, the second mask image block, and the second image encoding features. In fact, the second decoder uses the second image encoding features representing the visible image in the second image and the first image encoding features representing the visible image in the first image, and decodes the learnable parameters (supervision signals) of the second decoder using the second mask image block as the supervision signals, to predict the masked image block in the second image and obtain the second reconstructed image. Through the first image encoding features, the second decoder can extract useful information from the first image to assist in reconstructing the masked image block in the second image based on the useful information extracted from the first image.

[0083] It is worth noting that the first decoder and the second decoder can be used to reconstruct the details and corresponding spatial dimensions of the first image and the second image through network layers such as linear layers. For example, the first decoder and the second decoder can adopt a Transformer structure (a model that uses an attention mechanism to obtain contextual information).

[0084] It is worth noting that the first decoder and the second decoder in the initial image feature extraction model are only used in the training stage, and the first decoder and the second decoder can not be needed in the subsequent application stage. Therefore, the network structure of the first decoder and the second decoder can be selected according to actual conditions. For example, using a deep network structure of the first decoder and the second decoder can express the latent representation as more abstract semantic information, and using a shallow and narrow network structure of the first decoder and the second decoder can make the first decoder and the second decoder faster.

[0085] Thus, by using the initial image feature extraction model shown in Figure 2 , the initial image feature extraction model can be more effectively trained in the training stage by using the multi-modal features of the first image and the second image, so as to obtain the first encoder and the second encoder for the first image or the second image that are comprehensive and effective.

[0086] In some implementable embodiments, in step 150, a total loss function can be constructed based on the first image encoding features, the second image encoding features, the first reconstructed image, the second reconstructed image, the first image, and the second image.

[0087] Here, the first image encoding features are obtained by the initial image feature extraction model based on the encoder encoding the first unmasked image block. As shown in Figure 2As shown, the initial image feature extraction model processes the first unmasked image block into a first feature representation through a first representation layer, and encodes the first feature representation into a first image encoded feature through a first encoder. The second image encoded feature is obtained by the initial image feature extraction model based on encoding the second unmasked image block through the encoder. As shown, Figure 2 As shown, the initial image feature extraction model processes the second unmasked image block into a second feature representation through a second representation layer, and encodes the second feature representation into a second image encoded feature through a second encoder.

[0088] In some embodiments, a first loss value can be determined based on the first image encoded feature and the second image encoded feature, a second loss value can be determined based on the first reconstructed image and the first image, a third loss value can be determined based on the second reconstructed image and the second image, and a total loss function can be constructed based on the first loss value, the second loss value and the third loss value.

[0089] The first loss value can be calculated based on the cosine similarity between the first image encoded feature and the second image encoded feature in combination with the third contrastive loss function. In the training process, the cosine similarity between the matched first image encoded feature and the second image encoded feature can be maximized, and the cosine similarity between the unmatched first image encoded feature and the second image encoded feature can be minimized.

[0090] Exemplarily, the total loss function can be Loss 总 =aLoss1+bLoss2+cLoss3, where Loss 总 is the total loss value of the total loss function, Loss1 is the first loss value, Loss2 is the second loss value, Loss3 is the third loss value, a, b, c are loss weights respectively, and the values of a, b, c can be constants.

[0091] In some embodiments, the total loss function can be constructed according to the product between the first loss value and the first learnable parameter, the product between the second loss value and the second learnable parameter, and the product between the third loss value and the third learnable parameter.

[0092] For example, the total loss function can be Loss 总 =W1Loss1+W2Loss2+W3Loss3, where Loss 总 is the total loss value of the total loss function, Loss1 is the first loss value, Loss2 is the second loss value, Loss3 is the third loss value, W1 is the first learnable parameter, W2 is the second learnable parameter, and W3 is the third learnable parameter.

[0093] It is worth noting that the first learnable parameter, the second learnable parameter and the third learnable parameter are constantly optimized with the optimization of the parameters of the initial image feature extraction model. During the training process, the first loss value, the second loss value and the third loss value can be combined through grid search, and the space range of grid search can be reduced through the first learnable parameter, the second learnable parameter and the third learnable parameter, thereby saving computing resources.

[0094] Therefore, by adjusting the parameters of the initial image feature extraction model through the total loss function, the parameters of the initial image feature extraction model can be quickly optimized, and an accurate image feature extraction model trained is obtained.

[0095] Figure 3 is a flowchart of an image recognition method according to some embodiments. As shown in Figure 3 The present disclosure provides an image recognition method, which can be executed by an electronic device, specifically, an image recognition apparatus. The apparatus can be implemented in software and / or hardware, and configured in an electronic device. As shown in Figure 3 The method can include the following steps.

[0096] In step 310, a target image is obtained.

[0097] Here, the target image can be an image consistent with the type of the first image or the second image described in the above embodiments. Of course, the target image can also be a pair of images as described in the above embodiments. For example, the target image includes a third image and a fourth image obtained by taking the same scene in different shooting modes. It should be understood that the meanings of the third image and the fourth image are consistent with those of the first image and the second image in the above embodiments, and will not be repeated here.

[0098] In step 320, the target image pair is input into the image feature extraction model to obtain the image encoding features corresponding to the target image pair.

[0099] Here, the image feature extraction model is obtained by training according to the method for training the image feature extraction model provided in the above embodiments.

[0100] As some examples, when the target image is an image consistent with the type of the first image shown in the above embodiments, the image feature extraction model can include a first feature representation layer and a first encoder connected in sequence.

[0101] The first feature representation layer and the first encoder of the image feature extraction model in the image recognition method can be obtained based on the method for training the image feature extraction model provided in the above embodiments. The target image is input into the image feature extraction model, and the image feature extraction model processes the target image into a target feature representation through the first feature representation layer, and processes the target feature representation into an image coding feature through the first encoder.

[0102] As further examples, when the target image is an image consistent with the type of the second image shown in the above embodiments, the image feature extraction model can include a second feature representation layer and a second encoder connected in sequence.

[0103] The second feature representation layer and the second encoder of the image feature extraction model in the image recognition method can be obtained based on the method for training the image feature extraction model provided in the above embodiments. The target image is input into the image feature extraction model, and the image feature extraction model processes the target image into a target feature representation through the second feature representation layer, and processes the target feature representation into an image coding feature through the second encoder.

[0104] As further examples, when the target image includes a third image and a fourth image, the image feature extraction model can include a first feature representation layer and a first encoder connected in sequence, and a second feature representation layer and a second encoder connected in sequence.

[0105] The first feature representation layer, the first encoder, the second feature representation layer, and the second encoder of the image feature extraction model in the image recognition method can be obtained based on the method for training the image feature extraction model provided in the above embodiments. By inputting the target image into the image feature extraction model, the image feature extraction model processes the third image into a first target feature representation through the first feature representation layer, and processes the first target feature representation into a third image coding feature through the first encoder. The image feature extraction model processes the fourth image into a second target feature representation through the second feature representation layer, and processes the second target feature representation into a fourth image coding feature through the second encoder. The third image coding feature and the fourth image coding feature are the image coding feature of the target image.

[0106] In step 330, according to the image coding feature, an image recognition result of the target image is obtained.

[0107] Here, after obtaining the image coding feature, the image recognition result of the target image can be determined according to the image coding feature. The image recognition result includes the semantic category of the object in the target image and the probability corresponding to the semantic category.

[0108] As some examples, when the target image is an image consistent with the type of the first image shown in the above embodiments, the target image encoding feature corresponding to the target image can be input into the prediction layer to obtain the image recognition result. The prediction layer predicts the image recognition result of the target image based on the target image encoding feature. For example, the prediction layer can be a BP (Back Propagation) layer.

[0109] As some examples, when the target image is an image consistent with the type of the first image shown in the above embodiments, the target image encoding feature corresponding to the target image can be input into the prediction layer to obtain the image recognition result. The prediction layer predicts the image recognition result of the target image based on the target image encoding feature. For example, the prediction layer can be a BP (Back Propagation) layer.

[0110] As some examples, when the target image includes a third image and a fourth image, the third image encoding feature and the fourth image encoding feature can be input into the fusion layer to obtain the fusion feature, and the fusion feature is input into the prediction layer to obtain the image recognition result of the target image. The fusion layer can be used to splice the third image encoding feature and the fourth image encoding feature into the fusion feature, and the prediction layer predicts the image recognition result of the target image based on the fusion feature.

[0111] Figure 4 is a structural schematic diagram of an image recognition model according to some embodiments. The image recognition method provided by the embodiments of the present disclosure can be implemented by the image recognition model shown in Figure 4 As shown in Figure 4 When the target image includes a third image and a fourth image, the image recognition model can include a first image feature extraction model 410, a second image feature extraction model 420, a fusion layer 430, and a prediction layer 440.

[0112] The first image feature extraction model 410 can include a first feature representation layer 411 and a first encoder 412 connected in sequence, and the second image feature extraction model 420 can include a second feature representation layer 421 and a second encoder 422 connected in sequence. It should be noted that Figure 4 The first feature representation layer 411, the first encoder 412, the second feature representation layer 421, and the second encoder 422 shown in

[0113] The third image is input into the first feature representation layer 411, the first feature representation layer 411 processes the third image into a third feature representation, and the first encoder 412 encodes the third feature representation into a third image encoding feature. The fourth image is input into the second feature representation layer 421, the second feature representation layer 421 processes the fourth image into a fourth feature representation, and the second encoder 422 encodes the fourth feature representation into a fourth image encoding feature. The fusion layer 430 is used to fuse the third image encoding feature and the fourth image encoding feature to obtain a fused feature. The prediction layer 440 is used to obtain an image recognition result according to the fused feature output by the fusion layer 430.

[0114] Therefore, based on the image feature extraction model obtained by the method for training an image feature extraction model, accurate image features can be extracted from a target image, and an accurate image recognition result can be obtained.

[0115] Figure 5 is a structural schematic diagram of an apparatus for training an image feature extraction model according to some embodiments. As shown in Figure 5 , the apparatus 500 for training an image feature extraction model comprises:

[0116] The first acquisition module 501 is configured to acquire an image pair, wherein the image pair comprises a first image and a second image obtained by photographing a same scene in different photographing modes;

[0117] The first processing module 502 is configured to perform block processing on the first image, and perform mask processing on part of the image blocks after the block processing, to obtain first masked image blocks and first unmasked image blocks;

[0118] The second processing module 503 is configured to perform block processing on the second image, and perform mask processing on part of the image blocks after the block processing, to obtain second masked image blocks and second unmasked image blocks;

[0119] The reconstruction module 504 is configured to input the first masked image blocks, the first unmasked image blocks, the second masked image blocks, and the second unmasked image blocks into an initial image feature extraction model, to obtain first reconstructed images and second reconstructed images output by the initial image feature extraction model, wherein the first reconstructed images are obtained based on first image encoding features of the first unmasked image blocks, the first masked image blocks, and second image encoding features of the second unmasked image blocks, and the second reconstructed images are obtained based on second image encoding features of the second unmasked image blocks, the second masked image blocks, and first image encoding features of the first unmasked image blocks;

[0120] The constructing module 505 is configured to construct a total loss function based on the first reconstructed image, the second reconstructed image, the first image, and the second image.

[0121] The adjusting module 506 is configured to adjust parameters of the initial image feature extraction model based on the total loss function to obtain a trained image feature extraction model.

[0122] Optionally, the constructing module 505 is specifically configured to:

[0123] construct the total loss function based on the first image encoding feature, the second image encoding feature, the first reconstructed image, the second reconstructed image, the first image, and the second image.

[0124] Optionally, the constructing module 505 is specifically configured to:

[0125] determine a first loss value based on the first image encoding feature and the second image encoding feature;

[0126] determine a second loss value based on the first reconstructed image and the first image;

[0127] determine a third loss value based on the second reconstructed image and the second image;

[0128] construct the total loss function based on the first loss value, the second loss value, and the third loss value.

[0129] Optionally, the constructing module 505 is specifically configured to:

[0130] construct the total loss function according to a product between the first loss value and a first learnable parameter, a product between the second loss value and a second learnable parameter, and a product between the third loss value and a third learnable parameter.

[0131] Optionally, the initial image feature extraction model includes a first feature representation layer, a first encoder, and a first decoder connected in sequence, a second feature representation layer, a second encoder, and a second decoder connected in sequence.

[0132] The first feature representation layer is configured to perform feature representation processing on the first unmasked image block to obtain a first feature representation corresponding to the first unmasked image block.

[0133] The first encoder is configured to encode the first feature representation to obtain the first image encoding feature.

[0134] The second feature representation layer is configured to perform feature representation processing on the second unmasked image block to obtain a second feature representation corresponding to the second unmasked image block.

[0135] The second encoder is configured to encode the second feature representation to obtain a second image coding feature.

[0136] The first decoder is configured to obtain the first reconstructed image according to the first image coding feature, the first masked image block, and the second image coding feature.

[0137] The second decoder is configured to obtain the second reconstructed image according to the second image coding feature, the second masked image block, and the first image coding feature.

[0138] The functions performed by each functional module in the apparatus 500 have been described in detail in the method section, and will not be described here again.

[0139] Figure 6 is a structural schematic diagram of an image recognition apparatus according to some embodiments. As shown in Figure 6 The image recognition apparatus 600 includes:

[0140] The second acquisition module 601 is configured to acquire a target image.

[0141] The extraction module 602 is configured to input the target image into an image feature extraction model obtained according to the apparatus to obtain an image coding feature corresponding to the target image.

[0142] The recognition module 603 is configured to obtain an image recognition result of the target image according to the image coding feature.

[0143] The functions performed by each functional module in the apparatus 600 have been described in detail in the method section, and will not be described here again.

[0144] Reference will be made to Figure 7 which shows a structural schematic diagram of an electronic device 700 suitable for implementing the embodiments of the present disclosure. The terminal device in the embodiments of the present disclosure can include, but is not limited to, mobile terminals such as mobile phones, notebook computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Tablet Personal Computers), PMPs (Portable Multimedia Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), and the like, and fixed terminals such as digital TVs, desktop computers, and the like. Figure 7 The electronic device shown is only an example, and should not impose any limitation on the functions and use range of the embodiments of the present disclosure.

[0145] AsFigure 7 As shown, the electronic device 700 can include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 701 that can perform various appropriate actions and processes according to programs stored in a read-only memory (ROM) 702 or loaded into a random access memory (RAM) 703 from a storage device 708. Various programs and data required for the operation of the electronic device 700 are also stored in the RAM 703. The processing device 701, the ROM 702, and the RAM 703 are connected to each other through a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0146] In general, the following devices can be connected to the I / O interface 705: input devices 706 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; output devices 707 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 708 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 709. The communication devices 709 can allow the electronic device 700 to communicate wirelessly or wired with other devices to exchange data. Although Figure 7 The electronic device 700 is shown with various devices, but it should be understood that all of the illustrated devices are not required, and fewer or different devices can alternatively be implemented.

[0147] In particular, the processes described above with reference to the flowcharts can be implemented as a computer software program according to embodiments of the present disclosure. For example, embodiments of the present disclosure include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods illustrated by the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network through the communication devices 709, or installed from the storage devices 708, or installed from the ROM 702. When the computer program is executed by the processing device 701, the above-described functions defined in the methods of embodiments of the present disclosure are performed.

[0148] It is noted that the aforementioned computer-readable medium of the present disclosure can be a computer-readable signal medium or a computer-readable storage medium or any combination thereof. The computer-readable storage medium can be, for example and without limitation, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of the computer-readable storage medium can include, but are not limited to, an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the present disclosure, the computer-readable storage medium can be any tangible medium that contains or stores a program used by or in connection with an instruction execution system, apparatus, or device. In the present disclosure, the computer-readable signal medium can include a computer-readable program code transmitted by a computer-readable storage medium or carried by a carrier wave in a baseband or as part of a carrier wave. Such a propagated computer-readable signal medium can take various forms, including but not limited to electro-magnetic, optical, or any suitable combination thereof. The computer-readable signal medium can also be any computer-readable medium that can be used to carry or store a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained in the computer-readable medium can be transmitted by any suitable medium, including but not limited to wire, cable, RF (radio frequency), or the like, or any suitable combination of the foregoing.

[0149] In some embodiments, the electronic device can communicate using any currently known or future developed network protocol, such as HTTP (HyperText Transfer Protocol), and can be interconnected with digital data communications (e.g., a communications network) of any form or medium (e.g., a communications network). Examples of communications networks include local area networks ("LAN"), wide area networks ("WAN"), internetworks (e.g., the Internet), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future developed networks.

[0150] The aforementioned computer-readable medium can be included in the aforementioned electronic device; or can exist separately from the electronic device without being incorporated into the electronic device.

[0151] The computer readable medium carries one or more programs, when the one or more programs are executed by the electronic device, the electronic device is caused to: obtain an image pair, the image pair comprising a first image and a second image obtained by photographing a same scene in different photographing modes; perform block processing on the first image, and perform mask processing on part of the image blocks to obtain a first masked image block and a first unmasked image block; perform block processing on the second image, and perform mask processing on part of the image blocks to obtain a second masked image block and a second unmasked image block; input the first masked image block, the first unmasked image block, the second masked image block and the second unmasked image block into an initial image feature extraction model to obtain a first reconstructed image and a second reconstructed image output by the initial image feature extraction model, wherein the first reconstructed image is obtained based on first image coding features of the first unmasked image block, the first masked image block and second image coding features of the second unmasked image block, and the second reconstructed image is obtained based on second image coding features of the second unmasked image block, the second masked image block and first image coding features of the first unmasked image block; construct a total loss function based on the first reconstructed image, the second reconstructed image, the first image and the second image; and adjust parameters of the initial image feature extraction model based on the total loss function to obtain a trained image feature extraction model.

[0152] Alternatively, the computer readable medium carries one or more programs, when the one or more programs are executed by the electronic device, the electronic device is caused to: obtain a target image; input the target image into an image feature extraction model obtained by the method for training an image feature extraction model according to the above embodiment to obtain image coding features corresponding to the target image; and obtain an image recognition result of the target image based on the image coding features.

[0153] Computer program code for carrying out operations of the present disclosure can be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C++ or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider).

[0154] The computer program instructions can also be loaded onto a computer or other programmable information processing apparatus to cause a series of operations to be performed on the computer or other programmable information processing apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable information processing apparatus implement the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0155] The modules involved in the embodiments of the present disclosure can be implemented in the manner of software or hardware. Among them, the name of the module does not constitute a limitation to the module itself in some cases.

[0156] The functions described in the foregoing description can be implemented, at least in part, by one or more hardware logic components. For example, and without limitation, illustrative types of hardware logic components that can be used include Field-programmable Gate Arrays (FPGAs), Program-specific Integrated Circuits (ASICs), Program-specific Standard Products (ASSPs), System-on-a-chip systems (SOCs), Complex Programmable Logic Devices (CPLDs), etc.

[0157] In the context of this disclosure, a machine-readable medium can be a tangible medium that contains or stores a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include but is not limited to an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of the machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0158] The above description is only preferred embodiments of the present disclosure and the explanation of the technical principles of the application. Those skilled in the art should understand that the disclosure range involved in the present disclosure is not limited to the technical solutions formed by the specific combinations of the above technical features, and should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the technical solutions formed by replacing the above features with the technical features disclosed in the present disclosure (but not limited to) having similar functions.

[0159] In addition, although each operation is described in a particular order, this should not be understood as requiring the operations to be performed in the specific order shown or in a sequential order. In certain circumstances, multitasking and parallel processing can be advantageous. Similarly, although several implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Certain features described in the context of separate embodiments can also be combined in a single embodiment. Conversely, various features described in the context of a single embodiment can also be separated and implemented in multiple embodiments.

[0160] Although the subject matter has been described in language specific to structural features and / or methodological acts, it is to be understood that the subject defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are disclosed as example forms of implementing the claims. With respect to the devices in the above-described embodiments, the specific manner in which the various modules perform operations has been described in detail in the embodiments related to the method, and will not be described in detail here.

Claims

1. A method for training an image feature extraction model, characterized in that, The method comprises: obtaining an image pair, the image pair comprising a first image and a second image obtained by photographing a same scene by using different photographing manners; performing block processing on the first image, and performing mask processing on part of the image blocks after the block processing, to obtain a first masked image block and a first unmasked image block; performing block processing on the second image, and performing mask processing on part of the image blocks after the block processing, to obtain a second masked image block and a second unmasked image block; inputting the first masked image block, the first unmasked image block, the second masked image block and the second unmasked image block into an initial image feature extraction model, to obtain a first reconstructed image and a second reconstructed image output by the initial image feature extraction model, wherein the first reconstructed image is obtained based on first image coding features of the first unmasked image block, the first masked image block and second image coding features of the second unmasked image block, and the second reconstructed image is obtained based on second image coding features of the second unmasked image block, the second masked image block and first image coding features of the first unmasked image block; constructing a total loss function based on the first reconstructed image, the second reconstructed image, the first image and the second image; adjusting parameters of the initial image feature extraction model based on the total loss function, to obtain a trained image feature extraction model.

2. The method of claim 1, wherein, The method of constructing the total loss function based on the first reconstructed image, the second reconstructed image, the first image and the second image comprises: constructing the total loss function based on the first image coding features, the second image coding features, the first reconstructed image, the second reconstructed image, the first image and the second image.

3. The method of claim 2, wherein, The method of constructing the total loss function based on the first image coding features, the second image coding features, the first reconstructed image, the second reconstructed image, the first image and the second image comprises: determining a first loss value based on the first image coding features and the second image coding features; determining a second loss value based on the first reconstructed image and the first image; determining a third loss value based on the second reconstructed image and the second image; constructing the total loss function based on the first loss value, the second loss value and the third loss value.

4. The method of claim 3, wherein, The method of constructing the total loss function based on the first loss value, the second loss value and the third loss value comprises: constructing the total loss function according to a product between the first loss value and a first learnable parameter, a product between the second loss value and a second learnable parameter and a product between the third loss value and a third learnable parameter.

5. The method according to any one of claims 1 to 4, characterized in that, The initial image feature extraction model comprises a first feature representation layer, a first encoder and a first decoder connected in sequence, and a second feature representation layer, a second encoder and a second decoder connected in sequence. The first feature representation layer is configured to perform feature representation processing on the first unmasked image block to obtain a first feature representation corresponding to the first unmasked image block; The first encoder is configured to encode the first feature representation to obtain the first image coding feature; The second feature representation layer is configured to perform feature representation processing on the second unmasked image block to obtain a second feature representation corresponding to the second unmasked image block; The second encoder is configured to encode the second feature representation to obtain the second image coding feature; The first decoder is configured to obtain the first reconstructed image according to the first image coding feature, the first masked image block, and the second image coding feature; The second decoder is configured to obtain the second reconstructed image according to the second image coding feature, the second masked image block, and the first image coding feature.

6. An image recognition method characterized by, The method comprises the following steps: An image is obtained; The image is input into an image feature extraction model; An image coding feature of the image is obtained by using the image feature extraction model; An image recognition result of the image is obtained according to the image coding feature.

7. An apparatus for training an image feature extraction model, the apparatus comprising: The method comprises the following steps: A first image acquisition module is configured to acquire an image pair, wherein the image pair comprises a first image and a second image obtained by capturing a same scene by using different capturing manners; A first processing module is configured to perform block processing on the first image, and perform mask processing on part of the image blocks after the block processing to obtain first masked image blocks and first unmasked image blocks; A second processing module is configured to perform block processing on the second image, and perform mask processing on part of the image blocks after the block processing to obtain second masked image blocks and second unmasked image blocks; A reconstruction module is configured to input the first masked image blocks, the first unmasked image blocks, the second masked image blocks, and the second unmasked image blocks into an initial image feature extraction model to obtain first reconstructed images and second reconstructed images output by the initial image feature extraction model, wherein the first reconstructed images are obtained based on first image coding features of the first unmasked image blocks, the first masked image blocks, and second image coding features of the second unmasked image blocks, and the second reconstructed images are obtained based on second image coding features of the second unmasked image blocks, the second masked image blocks, and first image coding features of the first unmasked image blocks; A construction module is configured to construct a total loss function based on the first reconstructed images, the second reconstructed images, the first image, and the second image; An adjustment module is configured to adjust parameters of the initial image feature extraction model based on the total loss function to obtain a trained image feature extraction model.

8. An image recognition apparatus characterized by comprising: The method comprises the following steps: A second image acquisition module is configured to acquire a target image; An extraction module is configured to input the target image into an image feature extraction model; An image coding feature of the target image is obtained by using the image feature extraction model obtained by the device according to claim 7; An image recognition result of the target image is obtained according to the image coding feature. An identification module configured to obtain an image recognition result of the target image according to the image coding feature.

9. A computer readable medium having stored thereon a computer program, characterized in that The program is executed by the processing device to implement the steps of the method of any one of claims 1 to 5, or to implement the steps of the method of claim 6.

10. An electronic device, comprising: Comprising: A storage device having a computer program stored thereon; A processing device for executing the computer program in the storage device to implement the steps of the method of any one of claims 1 to 5, or to implement the steps of the method of claim 6.

Images