Image feature completion method and system
The image feature completion model trained by the MAE model solves the problem of insufficient flexibility in dealing with large areas of occlusion or erasing images in the prior art, and achieves a more efficient image completion effect.
Patent Information
- Application Number
- CN202411507122.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-28
- Publication Date
- 2025-03-14
AI Technical Summary
The prior art does not have enough flexibility and low resolution when processing images that are blocked or erased by the foreground, making it difficult to efficiently solve the completion preprocessing task.
The image feature completion model is obtained through MAE model training. During model training, the random positions of the input image are partially blocked, and the remaining visible image features are additionally encoded to generate an image feature completion model.
It reduces the difficulty of detecting target images, increases the flexibility of image recognition, and improves the completion effect of obscured target images.
Smart Images
Figure CN119624805A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of image processing technology, and in particular to an image feature completion method; in addition, the present invention also relates to an image feature completion system. Background Art
[0002] In the case where key parts of user image features are missing or artificially erased, the traditional image restoration method uses a broadcast diffusion mechanism to gradually fill the area to be repaired from the direction of the equal illumination line, and can use edge information to complete the damaged image. This method is suitable for repairing small-scale damaged images. When facing large or complex texture missing areas, the diffusion method will fail. Traditional image algorithms are not flexible enough and have low resolution, making it difficult to efficiently solve the preprocessing task of completing large areas of foreground occluded or erased targets in current scenarios. Summary of the invention
[0003] In order to solve the problems existing in the prior art, at least one embodiment of the present invention provides an image feature completion method and system, which greatly reduces the difficulty of detecting the target image, increases the flexibility of image recognition, and improves the completion effect of the occluded target image. To this end, at least one image feature completion system of the present invention.
[0004] In a first aspect, an embodiment of the present invention provides an image feature completion method, comprising:
[0005] Construct training and test sets for image data feature completion;
[0006] The image feature completion model is obtained by training the MAE model. During model training, random positions of the input image are partially masked, and position codes are added to the remaining visible image features.
[0007] The occluded image is completed using the image feature completion model.
[0008] In some embodiments, the present invention provides an image feature completion method, wherein partially masking a random position of an input image and adding position encoding to the remaining visible image features comprises:
[0009] The image is divided into multiple image blocks, 75% of the image blocks are randomly masked, and the feature vectors corresponding to the remaining 25% visible image blocks are additionally position-coded and input into the encoder for image feature learning.
[0010] In some embodiments, the present invention provides an image feature completion method, wherein the additional position encoding of the feature vector corresponding to the remaining 25% visible image block comprises:
[0011] Generate a corresponding learnable position code for the remaining 25 percent visible image blocks, where the dimension of the learnable position code is the same as the dimension of the embedding vector, and append the learnable position code to the corresponding image block feature vector.
[0012] In some embodiments, the present invention provides an image feature completion method that masks 75 percent of image blocks by mosaicking.
[0013] In some embodiments, the present invention provides an image feature completion method, and the loss function during model training is as follows:
[0014]
[0015] Among them, X i is the pixel corresponding to image X, Y i is the pixel corresponding to image Y.
[0016] In a second aspect, an embodiment of the present invention further provides an image feature completion system, including:
[0017] A construction module for constructing training sets and test sets for image data feature completion;
[0018] The training module is used to obtain the image feature completion model through MAE model training;
[0019] A masking module, used to partially mask random locations of the input image;
[0020] A position coding additional module is used to add position coding to the image features that remain visible after masking;
[0021] The image completion module is used to complete the obscured image through the image feature completion model.
[0022] In some embodiments, the present invention provides an image feature completion system, wherein the training module includes:
[0023] The segmentation module is used to segment the image into multiple image blocks.
[0024] In a third aspect, an embodiment of the present invention further provides an image feature completion device, comprising at least one processor; a memory coupled to the at least one processor, the memory storing executable instructions, and the executable instructions, when executed by the at least one processor, implement the steps of the method described in any one of the first aspects above.
[0025] In a fourth aspect, an embodiment of the present invention further provides a chip for executing the steps of the method in the first aspect. Specifically, the chip includes: a processor for calling and running a computer program from a memory, so that a device equipped with the chip is used to execute the steps of the method in the first aspect.
[0026] In a fifth aspect, an embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the method described in any one of the first aspects above are implemented.
[0027] It can be seen that an image feature completion method and device of an embodiment of the present invention obtains an image feature completion model through MAE model training. During model training, random positions of the input image are partially masked, and a corresponding learnable position code is generated for the remaining visible image blocks, and attached to the corresponding image block feature vector, which greatly reduces the difficulty of detecting the target image, increases the flexibility of image recognition, and improves the completion effect of the occluded target image. BRIEF DESCRIPTION OF THE DRAWINGS
[0028] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.
[0029] Figure 1 Shown is a flow chart of an image feature completion method in an embodiment of the present invention;
[0030] Figure 2 Shown is a schematic diagram of the framework of an image feature completion system in an embodiment of the present invention. Specific implementation plan
[0031] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0032] It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. In this article, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.
[0033] [Example 1]
[0034] The inventor of this solution found that in the prior art, in the case where key parts of user image features are missing or artificially erased, the traditional image restoration method uses a broadcast diffusion mechanism to gradually fill the area to be repaired from the direction of the equal illumination line, and can use edge information to complete the damaged image. This type of method is suitable for repairing small-scale damaged images. When faced with larger or complex texture missing areas, the diffusion method will fail. Traditional image algorithms are not flexible enough and have low resolution, making it difficult to efficiently solve the task of completing the preprocessing of large-area foreground occluded targets or erased targets in current scenarios. Embodiment 1 of the present invention provides the following solution:
[0035] like Figure 1 As shown, this embodiment provides an image feature completion method, the method comprising the following steps:
[0036] Step 101: construct a training set and a test set for image data feature completion.
[0037] It should be noted that the training set and test set are constructed based on the large-scale public dataset ImageNet. ImageNet is a huge image database jointly created by the Stanford University AI Laboratory and the Princeton University Vision and Graphics Laboratory. The ImageNet project is a large-scale visual database for visual object recognition software research. More than 14 million image URLs are manually annotated by ImageNet to indicate the objects in the pictures; in at least one million images, bounding boxes are also provided. ImageNet contains more than 20,000 categories.
[0038] Step 102, an image feature completion model is obtained by MAE model training. During model training, random positions of the input image are partially masked, and position codes are added to the remaining visible image features.
[0039] It should be noted that MAE is a generative pre-trained model that can perform self-supervised learning. This model applies the training ideas of the natural language processing model to the field of computer vision, and only uses unlabeled data sets for training. The model takes the complete image as input, actively and randomly masks some image blocks, and then uses the remaining unmasked image blocks to reconstruct the complete image.
[0040] In some embodiments, megabytes of binary image data are converted into an image, the image is segmented into multiple image blocks, 75 percent of the image blocks are randomly masked, feature vectors corresponding to the remaining 25 percent visible image blocks are positionally encoded, and input into an encoder for image feature learning.
[0041] In some embodiments, 75 percent of the image blocks are masked by mosaic, or 75 percent of the image blocks are masked by smearing.
[0042] It should be noted that the position encoding can adopt 1-D position encoding, which is a position encoding method used in Visual Transformer (ViT) that takes into account the linear position information of the image block. The basic principle of 1-D position encoding is to divide the image into several image blocks and assign a position code to the feature vector corresponding to the image block. This encoding method is simple and effective, and enables the model to understand the position relationship of the image blocks in the horizontal direction.
[0043] Specifically, the input image of the encoder is uniformly adjusted to a size of 224*224, and then divided into multiple regular, non-overlapping image blocks, each of which is 16*16*C in size, where C is the number of channels of the image. Each image block is mapped to a feature vector through a convolution kernel of size 16*16*C. Seventy-five percent of the image feature vectors are masked to reduce redundant information on the image and increase the difficulty of reconstructing the image. A corresponding 1-D position code is generated for the remaining 25% of visible image blocks. The dimension of the 1-D position code is the same as that of the embedding vector. The 1-D position code is attached to the corresponding image block feature vector to retain the position information and is input into the encoder at the same time. The features are extracted from each other through a multi-layer stacked self-attention mechanism and finally encoded into a hidden layer representation. The hidden layer representation and the mark of the masked image block are input into the Transformer decoder for decoding and reconstruction of the original image.
[0044] In some embodiments, the loss function during model training is as follows:
[0045]
[0046] Among them, X iis the pixel corresponding to image X, Y i is the pixel corresponding to image Y.
[0047] Step 103: Complete the obscured image using an image feature completion model.
[0048] [Example 2]
[0049] like Figure 2 As shown, this embodiment provides an image feature completion system, including:
[0050] The construction module 201 is used to construct a training set and a test set for image data feature completion.
[0051] It should be noted that the training set and test set are constructed based on the large-scale public dataset ImageNet. ImageNet is a huge image database jointly created by the Stanford University AI Laboratory and the Princeton University Vision and Graphics Laboratory. The ImageNet project is a large-scale visual database for visual object recognition software research. More than 14 million image URLs are manually annotated by ImageNet to indicate the objects in the pictures; in at least one million images, bounding boxes are also provided. ImageNet contains more than 20,000 categories.
[0052] The training module 202 is used to obtain an image feature completion model through MAE model training.
[0053] It should be noted that MAE is a generative pre-trained model that can perform self-supervised learning. This model applies the training ideas of the natural language processing model to the field of computer vision, and only uses unlabeled data sets for training. The model takes the complete image as input, actively and randomly masks some image blocks, and then uses the remaining unmasked image blocks to reconstruct the complete image.
[0054] A segmentation module 203, used to segment the image into multiple image blocks;
[0055] A masking module 204, used to partially mask random positions of the input image;
[0056] The position code adding module 205 is used to add position codes to the image features that remain visible after being masked.
[0057] In some embodiments, megabytes of binary image data are converted into an image, the image is segmented into multiple image blocks, 75 percent of the image blocks are randomly masked, feature vectors corresponding to the remaining 25 percent visible image blocks are positionally encoded, and input into an encoder for image feature learning.
[0058] In some embodiments, 75 percent of the image blocks are masked by mosaic, or 75 percent of the image blocks are masked by smearing.
[0059] It should be noted that the position encoding can adopt 1-D position encoding, which is a position encoding method used in Visual Transformer (ViT) that takes into account the linear position information of the image block. The basic principle of 1-D position encoding is to divide the image into several image blocks and assign a position code to the feature vector corresponding to the image block. This encoding method is simple and effective, and enables the model to understand the position relationship of the image blocks in the horizontal direction.
[0060] Specifically, the input image of the encoder is uniformly adjusted to a size of 224*224, and then divided into multiple regular, non-overlapping image blocks, each of which is 16*16*C in size, where C is the number of channels of the image. Each image block is mapped to a feature vector through a convolution kernel of size 16*16*C. Seventy-five percent of the image feature vectors are masked to reduce redundant information on the image and increase the difficulty of reconstructing the image. A corresponding 1-D position code is generated for the remaining 25% of visible image blocks. The dimension of the 1-D position code is the same as that of the embedding vector. The 1-D position code is attached to the corresponding image block feature vector to retain the position information and is input into the encoder at the same time. The features are extracted from each other through a multi-layer stacked self-attention mechanism and finally encoded into a hidden layer representation. The hidden layer representation and the mark of the masked image block are input into the Transformer decoder for decoding and reconstruction of the original image.
[0061] In some embodiments, the loss function during model training is as follows:
[0062]
[0063] Among them, X i is the pixel corresponding to image X, Y i is the pixel corresponding to image Y.
[0064] The image completion module 206 is used to complete the obscured image by using an image feature completion model.
[0065] [Example 3]
[0066] This embodiment provides an image feature completion system device, including:
[0067] At least one processor; a memory coupled to the at least one processor, the memory storing executable instructions, wherein the executable instructions, when executed by the at least one processor, implement the method steps of embodiment 1 of the present invention.
[0068] In an image feature completion system device provided by an embodiment of the present invention, a processor and a memory may be provided separately or integrated together.
[0069] For example, the memory may include a random access memory, a flash memory, a read-only memory, a programmable read-only memory, a non-volatile memory or a register, etc. The processor may be a central processing unit (CPU), etc. or a graphic processing unit (GPU). The memory may store executable instructions. The processor may execute the executable instructions stored in the memory to implement the various processes described herein.
[0070] It can be understood that the memory in this embodiment can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memories. Among them, the non-volatile memory can be a ROM (Read-Only Memory), a PROM (Programmable ROM), an EPROM (Erasable PROM), an EEPROM (Electrically EPROM), or a flash memory. The volatile memory can be a RAM (Random Access Memory), which is used as an external cache. By way of example but not limitation, many forms of RAM are available, such as SRAM (Static RAM), DRAM (Dynamic RAM), SDRAM (Synchronous DRAM), DDR SDRAM (Double Data Rate SDRAM), ESDRAM (Enhanced SDRAM), SLDRAM (Synchlink DRAM), and DRRAM (Direct Rambus RAM). The memories described herein are intended to include, but are not limited to, these and any other suitable types of memories.
[0071] In some embodiments, the memory stores the following elements, upgrade packages, executable units or data structures, or a subset thereof, or an extended set thereof: an operating system and an application program.
[0072] The operating system includes various system programs, such as a framework layer, a core library layer, a driver layer, etc., which are used to implement various basic services and process hardware-based tasks. The application program includes various application programs, which are used to implement various application services. The program for implementing the method of the embodiment of the present invention can be included in the application program.
[0073] In the embodiment of the present invention, the processor calls a program or instruction stored in the memory, specifically, a program or instruction stored in an application program, and the processor is used to execute the method steps of the embodiment 1 of the present invention.
[0074] [Example 4]
[0075] This embodiment provides a chip for executing the method of the above-mentioned embodiment 1 of the present invention. Specifically, the chip includes: a processor for calling and running a computer program from a memory, so that a device equipped with the chip is used to execute the method of the embodiment 1 of the present invention.
[0076] [Example 5]
[0077] This embodiment provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the method of Embodiment 1 of the present invention are implemented.
[0078] For example, machine-readable storage media may include, but are not limited to, various known and unknown types of non-volatile memory.
[0079] In summary, embodiments 1-5 of the present invention provide an image feature completion method and device, and an image feature completion model is obtained through MAE model training. During model training, random positions of the input image are partially masked, and a corresponding learnable position code is generated for the remaining visible image blocks, and attached to the corresponding image block feature vector, which greatly reduces the difficulty of detecting the target image, increases the flexibility of image recognition, and improves the completion effect of the occluded target image.
[0080] It will be appreciated by those skilled in the art that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.
[0081] In the embodiments of the present application, the disclosed systems, devices and methods can be implemented in other ways. For example, the division of units is only a logical function division, and there can be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system. In addition, the coupling between the various units can be direct coupling or indirect coupling. In addition, the various functional units in the embodiments of the present application can be integrated into a processing unit, or can be separate physical existences, etc.
[0082] It should be understood that in the various embodiments of the present application, the size of the serial number of each process does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.
[0083] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a machine-readable storage medium. Therefore, the technical solution of the present application can be embodied in the form of a software product, which can be stored in a machine-readable storage medium, which can include several instructions to enable an electronic device to perform all or part of the technical solution described in the embodiments of the present application. The above-mentioned storage medium may include various media that can store program codes, such as ROM, RAM, removable disk, hard disk, magnetic disk or optical disk.
[0084] The above contents are only specific implementation methods of the present application, and the protection scope of the present application is not limited thereto. Those skilled in the art may make changes or substitutions within the technical scope disclosed in the present application, and these changes or substitutions shall be within the protection scope of the present application.
Claims
1. A method for completing image features, characterized in that: include: Construct training and test sets for image data feature completion; The image feature completion model is obtained by training the MAE model. During model training, random positions of the input image are partially masked, and position codes are added to the remaining visible image features. The occluded image is completed using the image feature completion model.
2. The image feature completion method according to claim 1, characterized in that: The partially masking of random positions of the input image and additional position encoding of remaining visible image features comprises: The image is divided into multiple image blocks, 75% of the image blocks are randomly masked, and the feature vectors corresponding to the remaining 25% visible image blocks are additionally position-coded and input into the encoder for image feature learning.
3. The image feature completion method according to claim 2, characterized in that: The additional position coding of the feature vectors corresponding to the remaining 25% visible image blocks comprises: Generate a corresponding learnable position code for the remaining 25 percent visible image blocks, where the dimension of the learnable position code is the same as the dimension of the embedding vector, and append the learnable position code to the corresponding image block feature vector.
4. The image feature completion method according to claim 2, characterized in that: Seventy-five percent of the image blocks are masked by mosaicking.
5. The image feature completion method according to claim 1, characterized in that: The loss function during model training is as follows: Among them, X i is the pixel corresponding to image X, Y i is the pixel corresponding to image Y.
6. An image feature completion system, characterized in that: include: A construction module for constructing training sets and test sets for image data feature completion; The training module is used to obtain the image feature completion model through MAE model training; A masking module, used to partially mask random locations of the input image; A position coding additional module is used to add position coding to the image features that remain visible after masking; The image completion module is used to complete the obscured image through the image feature completion model.
7. The image feature completion system according to claim 6, characterized in that: The training module includes: The segmentation module is used to segment the image into multiple image blocks.
8. An image feature completion device, comprising at least one processor; a memory coupled to the at least one processor, the memory storing executable instructions, characterized in that: The executable instructions, when executed by the at least one processor, enable the steps of the method according to any one of claims 1 to 5 to be implemented.
9. A chip, characterized in that: It comprises a processor, which is used to call and run a computer program from a memory, so that a device equipped with the chip executes the steps of the method as claimed in any one of claims 1 to 5.
10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.