Image processing method based on neural network
Through object detection and feature extraction, LoRA training model is used to generate image models that meet the specified requirements, solving the problem of portrait image generation under the influence of noise, and achieving accurate face shape matching and image generation to ensure that the generated results meet user requirements.
Patent Information
- Application Number
- CN202410942250.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-15
- Publication Date
- 2024-11-15
AI Technical Summary
When the existing AIGC image generates portrait pictures, noises such as background, accessories on faces, hair, etc. in the pictures uploaded by the user affect the generation results, resulting in the generated portrait pictures that do not meet the face shape requirements of the user.
By obtaining the user's front face image and custom face template images, object detection and feature extraction are performed, and training models are used to generate training models that meet the specified image requirements, avoid noise influence, accurately match the template face shape, and image generation.
Accurate detection and optimization of user portrait face shapes is realized, ensuring that the generated portrait pictures more meet the user's face shape requirements, improving the accuracy and flexibility of image generation, and achieving fully automated batch image generation.
Smart Images

Figure CN118968583A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of artificial intelligence technology, and in particular to an image processing method based on a neural network. Background Art
[0002] LoRA (Low-Rank Adaptation of Large Language Models) is an efficient technology for fine-tuning large language models, aiming to reduce the number of parameters and computational requirements during training. In image generation tasks, LoRa can be adjusted for specific styles or details without the need to fully retrain the entire model. This greatly speeds up training and reduces resource consumption. With LoRA, image generation models can adapt to new data or changes in demand more quickly while maintaining high quality standards for generated images. For example, in dynamic scenes such as video games or movie special effects production, quickly generate image sequences that are consistent with the original style but have changed content.
[0003] Currently, portrait images are generated through AIGC image generation, usually based on the specified face shape images uploaded by the user, and the Lora file is used to generate portraits of a specified style. However, the images uploaded by the user usually contain noise such as the background, facial accessories, and hair. These factors will affect the quality of the generated portrait images, making the generated portrait images not conform to the face shape desired by the user. Summary of the invention
[0004] In order to solve the technical problems mentioned in the above background technology, the present invention proposes an image processing method based on a neural network to solve the above technical problems.
[0005] According to one aspect of the present invention, a neural network-based image processing method is proposed, comprising:
[0006] S1. Obtain user file information, and use the user file information to automatically perform Lora training to obtain a first training model;
[0007] S2, obtaining and preprocessing an image containing the front face of the user and a plurality of custom face shape template images, and inputting the image containing the front face of the user and the plurality of custom face shape template images into a target detection model to obtain the user's face shape and the plurality of template face shapes;
[0008] S3, extracting features from the user's face shape and the plurality of template face shapes, performing similarity calculation based on the extracted features to match the template face shape corresponding to the user's face shape, and modifying the template face shape according to the user's face shape;
[0009] S4. Importing the corrected template face image into the first training model for batch processing to generate a user portrait corresponding to the user file information, wherein the first training model is configured with a user-preset Lora file as a positive prompt word.
[0010] Through the above technical scheme, the model is pre-trained through Lora to obtain a training model that meets the requirements of the specified image. By performing target detection on several custom face template images imported by the user and the image of the user's front face, the face shape in the image can be detected more accurately, avoiding noise such as background, facial accessories, hair, etc. in the pictures uploaded by the user, making the face shape detection more accurate and convenient for subsequent feature extraction; similarity calculation is performed based on the extracted features, the user's face shape can be matched with the template face shape, and the corresponding template can be accurately matched. The template face shape is custom imported, and the corresponding template can be freely selected later to ensure the flexibility of template selection; the template image is imported into the model for image generation, and the training model generates the user's portrait according to the user's preset Lora file; the present application realizes fully automated batch processing of image generation, and performs detection, optimization and template matching on the face shape problems of the user's portrait, so that the generated portrait is more in line with the face shape of the user's imported front face image.
[0011] In some specific embodiments, the user file data in step S1 includes a training set consisting of images of a specific theme.
[0012] Through the above technical solution, the user file data includes specific theme images, and the specific theme images are used for training, so that the subsequently generated user portrait can be the theme specified by the user.
[0013] In some specific embodiments, the preprocessing of the image containing the front face of the user and the plurality of custom face template images in step S2 includes denoising and highlighting the image containing the front face of the user and the plurality of custom face template images.
[0014] In some specific embodiments, the preprocessing of the image containing the front face of the user and the plurality of custom face template images imported by the user in step S2 includes denoising and highlighting the image containing the front face of the user and the plurality of custom face template images imported by the user.
[0015] Through the above technical solution, the image is preprocessed to facilitate the accuracy of subsequent target detection.
[0016] In some specific embodiments, step S2 includes:
[0017] S21, obtaining a preset face data set, and dividing the preset face data set into a training set and a test set;
[0018] S22, training the target detection model based on the training set, verifying the trained target detection model based on the test set, and fine-tuning the target detection model according to the verification result;
[0019] S23, inputting the image containing the front face of the user and the plurality of custom face shape template images into the fine-tuned target detection model to obtain the user's face shape and a plurality of template face shapes from the custom face shape template images.
[0020] Through the above technical solutions, the target detection model is trained and fine-tuned in advance to ensure the accuracy of the model in recognizing face shapes when used in practice.
[0021] In some specific embodiments, the target detection model in step S22 is composed of a region proposal network and a prediction network, the region proposal network is used to generate potential face regions, and the prediction network is used to classify the potential face regions and refine the detected face boundaries.
[0022] Through the above technical solutions, the model structure can quickly detect face images and improve the recall rate and accuracy of face image detection.
[0023] In some specific embodiments, the user face shape and template face shape features extracted in step S3 both include a feature vector obtained by fusing the forehead width feature, the sacrum width feature and the chin width feature.
[0024] Through the above technical solution, based on the analysis of each face shape, the forehead width feature, the sacrum width feature and the chin width are set as the extracted features, which can accurately distinguish each face shape.
[0025] In some specific embodiments, the similarity calculation in step S3 includes cosine similarity calculation and Euclidean distance calculation.
[0026] In some specific embodiments, the calculation formula of the cosine similarity is:
[0027]
[0028] Among them, x i is the feature extracted from the user's face shape, y i The features extracted from the several template faces, n represents the dimension.
[0029] In some specific embodiments, the calculation formula of the Euclidean distance is:
[0030]
[0031] Among them, d(x,y) is the similarity, x i is the feature extracted from the user's face shape, y iThe features extracted from the several template faces, n represents the dimension.
[0032] Compared with the prior art, the beneficial effects of this application are:
[0033] A training model that meets the requirements of specified images is obtained through the Lora training model, and the face shape in the image is detected for several custom face template images imported by the user and the image of the user's front face, so as to avoid the influence of noise in the image, ensure the accuracy of face shape detection, and facilitate subsequent feature extraction; similarity calculation is performed based on the extracted features, and the user's face shape is matched with the template face shape, and the corresponding template is accurately matched, and the template face shape is customized by the user to ensure the flexibility of template selection; the template image is imported into the training model for image generation, and the training model generates the user's portrait according to the user's preset Lora file; the present application realizes fully automated batch processing of image generation, and detects, optimizes and matches the face shape problems of the user's portrait, so that the generated portrait is more in line with the face shape of the front face image imported by the user. BRIEF DESCRIPTION OF THE DRAWINGS
[0034] The accompanying drawings are included to provide a further understanding of the embodiments and are incorporated into and constitute a part of this specification. The accompanying drawings illustrate the embodiments and together with the description are used to explain the principles of the present invention. Other embodiments and many expected advantages of the embodiments will be readily appreciated as they become better understood by reference to the following detailed description. Other features, objects and advantages of the present application will become more apparent by reading the detailed description of the non-limiting embodiments made with reference to the following drawings:
[0035] Figure 1 is a flowchart of an image processing method based on a neural network according to an embodiment of the present application;
[0036] Figure 2 is a schematic diagram of the target detection model structure of an embodiment of the present application;
[0037] Figure 3 It is a structural diagram of a computer system suitable for implementing an electronic device of an embodiment of the present application. DETAILED DESCRIPTION
[0038] The present application will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the relevant invention, rather than to limit the invention. It should also be noted that, for ease of description, only the parts related to the relevant invention are shown in the accompanying drawings.
[0039] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.
[0040] According to an image processing method based on a neural network in one embodiment of the present application, Figure 1 FIG. 1 is a flowchart of an image processing method based on a neural network according to an embodiment of the present application. Figure 1 As shown, the method includes:
[0041] S1. Obtain user file information, and use the user file information to automatically perform Lora training to obtain a first training model;
[0042] Specifically, the user file data in step S1 includes a training set consisting of specific theme images. Through training with specific theme images, the user portrait generated subsequently can be the theme specified by the user.
[0043] S2, obtaining and preprocessing an image containing the front face of the user and a plurality of custom face shape template images, and inputting the image containing the front face of the user and the plurality of custom face shape template images into a target detection model to obtain the user's face shape and the plurality of template face shapes;
[0044] Specifically, the custom face template image in step S2 includes a preset default face template image. The user can directly select the default face template image, or import several custom face template images by themselves, which ensures the flexibility of template selection. The custom face template image can be set according to the size of the face.
[0045] Furthermore, the user's front face image, the preset default face template image, and several user-defined face template images imported by the user are pre-stored in a specified directory, so that the target detection model can perform face shape detection according to the corresponding graphics, and store the output face shape images in the specified directory according to the settings.
[0046] Specifically, the preprocessing of the image containing the front face of the user and the plurality of custom face template images in step S2 includes denoising and highlighting the image containing the front face of the user and the plurality of custom face template images. The image is preprocessed to facilitate the accuracy of subsequent target detection.
[0047] Specifically, step S2 includes:
[0048] S21, obtaining a preset face data set, and dividing the preset face data set into a training set and a test set;
[0049] S22, training the target detection model based on the training set, verifying the trained target detection model based on the test set, and fine-tuning the target detection model according to the verification result;
[0050] S23, inputting the image containing the front face of the user and the plurality of custom face shape template images into the fine-tuned target detection model to obtain the user's face shape and the plurality of template face shapes in the custom face shape template images. The target detection model is trained and fine-tuned in advance to ensure the accuracy of the model in recognizing face shapes in actual use.
[0051] Optionally, the target detection model includes YOLOv5-based target detection and DNN-based facial recognition classification.
[0052] Optionally, the target detection model in step S22 is composed of a region proposal network and a prediction network, wherein the region proposal network is used to generate potential face regions, and the prediction network is used to classify the potential face regions and refine the detected face boundaries. This model structure can quickly detect face images and improve the recall rate and accuracy of face image detection; wherein the region proposal network is composed of a high feature generation pyramid and a low feature generation pyramid, such as Figure 2 As shown in the figure, through the fusion of shallow features and deep features, the high feature generation pyramid receives the input image to generate basic features, and the conv4_3 and conv5_3 layers of its ResNet architecture provide multi-level semantic information for feature mapping. Secondly, the low feature generation pyramid is combined with the convolution layer, and the two parts are layered with each other. The low feature generation pyramid is used to generate low-level feature maps of different scales from the high feature generation pyramid, and the feature maps output by the high feature generation pyramid and the low feature generation pyramid are merged through the convolution layer.
[0053] The high feature generation pyramid combines a level of features, which uses a 1*1 convolution layer to perform compression and coupling operations on the input feature channel to merge these feature maps. And because the high feature generation pyramid receives feature maps of different scales as input, it includes upsampling operations to rescale deeper features to match the scale of the coupling operation. High-resolution feature mapping can be performed through the high feature generation pyramid, which can better extract features and improve the face detection performance when facing small objects.
[0054] The low-feature generation pyramid consists of a series of 3*3 convolutional layers with a stride of 2 to form a pyramid network. The input dataset of the subsequent convolutional layers consists of the feature maps of these convolutional layers. The outputs of the high-feature generation pyramid and the low-feature generation pyramid are combined to provide multi-scale features.
[0055] In order to integrate the multi-scale features of the low-feature generation pyramid and the high-feature generation pyramid into one convolutional layer, a prediction network is required. The prediction network first connects the functions of the same scale in the channel dimension, and then passes through their respective 3*3 convolutional layers, and finally sends the generated feature map to the subsequent convolutional layer. The prediction network is highly efficient and the model weight is light.
[0056] S3, extracting features from the user's face shape and the plurality of template face shapes, performing similarity calculation based on the extracted features to match the template face shape corresponding to the user's face shape, and modifying the template face shape according to the user's face shape;
[0057] Specifically, the user face shape and template face shape features extracted in step S3 both include a feature vector obtained by fusing the forehead width feature, the sacrum width feature, and the chin width feature. Based on the analysis of each face shape, the forehead width feature, the sacrum width feature, and the chin width are set as the extracted features, which can accurately distinguish each face shape.
[0058] Specifically, the similarity calculation includes cosine similarity calculation and Euclidean distance calculation;
[0059] Among them, the calculation formula of cosine similarity is:
[0060]
[0061] Among them, x i is the feature extracted from the user's face shape, y i The features extracted from the plurality of template faces, where n represents the dimension;
[0062] The calculation formula of Euclidean distance is:
[0063]
[0064] Among them, d(x,y) is the similarity, x i is the feature extracted from the user's face shape, y i The features extracted from the several template faces, n represents the dimension.
[0065] Specifically, correcting the template face shape according to the user's face shape includes obtaining the values of the forehead width, the sacrum width and the chin width of the user's face shape, and adjusting the values of the corresponding positions of the template face shape according to the values of the forehead width, the sacrum width and the chin width of the user's face shape.
[0066] S4. Importing the corrected template face image into the first training model for batch processing to generate a user portrait corresponding to the user file information, wherein the first training model is configured with a user-preset Lora file as a positive prompt word.
[0067] Specifically, the first training model is also configured with ADetailer enabled, Skipimg2img enabled, 2 units and redraw amplitude set, and ControlNet model-reference color turned on.
[0068] Through the above method, it is possible to obtain a training model that meets the requirements of a specified image through a Lora training model, perform target detection on several custom face template images imported by the user and the image of the user's front face to obtain the face shape in the image, avoid the influence of noise in the image, ensure the accuracy of face shape detection, and facilitate subsequent feature extraction; perform similarity calculation based on the extracted features, match the user's face shape with the template face shape, accurately match the corresponding template, and the template face shape is user-defined and imported to ensure the flexibility of template selection; import the template image into the training model for image generation, and the training model generates the user's portrait according to the user's preset Lora file; the present application realizes fully automated batch processing of image generation, and performs detection, optimization and template matching on the face shape problems of the user's portrait, so that the generated portrait is more consistent with the face shape of the front face image imported by the user.
[0069] Reference below Figure 3 , which shows a schematic diagram of the structure of a computer system 300 suitable for implementing an electronic device of an embodiment of the present application. Figure 3 The electronic device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.
[0070] like Figure 3 As shown, the computer system 300 includes a central processing unit (CPU) 301, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 302 or a program loaded from a storage part 308 into a random access memory (RAM) 303. In the RAM 303, various programs and data required for the operation of the system 300 are also stored. The CPU 301, the ROM 302, and the RAM 303 are connected to each other via a bus 304. An input / output (I / O) interface 305 is also connected to the bus 304.
[0071] The following components are connected to the I / O interface 305: an input section 306 including a keyboard, a mouse, etc.; an output section 307 including a liquid crystal display (LCD), etc. and a speaker, etc.; a storage section 308 including a hard disk, etc.; and a communication section 309 including a network interface card such as a LAN card, a modem, etc. The communication section 309 performs communication processing via a network such as the Internet. A drive 310 is also connected to the I / O interface 305 as needed. A removable medium 311, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 310 as needed, so that a computer program read therefrom is installed into the storage section 308 as needed.
[0072] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable storage medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 309, and / or installed from the removable medium 311. When the computer program is executed by the central processing unit (CPU) 301, the above functions defined in the method of the present application are executed. It should be noted that the computer-readable storage medium of the present application can be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium can be, for example, - but not limited to - an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to, an electrical connection with one or more conductors, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, device, or device. In the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable storage medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, device, or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wireless, wireline, optical cable, RF, etc., or any suitable combination of the foregoing.
[0073] Computer program code for performing the operations of the present application may be written in one or more programming languages or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).
[0074] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.
[0075] The modules involved in the embodiments of the present application may be implemented by software or by hardware.
[0076] As another aspect, the present application also provides a computer-readable storage medium, which may be included in the electronic device described in the above embodiment; or it may exist independently and not be assembled into the electronic device. The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed by the electronic device, the electronic device: obtains user file information, and uses the user file information to automatically perform Lora training to obtain a first training model; obtains and preprocesses an image containing a user's front face and several custom face template images, and respectively inputs the image containing the user's front face and the several custom face template images into a target detection model to obtain the user's face shape and several template face shapes; extracts features from the user's face shape and the several template face shapes, performs similarity calculation based on the extracted features to match the template face shape corresponding to the user's face shape, and corrects the template face shape according to the user's face shape; imports the template face shape corresponding to the matched user's face shape into the first training model for batch processing to generate a user portrait corresponding to the user file information, and the first training model is configured with a user-preset Lora file as a positive prompt word.
[0077] The above description is only a preferred embodiment of the present application and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the present application is not limited to the technical solution formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above invention concept. For example, the above features are replaced with the technical features with similar functions disclosed in this application (but not limited to) by each other to form a technical solution.
Claims
1. A neural network-based image processing method, characterized in that: include: S1. Obtain user file information, and use the user file information to automatically perform Lora training to obtain a first training model; S2, obtaining and preprocessing an image containing the front face of the user and a plurality of custom face shape template images, and inputting the image containing the front face of the user and the plurality of custom face shape template images into a target detection model to obtain the user's face shape and the plurality of template face shapes; S3, extracting features from the user's face shape and the plurality of template face shapes, performing similarity calculation based on the extracted features to match the template face shape corresponding to the user's face shape, and modifying the template face shape according to the user's face shape; S4. Importing the corrected template face image into the first training model for batch processing to generate a user portrait corresponding to the user file information, wherein the first training model is configured with a user-preset Lora file as a positive prompt word.
2. The image processing method based on neural network according to claim 1, characterized in that: The user file data in step S1 includes a training set consisting of images of a specific subject.
3. The image processing method based on neural network according to claim 1, characterized in that: The custom face template image in step S2 includes a preset default face template image and a face template image imported by a user.
4. The image processing method based on neural network according to claim 1, characterized in that: The preprocessing of the image containing the front face of the user and the plurality of custom face template images in step S2 includes denoising and highlighting the image containing the front face of the user and the plurality of custom face template images.
5. The image processing method based on neural network according to claim 1, characterized in that: Step S2 includes: S21, obtaining a preset face data set, and dividing the preset face data set into a training set and a test set; S22, training the target detection model based on the training set, verifying the trained target detection model based on the test set, and fine-tuning the target detection model according to the verification result; S23, inputting the image containing the front face of the user and the plurality of custom face shape template images into the fine-tuned target detection model to obtain the user's face shape and a plurality of template face shapes from the custom face shape template images.
6. The image processing method based on neural network according to claim 5, characterized in that: In step S22, the target detection model is composed of a region proposal network and a prediction network. The region proposal network is used to generate potential face regions, and the prediction network is used to classify the potential face regions and refine the detected face boundaries.
7. The image processing method based on neural network according to claim 1, characterized in that: The user face shape and template face shape features extracted in step S3 both include a feature vector obtained by fusing the forehead width feature, the sacrum width feature, and the chin width feature.
8. The image processing method based on neural network according to claim 1, characterized in that: The similarity calculation in step S3 includes cosine similarity calculation and Euclidean distance calculation.
9. The neural network-based image processing method according to claim 8, characterized in that: The calculation formula of the cosine similarity is: Among them, x i is the feature extracted from the user's face shape, y i The features extracted from the several template faces, n represents the dimension.
10. The neural network-based image processing method according to claim 8, characterized in that: The calculation formula of the Euclidean distance is: Among them, d(x,y) is the similarity, x i is the feature extracted from the user's face shape, y i The features extracted from the several template faces, n represents the dimension.
