A training method of an image processing model, a video image processing method and system

Abstract

The application relates to the field of image processing, in particular to a training method of an image processing model, a video image processing method and system. The method comprises the following steps: extracting a video frame image from a video as a to-be-processed image; performing cutting on the to-be-processed image according to a region of interest to obtain a cut image; and inputting the cut image into a twin network for processing to obtain a segmentation region image; wherein the training method of the twin network comprises the following steps: obtaining a plurality of sample images; performing target object cutting on the sample images to obtain sample cut images; performing triangular partition deformation on the sample cut images to obtain corresponding deformed sample images; marking segmentation objects of the sample images and the corresponding deformed sample images; inputting the marked sample cut images and the corresponding deformed sample images into the twin network respectively, training the twin network, and obtaining a trained twin network. The method can realize accurate segmentation and processing of a segmentation object in an image.

Description

Technical Field

[0001] This invention relates to the field of image processing, and more specifically, to a training method for an image processing model, a video image processing method, and a system. Background Technology

[0002] In the field of live video streaming, accurate detection and segmentation of target regions, especially human skin areas, is a crucial technology. Detected and segmented skin areas can be used for portrait beautification, including functions such as blemish removal, skin smoothing, and skin tone adjustment. However, current technologies still have many shortcomings in terms of accuracy, compatibility, real-time performance, and stability: existing algorithms perform poorly when dealing with changes in lighting, occlusion, and complex backgrounds in images, failing to accurately identify and segment target regions; current target segmentation techniques struggle to meet real-time requirements in practical applications; and handling different races, skin colors, and ambient lighting conditions remains a significant challenge when segmenting skin regions, often resulting in large errors in skin region identification and segmentation. Therefore, necessary improvements to target region detection and segmentation technologies are required. Summary of the Invention

[0003] The present invention aims to overcome at least one of the defects (deficiencies) of the prior art and provide a training method for an image processing model, a video image processing method and system for achieving accurate segmentation and processing of objects in an image.

[0004] According to a first aspect of this application, a method for training an image processing model is provided, the method comprising:

[0005] Acquire several sample images;

[0006] The sample image is cropped by cropping the target object to obtain a cropped sample image;

[0007] The cropped sample image is triangulated to obtain the corresponding deformed sample image;

[0008] The sample image and the corresponding deformed sample image are labeled with segmented objects;

[0009] The labeled cropped sample image and the corresponding deformed sample image are respectively input into the Siamese network, and the Siamese network is trained to obtain a trained Siamese network.

[0010] Understandably, training a Siamese network using cropped sample images and corresponding deformed sample images can accommodate images deformed due to uncontrollable reasons, thereby improving the robustness of the Siamese network. At the same time, it can also solve the problem of inaccurate segmentation of target regions in videos or images due to deformation, thus improving the reliability of video or image segmentation.

[0011] Optionally, the step of inputting the labeled cropped sample image and the corresponding deformed sample image into the Siamese network, respectively, and training the Siamese network to obtain a trained Siamese network includes:

[0012] The labeled cropped sample image is input into the Siamese network to obtain a first generated image, and the deformed sample image corresponding to the cropped sample image is input into the Siamese network to obtain a corresponding second generated image;

[0013] The loss value of the Siamese network is obtained based on the first generated image and the corresponding second generated image;

[0014] The parameters of the Siamese network are optimized based on the loss value to obtain a trained Siamese network.

[0015] Understandably, triangulating the sample image to obtain the corresponding deformed sample image, and then generating a second generated image from it, actually simulates the deformation and distortion of the segmented object region caused by network problems or target object motion problems in the video. Furthermore, training the Siamese network using the deformed sample image can enhance the Siamese network's ability to process deformed images, enabling the Siamese network to accurately segment the segmented object region when processing deformed images, thereby enhancing the robustness of the Siamese network.

[0016] Optionally, obtaining the loss value of the Siamese network based on the first generated image and the corresponding second generated image includes:

[0017] Perform an inverse deformation operation on the second generated image to obtain an inversely deformed second generated image;

[0018] The loss value of the Siamese network is obtained by comparing the first generated image and the second generated image after inverse deformation using global pixel comparison.

[0019] Understandably, the loss value of the Siamese network is obtained based on the similarity between the first generated image and the corresponding inversely deformed second generated image. The second generated image simulates the situation where the sample cropped image is deformed and then generated by the Siamese network. This loss value enables the Siamese network to segment the area where the segmented object of the deformed sample image is located, tending to the area where the segmented object of the original undeformed image is located, thereby enhancing the Siamese network's ability to process deformed images.

[0020] According to a second aspect of this application, a video image processing method is provided, the method comprising:

[0021] Extract video frame images from the video as images to be processed;

[0022] The image to be processed is cropped according to the region of interest to obtain a cropped image;

[0023] The cropped image is input into a Siamese network for processing to obtain a segmented region image; wherein the Siamese network is a Siamese network trained using the image processing model training method described in the first aspect.

[0024] Understandably, by using a trained Siamese network to process video frame images, stable and accurate segmentation and detection of the region where the segmented object is located can be achieved even in video frame images with complex backgrounds, improving the segmentation accuracy of the region where the segmented object is located. At the same time, it can also achieve efficient real-time computation of video processing, and can be compatible with the processing of images deformed by network problems or human motion in the video, enhancing the versatility of the method and ensuring stable operation under different video conditions.

[0025] Optionally, the region of interest is stored in a preset database;

[0026] Extract the specific location information of the target object in the image to be processed from the segmented region image;

[0027] Generate the region of interest (ROI) of the current video frame image based on the specific location information;

[0028] The region of interest (ROI) of the current video frame image is swapped with the ROI stored in the database and then stored as the ROI for the next video frame image.

[0029] Understandably, the segmented region image accurately represents the specific location of the segmented object in the image to be processed. It can generate the region of interest (ROI) of the current video frame image based on the specific location information of the segmented object, providing guidance for the cropping work of the next video frame image, thereby effectively narrowing the range of target object segmentation and improving the efficiency of Siamese network processing. By dynamically updating the ROI in the database, it is possible to more accurately extract small-area cropped images of video frames, thereby significantly improving the true resolution of the cropped images input to the Siamese network and reducing computational resource consumption. At the same time, it also ensures the continuity of the target in complex environments in the video, reduces the interference of background objects on the segmentation results, and improves the robustness of segmentation and detection.

[0030] Optionally, the method further includes:

[0031] Based on the segmented region image corresponding to the previous video frame, the motion vector of each pixel in the segmented region image corresponding to the current video frame is obtained by the fast optical flow algorithm.

[0032] Adjust the noise or discontinuous regions in the segmented region image corresponding to the current video frame image according to the motion vector to obtain the current segmented region image after processing.

[0033] Understandably, by smoothing the segmented region image corresponding to the current video frame image through the fast optical flow algorithm, the changes in the segmented region of the continuous video frame image segmentation can be effectively detected. This introduces short-term motion patterns into the static segmentation of video frame images, significantly reducing jitter and abrupt changes in the segmented region of the segmented object, reducing noise or discontinuous regions caused by segmentation errors, and improving the stability and continuity of the segmentation results.

[0034] Optionally, the method further includes:

[0035] The boundaries of the segmented region image are expanded by applying a morphological dilation operation;

[0036] Feathering the boundary yields a segmented region image with a faded boundary area.

[0037] Gaussian blur filtering is applied to the segmented region image with the faded boundary region to obtain the final segmented region image.

[0038] Understandably, by using morphological dilation, feathering, and Gaussian blur filtering, the boundaries of the target segmentation region can be expanded and processed, making the edges of the segmented object in the segmentation region image smoother and more natural. When applied to the segmentation of human skin, this helps to handle fine hairs, blurred edges, and other issues, achieving a more natural skin segmentation effect.

[0039] According to a third aspect of this application, a training system for an image processing model is provided, the system comprising:

[0040] The sample acquisition module is used to acquire several sample images;

[0041] The sample cropping module is used to crop the target object from the sample image to obtain a cropped sample image.

[0042] The deformation processing module is used to triangulate and deform the cropped sample image to obtain the corresponding deformed sample image.

[0043] The labeling module is used to label the segmented objects in the sample image and the corresponding deformed sample image;

[0044] The training module is used to input the labeled cropped sample image and the corresponding deformed sample image into the Siamese network, respectively, to train the Siamese network and obtain a trained Siamese network.

[0045] According to a fourth aspect of this application, a video image processing system is provided, the system comprising:

[0046] The acquisition module is used to extract video frame images from the video as images to be processed.

[0047] The cropping module is used to crop the image to be processed according to the region of interest to obtain a cropped image;

[0048] The generation module is used to input the cropped image into a Siamese network for processing to obtain a segmented region image; wherein the Siamese network is a Siamese network trained using the training method of the image processing model described in the first aspect.

[0049] According to a fifth aspect of this application, an electronic device is provided, comprising:

[0050] Memory, used to store one or more computer programs;

[0051] A processor, when the one or more computer programs are executed by the processor, implements the training method for the image processing model described in the first aspect above or a video image processing method as described in the second aspect above.

[0052] According to a sixth aspect of this application, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions, the computer instructions being configured to cause a processor to execute and implement the training method of the image processing model described in the first aspect or the video image processing method described in the second aspect.

[0053] Based on any of the above aspects, embodiments of this application provide a training method for an image processing model, a video image processing method, a system, an electronic device, and a storage medium. The method involves extracting video frame images from a video as images to be processed; cropping the images to be processed according to regions of interest to obtain cropped images; and inputting the cropped images into a Siamese network for processing to obtain segmented region images. The training of the Siamese network includes: acquiring several sample images; cropping the sample images to obtain sample cropped images; triangulating the sample cropped images to obtain corresponding deformed sample images; labeling the segmented objects in the sample images and the corresponding deformed sample images; and inputting the labeled sample cropped images and the corresponding deformed sample images into the Siamese network to train the Siamese network, thereby obtaining a trained Siamese network. This method offers the following benefits:

[0054] 1) Training the Siamese network based on the cropped sample image and the corresponding deformed sample image can enhance the Siamese network's ability to segment objects from deformed images, thereby improving the reliability and robustness of the Siamese network-based image processing model.

[0055] 2) In the field of video processing, especially in the scenario of live video streaming, it provides an efficient, stable, accurate and real-time target region segmentation method with good general performance;

[0056] 3) It can effectively solve the problems of poor accuracy and poor inter-frame stability for videos, especially live video. By constructing a Siamese network structure, it increases the training loss for stability optimization, optimizes the problem of decreased accuracy of target region segmentation due to image distortion during live video, and avoids the possibility of large differences in target segmentation regions between two frames. Attached Figure Description

[0057] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0058] Figure 1 This embodiment illustrates a training method for an image processing model and an application scenario of a video image processing method.

[0059] Figure 2 This is a flowchart of a training method for an image processing model provided in this embodiment.

[0060] Figure 3 This is a flowchart of a twin network training method provided in this embodiment.

[0061] Figure 4 This is a flowchart of a video image processing method provided in this embodiment.

[0062] Figure 5 This embodiment provides a flowchart of a method for obtaining a region of interest.

[0063] Figure 6 This embodiment provides a flowchart of a method for smoothing segmented region images.

[0064] Figure 7 This embodiment provides a flowchart of a method for feathering segmented region images.

[0065] Figure 8 This embodiment provides a schematic diagram of a training method for an image processing model.

[0066] Figure 9 This embodiment provides a schematic diagram of a video image processing system module.

[0067] Figure 10 This embodiment provides a schematic diagram of the electronic device. Detailed Implementation

[0068] The accompanying drawings are for illustrative purposes only and should not be construed as limiting the scope of this application. To better illustrate the following embodiments, some components in the drawings may be omitted, enlarged, or reduced, and do not represent the actual dimensions of the product; it is understandable to those skilled in the art that some well-known structures and their descriptions may be omitted in the drawings.

[0069] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0070] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0071] Live video streaming refers to a method of live broadcasting using the internet and streaming media technology. Currently, live video streaming is being applied to multiple fields and is developing rapidly. To achieve better broadcast results, streamers often use the beautification effects built into live streaming software, such as blemish removal, skin smoothing, and skin tone adjustment. However, for live streaming software to deliver better beautification effects, it needs to address a fundamental issue: accurately identifying the skin areas of the people in the live stream, especially the streamer's face. Currently, although many existing technologies can segment human skin areas, they often suffer from significant errors and noise when faced with complex live streaming backgrounds, varying ethnicities and skin tones, dynamic movements of the human body during the broadcast, and image distortion caused by network lag. This results in a decrease in beautification effects or inaccurate beautification of the target area. Therefore, necessary improvements are needed in the segmentation and processing of target areas, especially skin areas.

[0072] This embodiment provides a technical solution that can solve the above problems. The specific implementation of this application will be described in detail below with reference to the accompanying drawings.

[0073] Exemplary examples are illustrated in the diagram showing an application scenario of an image processing model training method and a video image processing method provided in this application. For instance... Figure 1 As shown, the application scenario includes at least a server 100 and a terminal 200 that can communicate with the server 100. The server 100 has image processing capabilities and can also train twin networks; the terminal 200 has video playback capabilities and can also acquire images.

[0074] Understandably, the server 100 can be an independent electronic device or a cluster of multiple electronic devices; the terminal 200 can be a smartphone terminal, personal computer, tablet computer, vehicle terminal, etc., but is not limited to these.

[0075] In one possible implementation, server 100 and terminal 200 may respectively execute the image processing model training method or video image processing method provided in the embodiments of this application. Alternatively, the image processing model training method or video image processing method provided in the embodiments of this application may be partially executed in server 100 and partially executed in terminal 200.

[0076] For example, such as Figure 2 As shown, this embodiment provides a training method for an image processing model, which may include the following steps:

[0077] S110. Obtain several sample images;

[0078] In this embodiment, a set of historical videos or images collected in advance can be used as sample images. The image set can encompass a large number of representative cases of image deformation or distortion. Preferably, some images are separate and discontinuous, and data augmentation is required to highlight the features of these images, so that the Siamese network can learn the processing methods of a large number of deformed and distorted images as easily as possible, thereby improving the robustness of the Siamese network.

[0079] S120. Cropping the target object from the sample image to obtain a cropped sample image;

[0080] In this embodiment, cropping the sample image to include the target object reduces the computational scope of the Siamese network, thereby improving the efficiency of Siamese network segmentation and saving computational resources. The identification of the target object is completed before cropping, and can be achieved through existing manual or machine recognition techniques, which will not be elaborated upon here. The target object can be a specific type of object, such as a person, an animated character, or a digital human.

[0081] S130. Triangulate the cropped sample image to obtain the corresponding deformed sample image.

[0082] In this embodiment, the collected sample images may be partially normal and require deformation processing to enhance their distortion features. Triangulation deformation constructs a triangular mesh based on the pixel set of the image, dividing the image into multiple triangular regions. In the field of image processing, it is commonly used as a deformation operation for images. Using the triangulation deformation method, images can be processed using simple mathematical and geometric principles, making it easier for Siamese networks to identify features and improving the processing efficiency of Siamese networks.

[0083] S140. Label the segmented objects of the sample image and the corresponding deformed sample image;

[0084] In this embodiment, the Siamese network is trained by pre-labeling the objects to be segmented in the sample images to form labeled training samples. This allows the Siamese network to learn segmentation methods for the objects from the labeled training samples, forming a supervised machine learning process. The segmented object is a part of the target object. Training the Siamese network aims to segment and recognize images, thereby segmenting the region where the segmented object is located in the image, and then performing subsequent processing on the region where the segmented object is located in the image. For example, when the target object is a person, the segmented object can be a person's skin, eyes, nose, etc.

[0085] S150. Input the labeled cropped sample image and the corresponding deformed sample image into the Siamese network respectively, and train the Siamese network to obtain a trained Siamese network.

[0086] In this embodiment, cropping the region containing the segmented object in a normal image is a necessary function of the method. However, in live video applications, the real-time movement of people in the video or the image distortion caused by network issues often results in the segmented object region containing other background images or an incomplete segmentation. This causes subsequent beautification functions to render too much or too little of the area that should be processed, resulting in an unnatural live broadcast effect. This application not only uses normal cropped images as training data for the Siamese network, but also artificially triangulates and deforms the normal cropped images to simulate the distortion of the segmented object region in the real-world image, thereby training the Siamese network to crop distorted images. Because live video broadcasts have high requirements for the accuracy and stability of target region segmentation, this application uses a Siamese network and trains it to obtain a well-trained target region segmentation model. Preferably, DeepLabV3+ is selected as the basic deep learning framework, MobileNetV3 is selected as the underlying model, and a Siamese network is used to build two branches with shared parameters.

[0087] In this embodiment, the labeled sample cropped image and the corresponding deformed sample image actually simulate the situation after the sample cropped image is deformed and distorted. By inputting the sample cropped image and the corresponding deformed sample image into the Siamese network, the Siamese network can learn the segmentation method of the sample cropped image in the normal situation for the segmentation object area, as well as the segmentation method of the corresponding deformed sample image in the deformed situation for the segmentation object area, thereby enhancing the reliability and versatility of the Siamese network.

[0088] Specifically, such as Figure 3 As shown, the step of inputting the labeled cropped sample image and the corresponding deformed sample image into the Siamese network, and training the Siamese network to obtain a trained Siamese network includes the following steps:

[0089] S151. Input the labeled sample cropped image into the Siamese network to obtain a first generated image, and input the deformed sample image corresponding to the sample cropped image into the Siamese network to obtain a corresponding second generated image;

[0090] In this embodiment, a normal sample cropped image is input into the Siamese network, reflecting the region where the segmented object is located under normal conditions. Understandably, the first generated image output from this input branch ultimately needs to be trained to generate a result that conforms to the region where the segmented object is located. Similarly, a corresponding deformed sample image is input into the Siamese network, reflecting the region where the segmented object is located under distorted image conditions. Understandably, the second generated image output from this input branch ultimately needs to be trained to approximate the result of the first generated image, thus demonstrating that the Siamese network can accurately segment the region where the segmented object is located even in distorted images.

[0091] In this embodiment, the sample cropped image and the corresponding deformed sample image are respectively input into the Siamese network, which enables the Siamese network to extract features from the two input branches separately and learn the information and methods in the two input branches independently, thereby avoiding the situation of information confusion and cross-processing chaos between the two input branches.

[0092] S152. Obtain the loss value of the Siamese network based on the first generated image and the corresponding second generated image;

[0093] In this embodiment, training the Siamese network to process distorted images is a continuous optimization process. By comparing the similarity between the first generated image and the corresponding second generated image, the difference between the two is obtained. Based on the difference, the loss value of the Siamese network is generated. Based on the loss value, the parameters and structure of the Siamese network are fine-tuned and optimized so that the two images obtained by the two branches in the next training will continue to become more similar, thereby improving the Siamese network's ability to process distorted images.

[0094] Specifically, obtaining the loss value of the Siamese network based on the first generated image and the corresponding second generated image includes:

[0095] Perform an inverse deformation operation on the second generated image to obtain an inversely deformed second generated image;

[0096] The loss value of the Siamese network is obtained by comparing the first generated image and the second generated image after inverse deformation using global pixel comparison.

[0097] Preferably, the loss value Loss is obtained using the following formula:

[0098] Loss=(F L (Net ori (img),F rewarp (Net warp (img warp )))

[0099] Where img represents the cropped sample image, Netori (img) represents the first generated image obtained by inputting the cropped sample image into the Siamese network; warp Represents the deformed sample image corresponding to the cropped sample image, Net warp (img warp ) represents the second generated image obtained by inputting the deformed sample image corresponding to the cropped sample image into the Siamese network; F rewarp (·) is the inverse deformation operation function for the image, F L (·) is a global pixel comparison operation function for the first generated image and the second generated image.

[0100] In this embodiment, the deformed sample image obtained by autonomously triangulating and deforming the sample cropped image is then processed by the Siamese network to generate a processed image. It also needs to be inversely deformed to restore the generated processed image to a normal state, so as to better compare it with the first generated image. The comparison operation compares each pixel of the first generated image and the second generated image, which can improve the accuracy of obtaining differences.

[0101] S153. Optimize the parameters of the Siamese network based on the loss value to obtain a trained Siamese network.

[0102] In this embodiment, optimization is stopped when the loss value reaches a threshold set to halt optimization. It is understood that the acquired sample images are typically divided into a training set and a test set. The training of a Siamese network usually includes a training process and a testing process. After optimizing the Siamese network using the loss value from the training set and completing the training operation, the test set needs to be input into the Siamese network for testing. After completing the testing, a trained Siamese network is obtained. It is understood that a trained Siamese network can handle both normal cropped images and distorted cropped images in real-world situations, and can be directly used in subsequent segmentation steps.

[0103] like Figure 4 As shown in the embodiments of this application, a video image processing method is also provided, the method comprising:

[0104] S210. Extract video frame images from the video as images to be processed;

[0105] Preferably, in live video, the video frame images to be processed are extracted as images to be processed. Extraction can be performed on each video frame image, allowing for segmentation of the target object within each frame. For live video, beautification of people is often required. The areas requiring beautification can be used as segmentation objects. For example, if skin beautification is needed, the video image processing method of this application can be used to segment the areas requiring beautification, resulting in more natural subsequent beautification processing.

[0106] S220. Cropping the image to be processed according to the region of interest to obtain a cropped image;

[0107] In this embodiment, considering the complex background of the live video scene and the small size of the person as the target object in the frame, in order to better eliminate the interference of other non-target objects, small figures, etc. in the background of the video screen on the effect, this application adopts the image cropping preprocessing method. In real-time live video, video frame images that need to be processed are acquired and used as material for the identification and segmentation of segmentation objects. It is understood that the region of interest (ROI) is generated based on the specific target location information obtained from the segmentation region image of the previous video frame. It is understood that cropping the target object region of the current video frame image based on the ROI can reduce the access to subsequent segmentation object identification and calculation, reduce interference from excessively large background areas, and thus improve the accuracy of identifying the segmentation object region. Simultaneously, the ROI recorded from the previous video frame image can, to a certain extent, ensure the continuity and stability of the target object and / or segmentation object region in the video. This is because, generally, the range of movement of the live streamer between adjacent frames is small, and the position of the target object and / or segmentation object region in adjacent frames will not move significantly. Therefore, the target object and / or segmentation object region between two frames has continuity. When applying the segmentation image results to beautification processing, it can ensure that the identified target object and / or segmentation object region has continuity, thus making the subsequent beautification processing more natural.

[0108] Understandably, at the beginning of the processing, the region of interest in the database is empty, and the cropping of the video frame image cannot be guided by the region of interest. Therefore, the cropped image is the full size of the video frame image, that is, the cropping box coordinates are [0,0,height,width]. More computing resources are needed to identify the target region in order to obtain an accurate target segmentation region. However, after obtaining the segmentation region image of the first video frame image, the corresponding region of interest can be generated based on the segmentation region image and the empty data in the database can be replaced for storage, thus guiding the cropping work of the next frame.

[0109] In this embodiment, the coordinates of the cropping box are generated based on the region of interest. The corresponding cropping box coordinates are obtained in advance from the video frame image, which facilitates the subsequent cropping work.

[0110] In this embodiment, the guiding region (img) mapped onto the current video frame image (img) is obtained based on the region of interest. a The formula is as follows:

[0111] img a =img[x left :x right ,y top :y bottom ]

[0112] Where x left x right y top and y bottom These represent the leftmost, rightmost, topmost, and bottommost coordinates of the region of interest mapped to the current video frame image (img).

[0113] According to the guidance area img a The image to be processed is cropped to obtain the cropped image;

[0114] Understandably, the region of interest (ROI) can guide the cropping process. Understandably, the target object is a part of the ROI. The ROI can locate the approximate location of the segmented object in the current video frame image. However, the segmented object in the next video frame image will have slight differences compared to the current video frame image. Therefore, the cropping process of the next video frame image can only use the ROI as a reference. The cropped image obtained by cropping guided by the ROI contains the entire target object.

[0115] Specifically, such as Figure 5 As shown, the region of interest is stored in a preset database; obtaining the region of interest includes the following steps:

[0116] S221. Extract the specific location information of the target object in the image to be processed based on the segmented region image;

[0117] In this embodiment, the Region of Interest (ROI) is a target area or object in the image that we are particularly interested in. These areas typically contain key location information that needs to be identified, analyzed, or processed.

[0118] S222. Generate the region of interest (ROI) of the current video frame image based on the specific location information;

[0119] In this embodiment, the region of interest (ROI) is generated from the segmented region image results obtained from each video frame image, and the ROI in the database is updated. This can provide an accurate ROI for cropping subsequent video frame images, reduce the computational load of subsequent video frame images, and improve the stability and accuracy of target tracking.

[0120] S223. The region of interest of the current video frame image is swapped with the region of interest stored in the database and then stored, and used as the region of interest corresponding to the next video frame image.

[0121] In this embodiment, the database always stores the regions of interest (ROIs) acquired from the previous video frame. Before processing begins, the database is empty. When the current ROI is acquired, the ROI from the previous video frame in the database needs to be replaced and overwritten. This simplifies the storage format and content, eliminating the need to store the correspondence between ROIs and video frame images, saving memory, and reducing the complexity of the corresponding processing. For different videos, in specific implementations, the ROIs in the database can be cleared before processing different videos to prevent the target object in the previous video from interfering with the target object in the next video.

[0122] S230. The cropped image is input into a Siamese network for processing to obtain a segmented region image; wherein the Siamese network is a Siamese network trained by the above training method.

[0123] In this embodiment, the trained Siamese network only needs to use one input branch to process the cropped image to obtain the segmented region image.

[0124] In this embodiment, the segmented region image obtained by the twin network usually has a certain error, especially at the boundary of the region, where there is some noise and jagged segmentation marks, which makes the entire live broadcast or beautification effect unnatural and affects the user's viewing experience. Therefore, it is necessary to post-process the obtained segmented region image.

[0125] Specifically, such as Figure 6 As shown, post-processing of the segmented region image may include the following steps:

[0126] S241. Based on the segmented region image corresponding to the previous video frame image, obtain the motion vector of each pixel in the segmented region image corresponding to the current video frame image using the fast optical flow algorithm.

[0127] In this embodiment, the fast optical flow algorithm is based on the fundamental assumptions of optical flow, including constant brightness, temporal continuity, and spatial consistency. It estimates object motion by analyzing pixel intensity changes between consecutive video frames. Its core lies in using information from adjacent frames to calculate the motion vector of each pixel over time. In this embodiment, it is used to calculate the motion vector of each pixel in adjacent frames. Applying the fast optical flow algorithm to the post-processing of segmentation results in this application effectively tracks changes in the segmented region image within consecutive video frames, introducing short-term motion patterns into static segmentation and significantly reducing jitter and abrupt changes in the segmented region image.

[0128] S242. Adjust the noise or discontinuous regions in the segmented region image corresponding to the current video frame image according to the motion vector to obtain the processed current segmented region image.

[0129] In this embodiment, the motion vector adjustment of the current segmented region image can reduce noise or discontinuous areas caused by segmentation processing errors, thereby improving the stability and continuity of the segmentation results. Preferably, this application uses a fast optical flow algorithm to calculate the optical flow field between the current video frame image and the previous video frame image for smoothing the current segmented region image.

[0130] Specifically, such as Figure 7 As shown, the post-processing of the segmented region image may further include the following steps:

[0131] S251. Expand the boundary of the segmented region image by applying a morphological dilation operation;

[0132] In this embodiment, by applying morphological dilation to expand the boundary of the segmented region image, the segmented region image can be slightly enlarged, amplifying the noise and uneven areas in the boundary.

[0133] S252. Feather the boundary to obtain a segmented region image with the boundary region faded.

[0134] In this embodiment, feathering dilation is used to expand the boundaries of the segmented region image, making the boundary transition of the segmented region image smoother and more natural. This is particularly helpful for segmenting skin regions, as it helps handle fine hairs, blurred edges, and other issues, achieving a more natural segmentation effect. In this embodiment, the dilated boundary region is feathered, causing the boundary transition area to gradually fade, thereby reducing the hardness of the boundary.

[0135] S253. Perform Gaussian blur filtering on the segmented region image with the faded boundary region to obtain the final segmented region image.

[0136] In this embodiment, Gaussian blur is an image processing technique that smooths an image by applying a normal distribution as a weighting function, reducing noise and details while highlighting key features. In this embodiment, Gaussian blur is used to further smooth boundary transition areas, ensuring the naturalness and consistency of the boundaries.

[0137] like Figure 8 As shown in the embodiments of this application, a training system for an image processing model is also provided. Optionally, the training system may include:

[0138] The module includes a sample acquisition module 311, a sample cropping module 312, a deformation processing module 313, a labeling module 314, and a training module 315, wherein:

[0139] The sample acquisition module 311 is used to acquire several sample images;

[0140] In this embodiment, the sample acquisition module 311 can be used to perform... Figure 2 For a detailed description of the sample acquisition module 311, please refer to the description of step S110 shown.

[0141] The sample cropping module 312 is used to crop the sample image to obtain a cropped sample image by cropping the target object.

[0142] In this embodiment, the sample cropping module 312 can be used to perform... Figure 2 For a detailed description of the sample trimming module 312, please refer to the description of step S120 shown.

[0143] The deformation processing module 313 is used to triangulate and deform the cropped sample image to obtain the corresponding deformed sample image.

[0144] In this embodiment, the deformation processing module 313 can be used to perform... Figure 2 For a detailed description of step S130, please refer to the description of step S130.

[0145] The labeling module 314 is used to label the segmented objects of the sample image and the corresponding deformed sample image;

[0146] In this embodiment, the marking module 314 can be used to perform... Figure 2 For a detailed description of the marking module 314 shown in step S140, please refer to the description of step S140.

[0147] The training module 315 is used to input the labeled cropped sample image and the corresponding deformed sample image into the Siamese network respectively, and train the Siamese network to obtain a trained Siamese network.

[0148] In this embodiment, the training module 315 can be used to execute Figure 2 For a detailed description of the training module 315, please refer to the description of step S150 shown.

[0149] like Figure 9 As shown in the illustration, this application also provides a video image processing system. Optionally, the system may include:

[0150] The module consists of three parts: acquisition module 411, trimming module 412, and generation module 413.

[0151] The acquisition module 411 is used to extract video frame images from the video as images to be processed;

[0152] In this embodiment, the acquisition module 411 can be used to perform... Figure 4 For a detailed description of the acquisition module 411, please refer to the description of step S210 shown.

[0153] The cropping module 412 is used to crop the image to be processed according to the region of interest to obtain a cropped image;

[0154] In this embodiment, the trimming module 412 can be used to perform... Figure 4 For a detailed description of the cutting module 412, please refer to the description of step S220 shown.

[0155] The generation module 413 is used to input the cropped image into a Siamese network for processing to obtain a segmented region image; wherein the Siamese network is a Siamese network trained using an image processing model training method;

[0156] In this embodiment, the generation module 413 can be used to execute Figure 4 For a detailed description of step S230, please refer to the description of step S230.

[0157] This application also provides an electronic device, the structure of which is as follows: Figure 10 As shown, the electronic device includes a memory 511, a processor 512, a communication module 513, and an input / output interface 514, etc. Optionally, the memory 511, the processor 512, the communication module 513, and the input / output interface 514 can be connected and communicate with each other through a bus 515.

[0158] The memory 511 is used to store one or more computer programs and to transfer the code of the computer programs to the processor 512; when the one or more computer programs are executed by the processor 512, a training method for an image processing model or a video image processing method in the embodiments of this application are implemented.

[0159] Optionally, the electronic device can be connected to a network via communication module 513 to exchange data with other devices, such as terminals or servers, through communication over the network. The electronic device can be various forms of digital computers, exemplarily such as desktop computers, servers, workbenches, mainframes, or other types of computers. The electronic device can also be various forms of mobile terminals, exemplarily such as smartphones, tablets, wearable devices (such as helmets, glasses, watches, etc.), and other similar mobile terminals.

[0160] Optionally, the electronic device can connect to desired input / output devices, such as a keyboard or display device, via the input / output interface 514. The electronic device itself may have a display device, and other display devices can also be connected externally via the input / output interface 514. Optionally, a storage device, such as a hard disk, can also be connected via the input / output interface 514 to store data from the electronic device, read data from the storage device, or store data from the storage device in the memory 511. It is understood that the input / output interface 514 can be a wired interface or a wireless interface. Depending on the actual application scenario, the device connected to the input / output interface 514 can be a component of the electronic device or an external device connected to the electronic device when needed.

[0161] Optionally, the memory 511 may be a volatile memory and / or a non-volatile memory. The volatile memory may be a random access memory, etc., and the non-volatile memory may be a read-only memory, a programmable read-only memory, an erasable programmable read-only memory, an electrically erasable programmable read-only memory, or a flash memory, etc.

[0162] Optionally, the computer program stored in the memory 511 can be divided into one or more modules, which are stored in the memory 511 and executed by the processor 512 to perform the method provided in this embodiment. The one or more modules can be a series of computer program instruction segments capable of performing specific functions, which describe the execution process of the computer program in the electronic device.

[0163] Optionally, the processor 512 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the processor 512 include, but are not limited to, a central processing unit, a graphics processing unit, a digital signal processor, various special-purpose artificial intelligence computing chips, various processors running machine learning model algorithms, and can also be any suitable controller, microcontroller, processor, etc. The processor 512 executes the various methods and processes of this embodiment, exemplarily, such as a method for training an image processing model or a video image processing method according to an embodiment of this application.

[0164] Optionally, the bus 515 may include a path for transmitting information. The bus 515 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. Depending on its function, the bus 515 may be divided into an address bus, a data bus, a control bus, etc.

[0165] In an optional implementation, this application embodiment also provides a computer storage medium storing a computer program thereon. When the computer program is executed by a computer, it enables the computer to perform the methods described in the above-described method embodiments. Part or all of the computer program can be loaded and / or installed on the memory 511 of an electronic device. When the computer program is executed by the processor 512, it can perform one or more steps of a training method for an image processing model or a video image processing method according to embodiments of this application.

[0166] Optionally, the computer-readable storage medium may be a random access memory, a read-only memory, a programmable read-only memory, an erasable programmable read-only memory, an electrically erasable programmable read-only memory, etc.

[0167] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the technical solution of the present invention, and are not intended to limit the specific implementation of the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the claims of the present invention should be included within the protection scope of the claims of the present invention.

Claims

1. A training method for an image processing model, characterized in that, The method includes: Acquire several sample images; The sample image is cropped by cropping the target object to obtain a cropped sample image; The cropped sample image is triangulated to obtain a corresponding deformed sample image; wherein, the triangulation deformation is to construct a triangular mesh on the pixel set of the cropped sample image, dividing the cropped sample image into multiple triangular regions; The sample image and the corresponding deformed sample image are labeled with segmented objects; The labeled cropped sample image is input into the Siamese network to obtain a first generated image, and the deformed sample image corresponding to the cropped sample image is input into the Siamese network to obtain a corresponding second generated image; Perform an inverse deformation operation on the second generated image to obtain an inversely deformed second generated image; The loss value of the Siamese network is obtained by comparing the first generated image and the second generated image with inverse deformation at global pixels. The parameters of the Siamese network are optimized based on the loss value to obtain a trained Siamese network.

2. A video image processing method, characterized in that, The method includes: Extract video frame images from the video as images to be processed; The image to be processed is cropped according to the region of interest to obtain a cropped image; The cropped image is input into a Siamese network for processing to obtain a segmented region image; wherein the Siamese network is a Siamese network trained using the training method described in claim 1.

3. The video image processing method according to claim 2, characterized in that, The region of interest is stored in a preset database; Extract the specific location information of the target object in the image to be processed from the segmented region image; Generate the region of interest (ROI) of the current video frame image based on the specific location information; The region of interest (ROI) of the current video frame image is swapped with the ROI stored in the database and then stored as the ROI for the next video frame image.

4. The video image processing method according to claim 2, characterized in that, The method further includes: Based on the segmented region image corresponding to the previous video frame, the motion vector of each pixel in the segmented region image corresponding to the current video frame is obtained by the fast optical flow algorithm. Based on the motion vector, the noise or discontinuous regions in the segmented region image corresponding to the current video frame image are adjusted to obtain the current segmented region image after processing.

5. A video image processing method according to any one of claims 2-4, characterized in that, The method further includes: The boundaries of the segmented region image are expanded by applying a morphological dilation operation; Feathering the boundary yields a segmented region image with a faded boundary area. Gaussian blur filtering is applied to the segmented region image with the faded boundary region to obtain the final segmented region image.

6. A video image processing system, characterized in that, The system includes: The acquisition module is used to extract video frame images from the video as images to be processed. The cropping module is used to crop the image to be processed according to the region of interest to obtain a cropped image; The generation module is used to input the cropped image into a Siamese network for processing to obtain a segmented region image; wherein the Siamese network is a Siamese network trained using the training method described in claim 1.

7. A video image processing system according to claim 6, characterized in that, The region of interest is stored in a preset database; Extract the specific location information of the target object in the image to be processed from the segmented region image; Generate the region of interest (ROI) of the current video frame image based on the specific location information; The region of interest (ROI) of the current video frame image is swapped with the ROI stored in the database and then stored as the ROI for the next video frame image.

8. An electronic device, characterized in that, include: Memory, used to store one or more computer programs; A processor, when the one or more computer programs are executed by the processor, implements the training method of the image processing model as described in claim 1 or a video image processing method as described in any one of claims 2-5.

9. A computer-readable storage medium storing computer instructions for causing a processor to execute and implement the training method of the image processing model as described in claim 1 or the video image processing method as described in any one of claims 2-5.

Images