Image splicing method, device and equipment and computer readable storage medium

Abstract

The invention discloses an image splicing method, device and equipment and a computer readable storage medium, and is applied to the field of data processing, and the method comprises the steps: carrying out the segmentation processing of a to-be-spliced overall image through a semantic segmentation method, and obtaining a semantic segmentation result; determining a target mask region based on the semantic segmentation result and the segmentation category; carrying out object expansion processing on the target mask area to obtain a binary mask expansion image; performing blurring processing on the binarized mask expanded image to obtain a target mask image; and generating a coverage image corresponding to the segmentation category, and performing image stitching processing by using the target mask image, the coverage image and the to-be-stitched overall image to obtain a target stitched image. According to the method, object expansion processing is performed on the target mask area to obtain a larger mask, ghosting and unnatural phenomena during splicing are reduced, and fuzzy processing is performed on the binarized mask expansion image to improve transition smoothness, so that a better splicing effect is obtained.

Description

Technical Field

[0001] The present invention relates to the field of data processing, and in particular to an image stitching method, device, equipment and computer-readable storage medium. Background Art

[0002] Image stitching solutions generally have two inputs, a base map and an overlay map, and the final output is a single image. Overlay maps are often generated by algorithms. For example, in the application scenario of virtual population shape generation, a series of lip shape maps can be generated based on audio. Currently, when stitching is performed, it is mainly based on image segmentation methods. It is necessary to first segment the foreground and background, which has high requirements for segmentation quality. When the image quality is average, the stitching effect is poor; or stitching is based on parameter models. This stitching method is sensitive to parallax and is prone to ghosting and obvious gaps.

[0003] Therefore, there is currently a technical problem of poor image stitching effect. Summary of the invention

[0004] In view of this, an object of the present invention is to provide an image stitching method, device, equipment and computer-readable storage medium, which solve the technical problem of poor image stitching effect in the prior art.

[0005] In order to solve the above technical problems, the present invention provides an image stitching method, comprising:

[0006] The semantic segmentation method is used to segment the whole spliced ​​image to obtain the semantic segmentation result;

[0007] Determine a target mask area based on the semantic segmentation result and the segmentation category;

[0008] Performing object enlargement processing on the target mask area to obtain a binary mask enlarged image;

[0009] Performing fuzzy processing on the binary mask enlarged image to obtain a target mask image;

[0010] A coverage map corresponding to the segmentation category is generated, and image stitching processing is performed using the target mask image, the coverage map and the overall image to be stitched to obtain a target stitched image.

[0011] Optionally, the determining the target mask area based on the semantic segmentation result and the segmentation category includes:

[0012] Determine the segmentation category according to actual splicing requirements;

[0013] Binarization is performed on the pixels in the semantic segmentation result based on the segmentation category to obtain the target mask area.

[0014] Optionally, the whole image to be stitched is a virtual human image to be stitched, and the determining the segmentation category according to actual stitching requirements includes:

[0015] When the actual splicing requirement is to splice the generated physiological parts, the segmentation category is determined to be the physiological part.

[0016] Optionally, performing object enlargement processing on the target mask area to obtain a binary mask enlarged image includes:

[0017] Define a structure element; wherein the structure element is a matrix composed of 0 and 1;

[0018] Performing pixel comparison processing on the structural element and the overall image to be spliced ​​including the target mask area to obtain a comparison result;

[0019] The pixel values ​​of the whole image to be spliced ​​are adjusted based on the comparison result to obtain the binary mask enlarged image.

[0020] Optionally, the blurring the binary mask enlarged image to obtain a target mask image includes:

[0021] Performing Gaussian blur filtering on the binary mask enlarged image to obtain a blurred mask;

[0022] The fuzzy mask is normalized to obtain the target mask image.

[0023] Optionally, after blurring the binary mask enlarged image to obtain a target mask image, the method further includes:

[0024] Performing global color adjustment on the target mask image by using histogram equalization to obtain a target adjusted image;

[0025] Performing noise reduction processing on the target adjustment image to obtain a final target mask image;

[0026] Accordingly, a coverage map corresponding to the segmentation category is generated, and image stitching processing is performed using the target mask image, the coverage map and the overall image to be stitched to obtain a target stitched image, including:

[0027] The final target mask image, the coverage image and the overall image to be stitched are used to perform image stitching processing to obtain the target stitching image.

[0028] Optionally, the above image stitching method is applied to a video, including:

[0029] Extracting video frames from the video to obtain multiple video frames; wherein the overall image to be spliced ​​includes the video frames;

[0030] Processing each of the video frames frame by frame using the image stitching method to obtain a target stitching image corresponding to each of the video frames;

[0031] According to the timestamp of the target stitching image corresponding to each of the video frames, all the target stitching images are stitched together to obtain a target stitching video frame;

[0032] The target spliced ​​video frames are encoded into a video format to obtain a target spliced ​​video.

[0033] Optionally, generating a coverage map corresponding to the segmentation category, and performing image stitching processing using the target mask image, the coverage map and the overall image to be stitched to obtain a target stitched image, including:

[0034] Multiplying the whole image to be stitched and the non-masked area pixel by pixel to determine an image of the non-masked area;

[0035] Multiplying the coverage map and the target mask image pixel by pixel to determine a mask area image;

[0036] The target stitching image is determined by adding the image of the unmasked area and the image of the masked area pixel by pixel.

[0037] The present application also provides an image stitching device, comprising:

[0038] A semantic segmentation module, used to segment the whole image to be spliced ​​using a semantic segmentation method to obtain a semantic segmentation result;

[0039] A mask determination module, used to determine a target mask area based on the semantic segmentation result and the segmentation category;

[0040] A mask expansion module, used for performing object expansion processing on the target mask area to obtain a binary mask expansion image;

[0041] A fuzzy processing module, used for performing fuzzy processing on the binary mask enlarged image to obtain a target mask image;

[0042] The stitching module is used to generate a coverage map corresponding to the segmentation category, and perform image stitching processing using the target mask image, the coverage map and the overall image to be stitched to obtain a target stitched image.

[0043] The present application also provides an electronic device, comprising:

[0044] Memory for storing computer programs;

[0045] A processor is used to implement the steps of the above-mentioned image stitching method when executing the computer program.

[0046] The present application also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned image stitching method are implemented.

[0047] The present application also provides a computer program, including a computer program / instruction, which implements the steps of the above-mentioned image stitching method when executed by a processor.

[0048] It can be seen that the present invention uses a semantic segmentation method to segment the whole image to be spliced ​​to obtain a semantic segmentation result; based on the semantic segmentation result and the segmentation category, the target mask area is determined; the target mask area is enlarged to obtain a binary mask enlarged image; the binary mask enlarged image is blurred to obtain a target mask image; a coverage map corresponding to the segmentation category is generated, and the target mask image, the coverage map and the whole image to be spliced ​​are used to perform image splicing to obtain a target spliced ​​image. Compared with the current splicing based on parameter models, which is prone to ghosting and obvious gaps, the beneficial effects of the present application are: the target mask area is enlarged to obtain a larger mask, thereby reducing ghosting and unnatural phenomena during splicing, and the binary mask enlarged image is blurred to achieve transition smoothing, thereby obtaining a better splicing effect.

[0049] In addition, the present invention also provides an image stitching device, equipment and computer-readable storage medium, which also have the above-mentioned beneficial effects. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0051] Figure 1 A flowchart of an image stitching method provided by an embodiment of the present invention;

[0052] Figure 2 A semantic segmentation result obtained by segmentation based on a semantic segmentation method provided in an embodiment of the present invention;

[0053] Figure 3 A schematic diagram of a target mask area provided by an embodiment of the present invention;

[0054] Figure 4A schematic diagram of ghosting provided by an embodiment of the present invention;

[0055] Figure 5 An example flowchart of a mask generation method provided by an embodiment of the present invention;

[0056] Figure 6 A comparison diagram before and after expansion provided by an embodiment of the present invention;

[0057] Figure 7 A comparison diagram of fuzzy processing provided by an embodiment of the present invention;

[0058] Figure 8 A schematic structural diagram of an image stitching device provided by an embodiment of the present invention;

[0059] Fig. 9 A schematic diagram of the structure of an image stitching device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0060] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0061] The method based on pixel grayscale is greatly affected by changes in brightness and contrast and has poor results; the method based on feature matching has a large amount of calculation and is not suitable for scenes with high requirements for real-time video stitching; the method based on parameter model is sensitive to parallax and easily produces ghost images and obvious gaps; the method based on image segmentation needs to segment the foreground and background first, and has high requirements for segmentation quality.

[0062] This technical solution is based on a face segmentation method based on deep learning. It has the following advantages:

[0063] Semantic segmentation based on deep learning can accurately separate faces and backgrounds, providing accurate segmentation masks for subsequent stitching; image segmentation masks can be generated in advance, and the computational complexity of the stitching scheme is very small, thus achieving the effect of real-time stitching; the transition of the mask is achieved based on Gaussian blur, solving the problem of stitching gaps.

[0064] Please refer to Figure 1 , Figure 1 A flowchart of an image stitching method provided by an embodiment of the present invention. The method may include:

[0065] S101, using a semantic segmentation method to segment the entire image to be spliced, to obtain a semantic segmentation result.

[0066] The execution subject of this embodiment is an electronic device. For example, the electronic device in this embodiment can be a computer, a mobile phone, etc. This embodiment does not limit the specific application scenario. For example, this embodiment can be applied to the generation and splicing of a certain area in a static image, or this embodiment can also be applied to the generation (lip shape generation) and splicing of dynamic videos in videos. The semantic segmentation method in this embodiment is a method for segmenting an image according to a determined segmentation category. This embodiment does not limit the specific semantic segmentation method. For example, the semantic segmentation method in this embodiment can be a fully convolutional neural network (FCN, Fully Convolutional Networks). FCN replaces the fully connected layer in the traditional CNN (Convolutional Neural Network) with a convolutional layer to achieve end-to-end pixel-level segmentation mapping; or the semantic segmentation method in this embodiment can be a BISENet (Bilateral Segmentation Network for Real-time Semantic Segmentation, bilateral segmentation network) network structure to achieve real-time semantic segmentation of humans: BiSeNet This structure contains two parallel branches: one branch focuses on spatial detail information, and the other branch focuses on semantic information. The two branches finally perform feature fusion to obtain a segmentation result with both spatial details and semantic information. BISENet uses a lightweight network structure that can achieve real-time semantic segmentation on the CPU and is suitable for mobile applications. Figure 2 , Figure 2 A semantic segmentation result obtained by segmentation based on a semantic segmentation method provided in an embodiment of the present invention is obtained from Figure 2 It can be seen that the boundaries between the parts are clear and the segmentation types are obvious. This embodiment does not limit the specific overall image to be spliced. For example, the overall image to be spliced ​​in this embodiment can be a virtual human image to be spliced; or the overall image to be spliced ​​in this embodiment can be an animal image to be spliced.

[0067] It should be further explained that the above-mentioned image stitching method can be applied to video, and may include: extracting video frames in the video to obtain multiple video frames; wherein the overall image to be stitched includes video frames; using the image stitching method to process each video frame frame by frame to obtain a target stitching image corresponding to each video frame; stitching all target stitching images according to the timestamp of the target stitching image corresponding to each video frame to obtain a target stitching video frame; encoding the target stitching video frame into a video format to obtain a target stitching video. This embodiment processes the video frame by frame to achieve the detailed steps of image stitching in the video scene: 1. Video frame extraction: use video processing software or library (such as OpenCV (Open Source Computer Vision Library, open source computer vision library)) to read the input video file. Decompose the video into separate frame sequences. 2. Face detection: use a face detection algorithm (such as Haar (Haar-likefeatures, digital image features for object recognition) features or deep learning models) to detect the position of the face for each frame of the image. 3. Face parsing: Apply BiSeNet (Bilateral Segmentation Network for Real-time Semantic Segmentation) network structure to the detected face area for semantic segmentation. Enhance the feature expression of key areas through the attention mechanism to obtain the segmentation mask of the face. 4. Mask expansion: Use morphological dilation operation to expand the segmentation mask to cover more face area. Select appropriate structural elements for morphological dilation. 5. Gaussian blur: Apply Gaussian blur to the expanded mask to smooth edges and create transition areas. Adjust the parameters of Gaussian blur to obtain the best blur effect. 6. Image stitching: Align the generated overlay (such as virtual population map) with the base map (original video frame). According to the mask after Gaussian blur, merge the corresponding area of ​​the overlay with the base map. Reassemble the stitched images into frames. Make sure that the stitched frames are visually consistent with other parts of the original video frame (such as the background). 7. Output video: Re-encode all processed frames into video format. Adjust the encoding parameters of the video, such as resolution, frame rate and bit rate, to ensure the video quality. Save the final video file in a desired format, such as MP4 (MPEG-4 Part 14, digital multimedia container) or AVI (Audio Video Interleave, video file format). In this embodiment, the video frames in the video can be spliced ​​frame by frame to obtain the target spliced ​​video, so that the image splicing method can be applied to the video.

[0068] S102, determining a target mask area based on the semantic segmentation result and the segmentation category.

[0069] This embodiment does not limit the specific segmentation category. For example, the segmentation category in this embodiment may be the mouth; or the segmentation category in this embodiment may be various physiological parts such as the eyes. For ease of understanding, this embodiment determines the target mask area based on the semantic segmentation results and segmentation categories. After obtaining the semantic segmentation results, each pixel is binarized. If the pixel belongs to the selected category, it is set to 1 (white). Other pixels are set to 0 (black). For example, if the current splicing requirement is to splice the generated mouth, then what is needed here is the mask of the mouth, that is, the target mask area, such as Figure 3 , Figure 3 A schematic diagram of a target mask area provided by an embodiment of the present invention, Figure 3 The target mask area in is the mouth.

[0070] It should be further explained that, in order to improve the accuracy of determining the target mask area, the above-mentioned determination of the target mask area based on the semantic segmentation result and the segmentation category may include: determining the segmentation category according to the actual splicing requirements; binarizing the pixels in the semantic segmentation result based on the segmentation category to obtain the target mask area. Determine the segmentation category according to the actual splicing requirements; binarize the pixels in the semantic segmentation result based on the segmentation category to obtain the target mask area. This embodiment does not limit the specific actual splicing requirements. For example, when the actual splicing requirement is to splice the eyes, the segmentation category is determined to be eyes. For example, when the actual splicing requirement is to splice the generated mouth, the segmentation category is determined to be mouth. This embodiment improves the accuracy of determining the target mask area by accurately determining the segmentation category by determining the specific actual requirements.

[0071] It should be further explained that the above-mentioned whole image to be spliced ​​is a virtual human image to be spliced, and the segmentation category is determined according to the actual splicing requirements, which may include: when the actual splicing requirement is to splice the generated physiological parts, determining the segmentation category as physiological parts. This embodiment does not limit specific physiological parts. For example, the physiological part in this embodiment may be a nose; or the physiological part in this embodiment may be an ear, or the physiological part in this embodiment may also be a hand. This embodiment can meet the needs of different users and improve the user experience by splicing various physiological parts.

[0072] S103, performing object enlargement processing on the target mask area to obtain a binary mask enlarged image.

[0073] This embodiment is not limited to the specific method of performing object expansion processing on the target mask area. For example, this embodiment can perform object expansion processing by threshold adjustment; or this embodiment can also perform object expansion processing through morphological operations. For example, this embodiment can perform object expansion processing based on a structural element. The structural element (also called a kernel or mask) in this embodiment is a small two-dimensional matrix used to define the "shape" and "size" of the expansion operation. The selection of the structural element has an important impact on the result of the expansion (enlargement). It is not determined randomly, but is designed according to specific application requirements and image features. The following are some considerations for defining structural elements: Shape: The structural element can be rectangular, circular, cross-shaped, etc. The choice of shape depends on the type of image features that you want to emphasize. Size: The size of the structural element (for example, 3x3, 5x5, 7x7, etc.) determines the scope of the expansion operation. Larger structural elements will result in a wider expansion effect, while smaller structural elements will affect a smaller range. It can be understood that the target mask area obtained based on segmentation is relatively small. If it is used directly for stitching, it will cause ghosting, such as Figure 4 , Figure 4 A schematic diagram of a ghosting provided by an embodiment of the present invention, from Figure 4 It can be seen that there are ghost images in the current image. This embodiment does not limit the specific process of object enlargement processing on the target mask area based on the structure element. For example, this embodiment can a) define a structure element, which is usually a small two-dimensional matrix, such as a 3x3 or 5x5 matrix, in which the value can be 0 or 1. b) Place the structure element at a position in the image (the image of the target mask area), and then compare the structure element with the image area at that position. c) If the structure element can "adapt" to the position (that is, the image pixels corresponding to all 1s in the structure element are also 1), then set the output pixel value of the position to 1, otherwise set it to 0. d) Move the structure element to the next position in the image, and repeat steps b) and c) until the entire image is scanned.

[0074] It should be further explained that, in order to improve the accuracy of the object enlargement process, the object enlargement process is performed on the target mask area to obtain a binary mask enlarged image, which may include:

[0075] S1031, define a structure element; wherein the structure element is a matrix composed of 0 and 1;

[0076] S1032, performing pixel comparison processing on the structural element and the whole image to be spliced ​​including the target mask area to obtain a comparison result;

[0077] S1033, adjusting the pixel values ​​of the whole image to be spliced ​​based on the comparison result to obtain a binary mask enlarged image.

[0078] In this embodiment, pixel comparison processing is performed on the structural element and the overall image to be spliced ​​including the target mask area, which means that the structural element is used to compare all areas of the overall image to be spliced ​​including the target mask area. If the pixels of the area corresponding to all 1s of the structural element are also 1, the pixel value of the position is set to 1, otherwise it is set to 0. In this embodiment, the mask is enlarged by using the structural element to perform pixel comparison with the overall image to be spliced ​​including the target mask area, thereby improving the accuracy of mask enlargement (expansion).

[0079] S104, performing blur processing on the binary mask enlarged image to obtain a target mask image.

[0080] This embodiment does not limit the specific method of blur processing. For example, the blur processing method in this embodiment may be a Gaussian blur processing method; or the blur processing method in this embodiment may also be a mean blur processing method.

[0081] S105, generating a coverage map corresponding to the segmentation category, and performing image stitching processing using the target mask image, the coverage map and the overall image to be stitched to obtain a target stitched image.

[0082] This embodiment does not limit the specific method of generating the overlay map corresponding to the segmentation type. For example, when the segmentation type is mouth, a face overlay map related to the mouth is generated; or, when the segmentation type is eyes, a face overlay map related to the eyes is generated. This embodiment uses the target mask image, the overlay map and the overall image to be spliced ​​to perform image splicing processing. The process of obtaining the target spliced ​​image can be to use the target mask image to perform pixel-by-pixel multiplication of the base image (the overall image to be spliced) and the target mask image to obtain a first pixel-by-pixel multiplication result map, and to perform pixel-by-pixel multiplication of the overlay map and the target mask image to obtain a second pixel-by-pixel multiplication result map, and to perform pixel-by-pixel addition of the first pixel-by-pixel multiplication result map and the second pixel-by-pixel multiplication result map to obtain the target spliced ​​image.

[0083] It should be further explained that, in order to improve the effect of blurring, the above-mentioned blurring of the binary mask enlarged image to obtain the target mask image may include: performing Gaussian blur filtering on the binary mask enlarged image to obtain a blurred mask; normalizing the blurred mask to obtain the target mask image. This embodiment can smooth the mask edge and generate a transition area by performing Gaussian blur filtering on the binary mask enlarged image. The kernel size and standard deviation of the Gaussian blur can be adjusted according to actual needs to obtain an ideal blurring effect. In the mask after Gaussian blurring, the pixel value of the transition area is between 0 and 1. The transition area can be used to achieve a smooth edge effect and avoid hard boundaries. The blurred mask is normalized to limit the pixel value range to between 0 and 1. This embodiment can smooth the mask edge and generate a transition area by blurring through Gaussian blurring, thereby improving the effect of blurring. The characteristic of Gaussian blurring is that it maintains the natural transition of the edge during the blurring process, so it is very suitable for image splicing and fusion.

[0084] It should be further explained that in order to improve the effect of the target mask image, after blurring the binary mask enlarged image to obtain the target mask image, it can also include: using histogram equalization to perform global color adjustment on the target mask image to obtain a target adjustment image; performing noise reduction on the target adjustment image to obtain the final target mask image; correspondingly, generating a coverage map corresponding to the segmentation category, and performing image stitching processing using the target mask image, the coverage map and the overall image to be stitched to obtain a target stitched image, including: performing image stitching processing using the final target mask image, the coverage map and the overall image to be stitched to obtain a target stitched image. This embodiment will perform color adjustment and noise reduction processing on the target mask image to improve the clarity of the target mask image, thereby improving the effect of the target mask image.

[0085] It should be further explained that, in order to improve the accuracy of image stitching, the above generation of a coverage map corresponding to the segmentation category, and the use of the target mask image, the coverage map and the overall image to be stitched for image stitching processing to obtain the target stitched image, can include: pixel-by-pixel multiplication of the overall image to be stitched and the non-mask area to determine the image of the non-mask area; pixel-by-pixel multiplication of the coverage map and the target mask image to determine the mask area image; pixel-by-pixel addition of the image of the non-mask area and the mask area image to determine the target stitched image. This embodiment provides a specific process for image stitching based on the target mask image, the coverage map, and the overall image to be stitched, which improves the accuracy of image stitching using this application.

[0086] An image stitching method provided by an embodiment of the present invention may include: S101, using a semantic segmentation method to perform segmentation processing on the entire image to be stitched to obtain a semantic segmentation result; S102, determining a target mask area based on the semantic segmentation result and the segmentation category; S103, performing object enlargement processing on the target mask area to obtain a binary mask enlarged image; S104, blurring the binary mask enlarged image to obtain a target mask image; S105, generating a coverage map corresponding to the segmentation category, and performing image stitching processing using the target mask image, the coverage map and the entire image to be stitched to obtain a target stitched image. The present application obtains a larger mask by performing object enlargement processing on the target mask area, thereby reducing ghosting and unnatural phenomena during stitching, and blurring the binary mask enlarged image to achieve smooth transition, thereby obtaining a better stitching effect. Furthermore, this embodiment improves the accuracy of determining the target mask area by accurately determining the segmentation category by determining specific actual needs; further, this embodiment achieves mask expansion by using structural elements to perform pixel comparison with the overall image to be spliced ​​including the target mask area, thereby improving the accuracy of mask expansion (expansion); further, this embodiment can smooth the mask edge and generate a transition area by performing blur processing through Gaussian blur, thereby improving the effect of blur processing.

[0087] In order to make the present invention easier to understand, please refer to Figure 5 , Figure 5 An example flowchart of a mask generation method provided by an embodiment of the present invention may specifically include:

[0088] S201, using face analysis technology to analyze the input face image (base image) to obtain a semantic segmentation result of the face.

[0089] The face analysis technology in this embodiment is a semantic segmentation technology, and the semantic segmentation technology can be a bilateral segmentation network. The input face image in this embodiment corresponds to the above-mentioned overall image to be spliced.

[0090] S202, determining a target mask area based on the semantic segmentation result and the segmentation category.

[0091] It is understandable that the semantic segmentation results are often multi-classified. You can select the corresponding segmentation category according to your needs and retain the corresponding category. After obtaining the semantic segmentation results, binarize each pixel. If the pixel belongs to the category you selected, set it to 1 (white). Other pixels are set to 0 (black). The use of binarization has the following effects: Accuracy: The binary mask can define the face area very accurately and provide a clear boundary for subsequent stitching. Simplified calculation: The binary mask simplifies the calculation process because it only contains two values ​​0 and 1, which helps to reduce the amount of calculation, especially in real-time application scenarios. Easy to process: The binary mask is easy to combine with image processing algorithms, such as morphological dilation and Gaussian blur, which can further optimize the stitching effect.

[0092] Real-time: The generation and processing of the binary mask can be completed quickly, which is suitable for the needs of real-time face replacement technology. In this application, the binary mask is an effective choice for real-time face replacement technology. Of course, there can be some extensions, such as using different masks for the three channels of RGB (Red Green Blue, color model). The RGB color model is a color standard widely used in the industry. Various colors are generated by changing the three color channels of red, green, and blue and superimposing them on each other.

[0093] S203, performing mask expansion on the target mask region by using morphological dilation to obtain a mask expansion region.

[0094] The process of performing mask expansion in this embodiment may include: (1) defining a binary structure element; (2) comparing the structure element with all regions in the target mask region, determining the pixel values ​​corresponding to all regions in the target mask region based on the comparison result, and obtaining the mask expansion region, which corresponds to the binary mask enlarged image mentioned above. For ease of understanding, please refer to Figure 6 , Figure 6 A comparison diagram before and after expansion provided by an embodiment of the present invention, from Figure 6 It can be seen that the area on the left before expansion is larger than the area on the right after expansion.

[0095] S204, using Gaussian blur to blur the mask extension area to obtain a target mask image.

[0096] It can be understood that the binary mask (mask extension area) obtained in the previous step has a pixel value of 1 in the target area and a pixel value of 0 in other areas. In this step, a Gaussian blur filter is applied to the mask extension area to smooth the mask edge and generate a transition area. This embodiment can also normalize the blurred mask to limit the pixel value range to between 0 and 1. For ease of understanding, please refer to Figure 7 , Figure 7 A comparison diagram of blur processing provided by an embodiment of the present invention, the right side shows the effect after blur processing, and the left side shows the effect without blur processing.

[0097] S205, multiplying the base image (face image) and the non-mask area pixel by pixel to determine the image of the non-mask area.

[0098] S206, generating a coverage map corresponding to the segmentation category, multiplying the coverage map and the target mask image pixel by pixel, and determining a mask region image.

[0099] S207, adding the image of the unmasked area and the image of the masked area pixel by pixel to determine a target stitched image.

[0100] In the embodiment of the present invention, the image stitching scheme has three inputs, a base map (input face image), an overlay map and a mask (target mask image), and finally outputs a map. The overlay map in this embodiment corresponds to the target mask area. For example, when the target mask area is the mouth, the overlay map is a mouth shape map generated according to the audio. The overlay map is often generated by an algorithm. For example, in the application scenario of virtual population shape generation, a series of mouth shape maps can be generated according to the audio. The process of image stitching according to the base map, overlay map and mask in this embodiment is "base map X (1-mask)" + "overlay map X mask"; wherein X represents pixel-by-pixel multiplication, + represents pixel-by-pixel addition, the base map is the above-mentioned input face image, and the mask is the obtained target mask image; that is, the base map and the non-mask area are multiplied pixel by pixel to determine the image of the non-mask area, the overlay map and the mask are multiplied pixel by pixel to determine the mask area image, and the image of the non-mask area and the mask area image are added pixel by pixel to determine the target stitching image.

[0101] The embodiment of the present invention performs morphological expansion on the mask to reduce the splicing gap, and uses the special effect that the human eye is insensitive to color transition to perform Gaussian blur on the binary mask expansion area, so as to obtain a better splicing effect.

[0102] The image stitching device provided by the embodiment of the present invention is introduced below. The image stitching device described below and the image stitching method described above can be referred to each other.

[0103] Please refer to Figure 8 , Figure 8 A schematic diagram of the structure of an image stitching device provided by an embodiment of the present invention may include:

[0104] The semantic segmentation module 100 is used to segment the whole image to be spliced ​​using a semantic segmentation method to obtain a semantic segmentation result;

[0105] A mask determination module 200, configured to determine a target mask region based on the semantic segmentation result and the segmentation category;

[0106] The mask expansion module 300 is used to perform object expansion processing on the target mask area to obtain a binary mask expansion image;

[0107] A fuzzy processing module 400 is used to perform fuzzy processing on the binary mask enlarged image to obtain a target mask image;

[0108] The stitching module 500 is used to generate a coverage map corresponding to the segmentation category, and perform image stitching processing using the target mask image, the coverage map and the overall image to be stitched to obtain a target stitched image.

[0109] Further, based on the above embodiment, the mask determination module 200 may include:

[0110] A segmentation category determination unit, used to determine the segmentation category according to actual splicing requirements;

[0111] A binarization processing unit is used to perform binarization processing on the pixel points in the semantic segmentation result based on the segmentation category to obtain the target mask area.

[0112] Further, based on the above embodiment, the segmentation category determination unit may include:

[0113] The segmentation category determination subunit is used to determine that the segmentation category is a physiological part when the actual splicing requirement is to splice the generated physiological parts.

[0114] Further, based on any of the above embodiments, the mask expansion module 300 may include:

[0115] A structure element definition unit, used to define a structure element; wherein the structure element is a matrix composed of 0 and 1;

[0116] A comparison result determination unit, configured to perform pixel comparison processing on the structural element and the overall image to be spliced ​​including the target mask area to obtain a comparison result;

[0117] The mask enlargement unit is used to adjust the pixel value of the whole image to be spliced ​​based on the comparison result to obtain the binary mask enlarged image.

[0118] Further, based on the above embodiment, the fuzzy processing module 400 may include:

[0119] A fuzzy processing unit, used for performing Gaussian fuzzy filtering on the binary mask enlarged image to obtain a fuzzy mask;

[0120] The normalization processing unit is used to perform normalization processing on the fuzzy mask to obtain the target mask image.

[0121] Further, based on any of the above embodiments, the image stitching device may further include:

[0122] A color adjustment module, used for performing global color adjustment on the target mask image by using histogram equalization to obtain a target adjusted image;

[0123] A final target mask image determination module is used to perform noise reduction processing on the target adjustment image to obtain a final target mask image;

[0124] Accordingly, the splicing module 500 includes:

[0125] The stitching unit is used to perform image stitching processing using the final target mask image, the coverage image and the overall image to be stitched to obtain the target stitching image.

[0126] Further, based on any of the above embodiments, the above image stitching method is applied to a video and may include:

[0127] A video frame extraction module, used to extract video frames from a video to obtain multiple video frames; wherein the overall image to be spliced ​​includes the video frames;

[0128] An image stitching module, used to process each of the video frames frame by frame using the image stitching method to obtain a target stitching image corresponding to each of the video frames;

[0129] A video frame stitching module, used for stitching all target stitching images according to the timestamp of the target stitching image corresponding to each video frame to obtain a target stitching video frame;

[0130] The encoding module is used to encode the target spliced ​​video frame into a video format to obtain a target spliced ​​video.

[0131] Further, based on any of the above embodiments, the splicing module 500 may include:

[0132] An image determination unit for the unmasked area is used to multiply the whole image to be spliced ​​and the unmasked area pixel by pixel to determine the image of the unmasked area;

[0133] A mask area image determination unit, configured to multiply the coverage image and the target mask image pixel by pixel to determine a mask area image;

[0134] The target stitched image determining unit is used to add the image of the unmasked area and the image of the masked area pixel by pixel to determine the target stitched image.

[0135] It should be noted that the order of the modules and units in the above-mentioned image stitching device can be changed without affecting the logic.

[0136] The image stitching device provided by the embodiment of the present invention may include: a semantic segmentation module 100, which is used to segment the whole image to be stitched using a semantic segmentation method to obtain a semantic segmentation result; a mask determination module 200, which is used to determine the target mask area based on the semantic segmentation result and the segmentation category; a mask expansion module 300, which is used to perform object enlargement processing on the target mask area to obtain a binary mask enlarged image; a blur processing module 400, which is used to blur the binary mask enlarged image to obtain a target mask image; a stitching module 500, which is used to generate a coverage map corresponding to the segmentation category, and use the target mask image, the coverage map and the whole image to be stitched to perform image stitching processing to obtain a target stitched image. The present application obtains a larger mask by performing object enlargement processing on the target mask area, thereby reducing ghosting and unnatural phenomena during stitching, and blurs the binary mask enlarged image to achieve smoothness of transition, thereby obtaining a better stitching effect. Furthermore, this embodiment improves the accuracy of determining the target mask area by accurately determining the segmentation category by determining specific actual needs; further, this embodiment achieves mask expansion by using structural elements to perform pixel comparison with the overall image to be spliced ​​including the target mask area, thereby improving the accuracy of mask expansion (expansion); further, this embodiment can smooth the mask edge and generate a transition area by performing blur processing through Gaussian blur, thereby improving the effect of blur processing.

[0137] An image stitching device provided by an embodiment of the present invention is introduced below. The image stitching device described below and the image stitching method described above can be referred to each other.

[0138] Please refer to Fig. 9 , Fig. 9 A structural schematic diagram of an image stitching device provided by an embodiment of the present invention may include:

[0139] A memory 10, used for storing computer programs;

[0140] The processor 20 is used to execute a computer program to implement the above-mentioned image stitching method.

[0141] The memory 10 , the processor 20 , and the communication interface 30 all communicate with each other via a communication bus 40 .

[0142] In the embodiment of the present invention, the memory 10 is used to store one or more programs, and the program may include program code, and the program code includes computer operation instructions. In the embodiment of the present invention, the memory 10 may store programs for implementing the following functions:

[0143] The semantic segmentation method is used to segment the whole spliced ​​image to obtain the semantic segmentation result;

[0144] Determine the target mask area based on the semantic segmentation results and segmentation categories;

[0145] Performing object enlargement processing on the target mask area to obtain a binary mask enlarged image;

[0146] Perform fuzzy processing on the binary mask enlarged image to obtain the target mask image;

[0147] Generate a coverage map corresponding to the segmentation category, and use the target mask image, the coverage map and the overall image to be stitched to perform image stitching processing to obtain a target stitching image.

[0148] In a possible implementation, the memory 10 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and an application required for at least one function, etc.; the data storage area may store data created during use.

[0149] In addition, the memory 10 may include a read-only memory and a random access memory, and provide instructions and data to the processor. A portion of the memory may also include an NVRAM. The memory stores an operating system and operating instructions, executable modules or data structures, or a subset thereof, or an extended set thereof, wherein the operating instructions may include various operating instructions for implementing various operations. The operating system may include various system programs for implementing various basic tasks and processing hardware-based tasks.

[0150] The processor 20 may be a central processing unit (CPU), an application specific integrated circuit, a digital signal processor, a field programmable gate array or other programmable logic device, a microprocessor or any conventional processor, etc. The processor 20 may call a program stored in the memory 10 .

[0151] The communication interface 30 may be an interface of a communication module, and is used to connect to other devices or systems.

[0152] Of course, it should be noted that Fig. 9 The structure shown does not constitute a limitation on the image stitching device in the embodiment of the present invention. In practical applications, the image stitching device may include Fig. 9 More or fewer components than shown, or combinations of certain components.

[0153] The computer-readable storage medium provided by the embodiment of the present invention is introduced below. The computer-readable storage medium described below and the image stitching method described above can be referred to each other.

[0154] The present invention also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned image stitching method are implemented.

[0155] The computer-readable storage medium may include: a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and other media that can store program codes.

[0156] The computer program provided by the embodiment of the present invention is introduced below. The computer program described below and the image stitching method described above can be referred to each other.

[0157] The present invention also provides a computer program, including a computer program / instruction, which implements the steps of the above-mentioned image stitching method when executed by a processor.

[0158] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0159] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0160] Finally, it should be noted that, in this article, relationships such as first and second, etc. are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprises" or any other variations are intended to cover non-exclusive inclusion, so that a process, method, article or device that includes a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.

[0161] The above is a detailed introduction to an image stitching method, device, equipment and computer-readable storage medium provided by the present invention. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. An image stitching method, characterized in that: include: The semantic segmentation method is used to segment the whole spliced ​​image to obtain the semantic segmentation result; Determine a target mask area based on the semantic segmentation result and the segmentation category; Performing object enlargement processing on the target mask area to obtain a binary mask enlarged image; Performing fuzzy processing on the binary mask enlarged image to obtain a target mask image; A coverage map corresponding to the segmentation category is generated, and image stitching processing is performed using the target mask image, the coverage map and the overall image to be stitched to obtain a target stitched image.

2. The image stitching method according to claim 1, characterized in that: The determining of the target mask area based on the semantic segmentation result and the segmentation category includes: Determine the segmentation category according to actual splicing requirements; Binarization is performed on the pixels in the semantic segmentation result based on the segmentation category to obtain the target mask area.

3. The image stitching method according to claim 2, characterized in that: The whole image to be spliced ​​is a virtual human image to be spliced, and the segmentation category is determined according to actual splicing requirements, including: When the actual splicing requirement is to splice the generated physiological parts, the segmentation category is determined to be the physiological part.

4. The image stitching method according to any one of claims 1 to 3, characterized in that: Performing object enlargement processing on the target mask area to obtain a binary mask enlarged image, including: Define a structure element; wherein the structure element is a matrix composed of 0 and 1; Performing pixel comparison processing on the structural element and the overall image to be spliced ​​including the target mask area to obtain a comparison result; The pixel values ​​of the whole image to be spliced ​​are adjusted based on the comparison result to obtain the binary mask enlarged image.

5. The image stitching method according to claim 1, characterized in that: The step of performing blur processing on the binary mask enlarged image to obtain a target mask image includes: Performing Gaussian blur filtering on the binary mask enlarged image to obtain a blurred mask; The fuzzy mask is normalized to obtain the target mask image.

6. The image stitching method according to claim 1, characterized in that: After the binary mask enlarged image is fuzzy processed to obtain the target mask image, the method further includes: Performing global color adjustment on the target mask image by using histogram equalization to obtain a target adjusted image; Performing noise reduction processing on the target adjustment image to obtain a final target mask image; Correspondingly, the generating of the coverage map corresponding to the segmentation category and performing image stitching processing using the target mask image, the coverage map and the overall image to be stitched to obtain a target stitched image includes: The final target mask image, the coverage image and the overall image to be stitched are used to perform image stitching processing to obtain the target stitching image.

7. The image stitching method according to claim 1, characterized in that: The above image stitching methods are applied to videos, including: Extracting video frames from the video to obtain multiple video frames; wherein the overall image to be spliced ​​includes the video frames; Processing each of the video frames frame by frame using the image stitching method to obtain a target stitching image corresponding to each of the video frames; According to the timestamp of the target stitching image corresponding to each of the video frames, all the target stitching images are stitched together to obtain a target stitching video frame; The target spliced ​​video frames are encoded into a video format to obtain a target spliced ​​video.

8. The image stitching method according to claim 1, characterized in that: Generating a coverage map corresponding to the segmentation category, and performing image stitching processing using the target mask image, the coverage map and the overall image to be stitched to obtain a target stitched image, including: Multiplying the whole image to be stitched and the non-masked area pixel by pixel to determine an image of the non-masked area; Multiplying the coverage map and the target mask image pixel by pixel to determine a mask area image; The target stitching image is determined by adding the image of the unmasked area and the image of the masked area pixel by pixel.

9. An image stitching device, characterized in that: include: A semantic segmentation module, used to segment the whole image to be spliced ​​using a semantic segmentation method to obtain a semantic segmentation result; A mask determination module, used to determine a target mask area based on the semantic segmentation result and the segmentation category; A mask expansion module, used for performing object expansion processing on the target mask area to obtain a binary mask expansion image; A fuzzy processing module, used for performing fuzzy processing on the binary mask enlarged image to obtain a target mask image; The stitching module is used to generate a coverage map corresponding to the segmentation category, and perform image stitching processing using the target mask image, the coverage map and the overall image to be stitched to obtain a target stitched image.

10. An electronic device, characterized in that: include: Memory for storing computer programs; A processor, configured to implement the steps of the image stitching method according to any one of claims 1 to 8 when executing the computer program.

11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the image stitching method according to any one of claims 1 to 8 are implemented.

12. A computer program, characterized in that The method comprises a computer program / instruction, wherein when the computer program / instruction is executed by a processor, the steps of the image stitching method according to any one of claims 1 to 8 are implemented.

Images