A virtual-real fusion image matting method and device, electronic equipment and storage medium

Abstract

The application discloses a virtual-real fusion image matting method and device, electronic equipment and storage medium. The method comprises the following steps: collecting an ultra-long-focus lens image of a current scene, and extracting pixel features of a specified part region of an object subject and a green screen region on the ultra-long-focus lens image to obtain a part standard feature set and a green screen standard feature; collecting a to-be-processed lens image of the current scene, and mapping coordinates of each pixel of the part standard feature set to a coordinate system of the to-be-processed lens image to obtain a part feature mapping region; performing image matting on the to-be-processed lens image to obtain an initial mask; matching the pixel features in the part feature mapping region with the part standard feature set, correcting the gray values of the pixels in the initial mask, and obtaining an optimized mask; matching the pixel features of the green screen region of the to-be-processed lens image with the green screen standard feature, performing color compensation on the green screen region of the to-be-processed lens image, and obtaining an optimized lens image; and performing image matting on the optimized lens image based on the optimized mask to obtain a final image matting image.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a method and apparatus for virtual-real fusion keying, electronic equipment, and storage medium. Background Technology

[0002] When creating content that blends reality and virtuality, it is usually necessary to first shoot virtual scenes against a green screen. This involves filming real objects such as people against a green screen in a real-world setting. The filmed images are then chroma keyed to obtain images containing only the subjects being filmed. These images are then blended with the virtual scene to create the content that blends reality and virtuality.

[0003] Currently, chroma keying techniques are commonly used for images shot in green screen environments. These techniques are based on the significant color differences between the foreground and background. However, real-world green screen environments can be affected by factors such as wear and tear from prolonged use, uneven paint application, or lighting conditions. This can lead to inaccuracies when using individual green areas for keying, resulting in some green areas not being removed.

[0004] To completely remove the green, the current practice is to increase the area covered by the keying process. However, while this ensures complete removal of the green, it can also lead to the removal of some detailed parts of the subject, such as individual strands of hair, resulting in a loss of detail and affecting the quality of the keying. Summary of the Invention

[0005] In view of the shortcomings of the prior art, this application provides a virtual-real fusion keying method and apparatus, electronic device and storage medium to solve the problem that the prior art cannot perform accurate keying.

[0006] To achieve the above objectives, this application provides the following technical solution:

[0007] The first aspect of this application provides a keying method for virtual-real fusion, including:

[0008] Capture super telephoto lens images of the current scene;

[0009] Extract the features of pixels in a specified area of ​​the main object in the super telephoto lens image and the features of pixels in the green screen area to obtain a set of standard features for the part and standard features for the green screen.

[0010] Acquire the lens image to be processed in the current scene, and map the coordinates of each pixel in the standard feature set of the part to the coordinate system of the lens image to be processed to obtain the part feature mapping region;

[0011] The image of the lens to be processed is keyed to obtain an initial mask;

[0012] By matching the features of pixels in the feature mapping region of the image to be processed with the standard feature set of the region, the grayscale values ​​of each pixel in the initial mask are corrected to obtain an optimized mask.

[0013] By matching the features of pixels in the green screen region of the image to be processed with the standard features of the green screen, color compensation is performed on the green screen region of the image to be processed to obtain an optimized image.

[0014] Based on the optimized mask, the optimized lens image is keyed to obtain the final keyed image.

[0015] Optionally, in the above-described keying method for virtual-real fusion, after acquiring the super telephoto lens image of the current scene, the method further includes:

[0016] The super telephoto lens image is subjected to bilateral filtering processing within a specified window, and the color values ​​of each pixel in the filtered super telephoto lens image are normalized.

[0017] Optionally, in the above-described keying method for virtual-real fusion, the step of extracting the pixel features of a specified region of the subject on the super telephoto lens image and the pixel features of the green screen region to obtain a set of standard features for the subject and standard features for the green screen includes:

[0018] The main object in the super telephoto lens image is keyed out to obtain a mask of the main object;

[0019] The specified area in the main mask of the object is located by edge detection, and the specified area is obtained.

[0020] The pixels in the specified region of the super telephoto lens image are sampled, and the RGB values, HSV values, and local texture features of each sampled pixel are extracted to form a standard feature set of the region.

[0021] Based on the object's main mask, the pixels of the super telephoto lens image within the green screen area are sampled, and the mean and variance of the RGB values ​​and the mean and variance of the HSV values ​​of each sampled pixel are calculated as standard features of the green screen.

[0022] Optionally, in the above-described keying method for virtual-real fusion, before acquiring the lens image to be processed in the current scene, the method further includes:

[0023] Switch the acquisition device to the target lens and set the same parameters as when capturing the super telephoto lens image;

[0024] Adjust the acquisition device so that the shooting center of the target lens coincides with the coordinate system center of the ultra-long cross-section lens;

[0025] The pre-stored distortion correction parameters are loaded to perform electronic compensation on the imaging optical path of the acquisition device.

[0026] Optionally, in the above-described method for merging virtual and real images, the step of mapping the coordinates of each pixel in the standard feature set of the part to the coordinate system of the lens image to be processed, to obtain the part feature mapping region, includes:

[0027] Obtain the relative coordinates of each pixel in the standard feature set of the part; wherein, the relative coordinates of the pixels in the standard feature set of the part are the coordinates relative to the specified origin on the super telephoto lens image;

[0028] Based on the coordinate system mapping relationship between the pre-calibrated target lens and the super telephoto lens, the relative coordinates of each pixel in the standard feature set of the part are mapped to the corresponding coordinates in the lens image to be processed, thereby obtaining the feature mapping region of the part.

[0029] Optionally, in the above-described keying method for virtual-real fusion, the step of matching the features of pixels in the feature mapping region of the image to be processed with the standard feature set of the region to correct the grayscale values ​​of each pixel in the initial mask, thereby obtaining an optimized mask, includes:

[0030] Each pixel in the feature mapping region of the lens image to be processed is traversed and used as the target pixel;

[0031] Extract the features of the target pixel and match the features of the target pixel with the standard feature set of the part to obtain the matching degree corresponding to the target pixel;

[0032] If the matching degree corresponding to the target pixel is greater than the matching degree threshold, the gray value of the target pixel in the initial mask is corrected from the minimum value to the maximum value to obtain the optimized mask.

[0033] Optionally, in the above-described keying method for virtual-real fusion, the step of matching the features of pixels in the green screen region of the image to be processed with the standard green screen features to perform color compensation on the green screen region of the image to be processed, thereby obtaining an optimized image, includes:

[0034] Extract the features of each pixel in the green screen region of the image to be processed;

[0035] Calculate the mean and variance of the features of all pixels in the green screen region of the image to be processed to obtain the current green screen features;

[0036] Compare the current green screen features with the standard green screen features;

[0037] If the deviation between the current green screen feature and the green screen standard feature is greater than a preset deviation threshold, then the color value of each pixel in each channel of the green screen area of ​​the image to be processed is compensated according to the color compensation amount of each channel; wherein, the color compensation amount of one channel is the deviation of the average color value of each pixel in the green screen area of ​​the image to be processed and the super telephoto lens image in that channel.

[0038] A second aspect of this application provides a keying device for virtual-real fusion, comprising:

[0039] The first image acquisition unit is used to acquire super telephoto lens images of the current scene;

[0040] A standard feature extraction unit is used to extract the features of pixels in a specified part of the main body of the object on the super telephoto lens image and the features of pixels in the green screen area, to obtain a set of standard features for the part and standard features for the green screen.

[0041] The second image acquisition unit is used to acquire the lens image to be processed in the current scene;

[0042] The region mapping unit is used to map the coordinates of each pixel in the standard feature set of the part to the coordinate system of the lens image to be processed, so as to obtain the part feature mapping region.

[0043] The initial keying unit is used to key the lens image to be processed to obtain an initial mask;

[0044] The mask optimization unit is used to modify the grayscale value of each pixel in the initial mask by matching the features of the pixels in the feature mapping area of ​​the lens image to be processed with the standard feature set of the part, so as to obtain an optimized mask.

[0045] The color compensation unit is used to perform color compensation on the green screen area of ​​the lens image to be processed by matching the features of the pixels in the green screen area of ​​the lens image to be processed with the green screen standard features, so as to obtain an optimized lens image.

[0046] The final keying unit is used to key the optimized lens image based on the optimized mask to obtain the final keyed image.

[0047] Optionally, the aforementioned keying device for virtual-real fusion further includes:

[0048] The preprocessing unit is used to perform bilateral filtering on the super telephoto lens image within a specified window, and to normalize the color values ​​of each pixel in the filtered super telephoto lens image.

[0049] Optionally, in the above-described virtual-real fusion keying device, the standard feature extraction unit includes:

[0050] The mask generation unit is used to perform image matting on the main object in the super telephoto lens image to obtain a mask of the main object.

[0051] The detection unit is used to locate a specified area in the main body mask of the object through edge detection, and obtain the specified area.

[0052] The part standard feature extraction unit is used to sample pixels in the specified part area of ​​the super telephoto lens image, and extract the RGB value, HSV value and local texture features of each sampled pixel to form a part standard feature set.

[0053] The green screen standard feature extraction unit is used to sample the pixels of the super telephoto lens image within the green screen area based on the object subject mask, and calculate the mean and variance of the RGB values ​​and the mean and variance of the HSV values ​​of each sampled pixel as green screen standard features.

[0054] Optionally, the aforementioned keying device for virtual-real fusion further includes:

[0055] The setting unit is used to switch the acquisition device to the target lens and set the same parameters as when capturing the super telephoto lens image.

[0056] An adjustment unit is used to adjust the acquisition device so that the shooting center of the target lens coincides with the coordinate system center of the ultra-long cross-section lens;

[0057] The correction unit is used to load pre-stored distortion correction parameters to electronically compensate the imaging optical path of the acquisition device.

[0058] Optionally, in the above-described keying device for virtual-real fusion, the region mapping unit includes:

[0059] A coordinate acquisition unit is used to acquire the relative coordinates of each pixel in the standard feature set of the part; wherein, the relative coordinates of the pixels in the standard feature set of the part are the coordinates relative to a specified origin on the super telephoto lens image;

[0060] The coordinate mapping unit is used to map the relative coordinates of each pixel in the standard feature set of the part to the corresponding coordinates in the lens image to be processed, based on the coordinate system mapping relationship between the pre-calibrated target lens and the super telephoto lens, so as to obtain the feature mapping region of the part.

[0061] Optionally, in the above-described keying device for blending virtual and real images, the mask optimization unit includes:

[0062] A traversal unit is used to traverse each pixel in the feature mapping region of the lens image to be processed, as the target pixel;

[0063] A matching degree calculation unit is used to extract the features of the target pixel and match the features of the target pixel with the standard feature set of the part to obtain the matching degree corresponding to the target pixel;

[0064] The grayscale correction unit is used to correct the grayscale value of the target pixel in the initial mask from the minimum value to the maximum value when the matching degree corresponding to the target pixel is greater than the matching degree threshold, so as to obtain an optimized mask.

[0065] Optionally, in the above-described keying device for blending virtual and real images, the color compensation unit includes:

[0066] A green screen feature extraction unit is used to extract the features of each pixel in the green screen area of ​​the image to be processed.

[0067] The feature calculation unit is used to calculate the mean and variance of the features of all pixels in the green screen area of ​​the image to be processed, so as to obtain the current green screen features.

[0068] A feature comparison unit is used to compare the current green screen feature with the green screen standard feature;

[0069] A pixel color compensation unit is used to compensate the color values ​​of each pixel in each channel of the green screen area of ​​the lens image to be processed according to the color compensation amount of each channel when the deviation between the current green screen feature and the green screen standard feature is greater than a preset deviation threshold; wherein, the color compensation amount of one channel is the deviation of the average color value of each pixel in the green screen area of ​​the lens image to be processed and the super telephoto lens image in that channel.

[0070] A third aspect of this application provides an electronic device, comprising:

[0071] Memory and processor;

[0072] The memory is used to store programs;

[0073] The processor is used to execute the program, which, when executed, is specifically used to implement the keying method for virtual-real fusion as described in any of the above.

[0074] The fourth aspect of this application provides a computer storage medium for storing a computer program, which, when executed by a processor, is used to implement the virtual-real fusion keying method as described in any of the preceding claims.

[0075] This application provides a virtual-real fusion keying method. It acquires a super-telephoto lens image of the current scene and extracts pixel features from a specified area of ​​the main subject and the green screen area, obtaining a standard feature set for the subject and standard features for the green screen. This allows for accurate capture of specified details and green screen features using the super-telephoto lens image. Next, it acquires the image of the scene to be processed and maps the coordinates of each pixel in the standard feature set to the coordinate system of the image to be processed, obtaining a feature mapping region. Then, it performs keying on the image to be processed to obtain an initial mask. Finally, by matching the features of pixels in the feature mapping region of the image to be processed with the standard feature set, it corrects the grayscale values ​​of each pixel in the initial mask, obtaining an optimized mask. This allows for the correction of pixels with initial keying errors through standard detail feature matching, effectively preserving object details. Similarly, by matching the pixel features of the green screen area in the image to be processed with standard green screen features, color compensation is applied to the green screen area of ​​the image to be processed, resulting in an optimized image. This color compensation of the green screen area allows for more accurate identification of green screen pixels, ensuring the effective removal of all green screen portions. Therefore, finally, keying is performed on the optimized image based on the optimized mask to obtain the final keyed image, which effectively removes the entire green screen while preserving the details of the subject, achieving precise keying. Attached Figure Description

[0076] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0077] Figure 1 A flowchart illustrating a virtual-real fusion keying method provided in an embodiment of this application;

[0078] Figure 2 A flowchart illustrating a method for extracting a set of standard features for body parts and standard features for green screens, provided in an embodiment of this application;

[0079] Figure 3 A flowchart illustrating a lens switching and parameter synchronization method provided in an embodiment of this application;

[0080] Figure 4 A flowchart illustrating a method for mapping a set of standard features of a part, provided in an embodiment of this application;

[0081] Figure 5A flowchart illustrating a method for optimizing an initial mask, as provided in an embodiment of this application;

[0082] Figure 6 A flowchart illustrating a method for color compensation in a green screen area, provided as an embodiment of this application;

[0083] Figure 7 A schematic diagram of the architecture of a virtual-real fusion keying device provided in an embodiment of this application;

[0084] Figure 8 This is a schematic diagram of the architecture of an electronic device provided in an embodiment of this application. Detailed Implementation

[0085] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0086] In this application, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0087] This application provides a keying method for virtual-real fusion, such as... Figure 1 As shown, it includes the following steps:

[0088] S101. Acquire super telephoto lens image of the current scene.

[0089] The current scene refers to the location under the green screen where the scene needs to be filmed. The super telephoto lens image is an image taken using a super telephoto lens.

[0090] It should be noted that super telephoto lenses can capture high-resolution images, thus clearly capturing fine details such as individual hair strands. Furthermore, the narrow field of view can effectively avoid areas of color abrupt change at the edges of green screens. Therefore, for a scene that needs to be captured, it is best to first use a super telephoto lens to obtain a super telephoto image of the current scene, so that accurate features of each part can be extracted from it.

[0091] Optionally, the subject can be placed in front of a uniformly lit green screen, and the camera can be switched to a super telephoto lens (i.e., a close-up shot) to focus on the main subject, ensuring that the captured image only includes the subject and a small amount of uniform green screen background, specifically, the green screen ratio can be ≤10%, thus obtaining a super telephoto image. In other words, the super telephoto image is only an image of a portion of the current scene.

[0092] Optionally, in another embodiment of this application, after performing step S101, image preprocessing is further performed, specifically including:

[0093] Perform bilateral filtering on the super telephoto lens image within a specified window, and normalize the color values ​​of each pixel in the filtered super telephoto lens image.

[0094] Specifically, bilateral filtering is performed on the super telephoto lens image within a specified window, such as a 5×5 window, to preserve edge details while reducing noise. Then, the RGB values ​​of each pixel in the filtered super telephoto lens image are normalized to the range of 0-1, thereby eliminating color differences caused by different lighting intensities and establishing a unified benchmark for subsequent feature comparison.

[0095] S102. Extract the pixel features of a specified part of the subject in the super telephoto lens image and the pixel features of the green screen area to obtain the part standard feature set and the green screen standard feature.

[0096] Among them, the designated area of ​​the main body of the object is the area that needs to be finely chroma keyed, such as the hair area.

[0097] Because super telephoto lens images can clearly capture fine details such as hair strands, and the narrow field of view can effectively avoid color abrupt changes at the edges of green screens, the features of pixels in a specified area and the features of pixels in the green screen area can be accurately extracted from super telephoto lens images. These features can then be used as standard features to optimize the specified area and the green screen area in normally acquired lens images, enabling precise differentiation between the two areas and thus accurate keying.

[0098] Optionally, in another embodiment of this application, one specific implementation of step S102 is as follows: Figure 2 As shown, it includes:

[0099] S201. Perform keying on the main subject of the super telephoto lens image to obtain the main subject mask.

[0100] To determine the green screen area and specific regions in the super telephoto lens image, the main subject in the image is first keyed out to obtain a subject mask. This keying can be accurately performed using conventional chroma keying techniques. The black areas with a grayscale value of 255 on the subject mask represent the subject area, while the white areas with a grayscale value of 0 represent the green screen area.

[0101] S202. Locate the specified area in the mask of the main body of the object through edge detection to obtain the specified area.

[0102] Specifically, Canny edge detection is used to identify the edges of the object on the object's mask based on an edge threshold. Then, the edges of specified parts within these parts are determined, thus obtaining the region of those specified parts. Optionally, the edge threshold can be between 50 and 150, thus balancing the completeness of edge detection with its robustness against interference.

[0103] S203. Sample pixels in a specified region of the super telephoto lens image, and extract the RGB values, HSV values, and local texture features of each sampled pixel to form a standard feature set for the region.

[0104] Pixels within a specified region of the super-telephoto lens image are randomly sampled. The number of samples can be determined based on statistical significance to ensure the representativeness of the feature set; for example, 1000-2000 pixels are randomly sampled. Then, the RGB value, HSV value (hue H: 0-360°, saturation S: 0-1, brightness V: 0-1), and local texture features (LBP operator, radius 1px, neighborhood 8 points, adapted for fine edge texture extraction) of each pixel are extracted to form a standard feature set for the region.

[0105] S204. Based on the object's main mask, sample the pixels of the super telephoto lens image within the green screen area, and calculate the mean and variance of the RGB values ​​and the mean and variance of the HSV values ​​of each sampled pixel as standard features of the green screen.

[0106] The green screen area of ​​the super telephoto lens image is determined based on the object's main mask. Multiple pixels are randomly sampled from the green screen area of ​​the super telephoto lens image, and the RGB and HSV values ​​of each pixel are extracted. Then, the mean of the RGB values ​​and the mean of the HSV values ​​of each pixel are calculated.

[0107] S103. Acquire the image of the current scene to be processed.

[0108] The image to be processed is the image captured using the target lens. The target lens is selected based on the actual shooting requirements of the current scene, and is typically a wider lens than a super telephoto lens. In other words, the image to be processed is the entire current scene that actually needs to be captured, so it needs to be processed to achieve image keying. Therefore, after capturing the current scene using a super telephoto lens and acquiring standard features, normal shooting of the current scene can then be performed to obtain the image to be processed.

[0109] Optionally, in order to ensure that the format of the lens image to be processed is consistent with that of the super telephoto lens image, the same preprocessing method can be used to process the lens image to be processed after it is acquired.

[0110] To acquire the lens image to be processed and ensure that features acquired from the super telephoto lens image can be used to optimize the lens image, lens switching and parameter synchronization are performed before acquiring the lens image to be processed. Optionally, in another embodiment of this application, as... Figure 3 As shown in the embodiment of this application, a lens switching and parameter synchronization method includes:

[0111] S301. Switch the acquisition device to the target lens and set the same parameters as when capturing images with a super telephoto lens.

[0112] Optionally, the super telephoto lens can be replaced with the target lens via a mechanical switching module or manual operation, while triggering the parameter locking module to retain the previous white balance and exposure parameters, ensuring color consistency in dual-lens shooting, thus setting the same parameters as when shooting images with the super telephoto lens.

[0113] S302. Adjust the acquisition device so that the shooting center of the target lens coincides with the coordinate system center of the ultra-long cross-section lens.

[0114] Specifically, based on the pre-stored coordinate mapping relationship between the target lens and the super telephoto lens, the camera automatically adjusts the shooting angle to ensure that the shooting center of the target lens coincides with the center of the subject captured by the super telephoto lens, thus guaranteeing an error of ≤1 pixel. Optionally, the pre-stored coordinate mapping relationship between the two lenses can be obtained through extrinsic parameter calibration using a calibration board.

[0115] S303. Load the pre-stored distortion correction parameters to perform electronic compensation on the imaging optical path of the acquisition device.

[0116] Specifically, before the target lens captures the image, pre-stored distortion correction parameters are automatically loaded to electronically compensate the imaging optical path, reducing the impact of edge color distortion on keying. Optionally, the distortion correction parameters can be based on a lens distortion model, and the radial distortion coefficients k1 and k2 and the tangential distortion coefficients p1 and p2 can be obtained through calibration experiments.

[0117] S104. Map the coordinates of each pixel in the standard feature set of the part to the coordinate system of the lens image to be processed to obtain the part feature mapping region.

[0118] In order to determine the region corresponding to the standard feature set of a part in the image to be processed, the coordinates of each pixel of the standard feature set of the part are determined in advance. Then, the coordinates of each pixel of the standard feature set of the part are mapped to the coordinate system of the target lens, that is, mapped to the coordinate system of the image to be processed, so as to obtain the coordinates of each pixel in the coordinate system of the target lens, and then determine the coordinates of these pixels to determine the part feature mapping region.

[0119] Optionally, in another embodiment of this application, one specific implementation of step S104 is as follows: Figure 4 As shown, it includes:

[0120] S401. Obtain the relative coordinates of each pixel in the standard feature set of the part.

[0121] In this system, the relative coordinates of pixels in the body part standard feature set are the coordinates relative to a specified origin on the super telephoto lens image. For example, a local coordinate system is established with the center of the person's eyes as the origin, and the relative coordinates (X, Y) of each pixel in the body part standard feature set are recorded. When mapping is required, the relative coordinates of each pixel in the pre-stored body part standard feature set are directly obtained.

[0122] S402. Based on the coordinate system mapping relationship between the pre-calibrated target lens and the super telephoto lens, the relative coordinates of each pixel in the part standard feature set are mapped to the corresponding coordinates in the lens image to be processed, thus obtaining the part feature mapping region.

[0123] Specifically, based on the focal length of the super telephoto lens and the focal length of the target lens, the focal length ratio of the two lenses is determined, and the coordinate system mapping relationship between the two lenses is determined based on the focal length ratio. Thus, based on the coordinate system mapping relationship of the two lenses, the relative coordinates of each pixel in the standard feature set of the part can be mapped to the corresponding coordinates in the image of the lens to be processed, thereby obtaining the part feature mapping region.

[0124] S105. Perform keying on the image to be processed to obtain the initial mask.

[0125] Specifically, a chroma key algorithm, similar to that used for keying images from super telephoto lenses, can be employed to key the image of the lens to be processed, generate an initial mask, and achieve preliminary separation of the green screen area.

[0126] S106. By matching the features of pixels in the part feature mapping area of ​​the lens image to be processed with the part standard feature set, the gray values ​​of each pixel in the initial mask are corrected to obtain an optimized mask.

[0127] Since the feature set of the part is matched with the feature of the pixel of the specified part, by matching the feature of the pixel in the part feature mapping area of ​​the lens image to be processed with the feature set of the part, the pixels in the part feature mapping area that do not belong to the specified part can be identified. Thus, the gray value of the pixel on the initial mask can be corrected, that is, the gray value belonging to the specified part is corrected to the gray value belonging to the green screen, thereby optimizing the initial mask.

[0128] Optionally, in another embodiment of this application, one specific implementation of step S106 is as follows: Figure 5 As shown, it includes:

[0129] S501. Traverse each pixel in the part feature mapping region of the lens image to be processed, and use it as the target pixel.

[0130] S502. Extract the features of the target pixel and match the features of the target pixel with the standard feature set of the part to obtain the matching degree corresponding to the target pixel.

[0131] It should be noted that the dimensionality of the extracted target pixel features is consistent with the dimensionality of the extracted location standard feature set. Therefore, specifically, the RGB values, HSV values, and LBP texture features of the target pixel can be extracted. This allows the target pixel features to be matched with the location standard feature set.

[0132] Optionally, the Euclidean distance algorithm can be used to perform similarity matching between the features of the target pixel and the standard feature set of the part, thereby obtaining the matching degree corresponding to the target pixel. Specifically, this can involve matching with the mean of each dimension of the standard feature set of the part, or matching with the features of pixels at the same position in the standard feature set of the part, etc.

[0133] S503. Determine whether the matching degree corresponding to the target pixel is greater than the matching degree threshold.

[0134] If the matching degree of the target pixel is greater than the matching degree threshold, it means that the target pixel is a pixel in the specified part. Therefore, if its gray value is the minimum value, it is initially judged to be a green screen, so step S504 is executed at this time.

[0135] S504. Correct the grayscale value of the target pixel in the initial mask from the minimum value to the maximum value to obtain the optimized mask.

[0136] This means correcting the grayscale value of the target pixel in the initial mask from 0 (originally determined to be a green screen) to 255 (determined to be a specified area), thereby achieving precise preservation of details in the specified area.

[0137] S107. By matching the features of the pixels in the green screen area of ​​the image to be processed with the standard features of the green screen, color compensation is performed on the green screen area of ​​the image to be processed to obtain an optimized image.

[0138] Since the green screen standard features contain the standard features of the green screen in the current scene, by matching the features of the pixels in the green screen area of ​​the image to be processed with the green screen standard features, color compensation is performed on the green screen area of ​​the image to be processed, so that the color of the green screen in the image to be processed approaches the standard value. This allows for more accurate identification of the pixels in the green screen area, ensuring that the green screen is completely removed.

[0139] Optionally, in another embodiment of this application, one specific implementation of step S107 is as follows: Figure 6 As shown, it includes:

[0140] S601. Extract the features of each pixel in the green screen area of ​​the image to be processed.

[0141] It should be noted that the features of each pixel in the green screen region of the extracted image of the shot to be processed are consistent with the feature dimensions used to determine the standard features of the green screen. Therefore, specifically, the RGB and HSV values ​​of the green screen region of the image of the shot to be processed are extracted.

[0142] S602. Calculate the mean and variance of the features of all pixels in the green screen area of ​​the image to be processed to obtain the current green screen features.

[0143] S603. Compare the current green screen features with the standard green screen features.

[0144] Specifically, the current green screen features are compared with the same dimension features in the green screen standard features.

[0145] S604. Determine whether the deviation between the current green screen features and the standard green screen features is greater than the preset deviation threshold.

[0146] If the deviation between the current green screen feature and any dimension of the green screen standard feature is greater than the preset deviation threshold, then step S605 is executed.

[0147] S605. Compensate the color values ​​of each pixel in each channel of the green screen area of ​​the image to be processed according to the color compensation amount of each channel.

[0148] The color compensation amount of one channel is the deviation of the average color value of each pixel in the green screen area of ​​the lens image to be processed and the super telephoto lens image.

[0149] In other words, the formula for compensating each pixel in the green screen area of ​​the image being processed is:

[0150] R_comp=R_wide-(Mean_green_wide.R-Mean_green.R).

[0151] G_comp=G_wide-(Mean_green_wide.G-Mean_green.G).

[0152] B_comp=B_wide-(Mean_green_wide.B-Mean_green.B).

[0153] Where R_wide, G_wide, and B_wide are the color values ​​of each pixel in the green screen region of the image to be processed before compensation, in the three channels. Mean_green_wide.R, Mean_green_wide.G, and Mean_green_wide.B are the average color values ​​of each pixel in the green screen region of the super telephoto lens image. Mean_green.R, Mean_green.G, and Mean_green.B are the average color values ​​of each pixel in the green screen region of the image to be processed.

[0154] S108. Based on the optimized mask, perform keying on the optimized lens image to obtain the final keyed image.

[0155] Specifically, the optimized mask and the optimized lens image are overlaid using the Alpha blending mode to remove the green screen background and output a final keyed image that retains details.

[0156] This application provides a virtual-real fusion keying method. It acquires a super-telephoto lens image of the current scene and extracts pixel features from a specified area of ​​the main subject and the green screen area, obtaining a standard feature set for the subject and standard features for the green screen. This allows for accurate capture of specified details and green screen features using the super-telephoto lens image. Next, it acquires the image of the scene to be processed and maps the coordinates of each pixel in the standard feature set to the coordinate system of the image to be processed, obtaining a feature mapping region. Then, it performs keying on the image to be processed to obtain an initial mask. Finally, by matching the features of pixels in the feature mapping region of the image to be processed with the standard feature set, it corrects the grayscale values ​​of each pixel in the initial mask, obtaining an optimized mask. This allows for the correction of pixels with initial keying errors through standard detail feature matching, effectively preserving object details. Similarly, by matching the pixel features of the green screen area in the image to be processed with standard green screen features, color compensation is applied to the green screen area of ​​the image to be processed, resulting in an optimized image. This color compensation of the green screen area allows for more accurate identification of green screen pixels, ensuring the effective removal of all green screen portions. Therefore, finally, keying is performed on the optimized image based on the optimized mask to obtain the final keyed image, which effectively removes the entire green screen while preserving the details of the subject, achieving precise keying.

[0157] Another embodiment of this application provides a keying device for virtual-real fusion, such as... Figure 7 As shown, it includes:

[0158] The first image acquisition unit 701 is used to acquire super telephoto lens images of the current scene.

[0159] The standard feature extraction unit 702 is used to extract the features of pixels in a specified part of the subject in the super telephoto lens image and the features of pixels in the green screen area, to obtain the part standard feature set and the green screen standard features.

[0160] The second image acquisition unit 703 is used to acquire the lens image to be processed in the current scene.

[0161] The region mapping unit 704 is used to map the coordinates of each pixel in the part standard feature set to the coordinate system of the lens image to be processed, so as to obtain the part feature mapping region.

[0162] The initial keying unit 705 is used to key the image of the lens to be processed to obtain an initial mask.

[0163] The mask optimization unit 706 is used to modify the grayscale values ​​of each pixel in the initial mask by matching the features of the pixels in the part feature mapping area of ​​the lens image to be processed with the part standard feature set, so as to obtain an optimized mask.

[0164] The color compensation unit 707 is used to perform color compensation on the green screen area of ​​the image to be processed by matching the features of the pixels in the green screen area of ​​the image to be processed with the standard features of the green screen, thereby obtaining an optimized image.

[0165] The final keying unit 708 is used to key the optimized lens image based on the optimized mask to obtain the final keyed image.

[0166] Optionally, in another embodiment of the image keying device for virtual-real fusion provided in this application, the device further includes:

[0167] The preprocessing unit is used to perform bilateral filtering on the super telephoto lens image within a specified window, and to normalize the color values ​​of each pixel in the filtered super telephoto lens image.

[0168] Optionally, in another embodiment of the virtual-real fusion keying device provided in this application, the standard feature extraction unit includes:

[0169] The mask generation unit is used to key out the main subject of the object in the super telephoto lens image and obtain the mask of the main subject.

[0170] The detection unit is used to locate a specified area in the mask of the main body of the object through edge detection, and obtain the specified area.

[0171] The part standard feature extraction unit is used to sample pixels in a specified part area of ​​an ultra-telephoto lens image and extract the RGB values, HSV values, and local texture features of each sampled pixel to form a part standard feature set.

[0172] The green screen standard feature extraction unit is used to sample pixels in the green screen area of ​​the super telephoto lens image based on the object subject mask, and calculate the mean and variance of the RGB values ​​and the mean and variance of the HSV values ​​of each sampled pixel as green screen standard features.

[0173] Optionally, in another embodiment of the image keying device for virtual-real fusion provided in this application, the device further includes:

[0174] The setting unit is used to switch the acquisition device to the target lens and set the same parameters as when shooting images with a super telephoto lens.

[0175] The adjustment unit is used to align the acquisition device with the center of the target lens's shooting position and the center of the coordinate system of the ultra-long cross-section lens.

[0176] The correction unit is used to load pre-stored distortion correction parameters to electronically compensate the imaging optical path of the acquisition device.

[0177] Optionally, in another embodiment of the virtual-real fusion keying device provided in this application, the region mapping unit includes:

[0178] The coordinate acquisition unit is used to acquire the relative coordinates of each pixel in the part standard feature set. The relative coordinates of the pixels in the part standard feature set are the coordinates relative to a specified origin on the super telephoto lens image.

[0179] The coordinate mapping unit is used to map the relative coordinates of each pixel in the part standard feature set to the corresponding coordinates in the lens image to be processed, based on the coordinate system mapping relationship between the pre-calibrated target lens and the super telephoto lens, so as to obtain the part feature mapping region.

[0180] Optionally, in another embodiment of the keying device for virtual-real fusion provided in this application, the mask optimization unit includes:

[0181] The traversal unit is used to traverse each pixel in the part feature mapping region of the lens image to be processed, as the target pixel.

[0182] The matching degree calculation unit is used to extract the features of the target pixel and match the features of the target pixel with the standard feature set of the part to obtain the matching degree of the target pixel.

[0183] The grayscale correction unit is used to correct the grayscale value of the target pixel in the initial mask from the minimum value to the maximum value when the matching degree of the target pixel is greater than the matching degree threshold, so as to obtain the optimized mask.

[0184] Optionally, in another embodiment of the keying device for virtual-real fusion provided in this application, the color compensation unit includes:

[0185] The green screen feature extraction unit is used to extract the features of each pixel in the green screen area of ​​the image to be processed.

[0186] The feature calculation unit is used to calculate the mean and variance of the features of all pixels in the green screen area of ​​the image to be processed, so as to obtain the current green screen features.

[0187] The feature comparison unit is used to compare the current green screen features with the standard green screen features.

[0188] The pixel color compensation unit is used to compensate the color values ​​of each pixel in each channel of the green screen area of ​​the image to be processed, according to the color compensation amount of each channel, when the deviation between the current green screen feature and the green screen standard feature exceeds a preset deviation threshold. The color compensation amount for one channel is the deviation of the average color value of each pixel in the green screen area of ​​the image to be processed and the super telephoto lens image in that channel.

[0189] It should be noted that the specific working process of each unit provided in the above embodiments of this application can be referred to the implementation process of the corresponding steps in the above method embodiments, and will not be repeated here.

[0190] Another embodiment of this application provides an electronic device, such as... Figure 8 As shown, it includes:

[0191] Memory 801 and processor 802.

[0192] Among them, memory 801 is used to store programs.

[0193] The processor 802 is used to execute the program stored in the memory 801. When the program is executed, it is specifically used to implement the virtual-real fusion keying method provided in any of the above embodiments.

[0194] Another embodiment of this application provides a computer storage medium for storing a computer program, which, when executed by a processor, is used to implement the virtual-real fusion keying method provided in any of the above embodiments.

[0195] Computer storage media, including both permanent and non-permanent, removable and non-removable media, can store information using any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0196] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0197] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A keying method for blending virtual and real images, characterized in that, include: Capture super telephoto lens images of the current scene; Extract the features of pixels in a specified area of ​​the main object in the super telephoto lens image and the features of pixels in the green screen area to obtain a set of standard features for the part and standard features for the green screen. Acquire the lens image to be processed in the current scene, and map the coordinates of each pixel in the standard feature set of the part to the coordinate system of the lens image to be processed to obtain the part feature mapping region; The image of the lens to be processed is keyed to obtain an initial mask; By matching the features of pixels in the feature mapping region of the image to be processed with the standard feature set of the region, the grayscale values ​​of each pixel in the initial mask are corrected to obtain an optimized mask. By matching the features of pixels in the green screen region of the image to be processed with the standard features of the green screen, color compensation is performed on the green screen region of the image to be processed to obtain an optimized image. Based on the optimized mask, the optimized lens image is keyed to obtain the final keyed image.

2. The method according to claim 1, characterized in that, After acquiring the super telephoto lens image of the current scene, the process also includes: The super telephoto lens image is subjected to bilateral filtering processing within a specified window, and the color values ​​of each pixel in the filtered super telephoto lens image are normalized.

3. The method according to claim 1, characterized in that, The process involves extracting pixel features from a specified region of the subject in the super telephoto lens image and pixel features from the green screen region to obtain a set of standard features for the subject and standard features for the green screen region, including: The main object in the super telephoto lens image is keyed out to obtain a mask of the main object; The specified area in the main mask of the object is located by edge detection, and the specified area is obtained. The pixels in the specified region of the super telephoto lens image are sampled, and the RGB values, HSV values, and local texture features of each sampled pixel are extracted to form a standard feature set of the region. Based on the object's main mask, the pixels of the super telephoto lens image within the green screen area are sampled, and the mean and variance of the RGB values ​​and the mean and variance of the HSV values ​​of each sampled pixel are calculated as standard features of the green screen.

4. The method according to claim 1, characterized in that, Before acquiring the image of the current scene to be processed, the process also includes: Switch the acquisition device to the target lens and set the same parameters as when capturing the super telephoto lens image; Adjust the acquisition device so that the shooting center of the target lens coincides with the coordinate system center of the ultra-long cross-section lens; The pre-stored distortion correction parameters are loaded to perform electronic compensation on the imaging optical path of the acquisition device.

5. The method according to claim 1, characterized in that, The step of mapping the coordinates of each pixel in the standard feature set of the part to the coordinate system of the lens image to be processed, to obtain the part feature mapping region, includes: Obtain the relative coordinates of each pixel in the standard feature set of the part; wherein, the relative coordinates of the pixels in the standard feature set of the part are the coordinates relative to the specified origin on the super telephoto lens image; Based on the coordinate system mapping relationship between the pre-calibrated target lens and the super telephoto lens, the relative coordinates of each pixel in the standard feature set of the part are mapped to the corresponding coordinates in the lens image to be processed, thereby obtaining the feature mapping region of the part.

6. The method according to claim 1, characterized in that, The process of matching the features of pixels in the feature mapping region of the image to be processed with the standard feature set of the region, and correcting the grayscale values ​​of each pixel in the initial mask to obtain an optimized mask includes: Each pixel in the feature mapping region of the lens image to be processed is traversed and used as the target pixel; Extract the features of the target pixel and match the features of the target pixel with the standard feature set of the part to obtain the matching degree corresponding to the target pixel; If the matching degree corresponding to the target pixel is greater than the matching degree threshold, the gray value of the target pixel in the initial mask is corrected from the minimum value to the maximum value to obtain the optimized mask.

7. The method according to claim 1, characterized in that, The step of matching the features of pixels in the green screen region of the image to be processed with the standard features of the green screen to perform color compensation on the green screen region of the image to be processed, thereby obtaining an optimized image, includes: Extract the features of each pixel in the green screen region of the image to be processed; Calculate the mean and variance of the features of all pixels in the green screen region of the image to be processed to obtain the current green screen features; Compare the current green screen features with the standard green screen features; If the deviation between the current green screen feature and the green screen standard feature is greater than a preset deviation threshold, then the color value of each pixel in each channel of the green screen area of ​​the image to be processed is compensated according to the color compensation amount of each channel; wherein, the color compensation amount of one channel is the deviation of the average color value of each pixel in the green screen area of ​​the image to be processed and the super telephoto lens image in that channel.

8. A keying device for blending virtual and real images, characterized in that, include: The first image acquisition unit is used to acquire super telephoto lens images of the current scene; A standard feature extraction unit is used to extract the features of pixels in a specified part of the main body of the object on the super telephoto lens image and the features of pixels in the green screen area, to obtain a set of standard features for the part and standard features for the green screen. The second image acquisition unit is used to acquire the lens image to be processed in the current scene; The region mapping unit is used to map the coordinates of each pixel in the standard feature set of the part to the coordinate system of the lens image to be processed, so as to obtain the part feature mapping region. The initial keying unit is used to key the lens image to be processed to obtain an initial mask; The mask optimization unit is used to modify the grayscale value of each pixel in the initial mask by matching the features of the pixels in the feature mapping area of ​​the lens image to be processed with the standard feature set of the part, so as to obtain an optimized mask. The color compensation unit is used to perform color compensation on the green screen area of ​​the lens image to be processed by matching the features of the pixels in the green screen area of ​​the lens image to be processed with the green screen standard features, so as to obtain an optimized lens image. The final keying unit is used to key the optimized lens image based on the optimized mask to obtain the final keyed image.

9. An electronic device, characterized in that, include: Memory and processor; The memory is used to store programs; The processor is used to execute the program, which, when executed, is specifically used to implement the virtual-real fusion keying method as described in any one of claims 1 to 7.

10. A computer storage medium, characterized in that, Used to store a computer program, which, when executed by a processor, is used to implement the keying method for virtual-real fusion as described in any one of claims 1 to 7.

Images