Image Processing Method, Apparatus, Device, Storage Medium, and Program Product

Abstract

The present application provides an image processing method, apparatus, electronic device, computer-readable storage medium, and computer program product; the method includes: obtaining an image to be processed, where the image to be processed includes a background object and a target object; constructing a one-dimensional transparency lookup table of the matte parameters based on at least one matte parameter of the background object; querying the one-dimensional transparency lookup table of the matte parameters for each pixel in the image to be processed to obtain a transparency channel image corresponding to the image to be processed; performing matte processing based on the transparency channel image to obtain a target image that removes the background object and includes the target object. Through the present application, the matte efficiency can be improved.

Description

Technical Field

[0001] This application relates to computer technology, and in particular, to an image processing method, apparatus, electronic device, computer-readable storage medium, and computer program product. Background Art

[0002] With the development of computer technology, matte extraction has been widely used in various aspects such as virtual production, interactive games, virtual live broadcasts, etc. For example, in current virtual production, the target object is extracted from the original video, and the extracted target object is merged with the background in the material video to produce a new video that conforms to the shooting scene, which can avoid repeatedly building different scenes when shooting videos and reduce the cost of scene arrangement.

[0003] However, the matte extraction solutions in the related art mainly rely on the multi-dimensional transparency lookup table set for the original image, and the transparent channel image is obtained through the multi-dimensional transparency lookup table of the original image, so as to perform matte extraction through the transparent channel image. However, the calculation process of the transparent channel image in this solution requires a large amount of computing resources, resulting in poor matte extraction efficiency. Summary of the Invention

[0004] Embodiments of this application provide an image processing method, apparatus, electronic device, computer-readable storage medium, and computer program product, which can fully and effectively utilize the one-dimensional transparency lookup table of each matte extraction parameter to implement matte extraction and improve the matte extraction efficiency.

[0005] The technical solution of the embodiments of this application is implemented as follows:

[0006] Embodiments of this application provide an image processing method, including:

[0007] Obtain an image to be processed, where the image to be processed includes a background object and a target object;

[0008] Based on at least one matte extraction parameter of the background object, construct a one-dimensional transparency lookup table of the matte extraction parameter;

[0009] Query the one-dimensional transparency lookup table of the matte extraction parameter for each pixel in the image to be processed to obtain a transparent channel image corresponding to the image to be processed;

[0010] Perform matte extraction processing based on the transparent channel image to obtain a target image that removes the background object and includes the target object.

[0011] Embodiments of this application provide an image processing apparatus, including:

[0012] An obtaining module, configured to obtain an image to be processed, where the image to be processed includes a background object and a target object;

[0013] A construction module, configured to construct a one-dimensional transparency query table of the matte extraction parameters based on at least one matte extraction parameter of the background object;

[0014] A query module, configured to query the one-dimensional transparency query table of the matte extraction parameters based on each pixel in the image to be processed, and obtain a transparency channel image corresponding to the image to be processed;

[0015] A processing module, configured to perform matte extraction processing based on the transparency channel image, and obtain a target image that removes the background object and includes the target object.

[0016] In the above technical solution, the construction module is further configured to determine a matte extraction range corresponding to the background object based on the matte extraction parameters of the background object;

[0017] Wherein, the transparency corresponding to the color parameter value included in the matte extraction range is completely transparent;

[0018] Based on the matte extraction range corresponding to the background object, construct the one-dimensional transparency query table of the matte extraction parameters, and the one-dimensional transparency query table includes the corresponding relationship between the color parameter value and the transparency.

[0019] In the above technical solution, when the matte extraction parameter is a hue matte extraction parameter, the matte extraction range corresponding to the background object includes a hue matte extraction range; when the matte extraction parameter is a saturation matte extraction parameter, the matte extraction range corresponding to the background object includes a saturation matte extraction range;

[0020] The construction module is further configured to determine a minimum hue value and a maximum hue value of the background object based on the hue matte extraction parameter of the background object, and use the minimum hue value and the maximum hue value as the end point values of the hue matte extraction range;

[0021] Based on the saturation matte extraction parameter of the background object, determine a minimum saturation value and a maximum saturation value of the background object, and use the minimum saturation value and the maximum saturation value as the end point values of the saturation matte extraction range.

[0022] In the above technical solution, the construction module is further configured to determine an initial minimum hue value, an initial maximum hue value and a hue gradient width of the background object based on the hue matte extraction parameter of the background object;

[0023] Increase the initial minimum hue value based on the hue gradient width to obtain the minimum hue value of the background object;

[0024] Decrease the initial maximum hue value based on the hue gradient width to obtain the maximum hue value of the background object;

[0025] Based on the saturation matte parameters of the background object, determine the initial minimum saturation value, the initial maximum saturation value, and the saturation gradient width of the background object;

[0026] Increase the initial minimum saturation value based on the saturation gradient width to obtain the minimum saturation value of the background object, and use the initial maximum saturation value as the maximum saturation value of the background object.

[0027] In the above technical solution, the construction module is further configured to obtain the hue gradient width of the background object, and determine the hue gradient range based on the hue matte range and the hue gradient width, where the transparency corresponding to the color parameter values included in the hue gradient range is neither completely transparent nor completely opaque;

[0028] Based on the hue matte range and the hue gradient range, construct a one-dimensional transparency lookup table for the hue matte parameters;

[0029] Obtain the saturation gradient width of the background object, and determine the saturation gradient range based on the saturation matte range and the saturation gradient width, where the transparency corresponding to the color parameter values included in the saturation gradient range is neither completely transparent nor completely opaque;

[0030] Based on the saturation matte range and the saturation gradient range, construct a one-dimensional transparency lookup table for the saturation matte parameters.

[0031] In the above technical solution, the query module is further configured to perform the following processing on any pixel in the to-be-processed image:

[0032] Query the one-dimensional transparency lookup table of the matte parameters based on the color parameters of the pixel to obtain the transparency of the pixel;

[0033] Combine the transparencies of multiple pixels according to the positional relationship of the pixels in the to-be-processed image to obtain the transparency channel image corresponding to the to-be-processed image.

[0034] In the above technical solution, when there are multiple matte parameters, the query module is further configured to query the one-dimensional transparency lookup tables of multiple matte parameters respectively based on the color parameters of the pixel to obtain the transparencies corresponding to the multiple matte parameters;

[0035] Take the maximum value of the transparencies corresponding to the multiple matte parameters as the transparency of the pixel.

[0036] In the above technical solution, the image to be processed is a color image; before querying the one-dimensional transparency lookup table of the matte parameters based on the color parameter values of each pixel in the image to be processed to obtain the transparency channel image corresponding to the image to be processed, the construction module is further configured to convert the image to be processed into a candidate color space to obtain a first converted image of the image to be processed in the candidate color space;

[0037] Convert the image to be processed into a black-and-white space to obtain a second converted image of the image to be processed in the black-and-white space;

[0038] Fuse the first converted image and the second converted image to obtain a third converted image corresponding to the image to be processed;

[0039] The query module is further configured to query the one-dimensional transparency lookup table of the matte parameters based on the color parameters of each pixel in the third converted image to obtain the transparency channel image corresponding to the image to be processed.

[0040] In the above technical solution, the color parameters of each pixel in the third converted image include a hue parameter, a saturation parameter, and a brightness parameter; the construction module is further configured to determine a saturation scaling factor based on the brightness parameter of each pixel in the second converted image;

[0041] Scale the saturation parameter of each pixel in the first converted image based on the saturation scaling factor to obtain the scaled saturation parameter;

[0042] Combine the brightness parameter of each pixel in the second converted image, the scaled saturation parameter, and the hue parameter of each pixel in the first converted image to obtain a third converted image corresponding to the image to be processed.

[0043] In the above technical solution, the processing module is further configured to perform morphological processing on the transparency channel image to obtain the transparency channel image after morphological processing;

[0044] Perform interference color removal processing on the image to be processed to obtain the image to be processed after interference color removal;

[0045] Perform fusion processing on the transparency channel image after morphological processing and the image to be processed after interference color removal to obtain a target image that removes the background object and includes the target object.

[0046] In the above technical solution, the pixel value of each pixel in the image to be processed includes a first color value, a second color value, and an interference color value; the processing module is further configured to perform the following processing on any pixel in the image to be processed:

[0047] Determine an interference color reference value of the pixel based on the first color value and the second color value of the pixel;

[0048] Combine the minimum value of the interference color of the pixel and the interference color reference value of the pixel, the first color value of the pixel, and the second color value to obtain the pixel after interference color removal;

[0049] Combine multiple pixels after interference color removal according to the positional relationship of the pixels in the image to be processed to obtain the image to be processed after interference color removal.

[0050] In the above technical solution, the processing module is further configured to use the absolute value of the difference between the first color value and the second color value of the pixel as the difference between the first color value and the second color value;

[0051] Determine the minimum value of the first color value and the second color value of the pixel;

[0052] The sum of the product of the difference and the interference color coefficient and the minimum value is used as the interference color reference value of the pixel.

[0053] An embodiment of the present application provides an electronic device for image processing, and the electronic device includes:

[0054] A memory for storing executable instructions;

[0055] A processor, when executing the executable instructions stored in the memory, implements the image processing method provided by the embodiment of the present application.

[0056] An embodiment of the present application provides a computer-readable storage medium storing executable instructions for causing a processor to implement the image processing method provided by the embodiment of the present application when executed.

[0057] An embodiment of the present application provides a computer program product including executable instructions, and the executable instructions implement the image processing method provided by the embodiment of the present application when executed by a processor.

[0058] The embodiment of the present application has the following beneficial effects:

[0059] By using at least one matte parameter of the background object in the image to be processed, a one-dimensional transparency lookup table for each matte parameter is constructed, and the one-dimensional transparency lookup table is quickly queried based on each pixel in the image to be processed, and matte processing is performed on the image to be processed based on the transparency channel image corresponding to the image to be processed, so as to fully and effectively utilize the one-dimensional transparency lookup table to achieve fast matte, improve the matte efficiency, and save a large amount of computing resources. Description of the Drawings

[0060] Figure 1 It is a schematic diagram of the application scenario of the image processing system provided by the embodiments of the present application;

[0061] Figure 2 It is a schematic diagram of the structure of the electronic device provided by the embodiments of the present application;

[0062] Figures 3A - 3C It is a schematic flowchart of the image processing method provided by the embodiments of the present application;

[0063] Figures 3D - 3E It is a schematic diagram of the one-dimensional transparency lookup table provided by the embodiments of the present application;

[0064] Figure 4A It is a schematic diagram of the matte extraction effect provided by the embodiments of the present application;

[0065] Figure 4B It is a schematic diagram of the virtual production scene provided by the embodiments of the present application;

[0066] Figure 5 It is a schematic flowchart of the high-quality fast green screen matte extraction algorithm based on separate lookup tables provided by the embodiments of the present application;

[0067] Figure 6 It is a schematic flowchart of the color space conversion provided by the embodiments of the present application;

[0068] Figure 7 It is a schematic flowchart of the table building process of the one-dimensional <hue - transparency> table provided by the embodiments of the present application;

[0069] Figure 8 It is a schematic diagram of the <hue - transparency> table provided by the embodiments of the present application;

[0070] Figure 9 It is a schematic flowchart of the table building process of the one-dimensional <saturation - transparency> table provided by the embodiments of the present application;

[0071] Figure 10 It is a schematic diagram of the <saturation - transparency> table provided by the embodiments of the present application;

[0072] Figure 11 It is a schematic flowchart of the process for calculating the alpha channel provided by the embodiments of the present application;

[0073] Figure 12A It is a schematic diagram of the two-dimensional table provided by the embodiments of the present application;

[0074] Figure 12B It is a schematic diagram of the <hue - transparency> table provided by the embodiments of the present application;

[0075] Figure 12C It is a schematic diagram of the <saturation - transparency> table provided by the embodiments of the present application;

[0076] Figure 13A It is a schematic diagram of the pixels of the black pants dyed green by the green screen in the embodiment of the present application;

[0077] Figure 13B It is a schematic diagram of the pixels of the skin not dyed green by the reflection of the green screen in the embodiment of the present application;

[0078] Figure 14 It is a schematic diagram of the post - processing flow in the embodiment of the present application;

[0079] Figure 15A It is a schematic diagram of the matte extraction result in the embodiment of the present application;

[0080] Figure 15B It is a schematic diagram of the matte extraction result in the embodiment of the present application. Detailed implementation manners

[0081] In order to make the objectives, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on the present application. All other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the scope of protection of the present application.

[0082] In the following description, the terms "first / second" involved are only used to distinguish similar objects and do not represent a specific order for the objects. It can be understood that "first / second" can be interchanged with a specific order or sequence when permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0083] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those of ordinary skill in the technical field to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0084] Before further elaborating on the embodiments of the present application, the nouns and terms involved in the embodiments of the present application are described. The nouns and terms involved in the embodiments of the present application are subject to the following explanations.

[0085] 1) Video: Composed of a series of image frames, that is, composed of an image sequence. The smoothness of the video can be represented by the frames per second (FPS, Frames Per Second). The more frames per second, the smoother the displayed action will be. FPS is a definition in the field of images and refers to the number of frames of an animation or video. Each image frame is a still image, and when played in sequence, a moving image is created. For example, 30 FPS means that 30 "still images" will be played per second.

[0086] 2) Matte extraction parameter: A parameter that determines whether a pixel in the image to be processed is part of the background. When a pixel in the image to be processed belongs to the background, the corresponding color parameter (or pixel value) of that pixel is the matte extraction parameter. For example, the color parameters (including saturation and hue) corresponding to the background object (such as a green screen) on the image to be processed are the matte extraction parameters of the background object.

[0087] 3) QR code green screen: A green screen printed with QR codes. The green screen includes three sides: the left plane of the QR code, the right plane of the QR code, and the bottom plane of the QR code. All QR codes on the QR code green screen have unique patterns and numbers. Using the corresponding QR code detection algorithm, all unobstructed QR codes can be detected, and the corner coordinates of the QR codes on the imaging image can be accurately obtained.

[0088] 4) Color parameter: The parameter value corresponding to each pixel in the color space. For example, in the RGB color space, red (R, Red), green (G, Green), and blue (B, Blue) all represent color parameters; in the RGBA color space, red (R, Red), green (G, Green), blue (B, Blue), and transparency (A, Alpha) all represent color parameters; in the HSY color space, hue (H, Hue), saturation (S, Saturation), and lightness (Y) all represent color parameters.

[0089] Among them, hue is the primary characteristic of color and the most accurate standard for distinguishing different colors. Under the illumination of light with different wavelengths, the human eye will perceive different colors, such as blue and red. These external manifestation characteristics of colors are called hue.

[0090] Saturation, also known as "purity", refers to the vividness of color. The higher the saturation, the purer the color and the more vivid the color. When mixed with other colors, the saturation of the color will decrease, and the color will become darker and lighter. When the saturation of the color drops to the lowest level, it will lose its hue and become achromatic (black, white, gray).

[0091] Lightness refers to the brightness of color, and all colors have different degrees of brightness. Among achromatic colors, white has the highest lightness, gray is in the middle, and black is the darkest. It should be noted that the change in the lightness of color often affects the purity. For example, when white is added to red, the lightness increases, but the purity decreases.

[0092] 5) Morphology: Extract the image components from the image that are meaningful for expressing and depicting the shape of the region, enabling the subsequent recognition work to capture the most essential shape features of the target object, such as boundaries and connected regions. Morphology is the basic theory of mathematical morphology image processing, and its basic operations include: binary erosion and dilation, binary opening and closing operations, skeleton extraction, ultimate erosion, hit-or-miss transform, morphological gradient, Top-hat transform, particle analysis, watershed transform, gray-scale erosion and dilation, gray-scale opening and closing operations, gray-scale morphological gradient, etc.

[0093] 6) Shader: It is used to implement image rendering and is an editable program that replaces the fixed rendering pipeline. In the shader, the processing flow for pixels can be defined. In this way, when the graphics processing unit (GPU) processes images in parallel, the shader can be used to accelerate the calculation and liberate the computing power of the central processing unit (CPU). The shader replaces the fixed rendering pipeline and can implement the relevant calculations in 3D graphics computing. Due to its editable nature, various image effects can be achieved without being restricted by the fixed rendering pipeline of the graphics card.

[0094] 7) Client: An application program running on the terminal that provides various services.

[0095] The embodiments of the present application provide an image processing method, apparatus, electronic device, computer-readable storage medium, and computer program product, which can make full and effective use of the one-dimensional transparency lookup table of each matte parameter to perform matting, improving the matting efficiency.

[0096] The image processing method provided by the embodiments of the present application can be implemented independently by the terminal or the server; it can also be implemented in cooperation by the terminal and the server. For example, the terminal independently undertakes the image processing method described below, or the terminal sends a matting request for the image to be processed to the server, and the server executes the image processing method according to the received matting request for the image to be processed, constructs a one-dimensional transparency lookup table of the matte parameter based on at least one matte parameter of the background object, queries the one-dimensional transparency lookup table of the matte parameter for each pixel in the image to be processed, obtains the alpha channel image corresponding to the image to be processed, performs matting processing based on the alpha channel image to obtain a target image that removes the background object and includes the target object, and executes applications such as virtual production, interactive games, and virtual live broadcast based on the target image.

[0097] The electronic device for image processing provided by the embodiments of the present application can be various types of terminals or servers. Among them, the server can be an independent physical server, a server cluster or a distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, Content Delivery Network (CDN), and big data and artificial intelligence platforms; the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, a smart TV, a smart vehicle-mounted device, etc., but is not limited thereto. The terminal and the server can be directly or indirectly connected through wired or wireless communication methods, and the present application does not limit this.

[0098] See Figure 1 , Figure 1 is a schematic diagram of the application scenario of the image processing system 10 provided by the embodiments of the present application. The terminal 200 is connected to the server 100 through the network 300. The network 300 can be a wide area network, a local area network, or a combination of the two.

[0099] The terminal 200 (running a client, such as a matte extraction client) can be used to obtain a matte extraction request for the image to be processed. For example, when the user opens the client running on the terminal and inputs the image that needs to be processed, the terminal automatically obtains the matte extraction request for the image to be processed (including the image to be processed for image processing).

[0100] In some embodiments, an image processing plug-in can be implanted in the client running on the terminal 200 to implement the image processing method locally on the client. For example, the terminal 200 calls the image processing plug-in to implement the image processing method. Based on at least one matte extraction parameter of the background object in the image to be processed, a one-dimensional transparency lookup table of the matte extraction parameter is constructed. Each pixel in the image to be processed queries the one-dimensional transparency lookup table of the matte extraction parameter to obtain the transparency channel image corresponding to the image to be processed. Matte extraction processing is performed based on the transparency channel image to obtain a target image that removes the background object and includes the target object, so as to be able to make full and effective use of the one-dimensional transparency lookup table of each matte extraction parameter to implement matte extraction, improve the matte extraction efficiency, and execute applications such as virtual production, interactive games, and virtual live broadcast based on the target image.

[0101] In some embodiments, after the terminal 200 obtains a matte extraction request for an image to be processed, it calls the image processing interface of the server 100 (which can be provided in the form of a cloud service, i.e., an image processing service). Based on the matte extraction request for the image to be processed, the server 100 implements an image processing method. Based on at least one matte extraction parameter of the background object in the image to be processed, a one-dimensional transparency query table of the matte extraction parameters is constructed. Each pixel in the image to be processed queries the one-dimensional transparency query table of the matte extraction parameters to obtain a transparency channel image corresponding to the image to be processed. Matte extraction processing is performed based on the transparency channel image to obtain a target image that removes the background object and includes the target object, so as to be able to make full and effective use of the one-dimensional transparency query table of each matte extraction parameter to implement matte extraction, improve the matte extraction efficiency, and execute applications such as virtual production, interactive games, and virtual live broadcast based on the target image.

[0102] In some embodiments, the terminal or the server can implement the image processing method provided in the embodiments of the present application by running a computer program (i.e., executable instructions). For example, the computer program can be a native program or software module in the operating system; it can be a local (Native) application (APP, Application), that is, a program that needs to be installed in the operating system to run, such as a video application (such as a video client running on the terminal); it can also be a small program, that is, a program that only needs to be downloaded to the browser environment to run; it can also be a small program that can be embedded in any APP. In short, the above computer program can be any form of application program, module, or plug-in.

[0103] In some embodiments, multiple servers can be organized into a blockchain, and the server 100 is a node on the blockchain. There can be an information connection between each node in the blockchain, and information can be transmitted between nodes through the above information connection. Among them, the data related to the image processing method provided in the embodiments of the present application (such as the logic of image processing, the target image) can be saved on the blockchain.

[0104] The structure of the electronic device provided in the embodiments of the present application will be described below. Refer to Figure 2 , Figure 2 FIG. is a schematic structural diagram of the electronic device 500 provided in the embodiments of the present application. Taking the electronic device 500 as a server or a terminal as an example, Figure 2 The electronic device 500 shown includes: at least one processor 510, a memory 550, at least one network interface 520, and a user interface 530. Each component in the electronic device 500 is coupled together through a bus system 540. It can be understood that the bus system 540 is used to realize the connection and communication between these components. The bus system 540 includes not only a data bus, but also a power bus, a control bus, and a status signal bus. However, for the sake of clarity, inFigure 2 Various buses are labeled as bus system 540 .

[0105] The processor 510 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., where the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0106] The memory 550 includes a volatile memory or a non-volatile memory, and may also include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), and the volatile memory may be a random access memory (RAM). The memory 550 described in the embodiment of the present application is intended to include any suitable type of memory. The memory 550 optionally includes one or more storage devices that are physically far away from the processor 510.

[0107] In some embodiments, the memory 550 can store data to support various operations, examples of which include programs, modules, and data structures, or a subset or superset thereof, as exemplarily described below.

[0108] Operating system 551, including system programs for processing various basic system services and performing hardware-related tasks, such as framework layer, core library layer, driver layer, etc., for implementing various basic services and processing hardware-based tasks;

[0109] A network communication module 552, used to reach other electronic devices via one or more (wired or wireless) network interfaces 520, exemplary network interfaces 520 include: Bluetooth, Wireless Compatibility Authentication (WiFi), and Universal Serial Bus (USB), etc.;

[0110] In some embodiments, the image processing device provided in the embodiments of the present application can be implemented in a software manner. Figure 2 An image processing device 555 stored in a memory 550 is shown, which may be software in the form of a program or a plug-in, and includes the following software modules: an acquisition module 5551, a construction module 5552, a query module 5553, and a processing module 5554. These modules are logical, and thus may be arbitrarily combined or further split according to the functions implemented. The functions of each module will be described below.

[0111] As described above, the video processing method provided by the embodiments of the present application can be implemented by various types of electronic devices. Refer to Figure 3A , Figure 3A which is a schematic flowchart of the image processing method provided by the embodiments of the present application, and will be described in conjunction with the steps shown in Figure 3A .

[0112] In step 101, an image to be processed is obtained, where the image to be processed includes a background object and a target object.

[0113] It should be noted that the image to be processed is an image to be cropped, and the target object needs to be cropped out from the image to be processed. The image to be processed can be an image obtained by photographing a background object (such as a green screen, a QR code green screen, etc.), that is, the image to be processed includes a background object and a target object (also known as the main body) that obscures part of the background object. Among them, the background object in the image to be processed is the object to be cropped out. The target object cropped out from the image to be processed is fused with the material to produce an image that conforms to the shooting scene, which can avoid repeatedly building different scenes during shooting and reduce the cost of scene arrangement. For example, if the material is a news hosting hall, the target object is fused with the news hosting hall to produce an image of the news hosting scene.

[0114] In step 102, based on at least one matte parameter of the background object, a one-dimensional transparency lookup table of the matte parameter is constructed.

[0115] Among them, the one-dimensional transparency lookup table includes the one-dimensional correspondence between the color parameter values of pixels and transparency. By one matte parameter of the background object, a one-dimensional transparency lookup table of this matte parameter (i.e., a one-dimensional transparency lookup table) is constructed. Subsequently, each pixel in the image to be processed quickly separates and queries the one-dimensional transparency lookup table, avoiding querying a multi-dimensional transparency lookup table and improving the query efficiency. It should be noted that the one-dimensional transparency lookup table can be a continuous table or a discrete table.

[0116] Refer to Figure 3B , Figure 3B which is a schematic flowchart of the image processing method provided by the embodiments of the present application, Figure 3B showing that Figure 3A step 102 can be implemented through steps 1021 - 1022: In step 1021, based on the matte parameter of the background object, a matte range corresponding to the background object is determined; among them, the transparency corresponding to the color parameter values included in the matte range is completely transparent; in step 1022, based on the matte range corresponding to the background object, a one-dimensional transparency lookup table of the matte parameter is constructed, and the one-dimensional transparency lookup table includes the correspondence between the color parameter values and transparency.

[0117] For example, based on the matte extraction parameters of the background object, the matte extraction range of the background object under the matte extraction parameters can be determined. That is, if the color parameter value of any pixel is within the matte extraction range, then the pixel is completely transparent. It should be noted that the one-dimensional transparency lookup table of the matte extraction parameters constructed through the matte extraction range corresponding to the background object includes the matte extraction range and the non-matte extraction range, where the transparency corresponding to the color parameter values included in the non-matte extraction range is not completely transparent. Subsequently, by querying the one-dimensional transparency lookup table with the color parameter value of the pixel, the transparency of the pixel can be queried.

[0118] For example, when the matte extraction parameter is the hue matte extraction parameter, based on the hue matte extraction parameter of the background object, the hue matte extraction range corresponding to the background object is determined. Among them, the transparency corresponding to the hue values included in the hue matte extraction range is completely transparent. Based on the hue matte extraction range corresponding to the background object, a one-dimensional transparency lookup table of the hue matte extraction parameter is constructed, and the one-dimensional transparency lookup table includes the correspondence between the hue value and the transparency; when the matte extraction parameter is the saturation matte extraction parameter, based on the saturation matte extraction parameter of the background object, the saturation matte extraction range corresponding to the background object is determined. Among them, the transparency corresponding to the saturation included in the saturation matte extraction range is completely transparent. Based on the saturation matte extraction range corresponding to the background object, a one-dimensional transparency lookup table of the saturation matte extraction parameter is constructed, and the one-dimensional transparency lookup table includes the correspondence between the saturation and the transparency.

[0119] In some embodiments, when the matte extraction parameter is the hue matte extraction parameter, the matte extraction range corresponding to the background object includes the hue matte extraction range; when the matte extraction parameter is the saturation matte extraction parameter, the matte extraction range corresponding to the background object includes the saturation matte extraction range; determining the matte extraction range corresponding to the background object based on the matte extraction parameter of the background object includes: determining the minimum hue value and the maximum hue value of the background object based on the hue matte extraction parameter of the background object, and using the minimum hue value and the maximum hue value as the endpoint values of the hue matte extraction range; determining the minimum saturation value and the maximum saturation value of the background object based on the saturation matte extraction parameter of the background object, and using the minimum saturation value and the maximum saturation value as the endpoint values of the saturation matte extraction range.

[0120] For example, when the matte extraction parameter is the hue matte extraction parameter, the minimum hue value h1 and the maximum hue value h2 corresponding to the background object are determined according to actual needs, and the minimum hue value h1 and the maximum hue value h2 are used as the endpoint values of the hue matte extraction range, that is, the hue matte extraction range [h1, h2] corresponding to the background object is obtained. When the hue value of the pixel is within the hue matte extraction range [h1, h2], the transparency of the pixel is completely transparent.

[0121] Such as Figure 3DThe one-dimensional transparency lookup table of the hue matte parameter (also known as the one-dimensional <hue-transparency> table) shown in the figure. The horizontal axis in the one-dimensional transparency lookup table of the hue matte parameter represents the hue value, and the vertical axis represents the transparency. Among them, the transparency corresponding to the hue matte range [h1, h2] is completely transparent, that is, f(x) = 0, and the transparency corresponding to other ranges, namely [0, h1) and (h1, 255], is completely opaque, that is, f(x) = 255.

[0122] For example, when the matte parameter is the saturation matte parameter, determine the minimum saturation value s1 and the maximum saturation value s2 corresponding to the background object according to actual needs, and use the minimum saturation value s1 and the maximum saturation value s2 as the end point values of the saturation matte range, that is, obtain the saturation matte range [s1, s2] corresponding to the background object. When the saturation of the pixel is within the saturation matte range [s1, s2], the transparency of the pixel is completely transparent.

[0123] Such as Figure 3E The one-dimensional transparency lookup table of the saturation matte parameter (also known as the one-dimensional <saturation-transparency> table) shown in the figure. The horizontal axis in the one-dimensional transparency lookup table of the saturation matte parameter represents the saturation, and the vertical axis represents the transparency. Among them, the transparency corresponding to the saturation matte range [s1, 255] is completely transparent, that is, g(x) = 0, and the transparency corresponding to other ranges, namely [0, s1), is completely opaque, that is, g(x) = 255.

[0124] In some embodiments, based on the hue matte parameter of the background object, determine the minimum hue value and the maximum hue value of the background object, including: based on the hue matte parameter of the background object, determine the initial minimum hue value, the initial maximum hue value and the hue gradient width of the background object; increase the initial minimum hue value based on the hue gradient width to obtain the minimum hue value of the background object; decrease the initial maximum hue value based on the hue gradient width to obtain the maximum hue value of the background object; based on the saturation matte parameter of the background object, determine the minimum saturation value and the maximum saturation value of the background object, including: based on the saturation matte parameter of the background object, determine the initial minimum saturation value, the initial maximum saturation value and the saturation gradient width of the background object; increase the initial minimum saturation value based on the saturation gradient width to obtain the minimum saturation value of the background object, and use the initial maximum saturation value as the maximum saturation value of the background object.

[0125] Among them, the transparency corresponding to the hue gradient width is neither completely transparent nor completely opaque, and the transparency corresponding to the saturation gradient width is neither completely transparent nor completely opaque. The hue matte range is reduced by the hue gradient width, and the saturation matte range is reduced by the saturation gradient width, so that the matte range is more accurate and the matte effect is improved.

[0126] Continuing with the above example, the initial minimum hue of the background object is h1, the initial maximum hue is h2, and the hue gradient width is r1. Based on the hue gradient width r1, the initial minimum hue is increased to obtain the minimum hue h1 + r1 / 2 of the background object. Based on the hue gradient width r1, the initial maximum hue is decreased to obtain the maximum hue h2 - r1 / 2 of the background object.

[0127] Continuing with the above example, the initial minimum saturation of the background object is s1, the initial maximum saturation is s2, and the saturation gradient width is r2. Based on the saturation gradient width r2, the initial minimum saturation is increased to obtain the minimum saturation s1 + r2 / 2 of the background object. Based on the saturation gradient width r2, the initial maximum saturation is decreased to obtain the maximum saturation s2 - r2 / 2 of the background object.

[0128] In some embodiments, based on the matte range corresponding to the background object, a one-dimensional transparency lookup table for matte parameters is constructed, including: obtaining the hue gradient width of the background object, and determining the hue gradient range based on the hue matte range and the hue gradient width, where the transparency corresponding to the color parameter values included in the hue gradient range is neither completely transparent nor completely opaque; constructing a one-dimensional transparency lookup table for hue matte parameters based on the hue matte range and the hue gradient range; obtaining the saturation gradient width of the background object, and determining the saturation gradient range based on the saturation matte range and the saturation gradient width, where the transparency corresponding to the color parameter values included in the saturation gradient range is neither completely transparent nor completely opaque; constructing a one-dimensional transparency lookup table for saturation matte parameters based on the saturation matte range and the saturation gradient range.

[0129] For example, through the hue gradient range and the saturation gradient range, it is possible to prevent pixels from going directly from completely transparent to completely opaque, or from completely opaque to completely transparent, so that pixels can transition from completely transparent to completely opaque, or from completely opaque to completely transparent, smoothing the matte process, improving the matte effect, and preventing the matte process from being too abrupt.

[0130] Such as Figure 8The one-dimensional transparency query table of the hue matte extraction parameter (also known as the one-dimensional <hue - transparency> table) shown. In the one-dimensional transparency query table of the hue matte extraction parameter, the horizontal axis represents the hue value, and the vertical axis represents the transparency. Among them, the transparency corresponding to the hue matte extraction range [h1 + r1 / 2, h2 - r1 / 2] is completely transparent, that is, f(x) = 0, where x represents the hue value. The transparency corresponding to the hue gradient ranges (h1 - r1 / 2, h1 + r1 / 2) and (h2 - r1 / 2, h2 + r1 / 2) is neither completely transparent nor completely opaque. The transparency corresponding to the other ranges [0, h1 - r1 / 2] and [h2 + r1 / 2, 255] is completely opaque, that is, f(x) = 255. Among them, the specific definition of the formula f(x) is as follows:

[0131]

[0132] As Figure 10 The one-dimensional transparency query table of the saturation matte extraction parameter (also known as the one-dimensional <saturation - transparency> table) shown. In the one-dimensional transparency query table of the saturation matte extraction parameter, the horizontal axis represents the saturation value, and the vertical axis represents the transparency. Among them, the transparency corresponding to the saturation matte extraction range [s2 + r2 / 2, 255] is completely transparent, that is, g(x) = 0, where x represents the saturation. The transparency corresponding to the saturation gradient range (s1 - r2 / 2, s1 + r2 / 2) is neither completely transparent nor completely opaque. The transparency corresponding to the other range [0, s1 - r2 / 2] is completely opaque, that is, g(x) = 255. Among them, the specific definition of the formula g(x) is as follows:

[0133]

[0134] In step 103, based on each pixel in the image to be processed, query the one-dimensional transparency query table of the matte extraction parameter to obtain the transparency channel image corresponding to the image to be processed.

[0135] For example, by quickly separating and querying the one-dimensional transparency query table through the color parameters of each pixel in the image to be processed, obtain the transparency of each pixel, avoid querying the multi-dimensional transparency query table, and improve the query efficiency.

[0136] In some embodiments, based on each pixel in the image to be processed, query the one-dimensional transparency query table of the matte extraction parameter to obtain the transparency channel image corresponding to the image to be processed, including: performing the following processing on any pixel in the image to be processed: query the one-dimensional transparency query table of the matte extraction parameter based on the color parameter of the pixel to obtain the transparency of the pixel; according to the positional relationship of the pixels in the image to be processed, combine the transparencies of multiple pixels to obtain the transparency channel image corresponding to the image to be processed.

[0137] For example, when the matte extraction parameter is the hue matte extraction parameter, the following processing is performed on any pixel in the image to be processed: query the one-dimensional transparency lookup table of the hue matte extraction parameter based on the hue value of the pixel to obtain the transparency of the pixel, and combine the transparencies of multiple pixels according to the positional relationship of the pixels in the image to be processed to obtain the transparency channel image corresponding to the image to be processed.

[0138] For example, when the matte extraction parameter is the saturation matte extraction parameter, the following processing is performed on any pixel in the image to be processed: query the one-dimensional transparency lookup table of the saturation matte extraction parameter based on the saturation of the pixel to obtain the transparency of the pixel, and combine the transparencies of multiple pixels according to the positional relationship of the pixels in the image to be processed to obtain the transparency channel image corresponding to the image to be processed.

[0139] In some embodiments, when there are multiple matte extraction parameters, querying the one-dimensional transparency lookup table of the matte extraction parameter based on the color parameter of the pixel to obtain the transparency of the pixel includes: querying the one-dimensional transparency lookup tables of multiple matte extraction parameters respectively based on the color parameter of the pixel to obtain the transparencies corresponding to the multiple matte extraction parameters; taking the maximum value of the transparencies corresponding to the multiple matte extraction parameters as the transparency of the pixel.

[0140] For example, when the matte extraction parameters of the background object include the hue matte extraction parameter and the saturation matte extraction parameter, query the one-dimensional transparency lookup table of the hue matte extraction parameter based on the hue value of the pixel to obtain the transparency a1 of the hue matte extraction parameter, query the one-dimensional transparency lookup table of the saturation matte extraction parameter based on the saturation of the pixel to obtain the transparency a2 of the saturation matte extraction parameter, and take the maximum value of the transparency a1 of the hue matte extraction parameter and the transparency a2 of the saturation matte extraction parameter as the transparency a of the pixel.

[0141] In some embodiments, the image to be processed is a color image; before querying the one-dimensional transparency lookup table of the matte extraction parameter based on the color parameter value of each pixel in the image to be processed to obtain the transparency channel image corresponding to the image to be processed, convert the image to be processed to a candidate color space to obtain a first converted image of the image to be processed in the candidate color space; convert the image to be processed to a black-and-white space to obtain a second converted image of the image to be processed in the black-and-white space; fuse the first converted image and the second converted image to obtain a third converted image corresponding to the image to be processed; querying the one-dimensional transparency lookup table of the matte extraction parameter based on each pixel in the image to be processed to obtain the transparency channel image corresponding to the image to be processed includes: querying the one-dimensional transparency lookup table of the matte extraction parameter based on the color parameter of each pixel in the third converted image to obtain the transparency channel image corresponding to the image to be processed.

[0142] For example, if the image to be processed is an RGB color image and the matte parameters are saturation matte parameters and hue matte parameters, it is necessary to convert the image to be processed to a candidate color space (i.e., the HLS color space) to obtain a first converted image of the image to be processed in the candidate color space (i.e., the HLS image), convert the image to be processed to a black-and-white space to obtain a second converted image of the image to be processed in the black-and-white space (i.e., the black-and-white image), fuse the first converted image and the second converted image to obtain a third converted image corresponding to the image to be processed (i.e., the HSY image), and finally query a one-dimensional transparency lookup table based on the color parameters (such as saturation and hue value) of each pixel in the third converted image to obtain a transparency channel image corresponding to the image to be processed.

[0143] In some embodiments, the color parameters of each pixel in the third converted image include a hue parameter, a saturation parameter, and a brightness parameter; fusing the first converted image and the second converted image to obtain a third converted image corresponding to the image to be processed includes: determining a saturation scaling factor based on the brightness parameter of each pixel in the second converted image; scaling the saturation parameter of each pixel in the first converted image based on the saturation scaling factor to obtain a scaled saturation parameter; combining the brightness parameter of each pixel in the second converted image, the scaled saturation parameter, and the hue parameter of each pixel in the first converted image to obtain a third converted image corresponding to the image to be processed.

[0144] Continuing with the above example, for each pixel on the RGB color image, the following processing is performed to convert to the HSY image: using the RGB-to-HLS algorithm, convert the image to be processed to the candidate color space to obtain the hue h and saturation s of each pixel; using the RGB-to-black-and-white image algorithm, convert the image to be processed to the black-and-white space to obtain the brightness Y (i.e., the brightness parameter) of each pixel; determine the saturation scaling factor based on the brightness parameter of each pixel in the second converted image, that is, calculate the minimum value m between the brightness Y and 255 - Y, and m / 255 is the saturation scaling factor; use m / 255 to scale the saturation parameter s, and the scaled s is the value of the saturation in the HSY color space. Take the hue parameter h of each pixel in the first converted image as the hue H in the HSY color space, the scaled s as the saturation S in the HSY color space, and Y as the Y in the HSY color space to obtain the converted HSY pixel. Combine all the converted HSY pixels to obtain a third converted image corresponding to the image to be processed (i.e., the HSY image).

[0145] In step 104, matte processing is performed based on the transparency channel image to obtain a target image that removes the background object and includes the target object.

[0146] For example, based on each pixel in the image to be processed, a one-dimensional transparency lookup table is quickly queried, and the image to be processed is cropped based on the transparency channel image corresponding to the image to be processed, so as to fully and effectively utilize the one-dimensional transparency lookup table to achieve fast image cropping, improve the image cropping efficiency, and save a large amount of computing resources.

[0147] See Figure 3C , Figure 3C which is a schematic flowchart of the image processing method provided by an embodiment of the present application. Figure 3C showing Figure 3A Step 104 shown in can be implemented through steps 1041 - 1043: In step 1041, morphological processing is performed on the transparency channel image to obtain the transparency channel image after morphological processing; in step 1042, interference color removal processing is performed on the image to be processed to obtain the image to be processed after interference color removal; in step 1043, the transparency channel image after morphological processing and the image to be processed after interference color removal are fused to obtain a target image that removes the background object and includes the target object.

[0148] It should be noted that the interference color is the color that affects the original color on the target object due to the color of the background object. For example, if the target object is a green screen, it may cause some pixels of the target object on the image to be processed to turn green. Therefore, it is necessary to perform de-greening processing on the image to be processed to obtain the image to be processed after de-greening.

[0149] For example, morphological processing such as dilation, erosion, and blurring is performed on the transparency channel image to obtain the transparency channel image after morphological processing. The transparency channel image after morphological processing and the image to be processed after interference color removal are combined and fused to obtain a target image that removes the background object and includes the target object, so as to be able to remove noise, holes, etc. on the transparency channel image and achieve an accurate image cropping function.

[0150] In some embodiments, the pixel value of each pixel in the image to be processed includes a first color value, a second color value, and an interference color value; performing interference color removal processing on the image to be processed to obtain the image to be processed after interference color removal includes: performing the following processing on any pixel in the image to be processed: determining the interference color reference value of the pixel based on the first color value and the second color value of the pixel; combining the minimum value of the interference color of the pixel and the interference color reference value of the pixel, the first color value of the pixel, and the second color value to obtain the pixel after interference color removal; combining multiple pixels after interference color removal according to the positional relationship of the pixels in the image to be processed to obtain the image to be processed after interference color removal.

[0151] Among them, the interference color reference value is used as a reference for defining the specific value of the interference color. The interference color reference value of a pixel. The determination process of the interference color reference value of a pixel is as follows: Take the absolute value of the difference between the first color value and the second color value of the pixel as the difference between the first color value and the second color value; Determine the minimum value among the first color value and the second color value of the pixel; The sum of the product of the difference and the interference color coefficient and the minimum value is used as the interference color reference value of the pixel.

[0152] The following takes the interference color being green as an example for illustration. The first color value is the grayscale value of the red channel (R) of the pixel, and the second color value is the grayscale value of the blue channel (B) of the pixel. If the grayscale value of the green channel (G) of a pixel is greater than the average value of the red channel and the blue channel, then this pixel will appear green. Therefore, as long as the green channel value of the pixel dyed green is reduced to the same value as the average value of the red channel and the blue channel, it will not look green. Then the green removal process (i.e., interference color removal) is as follows: For each RGB pixel (i.e., each pixel in the image to be processed), the following processing is performed: Calculate the minimum value and the maximum value of the R channel and the B channel, and record them as j and k respectively; Calculate d, d = k - j, that is, d represents the difference between the first color value and the second color value; Calculate the threshold T, T = j + t·d, where t is the green removal coefficient, which can be set to 0.5, and T represents the interference color reference value; Take the minimum value between the channel G and T as the G channel value after green removal, take the original B channel value as the B channel value after green removal, and the original R channel value as the R channel value after green removal, and combine the G channel value after green removal, the B channel value after green removal, and the R channel value after green removal to obtain the RGB pixel color after green removal, that is, the pixel after interference color removal.

[0153] Next, an exemplary application of the embodiments of the present application in an actual application scenario will be described.

[0154] The embodiments of the present application can be applied to various scenarios of image matting, such as live broadcast, star companion viewing scenario, virtual production, video editing service, production of virtual reality (VR) video, interactive games (allowing real people to interact with virtual scenes and providing immersive visual effects), etc.

[0155] Regarding the live broadcast and star companion viewing scenarios, the green screen matting technology can be used to achieve the Figure 4A shown visual effect, implant the extracted real person 401 into the virtual scene 402 to make the viewing experience better.

[0156] Regarding virtual production, virtual production is a technology that mixes real people with virtual scenes. Here, green screen cutout technology is needed. It should be noted that a good green screen cutout can retain the shadow of the human body on the green screen, which can achieve better picture effects. Using virtual production technology in variety shows, large-scale live broadcasts and other scenes can achieve very cool and flexible picture effects, bringing the ultimate special effects experience to live broadcasts and on-demand. Figure 4B The picture effect shown is to implant the real person 403 into the live broadcast scene 404.

[0157] Regarding video editing services, green screen cutout technology can be deployed as an online service for use in video and image cutout services, and applied to various post-editing applications.

[0158] In the related art, the cutout algorithm has various limitations. Although the cutout algorithm of the related art can correctly deduct the QR code green screen and can process 4K video in real time, the cutout algorithm of the related art is based on the central processing unit (CPU, Central Processing Unit) and will cause a very high load on the CPU when processing 4K video. When applied in virtual-real fusion, since other modules in the virtual-real fusion also need to use the CPU, it will lead to the preemption of CPU resources, which will lead to jamming. As for the green screen cutout algorithm of the related art, or the cutout algorithm in commercial software, it requires a very high-quality green screen, and cannot deduct the QR code green screen.

[0159] In order to solve the above problems, the embodiment of the present application proposes a high-quality and fast green screen cutout algorithm (i.e., image processing method) based on separate lookup table, which can facilitate the porting of shaders, can process 4K or higher resolution videos in real time, and only occupies a small amount of graphics processor (GPU, Graphics Processing Unit) resources; it has high robustness and can not only cut out the green screen, but also cut out the QR code green screen; it can process 1080P video cutout in real time using a single-core CPU.

[0160] It should be noted that since the QR code green screen is used in the fusion of virtual and real, it can bring many conveniences and cost advantages to the fusion of virtual and real. The fusion of virtual and real is to implant real people into virtual scenes. In order to obtain high-quality virtual and real fusion effects, it is necessary to support camera movement. The QR code on the QR code green screen can provide accurate feature points for the real-person video, so that the camera movement information of the real-person video can be calculated through the computer vision algorithm. Otherwise, a hardware locator is needed to obtain the camera movement information of the real-person video, which consumes hardware costs, and a special algorithm is required to synchronize the time difference between the real-person video and the hardware locator, which affects the virtual and real fusion effect.

[0161] The core point of the high-quality fast green screen matting algorithm based on separate look-up tables proposed in the embodiments of this application is to use two one-dimensional table look-ups to accelerate the speed of obtaining the alpha channel image and to be easily implemented on the GPU shader. It should be noted that for ultra-high-resolution videos, such as 4K, 5K, 6K, 8K videos, by using the shader for computing instead of the CPU, it is possible to avoid over-occupying the CPU.

[0162] The following will combine Figure 5 to illustrate the high-quality fast green screen matting algorithm based on separate look-up tables proposed in the embodiments of this application, which includes 4 parts: color space conversion, one-dimensional table building, look-up table to obtain the alpha channel, and post-processing. The following will specifically describe these 4 parts:

[0163] I. Color space conversion.

[0164] The embodiments of this application adopt the HSY color space, where the HSY color space is obtained by converting based on the HLS algorithm. For each pixel on the RGB image to be matted (i.e., the image to be processed), the following processing is performed to convert it into an image in the HSY color space. The following will combine Figure 6 to specifically describe the conversion process:

[0165] Step 11. Use the RGB to HLS algorithm (such as the RGB to HLS in opencv) to obtain the hue h and saturation s of each pixel.

[0166] Step 12. Use the RGB to black and white image algorithm to obtain the brightness Y of each pixel.

[0167] Step 13. Calculate the minimum value m between the brightness Y and 255 - Y.

[0168] Step 14. Use m to scale the saturation s: first magnify the saturation s by m times, and then shrink it by 255 times. The scaled s is the value of the saturation in the HSY color space.

[0169] Step 15. Take the hue h of each pixel obtained above as the hue H in the HSY color space; the scaled saturation s as the saturation S in the HSY color space; and Y as the Y in the HSY color space to obtain the converted HSY pixel.

[0170] II. One-dimensional table building.

[0171] In this step, it is necessary to establish a one-dimensional <hue - transparency> table and a one-dimensional <saturation - transparency> table according to the background color hue range [h1, h2], the minimum saturation value s0, the hue gradient width r1, and the saturation gradient width r2.

[0172] The following will combineFigure 7 Describe the process of creating a one-dimensional <hue - transparency> table:

[0173] Step 21: Initialize a one-dimensional table with a length of 256.

[0174] Step 22: For each element in the table, calculate the lookup value according to the formula f(x) to obtain the one-dimensional <hue - transparency> table:

[0175] Denote the subscript of the element as i (i.e., the hue value of the pixel). Substitute i into f(x) using formula (1) to calculate the lookup value of this element. The formula f(x) will use the matte extraction parameters: the hue range of the background color [h1, h2], and the hue gradient width r1. Here, 0 represents transparent, and 255 represents opaque.

[0176] Among them, the specific definition of the formula f(x) is shown in formula (1).

[0177] As Figure 8 shown, establish the one-dimensional <hue - transparency> table through the formula f(x).

[0178] The following is combined with Figure 9 Describe the process of creating a one-dimensional <saturation - transparency> table:

[0179] Step 31: Initialize a one-dimensional table with a length of 256.

[0180] Step 32: For each element in the table, calculate the lookup value according to the formula g(x) to obtain the one-dimensional <saturation - transparency> table:

[0181] Denote the subscript of the element as i (i.e., the saturation of the pixel). Substitute i into g(x) using formula (2) to calculate the lookup value of this element. The formula g(x) will use the matte extraction parameters: the minimum saturation s1, and the saturation gradient width r2. Here, 0 represents transparent, and 255 represents opaque.

[0182] Among them, the specific definition of the formula g(x) is shown in formula (2).

[0183] As Figure 10 shown, establish the one-dimensional <saturation - transparency> table through the formula g(x).

[0184] III. Obtain the alpha channel by looking up the table.

[0185] As Figure 11 shown, after obtaining the two one-dimensional tables, it is possible to perform a lookup operation on each frame of the image (i.e., the image in the HSY color space, abbreviated as the HSY image) in real time to obtain the alpha channel value (i.e., transparency) of each pixel in the image (i.e., the HSY pixel to be processed).

[0186] For the HSY pixel to be processed, the following processing is performed to obtain the transparency a of each HSY pixel to be processed:

[0187] Step 41: Obtain the hue value h of the HSY pixel to be processed.

[0188] Step 42, look up the one-dimensional <hue-transparency> table with h as the subscript to obtain the transparency a1.

[0189] Step 43: Obtain the saturation s of the HSY pixel to be processed.

[0190] Step 44, use s as the subscript to look up the one-dimensional <saturation-transparency> table to obtain the transparency a2.

[0191] Step 45: Take the maximum value from transparency a1 and transparency a2 to obtain the transparency a of the HSY pixel to be processed.

[0192] It should be noted that, by combining the transparency a of all HSY pixels to be processed, a transparent channel image of the HSY image is obtained, that is, the transparent channel image of the RGB image to be cut.

[0193] It should be noted that by looking up the table twice and finally obtaining the transparency, the final result is as follows Figure 12A The gradient effect of transparency with hue h and saturation s is essentially a two-dimensional table. However, directly looking up a two-dimensional table is slow (it is necessary to build a two-dimensional table through matrix multiplication, which is time-consuming. When looking up a table, the index of a one-dimensional table is 0-255, and the index of a two-dimensional table is 0-65535, so the table lookup is slow), and it is not easy to implement the algorithm on a GPU shader (the shader is calculated for pixels and cannot perform matrix operations on a two-dimensional table). Therefore, you can Figure 12B Look up the table for h, and then pass Figure 12C For the s table lookup, taking the maximum value of the two table lookups can get a completely equivalent two-dimensional table lookup effect. This is the process of separate table lookup.

[0194] 4. Post-processing.

[0195] If the grayscale value of the green channel (G) of a pixel is greater than the average of the red and blue channels, the pixel will be greenish. Figure 13A As shown in , the grayscale value of the green channel (G) of the pixel of the black pants dyed green by the green screen is greater than the average value of the red channel and the blue channel; Figure 13B As shown, the grayscale value of the green channel (G) of the skin pixel that is not stained green by the green screen reflection is less than the average value of the red channel and the blue channel.

[0196] Therefore, as long as the green channel value of the green-colored pixels is reduced to the same value as the average of the red channel and the blue channel, it will not look greenish. Further, in order to adjust the intensity of green removal, a parameter t is added to control the calculation of the average of the red channel and the blue channel.

[0197] The following combines Figure 14 to illustrate the overall process of post-processing:

[0198] Step 51. Remove the greenish tint of the subject caused by the green screen reflection in the RGB channels: For each RGB pixel, perform the following processing:

[0199] a) Calculate the minimum and maximum values of the R channel and the B channel, denoted as j and k respectively.

[0200] b) Calculate d, where d = k - j.

[0201] c) Calculate the threshold T, where T = j + t·d, and t is the green removal coefficient, which can be set to 0.5.

[0202] d) Use the minimum value of T and the G channel as the G channel value after green removal.

[0203] e) Use the original B channel value as the B channel value after green removal, the original R channel value as the R channel value after green removal, and combine them with the G channel value after green removal to obtain the RGB pixel color after green removal.

[0204] Step 52. Perform post-processing on the alpha channel.

[0205] Perform morphological processing such as dilation, erosion, and blurring on the alpha channel image to obtain the post-processed alpha channel image.

[0206] Step 53. Combine the RGB pixel color after green removal and the post-processed alpha channel image to obtain the ARGB image.

[0207] It should be noted that the image used to calculate the alpha channel does not necessarily have to be in the HSY color space. For example, it can also be three background parameters and a distance coefficient in YUV.

[0208] As Figure 15A shown, the embodiment of the present application extracts the target object 1502 from the green screen image 1501; as Figure 15B shown, the embodiment of the present application extracts the target object 1504 from the QR code green screen image 1503.

[0209] In summary, the high-quality fast green screen matting algorithm based on separate look-up tables proposed in the embodiments of the present application has the following beneficial effects: 1) It avoids accelerating the two-dimensional look-up table and can be very conveniently implemented on the GPU shader. This matting algorithm can not only process 4K videos in real time, but also avoid excessive CPU and GPU occupancy, ensuring that when applied to virtual-real fusion, it will not preempt hardware resources with other algorithms required for virtual-real fusion, avoiding lags, improving the rendering image quality, and enhancing the user experience; 2) After being accelerated by the GPU shader, it can even process 5K or 8K videos, achieving a more high-definition image quality experience.

[0210] So far, the exemplary applications and implementations of the electronic device provided in the embodiments of the present application have been combined to illustrate the image processing method provided in the embodiments of the present application. The embodiments of the present application also provide an image processing device. In actual applications, each functional module in the image processing device can be jointly implemented by the hardware resources of an electronic device (such as a terminal, a server), such as computing resources of a processor, etc., communication resources (such as various communication methods used to support optical cables, cellular networks, etc.), and a memory. Figure 2 The image processing device 555 stored in the memory 550 is shown. It can be software in the form of programs and plugins, etc. For example, software modules designed in programming languages such as C / C++, Java, application software designed in programming languages such as C / C++, Java, or dedicated software modules, application programming interfaces, plugins, cloud services, etc. in large software systems. The following gives examples of different implementation methods.

[0211] Among them, the image processing device 555 includes a series of modules, including an acquisition module 5551, a construction module 5552, a query module 5553, and a processing module 5554. The following continues to describe how each module in the image processing device 555 provided in the embodiments of the present application cooperates to implement the image processing solution.

[0212] The acquisition module 5551 is used to acquire an image to be processed, where the image to be processed includes a background object and a target object; the construction module 5552 is used to construct a one-dimensional transparency look-up table of the matting parameters based on at least one matting parameter of the background object; the query module 5553 is used to query the one-dimensional transparency look-up table of the matting parameters for each pixel in the image to be processed to obtain a transparency channel image corresponding to the image to be processed; the processing module 5554 is used to perform matting processing based on the transparency channel image to obtain a target image that removes the background object and includes the target object.

[0213] In some embodiments, the building block 5552 is further configured to determine a matte range corresponding to the background object based on the matte parameters of the background object; wherein, the transparency corresponding to the color parameter values included in the matte range is completely transparent; based on the matte range corresponding to the background object, construct a one-dimensional transparency lookup table of the matte parameters, and the one-dimensional transparency lookup table includes the correspondence between the color parameter values and the transparency.

[0214] In some embodiments, when the matte parameter is a hue matte parameter, the matte range corresponding to the background object includes a hue matte range; when the matte parameter is a saturation matte parameter, the matte range corresponding to the background object includes a saturation matte range; the building block 5552 is further configured to determine a minimum hue value and a maximum hue value of the background object based on the hue matte parameter of the background object, and use the minimum hue value and the maximum hue value as the endpoint values of the hue matte range; determine a minimum saturation value and a maximum saturation value of the background object based on the saturation matte parameter of the background object, and use the minimum saturation value and the maximum saturation value as the endpoint values of the saturation matte range.

[0215] In some embodiments, the building block 5552 is further configured to determine an initial minimum hue value, an initial maximum hue value, and a hue gradient width of the background object based on the hue matte parameter of the background object; increase the initial minimum hue value by the hue gradient width to obtain the minimum hue value of the background object; decrease the initial maximum hue value by the hue gradient width to obtain the maximum hue value of the background object; determine an initial minimum saturation value, an initial maximum saturation value, and a saturation gradient width of the background object based on the saturation matte parameter of the background object; increase the initial minimum saturation value by the saturation gradient width to obtain the minimum saturation value of the background object, and use the initial maximum saturation value as the maximum saturation value of the background object.

[0216] In some embodiments, the building block 5552 is further configured to obtain the hue gradient width of the background object, and determine a hue gradient range based on the hue matte range and the hue gradient width, where the transparency corresponding to the color parameter values included in the hue gradient range is neither completely transparent nor completely opaque; construct a one-dimensional transparency lookup table of the hue matte parameters based on the hue matte range and the hue gradient range; obtain the saturation gradient width of the background object, and determine a saturation gradient range based on the saturation matte range and the saturation gradient width, where the transparency corresponding to the color parameter values included in the saturation gradient range is neither completely transparent nor completely opaque; construct a one-dimensional transparency lookup table of the saturation matte parameters based on the saturation matte range and the saturation gradient range.

[0217] In some embodiments, the query module 5553 is further configured to perform the following processing on any pixel in the image to be processed: query the one-dimensional transparency lookup table of the matte parameters based on the color parameters of the pixel to obtain the transparency of the pixel; combine the transparencies of multiple pixels according to the positional relationship of the pixels in the image to be processed to obtain the transparency channel image corresponding to the image to be processed.

[0218] In some embodiments, when there are multiple matte parameters, the query module 5553 is further configured to query the one-dimensional transparency lookup tables of the multiple matte parameters respectively based on the color parameters of the pixel to obtain the transparencies corresponding to the multiple matte parameters; use the maximum value of the transparencies corresponding to the multiple matte parameters as the transparency of the pixel.

[0219] In some embodiments, the image to be processed is a color image; before querying the one-dimensional transparency lookup table of the matte parameters based on the color parameter values of each pixel in the image to be processed to obtain the transparency channel image corresponding to the image to be processed, the building block 5552 is further configured to convert the image to be processed to a candidate color space to obtain a first converted image of the image to be processed in the candidate color space; convert the image to be processed to a black-and-white space to obtain a second converted image of the image to be processed in the black-and-white space; fuse the first converted image and the second converted image to obtain a third converted image corresponding to the image to be processed; the query module 5553 is further configured to query the one-dimensional transparency lookup table of the matte parameters based on the color parameters of each pixel in the third converted image to obtain the transparency channel image corresponding to the image to be processed.

[0220] In some embodiments, the color parameters of each pixel in the third converted image include a hue parameter, a saturation parameter, and a brightness parameter; the construction module 5552 is further configured to determine a saturation scaling factor based on the brightness parameter of each pixel in the second converted image; scale the saturation parameter of each pixel in the first converted image based on the saturation scaling factor to obtain the scaled saturation parameter; and combine the brightness parameter of each pixel in the second converted image, the scaled saturation parameter, and the hue parameter of each pixel in the first converted image to obtain the third converted image corresponding to the image to be processed.

[0221] In some embodiments, the processing module 5554 is further configured to perform morphological processing on the transparent channel image to obtain the transparent channel image after morphological processing; perform interference color removal processing on the image to be processed to obtain the image to be processed after interference color removal; and perform fusion processing on the transparent channel image after morphological processing and the image to be processed after interference color removal to obtain a target image that removes the background object and includes the target object.

[0222] In some embodiments, the pixel value of each pixel in the image to be processed includes a first color value, a second color value, and an interference color value; the processing module 5554 is further configured to perform the following processing on any pixel in the image to be processed: determine an interference color reference value of the pixel based on the first color value and the second color value of the pixel; combine the minimum value of the interference color of the pixel and the interference color reference value, the first color value of the pixel, and the second color value to obtain the pixel after interference color removal; and combine multiple pixels after interference color removal according to the positional relationship of the pixels in the image to be processed to obtain the image to be processed after interference color removal.

[0223] In some embodiments, the processing module 5554 is further configured to use the absolute value of the difference between the first color value and the second color value of the pixel as the difference between the first color value and the second color value; determine the minimum value of the first color value and the second color value of the pixel; and use the sum of the product of the difference and the interference color coefficient and the minimum value as the interference color reference value of the pixel.

[0224] In summary, the image processing device proposed in the embodiments of the present application can construct a one-dimensional transparency lookup table for each matte parameter through at least one matte parameter of the background object in the image to be processed, quickly query the one-dimensional transparency lookup table based on each pixel in the image to be processed, and perform matte processing on the image to be processed based on the transparent channel image corresponding to the image to be processed, so as to fully and effectively utilize the one-dimensional transparency lookup table to achieve fast matting, improve the matting efficiency, and save a large amount of computing resources.

[0225] An embodiment of the present application provides a computer program product, which stores executable instructions, and the executable instructions are stored in a computer-readable storage medium. A processor of an electronic device reads the executable instructions from the computer-readable storage medium, and the processor executes the executable instructions, so that the electronic device executes the image processing method described above in the embodiments of the present application.

[0226] An embodiment of the present application provides a computer-readable storage medium storing executable instructions, where the executable instructions are stored, and when the executable instructions are executed by a processor, the processor will be caused to execute the image processing method provided in the embodiments of the present application. For example, as Figures 3A - 3C the shown image processing method.

[0227] In some embodiments, the computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface memory, optical disc, or CD-ROM; or may be various devices including one or any combination of the above memories.

[0228] In some embodiments, the executable instructions may be in the form of a program, software, software module, script, or code, and may be written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including being deployed as an independent program or being deployed as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0229] As an example, the executable instructions may or may not correspond to a file in the file system, and may be stored as part of a file that stores other programs or data. For example, they may be stored in one or more scripts in a Hyper Text Markup Language (HTML) document, stored in a single file dedicated to the program being discussed, or stored in multiple cooperating files (for example, files storing one or more modules, subroutines, or code portions).

[0230] As an example, the executable instructions may be deployed to execute on one electronic device, or on multiple electronic devices located at one location, or on multiple electronic devices distributed at multiple locations and interconnected by a communication network.

[0231] As described above, the above are only embodiments of the present application and are not used to limit the protection scope of the present application. Any modifications, equivalent replacements, and improvements made within the spirit and scope of the present application are all included in the protection scope of the present application.

Claims

1. An image processing method, characterized in that, The method includes: Obtaining an image to be processed, where the image to be processed includes a background object and a target object; Based on at least one matte parameter of the background object, constructing a one-dimensional transparency lookup table for each matte parameter; where the one-dimensional transparency lookup table includes a one-dimensional correspondence between the color parameter value of a pixel and transparency; Querying the one-dimensional transparency lookup table of the matte parameter based on each pixel in the image to be processed to obtain a transparency channel image corresponding to the image to be processed; Performing matte processing based on the transparency channel image to obtain a target image that removes the background object and includes the target object.

2. The method according to claim 1, wherein The constructing a one-dimensional transparency lookup table for each matte parameter based on at least one matte parameter of the background object includes: Based on the matte parameter of the background object, determining a matte range corresponding to the background object; Wherein, the transparency corresponding to the color parameter value included in the matte range is completely transparent; Based on the matte range corresponding to the background object, constructing a one-dimensional transparency lookup table for each matte parameter.

3. The method according to claim 2, wherein When the matte parameter is a hue matte parameter, the matte range corresponding to the background object includes a hue matte range; When the matte parameter is a saturation matte parameter, the matte range corresponding to the background object includes a saturation matte range; The determining a matte range corresponding to the background object based on the matte parameter of the background object includes: Based on the hue matte parameter of the background object, determining a minimum hue value and a maximum hue value of the background object, and using the minimum hue value and the maximum hue value as the end point values of the hue matte range; Based on the saturation matte parameter of the background object, determining a minimum saturation value and a maximum saturation value of the background object, and using the minimum saturation value and the maximum saturation value as the end point values of the saturation matte range.

4. The method according to claim 3, wherein The determining a minimum hue value and a maximum hue value of the background object based on the hue matte parameter of the background object includes: Based on the hue matte parameter of the background object, determining an initial minimum hue value, an initial maximum hue value, and a hue gradient width of the background object; Increasing the initial minimum hue value based on the hue gradient width to obtain the minimum hue value of the background object; Decreasing the initial maximum hue value based on the hue gradient width to obtain the maximum hue value of the background object; The determining a minimum saturation value and a maximum saturation value of the background object based on the saturation matte parameter of the background object includes: Based on the saturation matte parameter of the background object, determining an initial minimum saturation value, an initial maximum saturation value, and a saturation gradient width of the background object; Increasing the initial minimum saturation value based on the saturation gradient width to obtain the minimum saturation value of the background object, and using the initial maximum saturation value as the maximum saturation value of the background object.

5. The method according to claim 3, wherein Constructing a one-dimensional transparency lookup table for each of the matte parameters based on the matte range corresponding to the background object, includes: Obtaining the hue gradient width of the background object, and determining a hue gradient range based on the hue matte range and the hue gradient width, wherein the transparency corresponding to the color parameter values included in the hue gradient range is neither completely transparent nor completely opaque; Constructing a one-dimensional transparency lookup table for the hue matte parameter based on the hue matte range and the hue gradient range; Obtaining the saturation gradient width of the background object, and determining a saturation gradient range based on the saturation matte range and the saturation gradient width, wherein the transparency corresponding to the color parameter values included in the saturation gradient range is neither completely transparent nor completely opaque; Constructing a one-dimensional transparency lookup table for the saturation matte parameter based on the saturation matte range and the saturation gradient range.

6. The method according to claim 1, wherein The obtaining the alpha channel image corresponding to the image to be processed by querying the one-dimensional transparency lookup table of the matte parameter for each pixel in the image to be processed, includes: Performing the following processing on any pixel in the image to be processed: Querying the one-dimensional transparency lookup table of the matte parameter based on the color parameter of the pixel to obtain the transparency of the pixel; Combining the transparencies of multiple pixels according to the positional relationship of the pixels in the image to be processed to obtain the alpha channel image corresponding to the image to be processed.

7. The method according to claim 6, wherein When there are multiple matte parameters, the querying the one-dimensional transparency lookup table of the matte parameter based on the color parameter of the pixel to obtain the transparency of the pixel, includes: Querying the one-dimensional transparency lookup tables of multiple matte parameters respectively based on the color parameter of the pixel to obtain the transparencies corresponding to the multiple matte parameters; Taking the maximum value of the transparencies corresponding to the multiple matte parameters as the transparency of the pixel.

8. The method according to claim 1, wherein the image to be processed is a color image; before the obtaining the alpha channel image corresponding to the image to be processed by querying the one-dimensional transparency lookup table of the matte parameter for each pixel color parameter value in the image to be processed, the method further includes: Converting the image to be processed to a candidate color space to obtain a first converted image of the image to be processed in the candidate color space; Converting the image to be processed to a black-and-white space to obtain a second converted image of the image to be processed in the black-and-white space; Fusing the first converted image and the second converted image to obtain a third converted image corresponding to the image to be processed; The obtaining the alpha channel image corresponding to the image to be processed by querying the one-dimensional transparency lookup table of the matte parameter for each pixel in the image to be processed, includes: Querying the one-dimensional transparency lookup table of the matte parameter based on the color parameter of each pixel in the third converted image to obtain the alpha channel image corresponding to the image to be processed.

9. The method according to claim 8, wherein The color parameters of each pixel in the third transformed image include a hue parameter, a saturation parameter, and a brightness parameter; The fusing the first transformed image and the second transformed image to obtain a third transformed image corresponding to the image to be processed includes: Determining a saturation scaling factor based on the brightness parameter of each pixel in the second transformed image; Scaling the saturation parameter of each pixel in the first transformed image based on the saturation scaling factor to obtain the scaled saturation parameter; Combining the brightness parameter of each pixel in the second transformed image, the scaled saturation parameter, and the hue parameter of each pixel in the first transformed image to obtain a third transformed image corresponding to the image to be processed.

10. The method according to claim 1, wherein The performing a matting process based on the transparency channel image to obtain a target image removing the background object and including the target object includes: Performing a morphological process on the transparency channel image to obtain the transparency channel image after the morphological process; Performing an interference color removal process on the image to be processed to obtain the image to be processed after the interference color removal; Performing a fusion process on the transparency channel image after the morphological process and the image to be processed after the interference color removal to obtain a target image removing the background object and including the target object.

11. The method according to claim 10, wherein The pixel value of each pixel in the image to be processed includes a first color value, a second color value, and an interference color value; The performing an interference color removal process on the image to be processed to obtain the image to be processed after the interference color removal includes: Performing the following process on any pixel in the image to be processed: Determining an interference color reference value of the pixel based on the first color value and the second color value of the pixel; Combining the minimum value between the interference color of the pixel and the interference color reference value of the pixel, the first color value of the pixel, and the second color value of the pixel to obtain the pixel after the interference color removal; Combining a plurality of pixels after the interference color removal according to the positional relationship of the pixels in the image to be processed to obtain the image to be processed after the interference color removal.

12. The method according to claim 11, characterized in that The determining an interference color reference value of the pixel based on the first color value and the second color value of the pixel includes: Taking the absolute value of the difference between the first color value and the second color value of the pixel as the difference between the first color value and the second color value; Determining the minimum value between the first color value and the second color value of the pixel; Taking the sum of the product of the difference and the interference color coefficient and the minimum value as the interference color reference value of the pixel.

13. An image processing apparatus, characterized in that, The apparatus includes: An obtaining module, configured to obtain an image to be processed, where the image to be processed includes a background object and a target object; A constructing module, configured to construct a one-dimensional transparency query table for each of the matting parameters based on at least one matting parameter of the background object; wherein the one-dimensional transparency query table includes a one-dimensional corresponding relationship between a color parameter value of a pixel and transparency; A query module, configured to query a one-dimensional transparency query table of the matte parameters based on each pixel in the image to be processed, and obtain a transparency channel image corresponding to the image to be processed; A processing module, configured to perform matte processing based on the transparency channel image, and obtain a target image that removes the background object and includes the target object.

14. An electronic device, characterized in that, The electronic device includes: A memory, configured to store executable instructions; A processor, configured to implement the image processing method according to any one of claims 1 to 12 when executing the executable instructions stored in the memory.

15. A computer-readable storage medium, characterized in that, Stored with executable instructions, configured to implement the image processing method according to any one of claims 1 to 12 when being executed by a processor.

16. A computer program product comprising executable instructions, characterized in that, When the executable instructions are executed by a processor, the image processing method according to any one of claims 1 to 12 is implemented.

Images