Image processing method, device, equipment, storage medium and program product

Abstract

The present application provides an image processing method, device, electronic device, computer-readable storage medium and computer program product; the method includes: obtaining an image to be processed; performing position detection processing on the image to be processed to obtain the position coordinates of the visible graphic code and the position coordinates of the invisible graphic code; determining the cutout parameters of the visible graphic code in the image to be processed based on the position coordinates of the visible graphic code; performing parameter prediction processing on the invisible graphic code based on the cutout parameters of the visible graphic code in the image to be processed and the position coordinates of the invisible graphic code to obtain the cutout parameters of the invisible graphic code; performing cutout processing on the image to be processed based on the cutout parameters of the visible graphic code and the cutout parameters of the invisible graphic code to obtain a target image with the visible graphic code removed and including the target object. Through the present application, the cutout parameters of each graphic code can be fully and effectively adjusted to improve the cutout effect.

Description

Technical Field

[0001] The present application relates to computer technology, and in particular to an image processing method, device, electronic device, computer-readable storage medium and computer program product. Background Art

[0002] With the development of computer technology, cutouts have been widely used in virtual production, interactive games, virtual live broadcasts, etc. For example, in current virtual production, the target object is cut out from the original video, and the cutout target object is merged with the background in the source video to produce a new video that conforms to the shooting scene, which can avoid repeatedly building different scenes when shooting videos and reduce the cost of scene layout.

[0003] However, the cutout scheme in the related art mainly relies on the set fixed cutout parameters, so that the cutout is performed by the fixed cutout parameters. However, the cutout method of this scheme is too rigid, resulting in poor cutout effect. Summary of the invention

[0004] The embodiments of the present application provide an image processing method, device, electronic device, computer-readable storage medium and computer program product, which can fully and effectively utilize the cutout parameters of each graphic code to achieve accurate cutout and improve the cutout effect.

[0005] The technical solution of the embodiment of the present application is implemented as follows:

[0006] The present application provides an image processing method, including:

[0007] Acquire an image to be processed, wherein the image to be processed includes a visible graphic code and a target object that blocks an invisible graphic code;

[0008] Performing position detection processing on the image to be processed to obtain the position coordinates of the visible graphic code and the position coordinates of the invisible graphic code;

[0009] Determining the cutout parameters of the visible graphic code in the image to be processed based on the position coordinates of the visible graphic code;

[0010] Based on the cutout parameters of the visible graphic code in the image to be processed and the position coordinates of the invisible graphic code, parameter prediction processing is performed on the invisible graphic code to obtain the cutout parameters of the invisible graphic code;

[0011] Based on the cutout parameters of the visible graphic code and the cutout parameters of the invisible graphic code, the image to be processed is cutout to obtain a target image with the visible graphic code removed and including the target object.

[0012] The present application provides an image processing device, including:

[0013] An acquisition module, used for acquiring an image to be processed, wherein the image to be processed includes a visible graphic code and a target object that blocks an invisible graphic code;

[0014] A detection module, used for performing position detection processing on the image to be processed to obtain the position coordinates of the visible graphic code and the position coordinates of the invisible graphic code;

[0015] A determination module, used for determining the cutout parameters of the visible graphic code in the image to be processed based on the position coordinates of the visible graphic code;

[0016] A prediction module, configured to perform parameter prediction processing on the invisible graphic code based on the cutout parameters of the visible graphic code in the image to be processed and the position coordinates of the invisible graphic code, so as to obtain the cutout parameters of the invisible graphic code;

[0017] The cutout module is used to perform cutout processing on the image to be processed based on the cutout parameters of the visible graphic code and the cutout parameters of the invisible graphic code to obtain a target image with the visible graphic code removed and including the target object.

[0018] In the above technical solution, the detection module is further used to perform graphic code detection processing on the image to be processed to obtain the position coordinates of the visible graphic code in the image to be processed;

[0019] The invisible graphic code is processed for position prediction based on the position coordinates of the visible graphic code to obtain the position coordinates of the invisible graphic code.

[0020] In the above technical solution, the image to be processed includes a first graphic code panel and a second graphic code panel, the first graphic code panel includes at least one of the visible graphic codes and at least one of the invisible graphic codes, and the second graphic code includes at least one of the invisible graphic codes; the detection module is further used to determine the posture of the first graphic code panel based on the position coordinates of the visible graphic codes in the first graphic code panel;

[0021] Based on the three-dimensional coordinates of the invisible graphic code in the first graphic code panel and the posture of the first graphic code panel, the invisible graphic code in the first graphic code panel is projected and transformed to obtain the position coordinates of the invisible graphic code in the first graphic code panel;

[0022] Determine the posture of the second graphic code panel based on the posture of the first graphic code panel and the transformation matrix from the first graphic code panel to the second graphic code panel;

[0023] Based on the three-dimensional coordinates of the invisible graphic code in the second graphic code panel and the posture of the second graphic code panel, the invisible graphic code in the second graphic code panel is projected and transformed to obtain the position coordinates of the invisible graphic code in the second graphic code panel.

[0024] In the above technical solution, the determination module is further used to perform index construction processing on the image to be processed based on the position coordinates of the visible graphic code to obtain an index map of the visible graphic code, wherein the index map includes the index value of the visible graphic code;

[0025] Based on the color parameter value of each pixel in the image to be processed and the index value of the visible graphic code, the initial cutout parameters of the visible graphic code are updated to obtain the cutout parameters of the visible graphic code in the image to be processed.

[0026] In the above technical solution, the determination module is further used to obtain a single-channel image corresponding to the image to be processed, wherein the value of the pixel in the single-channel image is different from the index value;

[0027] The pixel values ​​of the area corresponding to the visible graphic code in the single-channel image are set as index values ​​of the visible graphic code to obtain an index map of the visible graphic code.

[0028] In the above technical solution, the determination module is further used to perform the following processing on any pixel in the image to be processed:

[0029] When the pixel value at the same position as the pixel in the index map is the index value, determining the color parameter value of the pixel;

[0030] Based on the color parameter value of the pixel, the initial cutout parameters of the visible graphic code corresponding to the pixel are updated to obtain the cutout parameters of the visible graphic code in the image to be processed.

[0031] In the above technical solution, the color characteristic value of the pixel includes a hue value and a saturation, and the initial cutout parameters include a minimum hue value, a maximum hue value, and a minimum saturation value; the determination module is further used to use the minimum hue value in the initial cutout parameters and the minimum value of the hue value of the pixel as the minimum hue value in the cutout parameters of the visible graphic code;

[0032] The maximum value of the hue in the initial cutout parameters and the maximum value of the hue values ​​of the pixels are used as the maximum value of the hue in the cutout parameters of the visible graphic code;

[0033] The minimum saturation value in the initial cutout parameters and the minimum saturation value in the pixel are used as the minimum saturation value in the cutout parameters of the visible graphic code.

[0034] In the above technical solution, the prediction module is further used to determine the invisible graphic code adjacent to the visible graphic code based on the position coordinates of the visible graphic code and the position coordinates of the invisible graphic code;

[0035] Based on the relationship between the cutout parameter equations of the visible graphic code and the adjacent invisible graphic code and the cutout parameter of the visible graphic code, the cutout parameter of the invisible graphic code is obtained.

[0036] In the above technical solution, the cutout module is further used to determine the transparent channel image corresponding to the image to be processed based on the cutout parameters of the visible graphic code and the cutout parameters of the invisible graphic code;

[0037] A cutout process is performed based on the transparent channel image to obtain a target image with the visible graphic code removed and including the target object.

[0038] In the above technical solution, the cutout module is further used to construct a transparency query table for each graphic code based on the cutout parameters of the visible graphic code and the cutout parameters of the invisible graphic code, and the graphic code is the visible graphic code or the invisible graphic code;

[0039] The transparency query table of each graphic code is queried based on the color parameter value of each pixel in the image to be processed to obtain a transparent channel image corresponding to the image to be processed.

[0040] In the above technical solution, the cutout parameters include the minimum hue value, the maximum hue value, and the minimum saturation value; the cutout module is also used to

[0041] Based on the minimum hue value and the maximum hue value in the cutout parameters, construct a corresponding hue curve in the transparency query table, wherein the hue curve includes a corresponding relationship between hue and transparency;

[0042] Based on the minimum saturation value in the cutout parameters, a saturation curve in the corresponding transparency query table is constructed, and the saturation curve includes a corresponding relationship between saturation and transparency.

[0043] In the above technical solution, when there is a gap between two adjacent graphic codes, before constructing the transparency query table of each graphic code based on the cutout parameters of the visible graphic code and the cutout parameters of the invisible graphic code, the cutout module is further used to determine the hue expansion radius and the saturation expansion range based on the gap;

[0044] The difference between the minimum hue value in the cutout parameter and the hue expansion radius is used as the updated minimum hue value;

[0045] The sum of the hue maximum value in the cutout parameters and the hue expansion radius is used as the updated hue maximum value;

[0046] The difference between the minimum saturation value in the cutout parameter and the saturation expansion range is used as the updated minimum saturation value;

[0047] Based on the updated minimum hue value, the updated maximum hue value, and the updated minimum saturation value, a corresponding transparency query table of the graphic code is constructed.

[0048] In the above technical solution, the cutout module is further used to perform the following processing on any pixel in the image to be processed:

[0049] Determine a transparency lookup table of the graphic code where the pixel is located;

[0050] Querying a transparency query table of the graphic code based on the color parameter value of the pixel to obtain the transparency of the pixel;

[0051] According to the positional relationship of pixels in the image to be processed, the transparency of a plurality of pixels is combined to obtain a transparent channel image corresponding to the image to be processed.

[0052] In the above technical solution, the cutout module is further used to perform morphological processing on the transparent channel image to obtain the transparent channel image after morphological processing;

[0053] Performing interference color removal processing on the image to be processed to obtain the image to be processed after the interference color is removed;

[0054] The transparent channel image after morphological processing and the image to be processed after interference color removal are fused to obtain a target image with the visible graphic code removed and including the target object.

[0055] An embodiment of the present application provides an electronic device for image processing, the electronic device comprising:

[0056] A memory for storing executable instructions;

[0057] The processor is used to implement the image processing method provided in the embodiment of the present application when executing the executable instructions stored in the memory.

[0058] An embodiment of the present application provides a computer-readable storage medium storing executable instructions for causing a processor to execute and implement the image processing method provided in the embodiment of the present application.

[0059] An embodiment of the present application provides a computer program product, including executable instructions, which, when executed by a processor, implement the image processing method provided in the embodiment of the present application.

[0060] The embodiments of the present application have the following beneficial effects:

[0061] By determining the cutout parameters of the visible graphic code and the position coordinates of the invisible graphic code, the cutout parameters of the invisible graphic code are predicted, and the processed image is cutout processed based on the cutout parameters of all graphic codes, so as to adaptively and accurately determine the cutout parameters of each graphic code, so as to fully and effectively utilize the cutout parameters of each graphic code to achieve precise cutout, improve the cutout effect, and save a lot of human resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] Figure 1A-1B is a schematic diagram of a QR code green screen provided in an embodiment of the present application;

[0063] Figure 2A is a schematic diagram of a QR code green screen provided in an embodiment of the present application;

[0064] Figure 2B is a schematic diagram of a small QR code provided in an embodiment of the present application;

[0065] Figure 2C is a schematic diagram of a large QR code provided in an embodiment of the present application;

[0066] Figure 3 is a schematic diagram of an application scenario of an image processing system provided in an embodiment of the present application;

[0067] Figure 4 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application;

[0068] Figure 5A-5C is a flowchart of an image processing method provided in an embodiment of the present application;

[0069] Fig. 6A is a schematic diagram of a virtual production scene provided in an embodiment of the present application;

[0070] Figure 6B is a schematic diagram of left and right 3D videos provided by the related art;

[0071] Figure 6C is a schematic diagram of a virtual scene provided in an embodiment of the present application;

[0072] Figure 7 It is a flowchart of a high-quality local adaptive real-time green screen cutout algorithm based on a QR code provided in an embodiment of the present application;

[0073] Figure 8is a schematic diagram of the color space (HSB) provided in an embodiment of the present application;

[0074] Fig. 9 is a schematic diagram of the color space (HSL) provided in an embodiment of the present application;

[0075] Fig.10 is a schematic diagram of the process of color space conversion provided by an embodiment of the present application;

[0076] Fig.11 It is a schematic diagram of the process of QR code detection provided in the embodiment of the present application;

[0077] Fig.12 It is a schematic diagram of the process of QR code panel calibration provided in an embodiment of the present application;

[0078] Figure 13A-13C It is a schematic diagram of the conversion of the QR code panel provided in the embodiment of the present application;

[0079] Fig.14 It is a schematic diagram of a process for calculating the coordinates of all corner points of a two-dimensional code provided in an embodiment of the present application;

[0080] Fig.15 It is a schematic diagram of a process for calculating local cutout parameters provided in an embodiment of the present application;

[0081] Fig.16 It is a schematic diagram of a visible two-dimensional code area index map provided in an embodiment of the present application;

[0082] Fig.17 It is a schematic diagram of the process of calculating the transparent channel provided in the embodiment of the present application;

[0083] Fig.18 is a schematic diagram of a two-dimensional code index map provided in an embodiment of the present application;

[0084] Fig.19A is a schematic diagram of a hue-transparency table provided in an embodiment of the present application;

[0085] Fig.19B is a schematic diagram of a saturation-transparency table provided in an embodiment of the present application;

[0086] Fig. 20A is a schematic diagram of pixels of black pants dyed green by a green screen provided by an embodiment of the present application;

[0087] Fig. 20B is a schematic diagram of a pixel of skin that is not stained green by green screen reflection provided by an embodiment of the present application;

[0088] Fig.21 It is a schematic diagram of the post-processing process provided by the embodiment of the present application;

[0089] Fig.22A is a schematic diagram of a cutout result provided in an embodiment of the present application;

[0090] Fig. 22B It is a schematic diagram of the cutout result provided by the related technology;

[0091] Fig.23A is a schematic diagram of a cutout result provided in an embodiment of the present application;

[0092] Fig. 23B It is a schematic diagram of the cutout result provided by the relevant technology. DETAILED DESCRIPTION

[0093] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings. The described embodiments should not be regarded as limiting the present application. All other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of this application.

[0094] In the following description, the terms "first\second" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0095] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0096] Before further describing the embodiments of the present application in detail, the nouns and terms involved in the embodiments of the present application are explained. The nouns and terms involved in the embodiments of the present application are subject to the following interpretations.

[0097] 1) Video: It is composed of a series of image frames, i.e., an image sequence. The smoothness of the video can be expressed by the number of frames transmitted per second (FPS). The more frames per second, the smoother the displayed action will be. FPS is a definition in the field of images, which refers to the number of pictures in an animation or video. Each image frame is a still image, and when played in sequence, a moving image is created. For example, 30FPS means that 30 "still images" will be played per second.

[0098] 2) Graphic code: including QR code and barcode, among which QR code is a black and white graphic that records data symbol information distributed in a plane (two-dimensional direction) according to a certain pattern using certain geometric figures. In the compilation of the code, the concept of "0" and "1" bit streams that constitute the internal logic basis of the computer is cleverly used, and several geometric shapes corresponding to binary are used to represent text numerical information. It is automatically read by image input devices or photoelectric scanning devices to realize automatic information processing. It has some common features of barcode technology: each code system has its own specific character set; each character occupies a certain width; it has certain verification functions, etc. It also has the function of automatically identifying information in different rows and processing graphic rotation change points.

[0099] Among them, the visible graphic code is the graphic code that can be seen by the human eye in the image to be processed; the invisible graphic code is the graphic code that exists in the real world but cannot be seen by the human eye in the image to be processed, that is, it is blocked by the target object and cannot be presented in the image to be processed.

[0100] 3) Cutout parameter: a parameter that determines whether a pixel in the image to be processed is a background. If a pixel in the image to be processed is a background, the background color corresponding to the pixel is the cutout parameter. For example, if there is no target object in the area corresponding to the visible graphic code on the image to be processed, the background color can be directly taken at the visible graphic code, and the background color of the visible graphic code is used as the cutout parameter of the visible graphic code.

[0101] 4) QR code green screen: Figure 1A As shown, a green screen 101 with a QR code printed on it. Figure 1B As shown, the green screen includes three surfaces: a QR code left plane 102, a QR code right plane 103, and a QR code bottom plane 104. All QR codes on the QR code green screen have unique patterns and numbers. Using the corresponding QR code detection algorithm, all unobstructed QR codes can be detected, and the corner coordinates of the QR codes on the image can be accurately obtained.

[0102] 5) Visible QR code: A QR code that can be correctly identified by the QR code detection algorithm, that is, a QR code that can be seen by the human eye in the image. It should be noted that when the QR code is blocked or part of the QR code exceeds the boundary of the image, the QR code detection algorithm will not be able to detect the QR code. Invisible QR code is a graphic code that cannot be seen by the human eye in the image, that is, it is blocked by the target object and cannot be presented in the image.

[0103] 6) Small QR code: Figure 2A As shown, the dark green area on the two-dimensional code green screen 201 forms a square 202. Figure 2B As shown, there are gaps 203 between the small two-dimensional codes. The corner points of the small two-dimensional code are the four vertices corresponding to the square where the small two-dimensional code is located.

[0104] 7) Large QR code: A rectangle that can divide the entire QR code green screen into multiple rectangular areas without leaving any gaps. Each rectangle (i.e. large QR code) has only one small QR code inside. The large QR code corresponds to the small QR code one by one. Figure 2C As shown, there are gaps between the large two-dimensional code 204. The corner points of the large two-dimensional code are the four vertices corresponding to the rectangle where the large two-dimensional code is located.

[0105] 8) Color parameters: parameter values ​​corresponding to each pixel in the color space. For example, in the RGB color space, red (R, Red), green (G, Green), and blue (B, Blue) all represent color parameters; in the RGBA color space, red (R, Red), green (G, Green), blue (B, Blue), and transparency (A, Alpha) all represent color parameters; in the HSY color space, hue (H, Hue), saturation (S, Saturation), and brightness (Y) all represent color parameters.

[0106] Among them, hue is the primary characteristic of color and the most accurate standard for distinguishing different colors. Under the irradiation of light of different wavelengths, the human eye will perceive different colors, such as blue, red, etc. The external characteristics of these colors are called hue.

[0107] Saturation, also known as "purity", refers to the vividness of a color. The higher the saturation, the purer the color and the brighter the color. When mixed with other colors, the saturation of the color will decrease, and the color will become darker and lighter. When the saturation of a color drops to the bottom, it will lose its hue and become achromatic (black, white, gray).

[0108] Brightness refers to the brightness of a color, and all colors have different degrees of brightness. Among achromatic colors, white has the highest brightness, gray is in the middle, and black is the darkest. It should be noted that changes in color brightness often affect purity. For example, when red is added with white, the brightness increases, but the purity decreases.

[0109] 9) Morphology: Extract image components that are meaningful for expressing and depicting the shape of the region from the image, so that the subsequent recognition work can grasp the most essential shape features of the target object, such as boundaries and connected areas. Morphology is the basic theory of mathematical morphological image processing, and its basic operations include: binary erosion and expansion, binary opening and closing operations, skeleton extraction, extreme erosion, hit-miss transformation, morphological gradient, Top-hat transformation, particle analysis, watershed transformation, gray value erosion and expansion, gray value opening and closing operations, gray value morphological gradient, etc.

[0110] The embodiments of the present application provide an image processing method, device, electronic device, computer-readable storage medium, and computer program product, which can fully and effectively adjust the cutout parameters of each graphic code to improve the cutout effect.

[0111] The image processing method provided in the embodiment of the present application can be implemented independently by a terminal or a server; it can also be implemented jointly by a terminal and a server, for example, the terminal independently undertakes the image processing method described below, or the terminal sends a cutout request for the image to be processed to the server, and the server executes the image processing method according to the received cutout request for the image to be processed, performs position detection processing on the image to be processed, obtains the position coordinates of the visible graphic code and the position coordinates of the invisible graphic code, determines the cutout parameters of the visible graphic code in the image to be processed based on the position coordinates of the visible graphic code, performs parameter prediction processing on the invisible graphic code based on the cutout parameters of the visible graphic code in the image to be processed and the position coordinates of the invisible graphic code, obtains the cutout parameters of the invisible graphic code, performs cutout processing on the image to be processed based on the cutout parameters of the visible graphic code and the cutout parameters of the invisible graphic code, obtains the target image with the visible graphic code removed and including the target object, and executes virtual production, interactive games, virtual live broadcast and other applications based on the target image.

[0112] The electronic device for image processing provided in the embodiments of the present application may be various types of terminals or servers, wherein the server may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms; the terminal may be a smart phone, tablet computer, laptop computer, desktop computer, smart speaker, smart watch, smart TV, smart car equipment, etc., but is not limited thereto. The terminal and the server may be directly or indirectly connected via wired or wireless communication, which is not limited in the present application.

[0113] See also Figure 3 , Figure 3 It is a schematic diagram of an application scenario of the image processing system 10 provided in an embodiment of the present application. The terminal 200 is connected to the server 100 via a network 300. The network 300 may be a wide area network or a local area network, or a combination of the two.

[0114] Terminal 200 (running a client, such as a cutout client) can be used to obtain a cutout request for an image to be processed. For example, when a user opens the client running on the terminal and inputs an image to be processed, the terminal automatically obtains a cutout request for the image to be processed (including the image to be processed for image processing).

[0115] In some embodiments, an image processing plug-in may be embedded in the client running in the terminal 200 to implement the image processing method locally on the client. For example, the terminal 200 calls the image processing plug-in to implement the image processing method, performs position detection processing on the image to be processed, obtains the position coordinates of the visible graphic code and the position coordinates of the invisible graphic code, determines the cutout parameters of the visible graphic code in the image to be processed based on the position coordinates of the visible graphic code, performs parameter prediction processing on the invisible graphic code based on the cutout parameters of the visible graphic code in the image to be processed and the position coordinates of the invisible graphic code, obtains the cutout parameters of the invisible graphic code, performs cutout processing on the image to be processed based on the cutout parameters of the visible graphic code and the cutout parameters of the invisible graphic code, obtains the target image with the visible graphic code removed and including the target object, so as to fully and effectively utilize the cutout parameters of each graphic code to achieve accurate cutout, improve the cutout effect, and perform virtual production, interactive games, virtual live broadcast and other applications based on the target image.

[0116] In some embodiments, after the terminal 200 obtains a cutout request for the image to be processed, the image processing interface of the server 100 (which can be provided in the form of a cloud service, i.e., an image processing service) is called. The server 100 implements an image processing method based on the cutout request for the image to be processed, performs position detection processing on the image to be processed, obtains the position coordinates of the visible graphic code and the position coordinates of the invisible graphic code, determines the cutout parameters of the visible graphic code in the image to be processed based on the position coordinates of the visible graphic code, performs parameter prediction processing on the invisible graphic code based on the cutout parameters of the visible graphic code in the image to be processed and the position coordinates of the invisible graphic code, obtains the cutout parameters of the invisible graphic code, performs cutout processing on the image to be processed based on the cutout parameters of the visible graphic code and the cutout parameters of the invisible graphic code, obtains the target image with the visible graphic code removed and including the target object, so as to fully and effectively utilize the cutout parameters of each graphic code to achieve precise cutout and improve the cutout effect, and execute virtual production, interactive games, virtual live broadcast and other applications based on the target image.

[0117] In some embodiments, the terminal or server can implement the image processing method provided by the embodiment of the present application by running a computer program (i.e., executable instructions). For example, the computer program can be a native program or software module in the operating system; it can be a native application (APP, Application), that is, a program that needs to be installed in the operating system to run, such as a video application (i.e., a video client running on the terminal); it can also be a small program, that is, a program that can be run only by downloading it to a browser environment; it can also be a small program that can be embedded in any APP. In short, the above-mentioned computer program can be an application, module or plug-in in any form.

[0118] In some embodiments, multiple servers may form a blockchain, and server 100 is a node on the blockchain. Information connections may exist between each node in the blockchain, and information may be transmitted between nodes through the above information connections. Among them, data related to the image processing method provided in the embodiment of the present application (such as the logic of image processing, target image) may be stored on the blockchain.

[0119] The structure of the electronic device provided by the embodiment of the present application is described below. Figure 4 , Figure 4 is a schematic diagram of the structure of an electronic device 500 provided in an embodiment of the present application, and takes the electronic device 500 as a server or a terminal as an example. Figure 4 The electronic device 500 shown includes: at least one processor 510, a memory 550, at least one network interface 520 and a user interface 530. The various components in the electronic device 500 are coupled together via a bus system 540. It is understood that the bus system 540 is used to achieve connection and communication between these components. In addition to the data bus, the bus system 540 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, the bus system 540 is not described in detail. Figure 4 Various buses are labeled as bus system 540 .

[0120] The processor 510 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., where the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0121] The memory 550 includes a volatile memory or a non-volatile memory, and may also include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), and the volatile memory may be a random access memory (RAM). The memory 550 described in the embodiment of the present application is intended to include any suitable type of memory. The memory 550 optionally includes one or more storage devices that are physically far away from the processor 510.

[0122] In some embodiments, the memory 550 can store data to support various operations, examples of which include programs, modules, and data structures, or a subset or superset thereof, as exemplarily described below.

[0123] Operating system 551, including system programs for processing various basic system services and performing hardware-related tasks, such as framework layer, core library layer, driver layer, etc., for implementing various basic services and processing hardware-based tasks;

[0124] A network communication module 552, used to reach other electronic devices via one or more (wired or wireless) network interfaces 520, exemplary network interfaces 520 include: Bluetooth, Wireless Compatibility Authentication (WiFi), and Universal Serial Bus (USB), etc.;

[0125] In some embodiments, the image processing device provided in the embodiments of the present application can be implemented in a software manner. Figure 4 The image processing device 555 stored in the memory 550 is shown, which can be software in the form of a program and a plug-in, etc., including the following software modules: an acquisition module 5551, a detection module 5552, a determination module 5553, a prediction module 5554, and a cutout module 5555. These modules are logical, and therefore can be arbitrarily combined or further split according to the functions implemented. The functions of each module will be described below.

[0126] As mentioned above, the video processing method provided in the embodiments of the present application can be implemented by various types of electronic devices. Figure 5A , Figure 5A is a flowchart of an image processing method provided in an embodiment of the present application, combined with Figure 5A The steps shown are explained.

[0127] In step 101, an image to be processed is obtained, wherein the image to be processed includes a visible graphic code and a target object that blocks an invisible graphic code.

[0128] It should be noted that the image to be processed is an image to be cut out, and the target object needs to be cut out from the image to be processed. The image to be processed can be an image obtained by photographing a QR code green screen, that is, the image to be processed includes at least one QR code panel, and the QR code panel includes at least one visible graphic code and a target object (also called a subject) that blocks the invisible graphic code. Among them, the graphic code included in the image to be processed can be a visible graphic code or an invisible graphic code. When the graphic code in the image to be processed is blocked, the graphic code detection algorithm will not be able to detect the graphic code.

[0129] Among them, the visible graphic code is the graphic code that can be seen by the human eye in the image to be processed; the invisible graphic code is the graphic code that exists in the real world but cannot be seen by the human eye in the image to be processed, that is, it is blocked by the target object and cannot be presented in the image to be processed.

[0130] In step 102, position detection processing is performed on the image to be processed to obtain the position coordinates of the visible graphic code and the position coordinates of the invisible graphic code.

[0131] For example, by performing position detection on the graphic codes in the image to be processed, the position coordinates of all the graphic codes in the image to be processed (i.e., the corner point coordinates of the graphic codes or the 2D coordinates of the graphic codes) are detected, so that the cutout parameters of all the graphic codes can be calculated based on the position coordinates of all the graphic codes.

[0132] See also Figure 5B , Figure 5B is a flowchart of an image processing method provided in an embodiment of the present application, Figure 5B Show Figure 5A Step 102 can be implemented through steps 1021-1022: in step 1021, a graphic code detection process is performed on the image to be processed to obtain the position coordinates of the visible graphic code in the image to be processed; based on the position coordinates of the visible graphic code, a position prediction process is performed on the invisible graphic code to obtain the position coordinates of the invisible graphic code.

[0133] It should be noted that visible graphic codes can be directly detected based on graphic code detection technology (such as a QR code detection algorithm, where each QR code corresponds to a serial number, and the QR code detection algorithm returns the QR code serial number and the corresponding corner point coordinates). If a QR code can be automatically detected, it means that there must be no subject in the area corresponding to the QR code on the image to be processed. For invisible graphic codes, the relative positions of the calibrated graphic codes are used, and the visible graphic codes are used to predict the position of the invisible graphic codes on the image to be processed (i.e., the position coordinates of the invisible graphic codes), thereby determining the position coordinates of all graphic codes in the image to be processed.

[0134] In some embodiments, the image to be processed includes a first graphic code panel and a second graphic code panel, the first graphic code panel includes at least one visible graphic code and at least one invisible graphic code, and the second graphic code includes at least one invisible graphic code; based on the position coordinates of the visible graphic code, the invisible graphic code is subjected to position prediction processing to obtain the position coordinates of the invisible graphic code, including: determining the posture of the first graphic code panel based on the position coordinates of the visible graphic code in the first graphic code panel; based on the three-dimensional coordinates of the invisible graphic code in the first graphic code panel and the posture of the first graphic code panel, the invisible graphic code in the first graphic code panel is subjected to projection transformation processing to obtain the position coordinates of the invisible graphic code in the first graphic code panel; based on the posture of the first graphic code panel and the transformation matrix from the first graphic code panel to the second graphic code panel, the posture of the second graphic code panel is determined; based on the three-dimensional coordinates of the invisible graphic code in the second graphic code panel and the posture of the second graphic code panel, the invisible graphic code in the second graphic code panel is subjected to projection transformation processing to obtain the position coordinates of the invisible graphic code in the second graphic code panel.

[0135] It should be noted that before predicting the position coordinates of the invisible graphic code by using the position coordinates of the visible graphic code, it is necessary to calibrate the relative positions between the graphic code panels in the image to be processed. For example, taking the first graphic code panel as a reference, calibrate the relative relationship of the second graphic code panel with respect to the first graphic code panel (i.e., the transformation matrix), so as to predict the position of the invisible graphic code on the image to be processed by using the calibrated relative positions between the graphic codes and the visible graphic code at the same time.

[0136] Among them, the first graphic code panel is simplified to panel 1, and the second graphic code panel is simplified to panel 2. The calibration process between panel 1 and panel 2 is as follows: through the graphic code detection algorithm, the 2D coordinates (i.e., corner point coordinates) of the graphic code in panel 1 and the 2D coordinates of the graphic code in panel 2 are detected; for each graphic code panel, the 2D coordinates of the visible graphic code, the camera intrinsic parameters, and the 3D coordinates of the visible graphic code are brought into SolvePNP to solve the posture of each graphic code panel, and the posture of panel 1 is recorded as P1, and the posture of panel 2 is recorded as P2, wherein the 3D coordinates of the graphic code are established with each graphic code as the center; using the method of calculating the transformation matrix between two postures, calculate the transformation matrix M12 from posture P1 to posture P2, and the transformation matrix M21 from posture P2 to posture P1.

[0137] Continuing with the above example, after calibrating the relative positions between the graphic code panels in the image to be processed, the coordinates of the intersections of all the graphic codes are calculated through the relative positions between the calibrated graphic codes. The calculation process is as follows: the 2D coordinates of the graphic code in panel 1 are calculated, and the 3D coordinates of the graphic code in panel 1 centered on panel 1, the posture of panel 1, and the internal parameters of the camera are brought into the projection transformation algorithm (the conversion relationship between the camera coordinates and the world coordinates) to obtain the 2D coordinates of the graphic code of panel 1 after projecting onto the imaging surface, so that the 2D coordinates of some invisible graphic codes in panel 1 can be supplemented; the 2D coordinates of the graphic code of panel 2 are inferred, and the transformation matrix M21 of panel 1 transformed to panel 2 is used to transform the posture of panel 2 centered on panel 2 to the posture of panel 2 centered on the camera, and the 3D coordinates of panel 2 centered on panel 2, the posture of panel 2 centered on the camera, and the internal parameters of the camera are brought into the projection transformation algorithm to obtain the 2D coordinates of the graphic code of panel 2 after projecting onto the imaging surface.

[0138] In step 103, based on the position coordinates of the visible graphic code, the cutout parameters of the visible graphic code in the image to be processed are determined.

[0139] For example, for visible graphic codes, the background color is automatically taken from the image to be processed based on the graphic code detection results, and the cutout parameters are calculated. It should be noted that graphic codes can be automatically detected. If a graphic code can be automatically detected, it means that there must be no target object in the area corresponding to the graphic code on the image to be processed. The cutout parameters can be obtained by directly taking the background color here, so that automatic color selection can be achieved and the complexity of user parameter adjustment can be simplified.

[0140] See also Figure 5C , Figure 5C is a flowchart of an image processing method provided in an embodiment of the present application, Figure 5C Show Figure 5A Step 103 can be implemented through steps 1031-1032: in step 1031, the image to be processed is indexed and constructed based on the position coordinates of the visible graphic code to obtain an index map of the visible graphic code, wherein the index map includes the index value of the visible graphic code; in step 1032, the initial cutout parameters of the visible graphic code are updated based on the color parameter value of each pixel in the image to be processed and the index value of the visible graphic code to obtain the cutout parameters of the visible graphic code in the image to be processed.

[0141] For example, an index map of the visible graphic code is constructed based on the position coordinates of the visible graphic code, so that the cutout parameters of the visible graphic code can be calculated based on the color parameter value of each pixel in the image to be processed (i.e., background color) and the index value of the visible graphic code. This allows automatic color picking and simplifies the complexity of user parameter adjustment.

[0142] In some embodiments, an index construction process is performed on the image to be processed based on the position coordinates of the visible graphic code to obtain an index map of the visible graphic code, including: obtaining a single-channel image corresponding to the image to be processed, wherein the value of the pixel in the single-channel image is different from the index value; setting the pixel value of the area corresponding to the visible graphic code in the single-channel image to the index value of the visible graphic code to obtain an index map of the visible graphic code.

[0143] For example, initialize a single-channel image of integer type with the same width and height as the image to be processed, and fill all pixels with a value v, for example, v is -1 (as long as v is not the same as the index value of any graphic code). For each visible graphic code, draw a convex quadrilateral formed by the corner points of all visible graphic codes to construct an index map of the visible graphic code: for a graphic code with a subscript (index value) of i, draw a solid convex polygon with the corner points of the visible graphic code as four vertices, and the polygon fill color value is i, that is, the pixel value of the area corresponding to the visible graphic code in the single-channel image is set to the index value of the visible graphic code. Among them, the index value is used to uniquely identify the visible graphic code.

[0144] In some embodiments, based on the color parameter value of each pixel in the image to be processed and the index value of the visible graphic code, the initial cutout parameters of the visible graphic code are updated to obtain the cutout parameters of the visible graphic code in the image to be processed, including: performing the following processing for any pixel in the image to be processed: when the pixel value at the same position as the pixel in the index map is the index value, determining the color parameter value of the pixel; based on the color parameter value of the pixel, updating the initial cutout parameters of the visible graphic code corresponding to the pixel to obtain the cutout parameters of the visible graphic code in the image to be processed.

[0145] For example, for each pixel on the image to be processed, the following traversal is performed: the value at the same position of the pixel on the index map is taken out. If the value at the same position of the pixel is v, that is, the value at the same position of the pixel is different from any index value, no operation is performed and it is skipped directly (indicating that this pixel does not belong to any graphic code); if the value at the same position of the pixel is not v, that is, the value at the same position of the pixel is the index value, then the cutout parameters of the graphic code to which the current pixel belongs need to be updated according to the color parameter value of the pixel.

[0146] In some embodiments, the color characteristic values ​​of a pixel include a hue value and a saturation, and the initial cutout parameters include a minimum hue value, a maximum hue value, and a minimum saturation value; based on the color parameter values ​​of the pixel, the initial cutout parameters of the visible graphic code corresponding to the pixel are updated to obtain the cutout parameters of the visible graphic code in the image to be processed, including: using the minimum hue value in the initial cutout parameters and the minimum value of the hue value of the pixel as the minimum hue value in the cutout parameters of the visible graphic code; using the maximum hue value in the initial cutout parameters and the maximum value of the hue value of the pixel as the maximum hue value in the cutout parameters of the visible graphic code; using the minimum saturation value in the initial cutout parameters and the minimum saturation value of the pixel as the minimum saturation value in the cutout parameters of the visible graphic code.

[0147] For example, when it is necessary to update the cutout parameters of the graphic code to which the current pixel belongs according to the color parameter value of the pixel, the cutout parameters h1i, h2i, and s1i corresponding to the graphic code to which the current pixel belongs (for example, the i-th graphic code) are updated according to the hue h and saturation s of the pixel. The specific updating method is: h1i is updated to min(h1i,h), h2i is updated to max(h2i,h), and s1i is updated to min(s1i,s).

[0148] In step 104, based on the cutout parameters of the visible graphic code in the image to be processed and the position coordinates of the invisible graphic code, parameter prediction processing is performed on the invisible graphic code to obtain the cutout parameters of the invisible graphic code.

[0149] It should be noted that the color of the middle graphic code is equal to the average color of the four adjacent graphic codes above, below, left and right, that is, the spatial relative relationship of the graphic codes, so the cutout parameters also satisfy this relationship. For invisible graphic codes, the cutout parameters of the invisible graphic codes can be calculated according to the spatial relative relationship of the graphic codes (that is, the position coordinates of the invisible graphic codes), thereby determining the cutout parameters of all graphic codes on the image to be processed.

[0150] In some embodiments, based on the cutout parameters of the visible graphic code in the image to be processed and the position coordinates of the invisible graphic code, parameter prediction processing is performed on the invisible graphic code to obtain the cutout parameters of the invisible graphic code, including: determining the invisible graphic code adjacent to the visible graphic code based on the position coordinates of the visible graphic code and the position coordinates of the invisible graphic code; obtaining the cutout parameters of the invisible graphic code based on the cutout parameter equation relationship between the visible graphic code and the adjacent invisible graphic code, and the cutout parameters of the visible graphic code.

[0151] For example, assuming that in space, with the i-th graphic code as the center, the four graphic codes above, below, left and right are j, k, m, and n respectively, then the cutout parameter of the i-th graphic code is considered to be equal to the average of the cutout parameters of these four graphic codes, that is, 4xj -x j -x k -x m -x n =0, where Indicates the minimum hue value of the i-th graphic code (visible graphic code), Indicates the maximum hue of the i-th graphic code. Indicates the minimum saturation value of the i-th graphic code.

[0152] Since there is at least one visible graphic code on the image to be processed, the entire equation must be a non-homogeneous positive definite equation group. By solving the equation group, the cutout parameters corresponding to all invisible graphic codes can be obtained.

[0153] In step 105, based on the cutout parameters of the visible graphic code and the cutout parameters of the invisible graphic code, the image to be processed is cutout to obtain a target image including the target object and without the visible graphic code.

[0154] For example, the image to be processed is cut out based on the cutout parameters of all graphic codes to adaptively and accurately determine the cutout parameters of each graphic code, thereby fully and effectively utilizing the cutout parameters of each graphic code to achieve precise cutout and improve the cutout effect.

[0155] In some embodiments, based on the cutout parameters of the visible graphic code and the cutout parameters of the invisible graphic code, a cutout process is performed on the image to be processed to obtain a target image with the visible graphic code removed and including the target object, including: based on the cutout parameters of the visible graphic code and the cutout parameters of the invisible graphic code, a transparent channel image corresponding to the image to be processed is determined; based on the transparent channel image, a cutout process is performed to obtain a target image with the visible graphic code removed and including the target object.

[0156] For example, by comparing the cutout parameters of each pixel on the image to be processed with those of all graphic codes, the transparency of each pixel is obtained, and the transparency of all pixels is combined to obtain a transparent channel image corresponding to the image to be processed. The image to be processed is then cutout based on the transparent channel image to obtain a target image with the visible graphic code removed and including the target object.

[0157] In some embodiments, based on the cutout parameters of the visible graphic code and the cutout parameters of the invisible graphic code, a transparent channel image corresponding to the image to be processed is determined, including: constructing a transparency query table for each graphic code based on the cutout parameters of the visible graphic code and the cutout parameters of the invisible graphic code, where the graphic code is a visible graphic code or an invisible graphic code; querying the transparency query table for each graphic code based on the color parameter value of each pixel in the image to be processed to obtain the transparent channel image corresponding to the image to be processed.

[0158] For example, by establishing a transparency query table (also called a separate lookup table) for each graphic code, the transparency query table includes the correspondence between color parameter values ​​and transparency, and then based on the color parameter value of each pixel in the image to be processed, the transparency query table of the graphic code where the pixel is located is queried to obtain the transparency of each pixel, and the transparency of all pixels is combined to obtain a transparent channel image corresponding to the image to be processed.

[0159] In some embodiments, the cutout parameters include a minimum hue, a maximum hue, and a minimum saturation; based on the cutout parameters of the visible graphic code and the cutout parameters of the invisible graphic code, a transparency query table for each graphic code is constructed, including: based on the minimum hue and the maximum hue in the cutout parameters, a hue curve in the corresponding transparency query table is constructed, and the hue curve includes the corresponding relationship between hue and transparency; based on the minimum saturation in the cutout parameters, a saturation curve in the corresponding transparency query table is constructed, and the saturation curve includes the corresponding relationship between saturation and transparency.

[0160] For example, Fig.19A As shown, a hue-transparency table (i.e., a hue curve in a transparency query table) is established based on the minimum hue value and the maximum hue value in the cutout parameters, wherein, as the subscript (hue) changes, when the hue is in the hue range [h1, h2] that needs to be subtracted (h1 corresponds to the minimum hue value in the cutout parameters, and h2 corresponds to the maximum hue value in the cutout parameters), the transparency is set to transparent (i.e., 0), and other areas are opaque (i.e., 255), with a gradient effect in the middle, and the gradient hue width is r1.

[0161] For example, Fig.19B As shown, a saturation-transparency table (i.e., a saturation curve in a transparency query table) is established based on the minimum saturation value in the cutout parameters, wherein, as the subscript (saturation) changes, when the saturation is greater than a given threshold s1 (corresponding to the minimum saturation value in the cutout parameters), the transparency is set to transparent (i.e., 0), and other areas are opaque (i.e., 255), with a gradient effect in the middle, and the gradient saturation width is r2.

[0162] In some embodiments, when there is a gap between two adjacent graphic codes, before constructing a transparency query table for each graphic code based on the cutout parameters of the visible graphic code and the cutout parameters of the invisible graphic code, the hue expansion radius and the saturation expansion range are determined based on the gap; the difference between the minimum hue value in the cutout parameters and the hue expansion radius is used as the updated minimum hue value; the sum of the maximum hue value in the cutout parameters and the hue expansion radius is used as the updated maximum hue value; the difference between the minimum saturation value in the cutout parameters and the saturation expansion range is used as the updated minimum saturation value; based on the cutout parameters of the visible graphic code and the cutout parameters of the invisible graphic code, a transparency query table for each graphic code is constructed, including: constructing a transparency query table for the corresponding graphic code based on the updated minimum hue value, the updated maximum hue value, and the updated minimum saturation value.

[0163] For example, Figure 2B As shown, when the graphic code is a small QR code, there is a gap between two adjacent QR codes, such as Figure 2C As shown, each small QR code corresponds to a large QR code. There is no gap between the large QR codes, so the green screen area on the image to be processed can be densely covered, thereby ensuring that in the subsequent cutout process, all pixels in the green screen area can be indexed into a QR code, thereby finding the cutout parameters of the corresponding QR code.

[0164] In order to ensure a flawless cutout process (to avoid cutout errors when converting small QR codes to large QR codes), the cutout parameters of all small QR codes need to be corrected. Specifically, the correction method is as follows: let the hue expansion radius in the hyperparameters be he and the saturation expansion range be se. Subtract he from the minimum hue value of each set of cutout parameters to obtain the updated minimum hue value, add he to the maximum hue value to obtain the updated maximum hue value, and subtract se from the minimum saturation value to obtain the updated minimum saturation value. Based on the updated minimum hue value, the updated maximum hue value and the updated minimum saturation value, a transparency query table for the corresponding large QR code is constructed to ensure that in the subsequent cutout process, all pixels in the large QR code can index into a QR code, thereby finding the cutout parameters of the corresponding QR code.

[0165] In some embodiments, a transparency query table of each graphic code is queried based on the color parameter value of each pixel in the image to be processed to obtain a transparent channel image corresponding to the image to be processed, including: performing the following processing for any pixel in the image to be processed: determining the transparency query table of the graphic code where the pixel is located; querying the transparency query table of the graphic code where the pixel is located based on the color parameter value of the pixel to obtain the transparency of the pixel; combining the transparencies of multiple pixels according to the positional relationship of the pixels in the image to be processed to obtain the transparent channel image corresponding to the image to be processed.

[0166] In some embodiments, a cutout process is performed based on the transparent channel image to obtain a target image with visible graphic codes removed and including a target object, including: performing morphological processing on the transparent channel image to obtain a transparent channel image after morphological processing; performing interference color removal processing on the image to be processed to obtain an image to be processed after the interference colors are removed; and performing fusion processing on the transparent channel image after morphological processing and the image to be processed after the interference colors are removed to obtain a target image with visible graphic codes removed and including the target object.

[0167] It should be noted that the interference color is the original color of the non-target object formed by the color reflection of the graphic code. For example, if the graphic code is green, it may cause some pixels on the image to be processed to be green. Therefore, it is necessary to perform green removal processing on the image to be processed to obtain the image to be processed after green removal.

[0168] The transparent channel image is subjected to morphological processing such as dilation, erosion, and blurring to obtain a transparent channel image after morphological processing, and the transparent channel image after morphological processing and the image to be processed after interference color removal are combined for fusion processing to obtain a target image with visible graphic codes removed and including the target object.

[0169] Taking the interference color as green as an example, if the grayscale value of the green channel (G) of a pixel is greater than the average value of the red channel and the blue channel, then the pixel will be greenish. Therefore, as long as the green channel value of the green-dyed pixel is reduced to the same value as the average value of the red channel and the blue channel, it will not look greenish. The green removal process is as follows: For each RGB pixel, the following processing is performed: Calculate the minimum and maximum values ​​of the R channel and the B channel, record them as j and k respectively; calculate d, d = kj; calculate the threshold T, T = j + t·d, where t is the green removal coefficient, which can be set to 0.5; use T and the minimum value of channel G as the G channel value after green removal, use the original B channel value as the B channel value after green removal, and the original R channel value as the R channel value after green removal, and combine them with the G channel value after green removal to obtain the RGB pixel color after green removal.

[0170] The following is an explanation of an exemplary application of an embodiment of the present application in a practical application scenario.

[0171] The embodiments of the present application can be applied to various cutout scenarios, such as virtual production, virtual reality (VR) video production, interactive games (allowing real people to interact with virtual scenes to provide immersive picture effects), virtual live broadcasts, etc.

[0172] Among them, virtual production technology is a technology that integrates real people into virtual scenes, such as Fig. 6AAs shown, the real person 601 is implanted into the virtual scene 602. Through virtual production, it is possible to avoid repeatedly building different scenes when shooting videos, reducing the cost of scene layout. In addition, due to the use of green screens, virtual scenes can be quickly changed and replaced, and the research and development efficiency is high. For virtual production, if a QR code green screen is used, the movement of the camera can be recognized through images, avoiding the use of hardware locators, thereby reducing the difficulty of development and hardware costs. Image-based camera recognition naturally carries positioning data to ensure that the shooting images are completely synchronized, and that the virtual and real are integrated and the camera movement effect is natural.

[0173] Regarding the production scenario of VR videos, due to the rise of the Metaverse and VR, video materials are needed to ensure that VR-related hardware and apps have content to play. Figure 6B As shown in the figure, the artificial intelligence (AI) technology for converting 2D to 3D can only convert a single-lens video into a left-right 3D video 603. The picture cannot rotate according to the user's rotation, and the best 3D180 / 3D360 video cannot be generated. The immersive feeling is average. For a real VR+360 video, the user can freely rotate the viewing angle while experiencing 3D, and get a realistic immersive experience. Therefore, the QR code green screen technology can be applied to VR video production to generate Figure 6C The realistic and immersive VR+3D+360 video shown in the figure integrates the real person 604 into the new virtual scene.

[0174] Regarding virtual live broadcast scenarios, by applying the QR code green screen-based cutout technology to virtual live broadcasts, it is possible to generate large-scale live events (such as concerts, sports meets, live events, etc.), program or video production (such as large-scale variety shows, documentaries, TV series, movies, personal short films, micro-movies, short videos), ordinary personal live broadcasts (such as selling goods, singing, dancing, games, etc.), and real-person interactive game live broadcasts (real people are integrated into the game to present virtual and real interactive game screens, and broadcast them live. Users can help or create difficulties for the anchor through barrages, rewards, and other operations to increase the fun).

[0175] In the related art, the cutout algorithm has various limitations. For example, the cutout effect of the main character is poor, resulting in jagged edges on the main body (i.e., the target object); some areas of the main body may be wrongly deducted, such as the area on the shoe that is dyed green due to the reflection of the green screen on the ground. Although there is a slight difference in color from the background green near the shoe, considering the different shades of green on the entire QR code green screen, as much green as possible will be deducted, resulting in the green area on the shoe being deducted; manual color selection is required, and the cutout quality is highly dependent on the set background color. If you want good cutout quality, you need to manually adjust the background color value. If the setting is not good, the cutout quality will be reduced at best, and the QR code green screen will not be deducted normally at worst; the non-green screen area cannot be automatically deducted.

[0176] In order to solve the above problems, the embodiment of the present application proposes a high-quality local adaptive real-time green screen cutout algorithm based on QR code (that is, the image processing method proposed in the embodiment of the present application). The algorithm can automatically set the background color according to the detected QR code, and adopt different cutout parameters for different regions according to the position of the QR code to improve the cutout quality and solve the above-mentioned problems such as poor edge quality, incorrect deduction of part of the main area, reliance on manual background color, and inability to automatically deduct non-green screen areas.

[0177] It should be noted that since the QR code green screen is used in the fusion of virtual and real, it can bring many conveniences and cost advantages to the fusion of virtual and real. The fusion of virtual and real is to implant real people into virtual scenes. In order to obtain high-quality virtual and real fusion effects, it is necessary to support camera movement. The QR code on the QR code green screen can provide accurate feature points for the real-person video, so that the camera movement information of the real-person video can be calculated through the computer vision algorithm. Otherwise, a hardware locator is needed to obtain the camera movement information of the real-person video, which consumes hardware costs, and a special algorithm is required to synchronize the time difference between the real-person video and the hardware locator, which affects the virtual and real fusion effect.

[0178] The high-quality local adaptive real-time green screen cutout algorithm based on QR codes proposed in the embodiment of the present application can divide the image into multiple local areas (areas where the QR codes are located) based on the boundary positions of the QR codes, so that each local area uses an independent set of cutout parameters to achieve local adaptive cutout and improve the cutout effect; since the detectable QR codes must be unobstructed, the observable QR codes can be automatically colored green based on the detection results of the QR codes, thereby simplifying the interaction and improving the ease of use; for invisible QR codes, their cutout parameters can be predicted based on the QR codes that can be seen around them, so as to ensure that each local area in the image uses an independent set of cutout parameters.

[0179] Among them, the key points of the high-quality local adaptive real-time green screen cutout algorithm based on QR code are:

[0180] 1) The image is divided into different local areas through QR codes, and each local area is adaptively cut out.

[0181] Since there are a large number of QR codes on the QR code green screen, these QR codes have divided the image into local areas (each large QR code divides the image into a separate local area). In this way, different cutout parameters can be used for each QR code area.

[0182] 2) For visible QR codes, the background color is automatically taken from the image and the cutout parameters are calculated based on the QR code detection results.

[0183] It should be noted that the QR code can be automatically detected. If a QR code can be automatically detected, it means that there must be no subject in the area corresponding to the QR code on the image. By directly taking the background color here, you can get the cutout parameters, so that automatic color selection can be achieved, simplifying the complexity of user parameter adjustment.

[0184] 3) For an obscured QR code, the cutout parameters of the obscured QR code can be calculated based on the spatial relative relationship of the QR code.

[0185] It should be noted that the color of the middle QR code is equal to the average color of the four adjacent QR codes above, below, left and right, that is, the spatial relative relationship of the QR codes, so the cutout parameters also satisfy this relationship. Therefore, the cutout parameters of the undetectable QR code can be predicted based on the cutout parameters of the detectable QR code.

[0186] Combine the following Figure 7 The high-quality local adaptive real-time green screen cutout algorithm based on QR code proposed in the embodiment of the present application includes five parts: HSY color space conversion, QR code detection, solving local cutout parameters, calculating transparent channel, and post-processing. The following describes these five parts in detail:

[0187] 1. HSY color space conversion.

[0188] For the separation lookup table green screen cutout algorithm, Figure 8 The image needs to be cut out in the color space (HSB) shown in the image, so the image needs to be converted to this color space first. Figure 8 A biconical color space.

[0189] However, HSB (also known as HSV) has various problems when it comes to engineering: for tools such as opencv and UE, the HSV color space defined internally is ambiguous, so direct use cannot produce a good cutout effect.

[0190] For most tools, such as Fig. 9 The color space (HSL) shown has no ambiguity in the definition of various tools, so the image can be converted to the above-mentioned biconical color space by post-processing the HSL color space.

[0191] By using the brightness L to reduce the saturation of colors close to white or black, HSL can be converted to HSB, that is, S 缩放后 =S 缩放前 *min(L,(255-L)) / 127.5. Replacing L with Y in YUV can achieve a higher cutout effect, so when scaling S, the Y value is used instead of the L value.

[0192] For each pixel on the RGB image to be processed, the following processing is performed to convert the image in the HSY color space. Fig.10 Detailed description of the conversion process:

[0193] Step 11. Use the RGB to HLS algorithm (such as RGB to HLS in opencv) to obtain the hue h and saturation s of each pixel.

[0194] Step 12. Use the RGB to black and white image algorithm to obtain the brightness Y of each pixel.

[0195] Step 13. Calculate the minimum value m between brightness Y and 255-Y.

[0196] Step 14. Use m to scale the saturation s: first enlarge the saturation s by m times, and then reduce it by 255 times. The scaled s is the saturation value in the HSY color space.

[0197] Step 15: The hue h of each pixel obtained above is used as the hue H in the HSY color space; the scaled saturation s is used as the saturation S in the HSY color space; and Y is used as the Y in the HSY color space to obtain the converted HSY pixel.

[0198] 2. QR code detection.

[0199] In this step, it is necessary to call the QR code detection algorithm to detect the QR code in real time and calibrate the relative positions between the QR codes. At the same time, the visible QR code (i.e., the visible QR code) is used to predict the position of the invisible QR code (i.e., the invisible QR code) on the image, so as to facilitate the subsequent calculation of the local cutout parameters. It should be noted that the QR code corner points used in QR code detection are all small QR code corner points.

[0200] Combine the following Fig.11 Describe the overall QR code detection process:

[0201] Step 21, QR code detection: For the image to be processed, use a QR code detection algorithm (such as the QR code detection algorithm in opencv, each QR code corresponds to a serial number, and the QR code detection algorithm returns the QR code serial number and the corresponding corner point coordinates) to detect all visible QR codes on the image and obtain the corner point coordinates of all visible QR codes.

[0202] Step 22, calibrate the relative position of the QR code panel: take the ground QR code panel as a reference, and calibrate the relative relationship of other QR code panels with respect to the ground QR code panel.

[0203] It should be noted that if the relative positions of two QR code panels are to be calibrated, both QR code panels need to be visible in the current screen and both QR codes need to be detectable, that is, the relative positions of the QR code panels need to be calibrated based on the corner coordinates of the visible QR codes.

[0204] Combine the following Fig.12 Describe the specific calibration process:

[0205] Step 31, in the aforementioned two-dimensional code detection process, the 2D coordinates (pixel coordinates of the two-dimensional code in the image, in units of image pixels) of each visible two-dimensional code on the two two-dimensional code panels can be obtained.

[0206] like Fig.12 As shown, the 2D coordinates (ie, corner point coordinates) of the 2D code in panel 1 and the 2D coordinates of the 2D code in panel 2 are detected through the 2D code detection algorithm.

[0207] Step 32: For each QR code panel, the 2D coordinates of the visible QR code, the camera intrinsic parameters, and the 3D coordinates of the visible QR code are brought into SolvePNP to solve the posture of each QR code panel.

[0208] like Fig.12 As shown, the posture of panel 1 is recorded as P1, and the posture of panel 2 is recorded as P2. The 3D coordinates of the two-dimensional code are established with each two-dimensional code as the center.

[0209] Step 33: Calculate the transformation matrix M12 from posture P1 to posture P2, and the transformation matrix M21 from posture P2 to posture P1 by using the method of calculating the transformation matrix between two postures.

[0210] like Fig.13A As shown, a QR code panel can be rotated and translated to another position in 3D space through a transformation matrix.

[0211] It should be noted that although not every image frame can see two QR code panels at the same time (for example, only one can be seen), as long as there is an image frame that can be calibrated before, the previous calibration result can be reused (because the relative position of the QR code green screen is fixed in the real world).

[0212] For example, if Fig. 13B As shown in the figure, if we see two QR code panels B and C, we can calculate the relative relationship between B and C. Fig. 13CAs shown, if we see two QR code panels A and C, we can calculate the relative relationship between A and C. Invert the transformation matrix of A→C to get the transformation matrix of C→A. Then, by combining the transformation matrix of B→C and the transformation matrix of C→A, we can get the transformation matrix of B→A. In this way, the transformations of A→C, B→C, and B→A can be used in the following process of calculating the 2D coordinates of all QR codes.

[0213] Step 23, calculate the 2D coordinates of all QR codes: use the relative spatial positions of the visible QR code panel (i.e., the visible QR code panel, the invisible QR code panel (i.e., the invisible QR code panel) to the visible QR code panel and the corner point coordinates of the visible QR code to calculate the spatial positions of all QR codes, project them onto the camera imaging surface, and calculate the 2D coordinates of all QR codes.

[0214] Assume that QR code panel C is visible, record it as panel 1, QR code panel A as panel 2, and QR code panel B as panel 3. Given the transformation matrix of panel 1 relative to panel 2, calculate the projected 2D coordinates of the QR code of panel 2 on the camera imaging surface.

[0215] Combine the following Fig.14 The following is a flowchart illustrating the process of calculating the 2D coordinates of all QR codes:

[0216] Step 41 : Calculate the 2D coordinates of the QR code in panel 1 .

[0217] Among them, the 3D coordinates of the QR code in panel 1 centered on panel 1, the posture P1 of panel 1, and the internal parameters of the camera are brought into the projection transformation algorithm (the conversion relationship between the camera coordinates and the world coordinates) to obtain the 2D coordinates of the QR code in panel 1 after being projected onto the imaging surface. In this way, the 2D coordinates of some undetected QR codes can be completed.

[0218] Step 42 , calculating the 2D coordinates of the QR code in panel 2 .

[0219] First, the transformation matrix M21 from panel 2 to panel 1 is used to transform the posture of panel 2 centered on panel 2 to the posture of panel 2 centered on the camera.

[0220] Then, the 3D coordinates of the QR code in panel 2 centered on panel 2, the posture of panel 2 centered on the camera, and the internal parameters of the camera are introduced into the projection transformation algorithm to obtain the 2D coordinates of the QR code in panel 2 after being projected onto the imaging surface.

[0221] Similarly, the transformation relationship between panel 1 and panel 3 or between panel 2 and panel 3 can be used to calculate the 2D coordinates of the two-dimensional code in panel 3 after being projected onto the imaging surface.

[0222] Through this calculation method, as long as the relative position of the QR code panel (transformation matrix) is calibrated in some previous image frames and at least one QR code can be detected, the 2D coordinates of all QR codes on the image to be cut out (i.e., the RGB image to be cut out) can be calculated.

[0223] Step 24: Calculate local cutout parameters based on the inferred 2D coordinates of all the two-dimensional codes and the corner coordinates of the visible two-dimensional codes detected by the two-dimensional code detection technology.

[0224] The 2D coordinates of all the two-dimensional codes calculated in step 23 are marked with detected or inferred marks and transmitted to the subsequent local cutout parameter solution module.

[0225] 3. Solve the local cutout parameters.

[0226] After obtaining the coordinates of the corner points of all QR codes, as well as which QR codes are detected and which QR code corner points are inferred, the corresponding cutout parameters of each QR code can be solved. It should be noted that the QR code corner points used in solving the cutout parameters are all small QR code corner points.

[0227] Combine the following Fig.15 The processing flow of solving local cutout parameters is described in detail:

[0228] Step 51: Quickly and automatically detect the cutout parameters corresponding to the visible QR code.

[0229] a) According to the steps of the aforementioned two-dimensional code detection, the mark of the two-dimensional code can be obtained, the coordinates of the corner points of all visible two-dimensional codes can be taken out, and the corresponding two-dimensional code numbers can be recorded.

[0230] b) Create an index map of the visible QR code area by drawing:

[0231] ① Initialize an integer single-channel image with the same width and height as the image to be deducted, and fill all pixels with the value v, for example, v is -1 (as long as v is not the same as the subscript of any QR code).

[0232] ② Draw a convex quadrilateral formed by the corner points of all visible QR codes in a polygonal manner to construct an index map of the visible QR code area:

[0233] For each visible QR code, the following drawing is performed on the visible QR code area index map: for the QR code with subscript i, a solid convex polygon with the corner points of the visible QR code as four vertices is drawn, and the polygon fill color value is i.

[0234] In order to better understand, Fig.16As shown, a visible two-dimensional code area index map (ie, a small two-dimensional code index map) is constructed for the cutout image, and the color value of each convex polygon 1601 in the small two-dimensional code index map is different.

[0235] c) Initialize the parameters of all visible QR codes.

[0236] It should be noted that the subsequent separation table lookup requires three parameters related to the background color (minimum hue value h1, maximum hue value h2, and minimum saturation value s1). Therefore, for each visible QR code (subscript i), its minimum hue value H1i is initialized to 255, its maximum hue value H2i is initialized to 0, and its minimum saturation value s1i is initialized to 255.

[0237] d) For each pixel in the HSY image, perform the following traversal:

[0238] ① Take out the value at the same position of the pixel on the QR code area index map, and obtain the index i of the QR code to which the current pixel belongs.

[0239] ② If the value of the pixel at the same position is v (i.e. -1), do not do anything and skip it directly (indicating that this pixel does not belong to any QR code); if the value of the pixel at the same position is not v, then according to the hue h and saturation s of the pixel, update the cutout parameters h1i, h2i, s1i corresponding to the i-th QR code. The specific updating method is: h1i is updated to min(h1i, h), h2i is updated to max(h2i, h), and s1i is updated to min(sli, s).

[0240] Step 52: Calculate the cutout parameters of all QR codes based on the cutout parameters of the visible QR code:

[0241] a) Construct the cutout parameter equations between the QR codes:

[0242] ① Assuming that in space, the i-th QR code is the center, and the four QR codes above, below, left and right are QR codes j, k, m, and n respectively, then the cutout parameter of the i-th QR code is considered to be equal to the average of the cutout parameters of these four QR codes, as shown in equation (1):

[0243] 4x i -x j -x k -x m -x n =0 (1)

[0244] in, Indicates the minimum hue value of the i-th QR code. Indicates the maximum hue of the i-th QR code. Indicates the minimum saturation value of the i-th QR code.

[0245] It should be noted that if the i-th QR code is located in the boundary area, it is only adjacent to 2 or 3 QR codes in the four directions of up, down, left, and right. Then the equation group should also change accordingly. Taking the case of two adjacent QR codes as an example, as shown in equation (2):

[0246] 2x i -x j -x k =0 (2)

[0247] Take three parameters as an example, as shown in equation (3):

[0248] 3x i -x j -x k -x m =0 (3)

[0249] The equation established with the i-th QR code as the center is marked as equation i.

[0250] ② For each visible QR code, perform the following processing: denote its subscript as j, replace the cutout parameter of the j-th QR code in equation j in the previous step with x j =[h1j h2j s1j], where h1j is the minimum hue value of the j-th two-dimensional code, h2j is the maximum hue value of the j-th two-dimensional code, and s1j is the minimum saturation value of the j-th two-dimensional code.

[0251] ③ Because at least one QR code must be visible by default, the entire system of equations must be a non-homogeneous positive definite system of equations. By solving the system of equations, the corresponding cutout parameters of all QR codes can be obtained; when solving the equations, the system of equations can be written in the form of a matrix, that is, MX=B, M will be a sparse matrix, and by inverting the sparse matrix (this will speed up the calculation a lot), X can be calculated as: X=M -1 B, X are the solutions to the equations, i.e. the cutout parameters of all QR codes

[0252] 4. Calculate the transparent channel.

[0253] Combine the following Fig.17 The following describes a method for calculating the transparent channel of the image to be cut out using the cutout parameters of all QR codes, wherein the QR code corner points used in calculating the transparent channel are all large QR code corner points:

[0254] Step 61: Calculate the 2D coordinates of all large QR codes after they are projected onto the camera imaging surface based on the postures of each panel and the 3D coordinates of the corner points of the large QR codes (each of which establishes a coordinate system by itself).

[0255] It should be noted that the calculation method of the 2D coordinates of the large QR code is the same as the method of calculating the 2D coordinates of the small QR code. By projecting the large QR code onto the imaging surface, the green screen area can be divided into multiple areas without leaving any gaps, and each area can correspond to a QR code, so that the area cutout can be realized.

[0256] Step 62: Use a drawing method to draw a QR code index map based on the 2D coordinates of all large QR codes. The drawing method is the same as the method of drawing an index map based on small QR codes.

[0257] It should be noted that since the 2D coordinates of the large QR code are used for drawing here, the drawn graphics can densely cover the green screen area on the screen without leaving any gaps, thereby ensuring that in the subsequent cutout process, all pixels in the green screen area can be indexed into a QR code subscript to find their corresponding parameters; as for the non-green screen area, it can be directly ignored and set to be completely transparent.

[0258] In order to better understand, Fig.18 As shown, a two-dimensional code index map (i.e., a large two-dimensional code index map) is constructed for the cutout image, and the color value of each convex polygon 1801 in the large two-dimensional code index map is different.

[0259] Step 63, table creation: based on the calculated cutout parameters of all QR codes, multiple sets of separate lookup tables (the number of separate lookup tables corresponds to the number of QR codes) of one-dimensional tables (two in each set, i.e., a hue-transparency table and a saturation-transparency table) are created.

[0260] For example, Fig.19A As shown, a hue-transparency table f(x) is established, in which, as the subscript (hue) changes, when the hue is in the hue range [h1,h2] that needs to be subtracted (h1 corresponds to the minimum hue value in the cutout parameter, and h2 corresponds to the maximum hue value in the cutout parameter), the transparency is set to transparent (i.e. 0), and the other areas are opaque (i.e. 255), and a gradient effect is applied in the middle, and the gradient hue width is r1.

[0261] For example, Fig.19B As shown, a saturation-transparency table g(x) is established, in which, as the subscript (saturation) changes, when the saturation is greater than a given threshold s1 (corresponding to the minimum saturation value in the cutout parameter), the transparency is set to transparent (i.e. 0), and the other areas are opaque (i.e. 255), and a gradient effect is applied in the middle, and the gradient saturation width is r2.

[0262] It should be noted that, for the sake of speed, only the QR codes that appear on the imaging surface can be tabled. In order to ensure that the cutout algorithm is flawless (to avoid cutout errors when converting small QR codes to large QR codes), the cutout parameters of all QR codes need to be corrected. Specifically, the correction method is as follows:

[0263] a) Let the hue expansion radius in the hyper parameters be he and the saturation expansion range be se. For each set of cutout parameters, subtract he from the minimum hue value, add he to the maximum hue value, and subtract se from the minimum saturation value.

[0264] b) For the updated cutout parameters of each QR code, a table is created in a separate table lookup manner, and a separate table lookup table is created for each QR code.

[0265] Step 64: Calculate the transparent channel. For each pixel on the HSY image, perform the following processing to calculate the grayscale value of its transparent channel:

[0266] a) On the two-dimensional code index map, take the value at the same position as the current pixel to obtain the identifier (ID) of the two-dimensional code to which the pixel belongs.

[0267] b) If the value of the current pixel at the same position is equal to the aforementioned initialized value v, it means that the pixel is not within the green screen range and does not need to be processed. The transparency of the current pixel can be directly set to 0, and no subsequent operations will be performed on the pixel.

[0268] c) If the value at the same position of the current pixel is not equal to the aforementioned initialized value v, the corresponding separation lookup table (two one-dimensional tables, namely the hue-transparency table and the saturation-transparency table) is retrieved according to the ID of the QR code.

[0269] d) Using the hue h of the current pixel, look up the hue-transparency table to obtain the transparency a1.

[0270] e) Using the saturation s of the current pixel, look up the saturation-transparency table to obtain the transparency a2.

[0271] f) Take the maximum value of a1 and a2 as the transparency of the pixel.

[0272] Step 65: Combine the transparency grayscale values ​​of all pixels to obtain a transparent channel image.

[0273] 5. Post-processing.

[0274] If the grayscale value of the green channel (G) of a pixel is greater than the average of the red and blue channels, the pixel will be greenish. Fig. 20A As shown in , the grayscale value of the green channel (G) of the pixel of the black pants dyed green by the green screen is greater than the average value of the red channel and the blue channel; Fig. 20B As shown, the grayscale value of the green channel (G) of the skin pixel that is not stained green by the green screen reflection is less than the average value of the red channel and the blue channel.

[0275] Therefore, as long as the green channel value of the green-dyed pixel is reduced to the same value as the average value of the red and blue channels, it will not look greenish. Furthermore, in order to be able to adjust the intensity of de-greening, a parameter t is added to control the calculation of the average value of the red and blue channels.

[0276] Combine the following Fig.21 Describe the overall process of post-processing:

[0277] Step 71: Remove the green cast of the main body caused by the reflection of the green screen in the RGB channel: For each RGB pixel, perform the following processing:

[0278] a) Calculate the minimum and maximum values ​​of the R channel and the B channel, denoted as j and k respectively.

[0279] b) Calculate d, d=kj.

[0280] c) Calculate the threshold T, T = j + t·d, where t is the green removal coefficient and can be set to 0.5.

[0281] d) Take the minimum value of T and channel G as the G channel value after removing the green color.

[0282] e) The original B channel value is used as the B channel value after de-greening, the original R channel value is used as the R channel value after de-greening, and combined with the G channel value after de-greening to obtain the RGB pixel color after de-greening.

[0283] Step 72: Post-process the transparent channel.

[0284] The transparent channel image is subjected to morphological processing such as dilation, erosion, blurring, etc. to obtain the post-processed transparent channel image.

[0285] Step 73: Combine the RGB pixel colors after green removal and the transparent channel image after post-processing to obtain an ARGB image.

[0286] It should be noted that, for each QR code, all kinds of green screen cutout methods can be used (it does not have to be a separate lookup table green screen cutout), and the image used to calculate the transparent channel does not have to be in the HSY color space, for example, it can also be three background parameters and a distance coefficient in YUV.

[0287] pass Fig.22A as well as Fig. 22B It can be seen that the algorithm provided in the embodiment of the present application can automatically deduct the non-QR code area 2201, thereby reducing the difficulty of green screen site layout and reducing costs. Fig.23A as well as Fig. 23B It can be seen that the algorithm provided in the embodiment of the present application has smoother edges without jagged edges or black edges; the main area that is dyed green by the reflection of the green screen will not be mistakenly deducted; there is no need to manually select the background color, and the algorithm can automatically detect the value of the background color based on the QR code, which has a lower threshold for use. Among them, the algorithm provided in the embodiment of the present application processes a single-frame 1080P image, and a single-core central processing unit (CPU) takes 16ms to reach 60FPS.

[0288] In summary, the high-quality local adaptive real-time green screen cutout algorithm based on QR code proposed in the embodiment of the present application has the following beneficial effects: 1) The use threshold is lower and more convenient. The cutout parameters of the visible QR code can be automatically extracted according to the detection result of the QR code, reducing the manual adjustment of parameters; the boundaries outside the green screen can be automatically removed according to the QR code, and there is no need to manually set the mask and manually remove the non-green screen area; 2) The picture is more detailed and accurate, and all QR codes have separate cutout parameters to ensure that the picture effect is more delicate, the cutout edge is more accurate, the fingers are clearly visible, and the edges of the characters are not Residual black edges and local areas of the subject that are stained green by the background reflection will not be mistakenly deducted; for the QR code blocked behind the subject, its cutout parameters can be automatically predicted based on the QR codes detected around it; 3) It runs fast and can process 1080P60FSP video in real time. It can quickly calculate the cutout parameter prediction method of the blocked QR code. Using only a single-core CPU, it can also solve the cutout parameters of all QR codes within 0.3ms without introducing delay. The fast table lookup and table building algorithm can use different cutout parameters for pixels in different positions without affecting real-time performance.

[0289] So far, the image processing method provided by the embodiment of the present application has been described in combination with the exemplary application and implementation of the electronic device provided by the embodiment of the present application. The embodiment of the present application also provides an image processing device. In actual applications, each functional module in the image processing device can be implemented by hardware resources of electronic devices (such as terminals and servers), such as computing resources such as processors, communication resources (such as for supporting various communication methods such as optical cables and cellular), and memory. Figure 4 An image processing device 555 stored in a memory 550 is shown, which may be software in the form of programs and plug-ins, for example, software modules designed in programming languages ​​such as C / C++ and Java, application software designed in programming languages ​​such as C / C++ and Java, or dedicated software modules in large software systems, application programming interfaces, plug-ins, cloud services, and other implementation methods. Different implementation methods are described below with examples.

[0290] The image processing device 555 includes a series of modules, including an acquisition module 5551, a detection module 5552, a determination module 5553, a prediction module 5554, and a cutout module 5555. The following further describes how the various modules in the image processing device 555 provided in the embodiment of the present application cooperate to implement the image processing solution.

[0291] The acquisition module 5551 is used to acquire the image to be processed, wherein the image to be processed includes a visible graphic code and a target object that blocks an invisible graphic code; the detection module 5552 is used to perform position detection processing on the image to be processed to obtain the position coordinates of the visible graphic code and the position coordinates of the invisible graphic code; the determination module 5553 is used to determine the cutout parameters of the visible graphic code in the image to be processed based on the position coordinates of the visible graphic code; the prediction module 5554 is used to perform parameter prediction processing on the invisible graphic code based on the cutout parameters of the visible graphic code in the image to be processed and the position coordinates of the invisible graphic code to obtain the cutout parameters of the invisible graphic code; the cutout module 5555 is used to perform cutout processing on the image to be processed based on the cutout parameters of the visible graphic code and the cutout parameters of the invisible graphic code to obtain a target image with the visible graphic code removed and including the target object.

[0292] In some embodiments, the detection module 5552 is also used to perform graphic code detection processing on the image to be processed to obtain the position coordinates of the visible graphic code in the image to be processed; and perform position prediction processing on the invisible graphic code based on the position coordinates of the visible graphic code to obtain the position coordinates of the invisible graphic code.

[0293] In some embodiments, the image to be processed includes a first graphic code panel and a second graphic code panel, the first graphic code panel includes at least one of the visible graphic code and at least one of the invisible graphic code, and the second graphic code includes at least one of the invisible graphic code; the detection module 5552 is also used to determine the posture of the first graphic code panel based on the position coordinates of the visible graphic code in the first graphic code panel; based on the three-dimensional coordinates of the invisible graphic code in the first graphic code panel and the posture of the first graphic code panel, the invisible graphic code in the first graphic code panel is projected and transformed to obtain the position coordinates of the invisible graphic code in the first graphic code panel; based on the posture of the first graphic code panel and the transformation matrix from the first graphic code panel to the second graphic code panel, the posture of the second graphic code panel is determined; based on the three-dimensional coordinates of the invisible graphic code in the second graphic code panel and the posture of the second graphic code panel, the invisible graphic code in the second graphic code panel is projected and transformed to obtain the position coordinates of the invisible graphic code in the second graphic code panel.

[0294] In some embodiments, the determination module 5553 is also used to perform index construction processing on the image to be processed based on the position coordinates of the visible graphic code to obtain an index map of the visible graphic code, wherein the index map includes the index value of the visible graphic code; based on the color parameter value of each pixel in the image to be processed and the index value of the visible graphic code, the initial cutout parameters of the visible graphic code are updated to obtain the cutout parameters of the visible graphic code in the image to be processed.

[0295] In some embodiments, the determination module 5553 is also used to obtain a single-channel image corresponding to the image to be processed, wherein the value of the pixel in the single-channel image is different from the index value; the pixel value of the area corresponding to the visible graphic code in the single-channel image is set to the index value of the visible graphic code to obtain an index map of the visible graphic code.

[0296] In some embodiments, the determination module 5553 is also used to perform the following processing for any pixel in the image to be processed: when the pixel value at the same position as the pixel in the index map is the index value, determine the color parameter value of the pixel; based on the color parameter value of the pixel, update the initial cutout parameters of the visible graphic code corresponding to the pixel to obtain the cutout parameters of the visible graphic code in the image to be processed.

[0297] In some embodiments, the color characteristic value of the pixel includes a hue value and a saturation, and the initial cutout parameters include a minimum hue value, a maximum hue value, and a minimum saturation value; the determination module 5553 is further used to use the minimum hue value in the initial cutout parameters and the minimum value of the hue values ​​of the pixel as the minimum hue value in the cutout parameters of the visible graphic code; use the maximum hue value in the initial cutout parameters and the maximum value of the hue value of the pixel as the maximum hue value in the cutout parameters of the visible graphic code; use the minimum saturation value in the initial cutout parameters and the minimum saturation value of the pixel as the minimum saturation value in the cutout parameters of the visible graphic code.

[0298] In some embodiments, the prediction module 5554 is also used to determine the invisible graphic code adjacent to the visible graphic code based on the position coordinates of the visible graphic code and the position coordinates of the invisible graphic code; and obtain the cutout parameters of the invisible graphic code based on the cutout parameter equation relationship between the visible graphic code and the adjacent invisible graphic code, and the cutout parameters of the visible graphic code.

[0299] In some embodiments, the cutout module 5555 is also used to determine the transparent channel image corresponding to the image to be processed based on the cutout parameters of the visible graphic code and the cutout parameters of the invisible graphic code; and perform cutout processing based on the transparent channel image to obtain a target image with the visible graphic code removed and including the target object.

[0300] In some embodiments, the cutout module 5555 is also used to construct a transparency query table for each graphic code based on the cutout parameters of the visible graphic code and the cutout parameters of the invisible graphic code, where the graphic code is the visible graphic code or the invisible graphic code; and query the transparency query table of each graphic code based on the color parameter value of each pixel in the image to be processed to obtain a transparent channel image corresponding to the image to be processed.

[0301] In some embodiments, the cutout parameters include a minimum hue, a maximum hue, and a minimum saturation; the cutout module 5555 is also used to construct a corresponding hue curve in the transparency query table based on the minimum hue and the maximum hue in the cutout parameters, and the hue curve includes the corresponding relationship between hue and transparency; based on the minimum saturation in the cutout parameters, a corresponding saturation curve in the transparency query table is constructed, and the saturation curve includes the corresponding relationship between saturation and transparency.

[0302] In some embodiments, when there is a gap between two adjacent graphic codes, before constructing the transparency query table of each graphic code based on the cutout parameters of the visible graphic code and the cutout parameters of the invisible graphic code, the cutout module 5555 is also used to determine the hue expansion radius and the saturation expansion range based on the gap; the difference between the minimum hue value in the cutout parameters and the hue expansion radius is used as the updated minimum hue value; the sum of the maximum hue value in the cutout parameters and the hue expansion radius is used as the updated maximum hue value; the difference between the minimum saturation value in the cutout parameters and the saturation expansion range is used as the updated minimum saturation value; and based on the updated minimum hue value, the updated maximum hue value and the updated minimum saturation value, a transparency query table of the corresponding graphic code is constructed.

[0303] In some embodiments, the cutout module 5555 is also used to perform the following processing on any pixel in the image to be processed: determine the transparency query table of the graphic code where the pixel is located; query the transparency query table of the graphic code based on the color parameter value of the pixel to obtain the transparency of the pixel; according to the positional relationship of the pixels in the image to be processed, combine the transparencies of multiple pixels to obtain a transparent channel image corresponding to the image to be processed.

[0304] In some embodiments, the cutout module 5555 is also used to perform morphological processing on the transparent channel image to obtain the transparent channel image after morphological processing; perform interference color removal processing on the image to be processed to obtain the image to be processed after the interference color is removed; perform fusion processing on the transparent channel image after morphological processing and the image to be processed after the interference color is removed to obtain a target image with the visible graphic code removed and including the target object.

[0305] In summary, the image processing device proposed in the embodiment of the present application can predict the cutout parameters of the invisible graphic code by determining the cutout parameters of the visible graphic code and the position coordinates of the invisible graphic code, and perform cutout processing on the processed image based on the cutout parameters of all graphic codes, so as to adaptively and accurately determine the cutout parameters of each graphic code, so as to fully and effectively utilize the cutout parameters of each graphic code to achieve precise cutout, improve the cutout effect, and save a lot of human resources.

[0306] The embodiment of the present application provides a computer program product, which stores executable instructions, and the executable instructions are stored in a computer-readable storage medium. The processor of the electronic device reads the executable instructions from the computer-readable storage medium, and the processor executes the executable instructions, so that the electronic device performs the above-mentioned image processing method of the embodiment of the present application.

[0307] The present application embodiment provides a computer-readable storage medium storing executable instructions, wherein the executable instructions are stored. When the executable instructions are executed by a processor, the processor will execute the image processing method provided by the present application embodiment, for example, Figure 5A-5C The image processing method is shown.

[0308] In some embodiments, the computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface storage, optical disk, or CD-ROM; or it may be various devices including one or any combination of the above memories.

[0309] In some embodiments, executable instructions may be in the form of a program, software, software module, script or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine or other unit suitable for use in a computing environment.

[0310] As an example, executable instructions may, but need not, correspond to a file in a file system, may be stored as part of a file that stores other programs or data, such as in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files storing one or more modules, subroutines, or code portions).

[0311] As an example, the executable instructions may be deployed to be executed on one electronic device, or on multiple electronic devices located at one site, or on multiple electronic devices distributed at multiple sites and interconnected by a communication network.

[0312] The above is only an embodiment of the present application and is not intended to limit the protection scope of the present application. Any modifications, equivalent substitutions and improvements made within the spirit and scope of the present application are included in the protection scope of the present application.

Claims

1. An image processing method, characterized in that: The method comprises: Acquire an image to be processed, wherein the image to be processed includes a visible graphic code and a target object that blocks an invisible graphic code, wherein the visible graphic code is a graphic code that can be seen by human eyes in the image to be processed, and the invisible graphic code is a graphic code that is blocked by the target object and cannot be presented in the image to be processed; Performing position detection processing on the image to be processed to obtain the position coordinates of the visible graphic code and the position coordinates of the invisible graphic code; Based on the position coordinates of the visible graphic code, determining a cutout parameter of the visible graphic code in the image to be processed, wherein the cutout parameter is a parameter that determines whether a pixel in the image to be processed is a background; Based on the cutout parameters of the visible graphic code in the image to be processed and the position coordinates of the invisible graphic code, parameter prediction processing is performed on the invisible graphic code to obtain the cutout parameters of the invisible graphic code; Based on the cutout parameters of the visible graphic code and the cutout parameters of the invisible graphic code, the image to be processed is cutout to obtain a target image with the visible graphic code removed and including the target object.

2. The method according to claim 1, characterized in that The performing position detection processing on the image to be processed to obtain the position coordinates of the visible graphic code and the position coordinates of the invisible graphic code includes: Performing graphic code detection processing on the image to be processed to obtain the position coordinates of the visible graphic code in the image to be processed; The invisible graphic code is processed for position prediction based on the position coordinates of the visible graphic code to obtain the position coordinates of the invisible graphic code.

3. The method according to claim 2, characterized in that The image to be processed includes a first graphic code panel and a second graphic code panel, the first graphic code panel includes at least one of the visible graphic codes and at least one of the invisible graphic codes, and the second graphic code includes at least one of the invisible graphic codes; The step of performing position prediction processing on the invisible graphic code based on the position coordinates of the visible graphic code to obtain the position coordinates of the invisible graphic code includes: determining the posture of the first graphic code panel based on the position coordinates of the visible graphic code in the first graphic code panel; Based on the three-dimensional coordinates of the invisible graphic code in the first graphic code panel and the posture of the first graphic code panel, the invisible graphic code in the first graphic code panel is projected and transformed to obtain the position coordinates of the invisible graphic code in the first graphic code panel; Determine the posture of the second graphic code panel based on the posture of the first graphic code panel and the transformation matrix from the first graphic code panel to the second graphic code panel; Based on the three-dimensional coordinates of the invisible graphic code in the second graphic code panel and the posture of the second graphic code panel, the invisible graphic code in the second graphic code panel is projected and transformed to obtain the position coordinates of the invisible graphic code in the second graphic code panel.

4. The method according to claim 1, characterized in that: The step of determining the cutout parameters of the visible graphic code in the image to be processed based on the position coordinates of the visible graphic code comprises: Performing index construction processing on the image to be processed based on the position coordinates of the visible graphic code to obtain an index map of the visible graphic code, wherein the index map includes an index value of the visible graphic code; Based on the color parameter value of each pixel in the image to be processed and the index value of the visible graphic code, the initial cutout parameters of the visible graphic code are updated to obtain the cutout parameters of the visible graphic code in the image to be processed.

5. The method according to claim 4, characterized in that The step of performing index construction processing on the image to be processed based on the position coordinates of the visible graphic code to obtain an index map of the visible graphic code includes: Acquire a single-channel image corresponding to the image to be processed, wherein the value of a pixel in the single-channel image is different from the index value; The pixel values ​​of the area corresponding to the visible graphic code in the single-channel image are set as index values ​​of the visible graphic code to obtain an index map of the visible graphic code.

6. The method according to claim 4, characterized in that The updating process of the initial cutout parameters of the visible graphic code based on the color parameter value of each pixel in the image to be processed and the index value of the visible graphic code to obtain the cutout parameters of the visible graphic code in the image to be processed includes: The following processing is performed for any pixel in the image to be processed: When the pixel value at the same position as the pixel in the index map is the index value, determining the color parameter value of the pixel; Based on the color parameter value of the pixel, the initial cutout parameters of the visible graphic code corresponding to the pixel are updated to obtain the cutout parameters of the visible graphic code in the image to be processed.

7. The method according to claim 6, characterized in that The color characteristic values ​​of the pixels include hue value and saturation, and the initial cutout parameters include minimum hue value, maximum hue value, and minimum saturation value; The updating process of the initial cutout parameters of the visible graphic code corresponding to the pixel based on the color parameter value of the pixel to obtain the cutout parameters of the visible graphic code in the image to be processed includes: The minimum hue value in the initial cutout parameters and the minimum hue value in the pixel are used as the minimum hue value in the cutout parameters of the visible graphic code; The maximum value of the hue in the initial cutout parameters and the maximum value of the hue values ​​of the pixels are used as the maximum value of the hue in the cutout parameters of the visible graphic code; The minimum saturation value in the initial cutout parameters and the minimum saturation value in the pixel are used as the minimum saturation value in the cutout parameters of the visible graphic code.

8. The method according to claim 1, characterized in that The step of performing parameter prediction processing on the invisible graphic code based on the cutout parameters of the visible graphic code in the image to be processed and the position coordinates of the invisible graphic code to obtain the cutout parameters of the invisible graphic code includes: Determining the invisible graphic code adjacent to the visible graphic code based on the position coordinates of the visible graphic code and the position coordinates of the invisible graphic code; Based on the relationship between the cutout parameter equations of the visible graphic code and the adjacent invisible graphic code and the cutout parameter of the visible graphic code, the cutout parameter of the invisible graphic code is obtained.

9. The method according to claim 1, characterized in that: The step of performing a cutout process on the image to be processed based on the cutout parameters of the visible graphic code and the cutout parameters of the invisible graphic code to obtain a target image with the visible graphic code removed and including the target object comprises: Determining a transparent channel image corresponding to the image to be processed based on the cutout parameters of the visible graphic code and the cutout parameters of the invisible graphic code; A cutout process is performed based on the transparent channel image to obtain a target image with the visible graphic code removed and including the target object.

10. The method according to claim 9, characterized in that The step of determining the transparent channel image corresponding to the image to be processed based on the cutout parameters of the visible graphic code and the cutout parameters of the invisible graphic code includes: Based on the cutout parameters of the visible graphic code and the cutout parameters of the invisible graphic code, constructing a transparency query table for each graphic code, the graphic code being the visible graphic code or the invisible graphic code; The transparency query table of each graphic code is queried based on the color parameter value of each pixel in the image to be processed to obtain a transparent channel image corresponding to the image to be processed.

11. The method according to claim 10, characterized in that The cutout parameters include a minimum hue value, a maximum hue value, and a minimum saturation value; The step of constructing a transparency query table for each graphic code based on the cutout parameters of the visible graphic code and the cutout parameters of the invisible graphic code comprises: Based on the minimum hue value and the maximum hue value in the cutout parameters, construct a corresponding hue curve in the transparency query table, wherein the hue curve includes a corresponding relationship between hue and transparency; Based on the minimum saturation value in the cutout parameters, a saturation curve in the corresponding transparency query table is constructed, and the saturation curve includes a corresponding relationship between saturation and transparency.

12. The method according to claim 10, characterized in that When there is a gap between two adjacent graphic codes, before constructing the transparency query table of each graphic code based on the cutout parameters of the visible graphic code and the cutout parameters of the invisible graphic code, the method further includes: Based on the gap, determining a hue enlargement radius and a saturation enlargement range; The difference between the minimum hue value in the cutout parameter and the hue expansion radius is used as the updated minimum hue value; The sum of the hue maximum value in the cutout parameters and the hue expansion radius is used as the updated hue maximum value; The difference between the minimum saturation value in the cutout parameter and the saturation expansion range is used as the updated minimum saturation value; The step of constructing a transparency query table for each graphic code based on the cutout parameters of the visible graphic code and the cutout parameters of the invisible graphic code comprises: Based on the updated minimum hue value, the updated maximum hue value, and the updated minimum saturation value, a corresponding transparency query table of the graphic code is constructed.

13. The method according to claim 10 or 11, characterized in that: The querying of the transparency query table of each graphic code based on the color parameter value of each pixel in the image to be processed to obtain the transparent channel image corresponding to the image to be processed includes: The following processing is performed for any pixel in the image to be processed: Determine a transparency lookup table of the graphic code where the pixel is located; Querying a transparency query table of the graphic code based on the color parameter value of the pixel to obtain the transparency of the pixel; According to the positional relationship of pixels in the image to be processed, the transparency of a plurality of pixels is combined to obtain a transparent channel image corresponding to the image to be processed.

14. The method according to claim 9, characterized in that The step of performing a cutout process based on the transparent channel image to obtain a target image with the visible graphic code removed and including the target object comprises: Performing morphological processing on the transparent channel image to obtain the transparent channel image after morphological processing; Performing interference color removal processing on the image to be processed to obtain the image to be processed after the interference color is removed; The transparent channel image after morphological processing and the image to be processed after interference color removal are fused to obtain a target image with the visible graphic code removed and including the target object.

15. An image processing device, characterized in that: The device comprises: an acquisition module, configured to acquire an image to be processed, wherein the image to be processed includes a visible graphic code and a target object that blocks an invisible graphic code, wherein the visible graphic code is a graphic code that can be seen by human eyes in the image to be processed, and the invisible graphic code is a graphic code that is blocked by the target object and cannot be presented in the image to be processed; A detection module, used for performing position detection processing on the image to be processed to obtain the position coordinates of the visible graphic code and the position coordinates of the invisible graphic code; A determination module, used to determine a cutout parameter of the visible graphic code in the image to be processed based on the position coordinates of the visible graphic code, wherein the cutout parameter is a parameter that determines whether a pixel in the image to be processed is a background; A prediction module, configured to perform parameter prediction processing on the invisible graphic code based on the cutout parameters of the visible graphic code in the image to be processed and the position coordinates of the invisible graphic code, so as to obtain the cutout parameters of the invisible graphic code; The cutout module is used to perform cutout processing on the image to be processed based on the cutout parameters of the visible graphic code and the cutout parameters of the invisible graphic code to obtain a target image with the visible graphic code removed and including the target object.

16. An electronic device, characterized in that: The electronic device comprises: A memory for storing executable instructions; A processor, configured to implement the image processing method according to any one of claims 1 to 14 when executing the executable instructions stored in the memory.

17. A computer-readable storage medium, characterized in that: Executable instructions are stored, and are used to implement the image processing method described in any one of claims 1 to 14 when executed by a processor.

18. A computer program product comprising executable instructions, characterized in that: When the executable instructions are executed by a processor, the image processing method according to any one of claims 1 to 14 is implemented.

Images