An image processing method and apparatus

Abstract

The application discloses an image processing method and device. In the embodiment of the application, a foreground image of an original image containing a performer is determined according to the original image currently obtained, and a fusion image of the foreground image and a virtual background image previously saved is determined; first position information of the performer in the fusion image is obtained based on a target detection algorithm; a light irradiation area corresponding to the performer is determined according to the first position information; a mask image corresponding to a light special effect is determined based on an image processing algorithm, according to a previously drawn image and the light irradiation area, and the fusion image and the mask image are fused to determine a target image. Since the light irradiation area corresponding to the performer can be determined according to the first position information of the performer on the virtual stage in the embodiment of the application, the light can be effectively gathered on the performer in real time, thereby improving the user experience.

Description

Technical Field

[0001] This application relates to the field of virtual reality technology, and in particular to an image processing method and device. Background Technology

[0002] With the rapid development of virtual reality technology, more and more people are watching virtual stages online.

[0003] In the existing technology, in the process of realizing a virtual stage, the image of the performer is captured by the image acquisition device and the image of the performer is presented in the pre-saved virtual stage background. However, it is impossible to realize the effect of adding lights that move with the performer. As a result, the lights cannot be focused on the performer in real time, resulting in low realism of the virtual stage and reduced immersion for users watching the virtual stage online, which affects the user experience. Summary of the Invention

[0004] This application provides an image processing method, apparatus, device, medium, and display device to solve the problem in the prior art where the light cannot be focused on the performer in real time, resulting in low realism of the virtual stage and affecting the user experience.

[0005] In a first aspect, this application provides an image processing method, the method comprising:

[0006] Based on the currently obtained original image containing the performer, determine the foreground image of the original image, and determine the fused image of the foreground image and the pre-saved virtual background image;

[0007] Obtain the performer's first position information in the fused image;

[0008] Based on the first location information, the lighting area corresponding to the performer is determined. Based on the pre-drawn image and the lighting area, the mask image corresponding to the lighting effect is determined. The fused image and the mask image are then fused to determine the target image.

[0009] Secondly, this application provides an image processing apparatus, the apparatus comprising:

[0010] The determination module is used to determine the foreground image of the original image based on the currently obtained original image containing the performer, and to determine the fused image of the foreground image and the pre-saved virtual background image;

[0011] The acquisition module is used to obtain the first position information of the performer in the fused image;

[0012] The determining module is further configured to determine the lighting area corresponding to the performer based on the first position information, determine the mask image corresponding to the lighting effect based on the pre-drawn image and the lighting area, and fuse the fused image with the mask image to determine the target image.

[0013] Thirdly, this application provides an electronic device, which includes a processor and a memory, wherein the memory is used to store program instructions, and the processor is used to execute the computer program stored in the memory to implement the steps of the above-described image processing method.

[0014] Fourthly, this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the above-described image processing method.

[0015] Fifthly, this application provides a display device, the display device comprising:

[0016] A display, the display being used for making a display;

[0017] The controller is used to perform:

[0018] The control display shows the performer and lighting effects. When the performer is in a first position, the lighting effects are in a second position. When the performer is in a third position, the lighting effects are in a fourth position. The first position and the second position are different positions, and the third position and the fourth position are different positions.

[0019] In this embodiment, based on the currently obtained original image containing the performer, a foreground image of the original image is determined, and a fused image of the foreground image and a pre-saved virtual background image is determined. Based on a target detection algorithm, the first position information of the performer in the fused image is obtained. Based on the first position information, the lighting area corresponding to the performer is determined. Based on an image processing algorithm, a mask image corresponding to the lighting effect is determined according to a pre-drawn image and the lighting area. The fused image is then fused with the mask image to determine the target image. Because this embodiment can determine the lighting area corresponding to the performer based on the performer's first position information on the virtual stage, it can effectively focus the light on the performer in real time, thereby improving the user experience. Attached Figure Description

[0020] To more clearly illustrate the technical solutions of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 This is a schematic diagram of an image processing method provided in an embodiment of this application;

[0022] Figure 2 This application provides a schematic diagram of a process for acquiring a fused image, as illustrated in some embodiments.

[0023] Figure 3 A schematic diagram of a fused image provided for some embodiments of this application;

[0024] Figure 4 A schematic diagram illustrating the determination of a performer's position information in a fused image, provided for some embodiments of this application;

[0025] Figure 5 A schematic diagram illustrating a process for determining a light-illuminated area, provided for some embodiments of this application;

[0026] Figure 6 A schematic diagram of a mask image for a lighting effect provided for some embodiments of this application;

[0027] Figure 7 A schematic diagram illustrating the softening of a mask image for lighting effects, provided for some embodiments of this application;

[0028] Figure 8 A schematic diagram of a target image provided for some embodiments of this application;

[0029] Figure 9 A schematic diagram illustrating a process for determining a target image provided in some embodiments of this application;

[0030] Figure 10 This application provides a schematic diagram of the structure of an image processing apparatus according to some embodiments;

[0031] Figure 11 A schematic diagram of an electronic device structure is provided for some embodiments of this application;

[0032] Figure 12 This is a schematic diagram of a display device structure provided for some embodiments of this application. Detailed Implementation

[0033] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art are within the scope of protection of this application.

[0034] In this embodiment, based on the currently obtained original image containing the performer, a foreground image of the original image is determined, and a fused image of the foreground image and a pre-saved virtual background image is determined; the first position information of the performer in the fused image is obtained; based on the first position information, the lighting area corresponding to the performer is determined; based on the pre-drawn image and the lighting area, a mask image corresponding to the lighting effect is determined; and the fused image and the mask image are fused together to determine the target image. Since this embodiment can determine the lighting area corresponding to the performer based on the performer's first position information on the virtual stage, it can effectively focus the light on the performer in real time, thereby improving the user experience.

[0035] In order to focus the light on the performer in real time, thereby enhancing the user's immersion and experience, this application provides an image processing method, apparatus, device, medium, and display device.

[0036] Figure 1 This is a schematic diagram of an image processing method provided in an embodiment of this application. The process includes the following steps:

[0037] S101: Based on the currently obtained original image containing the performer, determine the foreground image of the original image, and determine the fused image of the foreground image and the pre-saved virtual background image.

[0038] The image processing method provided in this embodiment of the invention is applied to an electronic device, which may be an image acquisition device or a server or other electronic device.

[0039] In this application, to construct a virtual stage, an original image containing the performer is first obtained, and then a sub-image of the performer in the original image is merged with a pre-saved virtual background image. Specifically, after obtaining the original image containing the performer, the electronic device obtains the foreground image of the original image, and determines the merged image of the foreground image and the pre-saved virtual background image based on the foreground image and the pre-saved virtual background image. The original image is an image captured from a real-world scene containing the performer and the actual environment in which the performer is located. The sub-image of the performer in the original image is the foreground image of the original image. The pre-saved virtual image is set according to requirements, and the foreground image contains the performer's features.

[0040] Specifically, in determining the foreground image based on the original image, a portrait segmentation algorithm can be used to segment the original image and obtain a mask image. In this application, the portrait segmentation algorithm can be the Yolact portrait segmentation algorithm. After obtaining the mask image of the original image, the mask image is binarized. For ease of distinction, the image obtained after binarizing the mask image is called the first binary image. A NOT operation is performed on the first binary image to obtain the second binary image. In the first binary image, the pixel value of the pixels located on the portrait is 1, and the pixel value of the pixels located on the background is 0. In the second binary image, the pixel value of the pixels located on the portrait is 0, and the pixel value of the pixels located on the background is 1. After obtaining the first and second binary images, in order to retain the sub-image of the performer in the original image, where the sub-image is the image of the corresponding part of the performer in the original image, a AND operation is performed between the original image and the first binary image to obtain the foreground image. To obtain the virtual background for the virtual stage, a bitwise AND operation is performed between the second binary image and a pre-saved virtual background image to update the pre-saved virtual background image. This update aims to determine the area in the pre-saved virtual background image where the performer is overlaid. After updating the pre-saved virtual background image, the foreground image and the pre-saved virtual background image are fused using an image processing algorithm to obtain a merged image.

[0041] S102: Obtain the first position information of the performer in the fused image.

[0042] To obtain the performer's position information, this application uses a target detection algorithm to determine the performer's first position information in the fused image. This first position information can be the coordinates of the performer's center point in the fused image, or it can be the information of the bounding box corresponding to the performer in the fused image. In this application, the target detection algorithm can be the YOLOv5 target detection algorithm.

[0043] S103: Based on the first position information, determine the lighting area corresponding to the performer, determine the mask image corresponding to the lighting effect based on the pre-drawn image and the lighting area, and fuse the fused image with the mask image to determine the target image.

[0044] To ensure the light is focused on the performer, after determining the performer's first position information, a corresponding lighting area is determined based on this first position information. This lighting area includes the first position information and ensures the light is focused on the performer. Based on an image processing algorithm, a mask image corresponding to the lighting effect is determined according to a pre-drawn image and the lighting area. To integrate the lighting effect into the fused image, this application fuses the fused image with the mask image to determine the target image.

[0045] In this embodiment of the application, the lighting area corresponding to the performer can be determined based on the performer's first position information on the virtual stage. Therefore, it is possible to effectively focus the light on the performer in real time, thereby improving the user experience.

[0046] To determine the fused image, based on the above embodiments, in this embodiment of the application, determining the fused image of the foreground image and the pre-saved virtual background image includes:

[0047] The sum of the pixel values ​​of the pixels in the foreground image and the corresponding pixel values ​​in the pre-saved virtual background image is determined as the pixel value of the corresponding pixel in the fused image.

[0048] After obtaining the foreground image and the pre-saved virtual background image, in order to obtain the fused image, the pixel values ​​of the corresponding pixels in the foreground image and the pre-saved virtual background image are superimposed. Specifically, the sum of the pixel values ​​of the pixels in the foreground image and the pixel values ​​of the corresponding pixels in the pre-saved virtual background image is determined as the pixel value of the corresponding pixel in the fused image. The fused image retains both the features of the performer in the original image and the features of the pre-saved virtual background in the pre-saved virtual background image.

[0049] Figure 2 This is a schematic diagram illustrating a process for acquiring a fused image, provided in some embodiments of this application. Figure 3 This application provides a schematic diagram of a fused image for some embodiments, and now focuses on... Figure 2 and Figure 3 Explanation:

[0050] The image acquisition device obtains the original image containing the performer and performs image segmentation on the original image to obtain a human face mask image, which is the mask image of the original image. The mask image is binarized to obtain a first binary image. The first binary image is ANDed with the original image to obtain a foreground composite image, which is the foreground image. The first binary image is NOTed to obtain a second binary image. In order to ensure that the features of the foreground image in the fused image are not affected by the features of the pre-saved virtual background image, thereby improving the image fusion effect, the second binary image is ANDed with the pre-saved virtual background image to obtain a background composite image, which is to update the pre-saved virtual background image. The foreground image is then fused with the updated pre-saved virtual background image to obtain a human-scene fused composite image, which is the fused image.

[0051] The fused image contains features of the performers in the original image as well as features of a pre-saved virtual background image.

[0052] To determine the performer's first position information, based on the above embodiments, the method in this application embodiment further includes:

[0053] If, based on the performer's first location information, it is determined that there are other performers whose distance from the performer is within a set range, then the performer's first location information is updated based on the second location information of the other performers.

[0054] During a performance, at least two performers may be relatively close to each other. If the distance between two performers is within a predetermined range, their position information can be considered identical. For any given performer, to determine if there are other performers with the same position information, the distance between performers is determined based on their position information. This determines whether other performers exist within the predetermined distance. If so, their position information is considered identical to the first performer's. For ease of distinction, this other performer's position information is referred to as "second position information." Therefore, to accurately determine the lighting area, the first position information of the first performer is updated based on the second position information of these other performers.

[0055] Figure 4 This application provides a schematic diagram illustrating the determination of a performer's position information in a fused image, based on some embodiments. Figure 4 Explanation:

[0056] After determining the performer's first position information in the merged image, if this first position information is the position information of the bounding rectangle corresponding to the area where the performer is located in the merged image, the position information of the rectangle can be the coordinates of the top-left vertex and the coordinates of the bottom-right vertex of the rectangle. The coordinates of the top-left vertex are denoted as (x...). min ,y min The coordinates of the lower right corner vertex are denoted as (x...). max ,y max ).

[0057] To determine the mask image corresponding to the lighting effect, and thus add the lighting effect to the blended image, based on the above embodiments, the method in this application embodiment further includes:

[0058] Identify the performer's identification information;

[0059] Based on the preset correspondence between identification information and light colors, the target light color corresponding to the performer is determined;

[0060] Based on the target light color and the area illuminated by the light, determine the mask image corresponding to the light effect.

[0061] In this application, since there may be one performer or at least two performers on the stage, if there are at least two performers, the light color corresponding to the at least two performers can be set to the same color, or the light color corresponding to each of the at least two performers can be set to a different color.

[0062] If there are at least two performers, and there is a need to set different light colors for each performer, then to distinguish different performers by different light colors, the performer's identification information can be identified for each performer. In this application, the performer's identification information can be identified based on object detection algorithms and facial recognition technology. This identification information can be the performer's ID information, which can be the performer's identity information or a performer's number. Specifically, a correspondence between facial information and the corresponding ID information is pre-stored. After the performer's facial information is identified based on the object detection algorithm and facial recognition technology, the performer's ID information is determined according to the pre-stored correspondence between facial information and the corresponding ID information.

[0063] The identification of the performer's identity information based on the target detection algorithm is existing technology and will not be elaborated here.

[0064] After identifying the performer's identification information, since a pre-defined correspondence between the identification information and the light color is established, the target light color corresponding to the performer can be determined based on the performer's identification information and this correspondence. Specifically, in this application, to determine the mask image corresponding to the lighting effect, a completely black image is pre-configured. This completely black image is a pre-drawn image in which the pixel value of all pixels is 0. Based on the pre-drawn image, the light illumination area, and the target light color, the mask image corresponding to the lighting effect is determined. The width, height, and number of channels of the pre-drawn image are the same as the width, height, and number of channels of the fused image.

[0065] For example, given two performers, performer 1 is identified by identifier A, and performer 2 by identifier B, based on a pre-defined correspondence between identifiers and light colors, the target light color corresponding to identifier A is determined to be red, and the target light color corresponding to identifier B is determined to be green. Therefore, based on a pre-drawn image, the light illumination area corresponding to performer 1, and the red light color, the mask image corresponding to the required lighting effect for performer 1 is determined. Similarly, based on a pre-drawn image, the light illumination area corresponding to performer 2, and the green light color, the mask image corresponding to the required lighting effect for performer 2 is determined.

[0066] Furthermore, since the first position information of the performers changes in real time during the performance, taking performer No. 1 as an example, for each first position information of performer No. 1, the lighting area corresponding to performer No. 1 is determined based on the first position information, and the mask image corresponding to the required lighting effects of performer No. 1 is determined based on the pre-drawn image, the lighting area corresponding to performer No. 1, and the color of the red light. The mask image is different depending on the first position information.

[0067] If there are at least two performers, and during the performance, at least two performers may be in close proximity, meaning there may be instances where the first position information of these at least two performers is the same, then for any performer, if there is another performer with the same first position information, and the target light color corresponding to that performer is the same as the target light color of that other performer, then the target light color corresponding to that performer will not interfere with the other performer; if there is another performer with the same first position information, and the target light color corresponding to that performer is different from the target light color of that other performer, then the target light color corresponding to that performer will interfere with the other performer.

[0068] To avoid interference between the target light colors of different performers, the light colors of all performers located at the first position information are unified. In this application, since there may be lead and supporting roles among the stage performers, the priority of the performers can be preset. The priority can be set according to the importance of the performer in the performance process, where the higher the importance of the performer, the higher the priority. To achieve the uniformity of light colors, the target light color corresponding to the performer with the highest priority among the multiple performers at the first position information can be determined as the target light color of all multiple performers. That is, after determining the performer with the highest priority, the light colors of all other performers are updated to the target light color of the performer with the highest priority. Alternatively, the priority of light colors can be preset, and to achieve the uniformity of light colors, the light color with the highest light color level among the target light colors of the multiple performers can be determined as the common target light color of all multiple performers at the first position information.

[0069] If there are at least two performers and there is no need to set different light colors for each performer, since it is not necessary to distinguish different performers by different light colors, the light colors for the at least two performers can be set to the same color. The light color is set according to the requirements. For each performer, the light illumination area corresponding to the performer is determined based on the performer's first position information. The mask image corresponding to the light effect is determined based on the pre-drawn image, the light illumination area, and the set light color.

[0070] To determine the mask image, based on the above embodiments, in this embodiment, determining the mask image corresponding to the lighting effect according to the pre-drawn image and the lighting area includes:

[0071] Based on the short side of the rectangle corresponding to the area occupied by the performer in the fused image and the preset ratio value carried in the first location information, the area where the ellipse corresponding to the light illumination area is located in the pre-drawn image is determined.

[0072] The illumination area is determined based on the information about the location of the light source in the pre-drawn image and the elliptical region.

[0073] Based on the area illuminated by the light and the pre-drawn image, determine the mask image corresponding to the light effect.

[0074] In the process of determining the mask image corresponding to the lighting effect, the mask image corresponding to the lighting effect is determined based on the pre-drawn image, the lighting area, and the lighting color. Specifically, the area where the light is located is drawn in the pre-drawn image, and the color of the area where the light is located is set to the lighting color. The area where the light is located can be composed of an elliptical area and a polygonal area.

[0075] In this application, in order to accurately determine the elliptical region corresponding to the light-illuminated area in the pre-drawn image, and to determine the mask image corresponding to the light effect based on the light-illuminated area and the pre-drawn image, based on the above embodiments, in this application embodiment, determining the region where the ellipse corresponding to the light-illuminated area in the pre-drawn image is located based on the shorter side of the rectangle corresponding to the area occupied by the performer in the fused image carried in the first position information and a preset ratio value includes:

[0076] The length of the shorter side of the rectangle is determined as the length of the minor axis of the ellipse;

[0077] The first product of the minor axis length and a preset ratio value is determined as the major axis length of the ellipse;

[0078] The designated position within the rectangle is defined as the center of the ellipse;

[0079] Based on the length of the minor axis of the ellipse, the radius of the major axis of the ellipse, and the center of the ellipse, the elliptical region corresponding to the illuminated area in the pre-drawn image is determined.

[0080] In this application, the elliptical region corresponding to the illuminated area can be determined in the pre-drawn image based on the elliptical function of the OpenCV vision and machine learning software library, that is, the ellipse is drawn in the pre-drawn image. Specifically, in the process of drawing the ellipse, the length of the short side of the bounding rectangle of the performer in the fused image can be determined as the length of the minor axis of the ellipse, and the first product of the length of the minor axis and a preset ratio value can be determined as the length of the major axis of the ellipse. The preset ratio value is a value greater than 1, for example, the preset ratio value can be 1.5. The set position of the bounding rectangle is determined as the center of the ellipse. The set position can be the position of the center point of any edge of the bounding rectangle, or it can be the center of the bounding rectangle. The any edge can be any long side or any short side of the bounding rectangle. In this application, the center point of the bottom edge of the bounding rectangle can be determined as the center of the ellipse. Based on the length of the minor axis of the ellipse, the length of the major axis of the ellipse, and the center of the ellipse, the region where the ellipse corresponding to the illuminated area in the pre-drawn image is located is determined. The region where the ellipse is located is the projection of the light onto the ground.

[0081] Furthermore, in practical applications, lights are generally installed above the stage to ensure that the light shines from top to bottom. Therefore, in order to increase the realism of the virtual stage, after determining the area of ​​the ellipse corresponding to the area illuminated by the light, a polygonal area is determined based on the elliptical area. In this application, a polygon can be drawn based on the OpenCV polygon function cvPolyline. Specifically, the polygon can be composed of four sides. In order to determine the polygon, the position information of the four vertices of the polygon in the pre-drawn image can be determined in advance. Among them, two vertices of the polygon can be the two vertices of the major axis of the ellipse. Any two points on the top edge of the pre-drawn image can be determined as the other two vertices of the polygon. In this application, pixels at a set length from the midpoint of the top edge of the pre-drawn image can be used as the other two vertices of the polygon. The set length can be set according to the requirements, for example, it can be a length of 20 pixels.

[0082] Figure 5 This application provides a schematic diagram illustrating a process for determining a light-illuminated area in some embodiments. Figure 6 This is a schematic diagram of a mask image for a lighting effect provided in some embodiments of this application. Figure 7 This application provides a schematic diagram of mask image softening for lighting effects in some embodiments, and now focuses on... Figure 5 , Figure 6 as well as Figure 7 Explanation:

[0083] The illuminated area consists of an elliptical region and a polygonal region. In determining the elliptical region, the length of the shorter side of the rectangle is defined as the minor axis of the ellipse. The coordinates of the top-left vertex of the rectangle corresponding to the performer are denoted as (x...). min ,y min The coordinates of the lower right corner vertex are denoted as (x...). max ,y max Therefore, the length of the shorter side of the rectangle is x. max -x min Therefore, the length of the minor axis of the ellipse is x. max -x min , 1.5*(x max -x min The length of the major axis of the ellipse is determined by the coordinates of the center point of the bottom edge of the rectangle, which is then defined as the center of the ellipse. Therefore, the elliptical region of the illuminated area can be determined based on the length of the minor axis of the ellipse, the length of the major axis of the ellipse, and the coordinates of the center point of the ellipse.

[0084] In determining the polygonal region, the two vertices of the major axis of the ellipse are defined as the two vertices of the polygon; that is, the coordinates of the two vertices of the polygon are... as well as Take a pixel 20 pixels away from the midpoint of the top edge of the pre-drawn image as the remaining two vertices of the polygon. Since the width of the pre-drawn image is W, the coordinates of the remaining two vertices are... as well as Therefore, the polygonal region is determined based on these four vertices.

[0085] After determining the elliptical and polygonal regions, the elliptical and polygonal regions are drawn in the pre-drawn image to obtain the mask image of the lighting effect.

[0086] In order to make the edges of the lighting effects smoother and thus improve the image quality of the target image, after determining the mask image corresponding to the lighting effects, the mask image is filtered. In this application, the matrix of the mask image can be determined and the matrix can be convolved to obtain a mask image of the lighting effects with smooth edges, that is, to obtain the mask image of the lighting effects.

[0087] In addition, the light colors can be set according to needs. Since there may be one performer on the stage or at least two performers, if there are at least two performers, the light colors corresponding to the at least two performers can be set to the same color, or the light colors corresponding to each of the at least two performers can be set to different colors.

[0088] If there is a performer on the stage, and the color of the area illuminated by the light corresponding to the performer is set to white, then the parameters corresponding to the edge colors of the determined ellipse and polygon are set to (255, 255, 255), and the ellipse and polygon are filled with white. These parameters can be set according to requirements.

[0089] If there are two performers on stage, and the color of the lighting area corresponding to performer number one is set to white, and the color of the lighting area corresponding to performer number two is set to red, then the parameters corresponding to the edge colors of the ellipse and polygon in the lighting area corresponding to performer number one are set to (255, 255, 255), and the parameters corresponding to the edge colors of the ellipse in the lighting area corresponding to performer number two are set to (255, 0, 0). The ellipse and polygon corresponding to performer number one are filled with white, and the ellipse and polygon corresponding to performer number two are filled with red. These parameters can be set according to requirements.

[0090] To determine the target image, based on the above embodiments, in this embodiment of the application, fusing the fused image with the mask image to determine the target image includes:

[0091] Based on the fused image and the mask image, a first floating-point matrix of the fused image and a second floating-point matrix of the mask image are determined, wherein the first floating-point matrix and the second floating-point matrix respectively carry pixel value information corresponding to each pixel point in the fused image and the mask image;

[0092] The target image is determined based on the first floating-point matrix, the second floating-point matrix, the first weight corresponding to the pre-set fused image, and the second weight corresponding to the pre-set mask image.

[0093] To improve the quality of the target image, this application sets different transparency levels for the fused image and the mask image when fusing them. That is, a first weight corresponding to the fused image and a second weight corresponding to the mask image are preset, wherein the first weight is greater than the second weight, and the sum of the first weight and the second weight is 1. If the second weight is too large, it may cause the lighting effect to affect the quality of the target image, fail to highlight the performer, and result in a poor viewing experience for the user. Therefore, in this application, the second weight is set to a value less than 0.1. Specifically, the first weight can be set to 0.92 and the second weight can be set to 0.08.

[0094] To determine the target image, a first matrix of the fused image and a second matrix of the mask image are determined based on the fused image and the mask image. Since the first and second weights are double-precision type data, the first and second matrices are converted into a first floating-point matrix of the fused image and a second floating-point matrix of the mask image. The first and second floating-point matrices respectively carry pixel value information corresponding to each pixel in the fused image and the mask image. To determine the target image, the target image is determined based on the first floating-point matrix, the second floating-point matrix, the first weight, and the second weight. Specifically, based on the Merge... i,j =Img i,j *α+Mask i,j *β, where i∈{0,1....w},j∈{0,1....h}, determines the third floating-point matrix of the target image, where the Merge i,j Img is the pixel value of the pixel in the i-th row and j-th column of the third floating-point matrix corresponding to the target image. i,j Mask is the pixel value of the pixel in the i-th row and j-th column of the first floating-point matrix.i,j Let be the pixel value of the pixel in the i-th row and j-th column of the second floating-point matrix, α be the first weight, β be the second weight, w be the width of the fused image or the mask image, and h be the height of the fused image or the mask image.

[0095] Therefore, after determining the third floating-point matrix of the target image, in order to display the target image, the data in the third floating-point matrix is ​​converted into UNIT8 type, and the target image is finally determined.

[0096] Figure 8 This is a schematic diagram of a target image provided for some embodiments of this application. Figure 9 This application provides a schematic diagram of a process for determining a target image, which is now discussed in some embodiments. Figure 8 or Figure 9 Explanation:

[0097] First, after obtaining the original image, a mask image is obtained based on a human image segmentation algorithm. Based on the mask image, the original image, and a pre-saved virtual background image, the performer is fused with the virtual background to obtain a fused image, which is the foreground image of the original image. This foreground image is then fused with the pre-saved virtual background image to obtain a fused image. Based on a target detection algorithm, the human body coordinates are obtained, which is the first position information of the performer in the fused image. If there are two or more performers and there is a need to set different light colors for each performer, a human body tracking algorithm is used to track the movement of the human body, identify the performer's identification information, and determine the light color corresponding to the performer based on the pre-set correspondence between the identification information and the light color. Based on an image processing algorithm, a mask image corresponding to the light effect is determined based on the light color and the area illuminated by the light. Based on an image processing algorithm, a layer with transparency is generated, which means that a second weight and a first weight are pre-set for the mask image and the fused image. Based on an image processing algorithm, multi-layer fusion is performed, which means that the target image is determined based on the mask image, the fused image, and the second and first weights.

[0098] The target image contains features of the performers from the original image, as well as features from a pre-saved virtual background image and features of lighting effects.

[0099] Figure 10 This application provides a schematic diagram of an image processing apparatus structure, which includes:

[0100] The determining module 1001 is used to determine the foreground image of the currently obtained original image containing the performer, and to determine the fused image of the foreground image and the pre-saved virtual background image;

[0101] The acquisition module 1002 is used to obtain the first position information of the performer in the fused image;

[0102] The determining module 1001 is further configured to determine the lighting area corresponding to the performer based on the first position information, determine the mask image corresponding to the lighting effect based on the pre-drawn image and the lighting area, and fuse the fused image with the mask image to determine the target image.

[0103] In one possible implementation, the device further includes:

[0104] The update module 1003 is used to update the first position information of the performer based on the first position information of the performer, if it is determined that there are other performers whose distance from the performer is within a set range, and based on the second position information of the other performers.

[0105] In one possible implementation, the determining module 1001 is further configured to identify the performer's identification information; determine the target light color corresponding to the performer based on a preset correspondence between the identification information and the light color; and determine the mask image corresponding to the light effect based on the target light color and the light illumination area.

[0106] In one possible implementation, the determining module 1001 is specifically used to determine the sum of the pixel values ​​of the pixels in the foreground image and the pixel values ​​of the corresponding pixels in the pre-saved virtual background image as the pixel value of the corresponding pixel in the fused image.

[0107] In one possible implementation, the determining module 1001 is specifically configured to determine the region where the ellipse corresponding to the light illumination area in the pre-drawn image is located, based on the short side of the rectangle corresponding to the area occupied by the performer in the fused image carried in the first position information and a preset ratio value; determine the light illumination area based on the information of the position of the light set in the pre-drawn image and the elliptical region; and determine the mask image corresponding to the light effect based on the light illumination area and the pre-drawn image.

[0108] In one possible implementation, the determining module 1001 is specifically used to determine the length of the short side of the rectangle as the minor axis length of the ellipse; determine the first product of the minor axis length and a preset ratio value as the major axis length of the ellipse; determine the set position in the rectangle as the center of the ellipse; and determine the region where the ellipse corresponding to the light-illuminated area in the pre-drawn image is located based on the minor axis length, the major axis radius, and the center of the ellipse.

[0109] In one possible implementation, the determining module 1001 is specifically configured to determine a first floating-point matrix of the fused image and a second floating-point matrix of the mask image based on the fused image and the mask image, wherein the first floating-point matrix and the second floating-point matrix respectively carry pixel value information corresponding to each pixel point in the fused image and the mask image; and determine a target image based on the first floating-point matrix, the second floating-point matrix, a pre-set first weight corresponding to the fused image, and a pre-set second weight corresponding to the mask image.

[0110] Figure 11 This application provides a schematic diagram of an electronic device structure based on some embodiments. In addition to the above embodiments, this application also provides an electronic device, such as... Figure 11 As shown, it includes: processor 1101, communication interface 1102, memory 1103 and communication bus 1104, wherein processor 1101, communication interface 1102 and memory 1103 communicate with each other through communication bus 1104.

[0111] The memory 1103 stores a computer program, which, when executed by the processor 1101, causes the processor 1101 to perform the following steps:

[0112] Based on the currently obtained original image containing the performer, determine the foreground image of the original image, and determine the fused image of the foreground image and the pre-saved virtual background image;

[0113] Obtain the performer's first position information in the fused image;

[0114] Based on the first location information, the lighting area corresponding to the performer is determined. Based on the pre-drawn image and the lighting area, the mask image corresponding to the lighting effect is determined. The fused image and the mask image are then fused to determine the target image.

[0115] Furthermore, the processor 1101 is also configured to update the first position information of the performer based on the first position information of the performer, if it is determined that there are other performers whose distance from the performer is within a set range, based on the second position information of the other performers.

[0116] Furthermore, the processor 1101 is also used to identify the performer's identification information; determine the target light color corresponding to the performer according to a preset correspondence between the identification information and the light color; and determine the mask image corresponding to the light effect according to the target light color and the light illumination area.

[0117] Furthermore, the processor 1101 is also configured to determine the pixel value of the corresponding pixel in the fused image as the sum of the pixel value of the pixel in the foreground image and the pixel value of the corresponding pixel in the pre-saved virtual background image.

[0118] Furthermore, the processor 1101 is also configured to determine the region where the ellipse corresponding to the light illumination area in the pre-drawn image is located based on the short side of the rectangle corresponding to the area occupied by the performer in the fused image carried in the first position information and a preset ratio value; determine the light illumination area based on the information of the position of the light set in the pre-drawn image and the elliptical region; and determine the mask image corresponding to the light effect based on the light illumination area and the pre-drawn image.

[0119] Furthermore, the processor 1101 is also configured to determine the length of the short side of the rectangle as the minor axis length of the ellipse; determine the first product of the minor axis length and a preset ratio value as the major axis length of the ellipse; determine the set position in the rectangle as the center of the ellipse; and determine the region where the ellipse corresponding to the light-illuminated area in the pre-drawn image is located based on the minor axis length of the ellipse, the major axis radius of the ellipse, and the center of the ellipse.

[0120] Furthermore, the processor 1101 is also configured to determine a first floating-point matrix of the fused image and a second floating-point matrix of the mask image based on the fused image and the mask image, wherein the first floating-point matrix and the second floating-point matrix respectively carry pixel value information corresponding to each pixel point in the fused image and the mask image; and determine a target image based on the first floating-point matrix, the second floating-point matrix, a pre-set first weight corresponding to the fused image, and a pre-set second weight corresponding to the mask image.

[0121] Communication interface 1102 is used for communication between the above-mentioned electronic device and other devices.

[0122] The memory may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0123] The processors mentioned above can be general-purpose processors, including central processing units, network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits, field-programmable gate arrays or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0124] Figure 12 This application provides a schematic diagram of the structure of a display device 1200 according to some embodiments. The display device 1200 includes:

[0125] Display 1201, the display being used for displaying;

[0126] Controller 1202, the controller is used to perform:

[0127] The control display 1201 displays the performer and lighting effects. When the performer is in a first position, the lighting effects are in a second position. When the performer is in a third position, the lighting effects are in a fourth position. The first position and the second position are different positions, and the third position and the fourth position are different positions.

[0128] In one possible implementation, the controller 1202 is used to perform:

[0129] The performers and the lighting effects are composited onto the same display screen.

[0130] Since the principle of the above-mentioned display device in solving the problem is similar to that of the image processing method, the corresponding functions of the above-mentioned display device are described in the above embodiments, and repeated parts will not be described again.

[0131] Based on the above embodiments, this application provides a computer-readable storage medium storing a computer program executable by a processor. When the program is run on the processor, the processor executes the following steps:

[0132] Based on the currently obtained original image containing the performer, determine the foreground image of the original image, and determine the fused image of the foreground image and the pre-saved virtual background image;

[0133] Obtain the performer's first position information in the fused image;

[0134] Based on the first location information, the lighting area corresponding to the performer is determined. Based on the pre-drawn image and the lighting area, the mask image corresponding to the lighting effect is determined. The fused image and the mask image are then fused to determine the target image.

[0135] In one possible implementation, the method further includes:

[0136] If, based on the performer's first location information, it is determined that there are other performers whose distance from the performer is within a set range, then the performer's first location information is updated based on the second location information of the other performers.

[0137] In one possible implementation, the method further includes:

[0138] Identify the performer's identification information;

[0139] Based on the preset correspondence between identification information and light colors, the target light color corresponding to the performer is determined;

[0140] Based on the target light color and the area illuminated by the light, determine the mask image corresponding to the light effect.

[0141] In one possible implementation, determining the fused image of the foreground image and the pre-saved virtual background image includes:

[0142] The sum of the pixel values ​​of the pixels in the foreground image and the corresponding pixel values ​​in the pre-saved virtual background image is determined as the pixel value of the corresponding pixel in the fused image.

[0143] In one possible implementation, determining the mask image corresponding to the lighting effect based on the pre-drawn image and the illuminated area includes:

[0144] Based on the short side of the rectangle corresponding to the area occupied by the performer in the fused image and the preset ratio value carried in the first location information, the area where the ellipse corresponding to the light illumination area is located in the pre-drawn image is determined.

[0145] The illumination area is determined based on the information about the location of the light source in the pre-drawn image and the elliptical region.

[0146] Based on the area illuminated by the light and the pre-drawn image, determine the mask image corresponding to the light effect.

[0147] In one possible implementation, determining the region where the ellipse corresponding to the illuminated area in the pre-drawn image is located, based on the shorter side of the rectangle corresponding to the area occupied by the performer in the fused image carried in the first location information and a preset proportion value, includes:

[0148] The length of the shorter side of the rectangle is determined as the length of the minor axis of the ellipse;

[0149] The first product of the minor axis length and a preset ratio value is determined as the major axis length of the ellipse;

[0150] The designated position within the rectangle is defined as the center of the ellipse;

[0151] Based on the length of the minor axis of the ellipse, the radius of the major axis of the ellipse, and the center of the ellipse, the region where the ellipse corresponding to the illuminated area in the pre-drawn image is located is determined.

[0152] In one possible implementation, fusing the fused image with the mask image to determine the target image includes:

[0153] Based on the fused image and the mask image, a first floating-point matrix of the fused image and a second floating-point matrix of the mask image are determined, wherein the first floating-point matrix and the second floating-point matrix respectively carry pixel value information corresponding to each pixel point in the fused image and the mask image;

[0154] The target image is determined based on the first floating-point matrix, the second floating-point matrix, the first weight corresponding to the pre-set fused image, and the second weight corresponding to the pre-set mask image.

[0155] In this embodiment of the application, the lighting area corresponding to the performer can be determined based on the performer's first position information on the virtual stage. Therefore, it is possible to effectively focus the light on the performer in real time, thereby improving the user experience.

[0156] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0157] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0158] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0159] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0160] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. An image processing method, characterized in that, The method includes: Based on the currently obtained original image containing the performer, determine the foreground image of the original image, and determine the fused image of the foreground image and the pre-saved virtual background image; Obtain the performer's first position information in the fused image; Based on the first location information, the lighting area corresponding to the performer is determined. Based on the pre-drawn image and the lighting area, the mask image corresponding to the lighting effect is determined. The fused image and the mask image are then fused to determine the target image. Identify the performer's identification information; Based on the preset correspondence between identification information and light colors, the target light color corresponding to the performer is determined; If there are multiple performers and the performers are prioritized during the performance, then the target light color corresponding to the performer with the highest priority among the multiple performers at the first location information is determined as the target light color of the multiple performers. If there are multiple performers and a priority is set for the light colors, the light color with the highest priority among the target light colors of the multiple performers is determined as the common target light color of the multiple performers at the first location information. Based on the target light color and the area illuminated by the light, determine the mask image corresponding to the light effect; The step of determining the mask image corresponding to the lighting effect based on the pre-drawn image and the lighting area includes: The length of the short side of the rectangle is determined as the minor axis length of the ellipse, wherein the rectangle is the rectangle corresponding to the area occupied by the performer in the fused image carried in the first position information; the first product of the minor axis length and a preset ratio value is determined as the major axis length of the ellipse; the set position in the rectangle is determined as the center of the ellipse; based on the minor axis length, the major axis radius, and the center of the ellipse, the region where the ellipse corresponding to the light-illuminated area in the pre-drawn image is located is determined; In the pre-drawn image, a polygonal region is drawn, where two vertices of the polygon are the two vertices of the major axis of the ellipse, and two pixels at a set distance from the midpoint of the uppermost edge of the pre-drawn image are taken as the other two vertices of the polygon. The region containing the ellipse and the region of the polygon constitute the light illumination area; Based on the area illuminated by the light and the pre-drawn image, determine the mask image corresponding to the light effect.

2. The method according to claim 1, characterized in that, The method further includes: If, based on the performer's first location information, it is determined that there are other performers whose distance from the performer is within a set range, then the performer's first location information is updated based on the second location information of the other performers.

3. The method according to claim 1, characterized in that, The fused image determined by the foreground image and the pre-saved virtual background image includes: The sum of the pixel values ​​of the pixels in the foreground image and the corresponding pixel values ​​in the pre-saved virtual background image is determined as the pixel value of the corresponding pixel in the fused image.

4. The method according to claim 1, characterized in that, The step of fusing the fused image with the mask image to determine the target image includes: Based on the fused image and the mask image, a first floating-point matrix of the fused image and a second floating-point matrix of the mask image are determined, wherein the first floating-point matrix and the second floating-point matrix respectively carry pixel value information corresponding to each pixel point in the fused image and the mask image; The target image is determined based on the first floating-point matrix, the second floating-point matrix, the first weight corresponding to the pre-set fused image, and the second weight corresponding to the pre-set mask image.

5. An electronic device, characterized in that, The electronic device includes a processor and a memory, the memory being used to store a computer program, and the processor being used to execute the computer program stored in the memory to implement the steps of the image processing method according to any one of claims 1-4.

6. A display device, characterized in that, The display device includes: A display, the display being used for making a display; The controller is used to perform: The control display shows the performer and lighting effects. When the performer is in a first position, the lighting effects are in a second position. When the performer is in a third position, the lighting effects are in a fourth position. The first position and the second position are different positions, and the third position and the fourth position are different positions. The controller is used to perform: The performer and the lighting effects are formed on the same display screen using the image processing method described in any one of claims 1-4.

Images