Image processing method and related equipment

Abstract

The invention discloses an image processing method and related equipment. The method comprises the following steps: respectively extracting a first brightness image corresponding to a target object image and a second brightness image corresponding to a reference image; performing down-sampling on the first brightness image and the second brightness image to obtain target first brightness information and target second brightness information; determining target brightness information based on the target first brightness information and the target second brightness information; performing feature extraction on the target brightness information through a brightness parameter perception model to obtain a target brightness adjustment parameter; performing brightness adjustment on the first brightness image according to the target brightness adjustment parameter to obtain a target brightness image; and carrying out image fusion on the target brightness image, channel images under other channels except the brightness channel in the target object image and the reference image to obtain a target composite image. According to the method, the image synthesis effect is improved, and the visual harmony of the synthesized image is enhanced; and the data processing efficiency is improved, and real-time processing can be carried out.

Description

Technical Field

[0001] This application relates to the field of computer technology, specifically to an image processing method and related equipment. Background Technology

[0002] With the development of computer technology, the application scope of image processing technology is becoming increasingly wide, and image compositing is a hot topic in the field. Image compositing technology is the technique of combining different images into one image. Specifically, it can combine parts of different images into a single image to achieve special effects. For example, image compositing technology can be used to replace the background of video streams in live streaming or video conferencing, and it can also be used to replace faces.

[0003] In current related technologies, image synthesis is usually performed by direct overlay. However, directly synthesizing images with inconsistent brightness may result in unnatural image fusion effects. For example, in a video conferencing scenario, directly replacing the background of a person's image may result in a large difference in brightness between the replaced background and the image, leading to unnatural blending of the foreground and background and obvious traces of image synthesis. Summary of the Invention

[0004] This application provides an image processing method and related equipment. The related equipment may include an image processing device, an electronic device, a computer-readable storage medium, and a computer program product. It can solve the problem of inconsistent brightness in synthesized images, which is beneficial to improving the image synthesis effect and enhancing the visual harmony of the synthesized images. It can also greatly reduce the amount of data processing and improve the data processing efficiency. It can perform real-time processing on images and has strong applicability to various scenarios.

[0005] This application provides an image processing method, including:

[0006] The target object image to be synthesized, the reference image, and the brightness parameter perception model are obtained. The target object image and the reference image are extracted in the brightness channel respectively to obtain the first brightness image corresponding to the target object image and the second brightness image corresponding to the reference image.

[0007] The first brightness image and the second brightness image are downsampled respectively to obtain the first brightness information and the second brightness information of the target;

[0008] Based on the first brightness information and the second brightness information of the target, the target brightness information is determined;

[0009] The target brightness adjustment parameters are obtained by extracting features from the target brightness information using the brightness parameter perception model.

[0010] According to the target brightness adjustment parameters, the first brightness image is subjected to brightness adjustment processing to obtain the target brightness image;

[0011] The target brightness image, the channel images of the target object image (excluding the brightness channel), and the reference image are subjected to image fusion processing to obtain and output the target composite image.

[0012] Accordingly, embodiments of this application provide an image processing apparatus, including:

[0013] The acquisition unit is used to acquire the target object image to be synthesized, the reference image, and the brightness parameter perception model, and to extract the image in the brightness channel of the target object image and the reference image respectively to obtain the first brightness image corresponding to the target object image and the second brightness image corresponding to the reference image.

[0014] The downsampling unit is used to perform downsampling processing on the first brightness image and the second brightness image respectively to obtain the first brightness information and the second brightness information of the target;

[0015] The determining unit is configured to determine target brightness information based on the first target brightness information and the second target brightness information;

[0016] The feature extraction unit is used to extract features from the target brightness information through the brightness parameter perception model to obtain the target brightness adjustment parameters;

[0017] A brightness adjustment unit is used to perform brightness adjustment processing on the first brightness image according to the target brightness adjustment parameters to obtain a target brightness image;

[0018] The synthesis unit is used to perform image fusion processing on the target brightness image, the channel images of the target object image other than the brightness channel, and the reference image to obtain and output the target synthesized image.

[0019] Optionally, in some embodiments of this application, the downsampling unit may include a first selection subunit, a first construction subunit, a second selection subunit, and a second construction subunit, as follows:

[0020] The first selection subunit is used to select a plurality of first target pixels from the pixels of the first brightness image;

[0021] The first construction subunit is used to construct target first brightness information based on the plurality of first target pixels;

[0022] The second selection subunit is used to select a plurality of second target pixels from the pixels of the second brightness image;

[0023] The second construction subunit is used to construct target second brightness information based on the plurality of second target pixels.

[0024] Optionally, in some embodiments of this application, the feature extraction unit may include an extraction subunit, a parameter acquisition subunit, and a smoothing subunit, as follows:

[0025] The extraction subunit is used to extract features from the target brightness information through the brightness parameter perception model to obtain initial brightness adjustment parameters;

[0026] The parameter acquisition subunit is used to acquire the reference brightness adjustment parameters of the preceding image associated with the target object image;

[0027] The smoothing subunit is used to smooth the initial brightness adjustment parameters based on the reference brightness adjustment parameters to obtain the target brightness adjustment parameters.

[0028] Optionally, in some embodiments of this application, the extraction subunit may be specifically used to perform convolution processing on the target brightness information through the brightness parameter perception model to obtain convolutional feature information; perform pooling processing on the convolutional feature information to obtain pooled feature information; and perform fully connected processing on the pooled feature information to obtain initial brightness adjustment parameters.

[0029] Optionally, in some embodiments of this application, the smoothing subunit may be used to determine the parameter weights of the reference brightness adjustment parameter and the initial brightness adjustment parameter respectively; based on the parameter weights, the reference brightness adjustment parameter and the initial brightness adjustment parameter are fused to obtain the target brightness adjustment parameter.

[0030] Optionally, in some embodiments of this application, the determining unit may include a first fusion subunit and a normalization subunit, as follows:

[0031] The first fusion subunit is used to fuse the first brightness information of the target and the second brightness information of the target to obtain fused brightness information;

[0032] The normalization subunit is used to normalize the fused brightness information to obtain the target brightness information.

[0033] Optionally, in some embodiments of this application, the synthesis unit may include a second fusion subunit and a third fusion subunit, as follows:

[0034] The second fusion subunit is used to fuse the target brightness image with the channel images of the target object image in other channels besides the brightness channel to obtain the brightness-processed object image;

[0035] The third fusion subunit is used to perform image fusion between the brightness-processed object image and the reference image to obtain and output the target composite image.

[0036] Optionally, in some embodiments of this application, the image processing apparatus may further include a training data acquisition unit, a first adjustment unit, a downsampling processing unit, an extraction unit, a second adjustment unit, and a parameter adjustment unit, as follows:

[0037] The training data acquisition unit is used to acquire training data, which includes multiple sample images, each containing a sample object; and to perform image segmentation on the sample images to obtain a first region image containing the sample object and a second region image not containing the sample object.

[0038] The first adjustment unit is used to perform brightness adjustment processing on the brightness image of the first region image to obtain the sample first brightness image;

[0039] The downsampling processing unit is used to perform downsampling processing on the brightness images corresponding to the first brightness image of the sample and the second region image, respectively, to obtain the first brightness information and the second brightness information of the target sample; and to determine the brightness information of the target sample based on the first brightness information and the second brightness information of the target sample.

[0040] The extraction unit is used to extract features from the brightness information of the target sample through the brightness parameter perception model to obtain the sample brightness adjustment parameters;

[0041] The second adjustment unit is used to perform brightness adjustment processing on the first brightness image of the sample according to the sample brightness adjustment parameters to obtain the sample brightness image;

[0042] The parameter adjustment unit is used to adjust the parameters of the brightness parameter perception model according to the sample brightness image and the sample image to obtain the trained brightness parameter perception model.

[0043] Optionally, in some embodiments of this application, the first adjustment unit may include a normalization processing subunit, a third selection subunit, an adjustment subunit, and an inverse normalization processing subunit, as follows:

[0044] The normalization processing subunit is used to normalize the brightness image corresponding to the first region image to obtain a normalized brightness image.

[0045] The third selection subunit is used to select brightness adjustment parameters from a preset brightness adjustment parameter range;

[0046] The adjustment subunit is used to perform brightness adjustment processing on the normalized brightness image based on the brightness adjustment parameters to obtain the adjusted brightness image;

[0047] The inverse normalization processing subunit is used to perform inverse normalization processing on the adjusted brightness image to obtain the sample first brightness image.

[0048] Optionally, in some embodiments of this application, the parameter adjustment unit may include an image fusion subunit, a loss calculation subunit, and a parameter adjustment subunit, as follows:

[0049] The image fusion subunit is used to fuse the sample brightness image, the channel images of the other channels in the first region image except the brightness channel, and the second region image to obtain a sample composite image.

[0050] A loss calculation subunit is used to calculate image loss information between the sample image and the synthesized sample image;

[0051] The parameter adjustment subunit is used to adjust the parameters of the brightness parameter perception model based on the image loss information to obtain the trained brightness parameter perception model.

[0052] In this application embodiment, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0053] An electronic device provided in this application includes a processor and a memory. The memory stores multiple instructions, and the processor loads the instructions to execute the steps in the image processing method provided in this application.

[0054] This application also provides a computer-readable storage medium storing a computer program thereon, wherein the computer program, when executed by a processor, implements the steps in the image processing method provided in this application.

[0055] Furthermore, embodiments of this application also provide a computer program product, including a computer program or instructions, which, when executed by a processor, implement the steps in the image processing method provided in embodiments of this application.

[0056] This application provides an image processing method and related equipment, which can acquire a target object image to be synthesized, a reference image, and a brightness parameter perception model. The method extracts images from the target object image and the reference image using their respective brightness channels to obtain a first brightness image corresponding to the target object image and a second brightness image corresponding to the reference image. It then performs downsampling processing on the first brightness image and the second brightness image to obtain target first brightness information and target second brightness information. Based on the target first brightness information and the target second brightness information, it determines target brightness information. Using the brightness parameter perception model, it extracts features from the target brightness information to obtain target brightness adjustment parameters. According to the target brightness adjustment parameters, it performs brightness adjustment processing on the first brightness image to obtain a target brightness image. Finally, it performs image fusion processing on the target brightness image, channel images from other channels of the target object image (excluding the brightness channel), and the reference image to obtain and output a target synthesized image.

[0057] This application can downsample the brightness images of both the target image and the reference image to obtain first and second brightness information of the target, thereby determining the target brightness information. This target brightness information is then input into a brightness parameter perception model to obtain target brightness adjustment parameters. Based on these parameters, the brightness of the first brightness image is adjusted, and then the image is synthesized with the reference image. This ensures brightness matching between the target image and the reference image, resolving the problem of brightness inconsistency in the synthesized image, improving the synthesis effect, and enhancing the visual harmony of the synthesized image. Furthermore, this brightness adjustment process only processes the brightness channel image and downsamples the brightness image, significantly reducing the amount of data processing and improving data processing efficiency. The model input is the downsampled target brightness information, and the output is the brightness adjustment parameters. This model algorithm is very lightweight, capable of real-time image processing, and has strong applicability to various scenarios. Attached Figure Description

[0058] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0059] Figure 1a This is a schematic diagram of a scene illustrating the image processing method provided in an embodiment of this application;

[0060] Figure 1b This is a flowchart of the image processing method provided in the embodiments of this application;

[0061] Figure 1c This is a model structure diagram of the image processing method provided in the embodiments of this application;

[0062] Figure 1d This is a schematic diagram of a page illustrating the image processing method provided in an embodiment of this application;

[0063] Figure 1e These are illustrative diagrams illustrating the image processing method provided in the embodiments of this application;

[0064] Figure 1f This is another flowchart of the image processing method provided in the embodiments of this application;

[0065] Figure 1g This is another illustrative diagram of the image processing method provided in the embodiments of this application;

[0066] Figure 2 This is another flowchart of the image processing method provided in the embodiments of this application;

[0067] Figure 3 This is a schematic diagram of the structure of the image processing apparatus provided in the embodiments of this application;

[0068] Figure 4 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0069] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0070] This application provides an image processing method and related equipment. The related equipment may include an image processing apparatus, an electronic device, a computer-readable storage medium, and a computer program product. Specifically, the image processing apparatus may be integrated into an electronic device, which may be a terminal or a server, etc.

[0071] It is understood that the image processing method of this embodiment can be executed on a terminal, on a server, or jointly by a terminal and a server. The above examples should not be construed as limiting this application.

[0072] like Figure 1aAs shown, an image processing method is executed jointly by a terminal and a server, as an example. The image processing system provided in this application includes a terminal 10 and a server 11, etc.; the terminal 10 and the server 11 are connected via a network, such as a wired or wireless network, etc., wherein the image processing device can be integrated into the server.

[0073] Server 11 can be used to: acquire a target object image to be synthesized, a reference image, and a brightness parameter perception model; extract images from the target object image and the reference image in the brightness channel to obtain a first brightness image corresponding to the target object image and a second brightness image corresponding to the reference image; perform downsampling processing on the first brightness image and the second brightness image to obtain target first brightness information and target second brightness information; determine target brightness information based on the target first brightness information and the target second brightness information; extract features from the target brightness information through the brightness parameter perception model to obtain target brightness adjustment parameters; perform brightness adjustment processing on the first brightness image according to the target brightness adjustment parameters to obtain a target brightness image; perform image fusion processing on the target brightness image, channel images in other channels of the target object image besides the brightness channel, and the reference image to obtain a target synthesized image; and send the target synthesized image to terminal 10 for display. Among them, server 11 can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms.

[0074] Terminal 10 can be used to: send the target object image and reference image to server 11, so that server 11 can perform image compositing on the target object image and reference image; terminal 10 can also be used to receive the target composite image sent by server 11 and display the target composite image. Terminal 10 may include a mobile phone, vehicle terminal, aircraft, tablet computer, laptop computer, or personal computer (PC), etc. A client can also be set on terminal 10, which can be an application client or a browser client, etc.

[0075] The image processing and other steps performed on the server 11 can also be executed by the terminal 10.

[0076] The image processing method provided in this application relates to computer vision technology and machine learning in the field of artificial intelligence.

[0077] Artificial intelligence (AI) is the theory, methods, technology, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to have perception, reasoning, and decision-making capabilities. AI technology is a comprehensive discipline involving a wide range of fields, encompassing both hardware and software technologies. AI software technology mainly includes computer vision, speech processing, natural language processing, as well as machine learning / deep learning, autonomous driving, and intelligent transportation.

[0078] Computer vision (CV) is a science that studies how to enable machines to "see." More specifically, it refers to machine vision, which uses cameras and computers to replace human eyes for target recognition and measurement, and further processes images to create images more suitable for human observation or transmission to instruments. As a scientific discipline, computer vision studies related theories and technologies, attempting to build artificial intelligence systems capable of extracting information from images or multidimensional data. Computer vision technologies typically include image processing, image recognition, image semantic understanding, image retrieval, OCR, video processing, video semantic understanding, video content / behavior recognition, 3D object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous localization and mapping (SLAM), autonomous driving, intelligent transportation, and other technologies. It also includes common biometric recognition technologies such as facial recognition, palmprint recognition, and fingerprint recognition.

[0079] Machine Learning (ML) is a multidisciplinary field involving probability theory, statistics, approximation theory, convex analysis, and algorithm complexity theory. It specifically studies how computers can simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to endow computers with intelligence; its applications span all areas of artificial intelligence. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and instructional learning.

[0080] The image processing method provided in this application also relates to the field of cloud computing in cloud technology, specifically to cloud conferencing in the field of cloud computing.

[0081] Cloud technology refers to a hosting technology that unifies hardware, software, and network resources within a wide area network (WAN) or local area network (LAN) to achieve data computation, storage, processing, and sharing. Cloud technology is a collective term for network technology, information technology, integration technology, management platform technology, and application technology applied based on the cloud computing business model. It can form resource pools, be used on demand, and is flexible and convenient. Cloud computing technology will become a crucial support. Backend services of technical network systems require substantial computing and storage resources, such as video websites, image websites, and many portal websites. With the rapid development and application of the internet industry, every item may have its own identification mark in the future, requiring transmission to backend systems for logical processing. Data of different levels will be processed separately, and various industry data will require robust system support, which can only be achieved through cloud computing.

[0082] Cloud computing is a computing model that distributes computing tasks across a resource pool composed of a large number of computers, enabling various application systems to obtain computing power, storage space, and information services as needed. The network providing these resources is called the "cloud." From the user's perspective, the resources in the "cloud" are infinitely scalable, readily available, on-demand, expandable, and pay-as-you-go. As the foundational providers of cloud computing capabilities, they establish cloud resource pools (referred to as cloud platforms, generally called IaaS (Infrastructure as a Service) platforms), deploying various types of virtual resources within these pools for external customers to choose from. The cloud resource pool primarily includes: computing devices (virtualized machines containing operating systems), storage devices, and network devices. Logically, a PaaS (Platform as a Service) layer can be deployed on top of the IaaS layer, and a SaaS (Software as a Service) layer can be deployed on top of the PaaS layer. Alternatively, SaaS can be directly deployed on top of IaaS. PaaS is a platform for running software, such as databases and web (world wide area network) containers. SaaS encompasses various types of business software, such as web portals and bulk SMS messaging tools. Generally speaking, SaaS and PaaS are upper-layered compared to IaaS.

[0083] Cloud conferencing is an efficient, convenient, and low-cost form of conferencing based on cloud computing technology. Users only need to use an internet interface for simple and easy-to-use operations to quickly and efficiently share voice, data files, and video with teams and clients around the world. The complex technologies involved in data transmission and processing during the meeting are handled by the cloud conferencing service provider. Currently, cloud conferencing in China mainly focuses on services based on the SaaS (Software as a Service) model, including telephone, internet, and video services. Video conferencing based on cloud computing is called cloud conferencing. In the era of cloud conferencing, data transmission, processing, and storage are all handled by the computer resources of the video conferencing provider. Users no longer need to purchase expensive hardware or install cumbersome software; they only need to open a browser and log in to the corresponding interface to conduct efficient remote meetings. Cloud conferencing systems support dynamic multi-server cluster deployment and provide multiple high-performance servers, greatly improving meeting stability, security, and availability. In recent years, video conferencing has gained popularity due to its ability to significantly improve communication efficiency, continuously reduce communication costs, and upgrade internal management levels, and has been widely applied in various fields such as transportation, finance, telecommunications, education, enterprises, and the Internet of Vehicles. Undoubtedly, with the application of cloud computing, video conferencing has become more attractive in terms of convenience, speed, and ease of use, which will surely stimulate a new wave of video conferencing applications.

[0084] The following sections provide detailed descriptions of each example. It should be noted that the order in which the embodiments are described is not intended to limit the preferred order of the embodiments.

[0085] This embodiment will be described from the perspective of an image processing device, which can be integrated into an electronic device, such as a server or a terminal.

[0086] It is understood that in the specific embodiments of this application, data such as user information are involved. When the above embodiments of this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0087] like Figure 1b As shown, the specific process of this image processing method can be as follows:

[0088] 101. Obtain the target object image to be synthesized, the reference image, and the brightness parameter perception model, and extract the image under the brightness channel of the target object image and the reference image respectively to obtain the first brightness image corresponding to the target object image and the second brightness image corresponding to the reference image.

[0089] In this context, the target image and the reference image are the images to be synthesized. In some embodiments, the target image may be the unsegmented original image. In other embodiments, the target image may be an image segmented from a certain image, such as by segmenting the region of interest (ROI) of an image. The image corresponding to the segmented ROI is the target image, which contains the object of user interest. For example, the target image may be an image obtained by performing portrait segmentation on a video frame, where the portrait region in the video frame is the ROI, and specifically, the foreground region of the video frame. The reference image may specifically be a background image.

[0090] Specifically, image segmentation processing can be performed on the target object in the original image to obtain the target object image. The original image can be a video frame or a simple image; this embodiment does not impose any limitations on this. The segmented target object regions constitute the target object image.

[0091] In a specific scenario, if the background of a person in a live broadcast or video conference is to be replaced, the target image can be the person's image area in the live broadcast or video conference. Specifically, the live broadcast or video conference can be segmented to obtain the foreground area as the target image; while the reference image can be a virtual background image to be composited with the target image.

[0092] In this embodiment, the brightness parameter perception model can be a neural network model, such as a Visual Geometry Group Network (VGGNet), a Residual Network (ResNet), or a Dense Convolutional Network (DenseNet), etc. However, it should be understood that the neural network in this embodiment is not limited to the types listed above.

[0093] In a specific embodiment, the target image and the reference image can be images in RGB (Red Green Blue) color mode. When the target image and the reference image are RGB images, color space conversion processing can be performed on the target image and the reference image to convert them from the RGB color space to the YUV color space.

[0094] In the YUV color space, Y represents the luminance component and UV represents the chrominance component. The color of a pixel can be determined by the luminance and chrominance components.

[0095] In this embodiment, the first brightness image is the brightness image of the target object image, that is, the first brightness image is the image of the target object image in the Y channel; the second brightness image is the brightness image of the reference image, that is, the second brightness image is the image of the reference image in the Y channel, where the Y channel is the brightness channel.

[0096] 102. Perform downsampling processing on the first brightness image and the second brightness image respectively to obtain the first brightness information and the second brightness information of the target.

[0097] Downsampling can reduce the amount of subsequent data processing and significantly improve data processing efficiency. There are several ways to downsampling. For example, downsampling can be performed by proportionally reducing the width and height of the feature map, such as changing the brightness image from (W, H) to (W / 2, H / 2); another example is downsampling by randomly selecting a number of pixels from the brightness image as the downsampled brightness image.

[0098] Optionally, in this embodiment, the step "downsampling the first brightness image and the second brightness image respectively to obtain the first brightness information and the second brightness information of the target" may include:

[0099] Select a plurality of first target pixels from the pixels of the first brightness image;

[0100] Based on the plurality of first target pixels, construct the target first brightness information;

[0101] Select multiple second target pixels from the pixels of the second brightness image;

[0102] Based on the plurality of second target pixels, target second brightness information is constructed.

[0103] The selection method can be random or according to a preset method, which can be determined based on the actual situation. This embodiment does not impose any restrictions on this. For example, the preset method could be to select one pixel every N pixels.

[0104] In this process, the selected pixels in the first brightness image are designated as the first target pixels, and the selected pixels in the second brightness image are designated as the second target pixels.

[0105] Specifically, the first brightness image can be reconstructed based on multiple first target pixels to obtain a reconstructed first brightness image, where the target first brightness information is the reconstructed first brightness image. This reconstruction method can involve rearranging the selected multiple first target pixels according to a matrix shape. For example, based on selecting K first target pixels, a matrix H*W can be constructed, where matrix H*W represents the target first brightness information, and H*W = K.

[0106] Specifically, the second brightness image can be reconstructed based on multiple second target pixels to obtain a reconstructed second brightness image, where the target second brightness information is the reconstructed second brightness image. This reconstruction method can involve rearranging the selected multiple second target pixels according to a matrix shape. For example, based on selecting K second target pixels, a matrix H*W can be constructed, where matrix H*W represents the target second brightness information, and H*W = K.

[0107] In a specific scenario, the downsampling process for the first brightness image and the second brightness image can be as follows:

[0108] Take the set of coordinates of the pixels in the first brightness image, then randomly select K coordinates from the coordinate set, take the pixels corresponding to these K coordinates (i.e. the first target pixels mentioned above) as the downsampled first brightness image, and adjust the shape to H*W to obtain the target first brightness information, where H*W=K;

[0109] Take the set of coordinates of the pixels in the second brightness image, then randomly select K coordinates from the coordinate set, and use the pixels corresponding to these K coordinates (i.e. the second target pixels mentioned above) as the downsampled second brightness image, and adjust the shape to H*W, thereby obtaining the target second brightness information, where H*W=K.

[0110] 103. Based on the first brightness information and the second brightness information of the target, determine the target brightness information.

[0111] One method is to fuse the first brightness information and the second brightness information of the target to obtain the target brightness information. There are various fusion methods, such as stitching.

[0112] Optionally, in this embodiment, the step "determining target brightness information based on the first target brightness information and the second target brightness information" may include:

[0113] The first brightness information and the second brightness information of the target are fused to obtain the fused brightness information;

[0114] The fused brightness information is normalized to obtain the target brightness information.

[0115] By normalizing the data, the range can be scaled down to 0 to 1, which facilitates the processing of the brightness parameter perception model in the future.

[0116] 104. Using the brightness parameter perception model, feature extraction is performed on the target brightness information to obtain the target brightness adjustment parameters.

[0117] Among them, a brightness parameter perception model can be used to extract features from the target brightness information, and the target brightness adjustment parameters can be predicted based on the extracted feature information.

[0118] Specifically, the target brightness information includes the brightness data distribution information of the target image and the reference image. Through model training, the brightness parameter perception model can learn "how to transform the brightness of one image to be closer to the style of the other image based on the brightness distribution information of the two images". When the target brightness information is input into the brightness parameter perception model, the model can output parameters for adjusting the image brightness of the target image, i.e., output target brightness adjustment parameters.

[0119] The target brightness adjustment parameter is used to adjust the brightness of the first brightness image (or the target object image) so that the brightness of the target object image and the reference image are more matched, thereby improving the image synthesis effect.

[0120] Specifically, the target image can be an image from a video. In a specific scenario, image compositing processing needs to be performed on each frame of the video. The background region in each video frame is replaced with a reference image (specifically, a virtual background image), and the foreground region of each video frame is combined with the reference image to create a new video frame. To improve the realism of the composite video frame, the image processing method provided in this application can be used to predict the brightness adjustment parameters of the foreground region. The brightness of the foreground region can then be adjusted based on these parameters to match the brightness of the adjusted foreground region with that of the virtual background image. Since video frames are continuous, but model predictions may contain abrupt changes, the brightness of a particular frame may differ significantly from that of its neighboring frames, causing brightness jitter in the video. This application provides a method to eliminate brightness jitter by using the brightness adjustment parameters corresponding to adjacent video frames to smooth the brightness adjustment parameters output by the model, thus ensuring that the final brightness adjustment parameter value does not change significantly.

[0121] Optionally, in this embodiment, the step "extracting features from the target brightness information using the brightness parameter perception model to obtain target brightness adjustment parameters" may include:

[0122] The initial brightness adjustment parameters are obtained by extracting features from the target brightness information using the brightness parameter perception model.

[0123] Obtain the reference brightness adjustment parameters of the preceding image associated with the target object image;

[0124] Based on the reference brightness adjustment parameters, the initial brightness adjustment parameters are smoothed to obtain the target brightness adjustment parameters.

[0125] The preceding image is an image that has a temporal relationship with the target object image. Specifically, the preceding image is an adjacent video frame to the video frame to which the target object image belongs, and can be the previous frame of the video frame to which the target object image belongs. The reference brightness adjustment parameters of the preceding image are obtained in the process of "adjusting the brightness of the object image corresponding to the preceding image in order to perform image compositing with the virtual background image".

[0126] The specific process of synthesizing the object image and virtual background image corresponding to the preceding image is as follows: Image segmentation is performed on the object region in the preceding image to obtain the object image corresponding to the preceding image. Then, the brightness images of the object image and the virtual background image are downsampled to obtain downsampled brightness images of the object image and the virtual background image. These downsampled brightness images are then fused to obtain fused brightness information. Features are extracted from the fused brightness information using a brightness parameter perception model to obtain initial brightness adjustment parameters. Based on the brightness adjustment parameters corresponding to the previous video frame of the preceding image, these initial brightness adjustment parameters are smoothed to obtain the final brightness adjustment parameters, which are the reference brightness adjustment parameters for the preceding image. The brightness image of the object image is then adjusted using these parameters to obtain the brightness-adjusted object image. Finally, the brightness-adjusted object image and the virtual background image are synthesized, resulting in the preceding image after background replacement.

[0127] Optionally, in this embodiment, the step "extracting features from the target brightness information using the brightness parameter perception model to obtain initial brightness adjustment parameters" may include:

[0128] The target brightness information is processed by convolution through the brightness parameter perception model to obtain convolution feature information;

[0129] The convolutional feature information is pooled to obtain pooled feature information;

[0130] The pooled feature information is processed by a fully connected layer to obtain the initial brightness adjustment parameters.

[0131] The pooling process can include max pooling, average pooling, and generalized mean pooling, etc., and this embodiment does not limit it. Pooling can reduce the dimensionality of feature information.

[0132] In a specific scenario, the model structure of the brightness parameter perception model is as follows: Figure 1c As shown, the specific description is as follows:

[0133] The learnable parameters of the brightness parameter-aware model consist of four convolutional layers and one fully connected layer. The convolutional layers employ separable convolutions to accelerate processing on the CPU (Central Processing Unit). Each separable convolutional layer contains one depthwise convolutional layer and one pointwise convolutional layer. Specifically, the depthwise convolutional layer has a 3×3 kernel size and a stride of 2, while the pointwise convolutional layer has a 1×1 kernel size and a stride of 1. All four separable convolutional layers can use the same parameter settings, with 16 channels per kernel. Furthermore, in the brightness parameter-aware model, a ReLU (Rectified Linear Unit) layer can be appended after each separable convolutional layer, followed by a global average pooling layer after the fourth ReLU layer. After the target brightness information is input into the brightness parameter perception model, it undergoes layer-by-layer processing through separable convolutional layer 1, corrected linear unit 1, separable convolutional layer 2, corrected linear unit 2, separable convolutional layer 3, corrected linear unit 3, separable convolutional layer 4, and corrected linear unit 4 to obtain convolutional feature information. This convolutional feature information is then input into a global average pooling layer, which, after passing through the global average pooling layer, yields a 1*16 dimensional feature (i.e., the pooled feature information mentioned above). Finally, it passes through a fully connected layer with an input dimension of 16 and an output dimension of 1, and a Clip (Contrastive Language-Image Pre-training) layer to output the brightness adjustment parameter x. The brightness adjustment parameter can be set to a range of 0.3 to 1.5 based on manual experience; values ​​smaller than this range will become 0.3, and values ​​larger than this range will become 1.5.

[0134] It should be noted that the parameter settings, network structure and number of layers of the model are not limited to the settings mentioned above. The brightness parameter perception model can also adopt other model architectures, such as fully connected neural networks, Transformers, etc.

[0135] Optionally, in this embodiment, the step "smoothing the initial brightness adjustment parameters based on the reference brightness adjustment parameters to obtain the target brightness adjustment parameters" may include:

[0136] The parameter weights of the reference brightness adjustment parameter and the initial brightness adjustment parameter are determined respectively;

[0137] Based on the parameter weights, the reference brightness adjustment parameters and the initial brightness adjustment parameters are fused to obtain the target brightness adjustment parameters.

[0138] The parameter weights of the reference brightness adjustment parameter and the initial brightness adjustment parameter can be set according to the actual situation. Specifically, the sum of the parameter weights of the two is 1.

[0139] One method for this fusion process is to perform a weighted summation based on parameter weights.

[0140] Specifically, the current frame to which the target object image belongs is frame t, and its initial brightness adjustment parameters can be smoothed using the weighted moving average method of the following formula (1):

[0141] s t =a*s t-1 +(1-a)*x (1)

[0142] Among them, s t-1 Let represent the reference brightness adjustment parameters of the preceding image (specifically, the brightness adjustment parameters after smoothing in frame t-1), x represent the initial brightness adjustment parameters, a represent the parameter weights of the reference brightness adjustment parameters, 1-a represent the parameter weights of the initial brightness adjustment parameters, and s represent the reference brightness adjustment parameters. t This represents the target brightness adjustment parameter (i.e., the brightness adjustment parameter after smoothing in frame t).

[0143] Here, 'a' can be a manually defined weight, ranging from 0 to 1. The larger 'a' is, the stronger the smoothing effect. In this embodiment, the value of 'a' can be set to 0.9.

[0144] It is understandable that the smoothing method for the initial brightness adjustment parameters is not limited to the exponential moving average (i.e., the weighted moving average mentioned above), but can also be smoothed by methods such as ridge regression.

[0145] Ridge regression is a least squares estimation method used for biased estimation regression in collinear data analysis. It sacrifices the unbiasedness of least squares, at the cost of losing some information and reducing accuracy, to obtain regression coefficients that are more realistic and reliable.

[0146] 105. According to the target brightness adjustment parameters, perform brightness adjustment processing on the first brightness image to obtain the target brightness image.

[0147] In this embodiment, there are various methods for brightness adjustment, and this embodiment does not limit the methods used. For example, brightness adjustment can be performed through gamma transformation, LUT enhancement, linear enhancement, or other set nonlinear enhancement functions.

[0148] Here, LUT (Lookup Table) represents a mapping relationship. Through LUT, an input pixel array can be transformed into another pixel array. Specifically, different brightness adjustment parameters can correspond to different LUT tables. Each LUT table includes the mapping relationship between the input brightness value (i.e., the pixel value of a pixel in the first brightness image) and the output brightness value (i.e., the pixel value of a pixel in the target brightness image). In this embodiment, the LUT table corresponding to the target brightness adjustment parameter can be determined, and then the target brightness image can be determined based on the pixel values ​​of the pixels in the first brightness image and the LUT table.

[0149] Linear enhancement can be achieved by linearly transforming the pixel values ​​of pixels in the first brightness image based on the target brightness adjustment parameters, thereby obtaining the target brightness image.

[0150] Optionally, in this embodiment, the step "performing brightness adjustment processing on the first brightness image according to the target brightness adjustment parameters to obtain the target brightness image" may include:

[0151] Based on the target brightness adjustment parameters, the first brightness image is subjected to gamma transformation processing to obtain the target brightness image.

[0152] Specifically, for each pixel in the first brightness image, based on the target brightness adjustment parameters, a gamma transform is performed on the pixel value of the pixel to obtain the pixel value of the corresponding pixel in the target brightness image.

[0153] Specifically, the gamma transformation can be performed using the following formula (2):

[0154]

[0155] Among them, S t This represents the target brightness adjustment parameter, where I represents the first brightness image. adj This represents the adjusted value, i.e., the target brightness image.

[0156] Specifically, the normalized first brightness image can be adjusted according to the target brightness adjustment parameters to obtain the processed first brightness image; then the processed first brightness image can be inversely normalized to obtain the target brightness image.

[0157] 106. Perform image fusion processing on the target brightness image, the channel images of the target object image other than the brightness channel, and the reference image to obtain and output the target composite image.

[0158] Optionally, in this embodiment, the step of "performing image fusion processing on the target brightness image, the channel images of the target object image excluding the brightness channel, and the reference image to obtain and output the target composite image" may include:

[0159] The target brightness image is fused with the channel images of the target object image in other channels besides the brightness channel to obtain the object image after brightness processing.

[0160] The brightness-processed object image is fused with the reference image to obtain and output the target composite image.

[0161] Among them, the channel images of the target object image, excluding the luminance channel, can include images under the UV channels.

[0162] Specifically, the image fusion of the brightness-processed object image and the reference image can include: converting the brightness-processed object image from the YUV color space to the RGB color space, and then fusing the converted object image with the reference image. The image fusion method can be: overlaying the converted object image on the reference image.

[0163] Specifically, the image processing method provided in this application can be used in any video editing or online video conferencing or live streaming scenario, including but not limited to conferencing software, live streaming software, video processing software, etc., with specific application scenarios as follows: Figure 1d As shown. Figure 1d The software settings page is shown, providing a function to replace the original image background with a virtual background. The settings page can include a virtual background and a beautification module, which can include virtual background controls, beautification controls, filter controls, headwear controls, and border controls. When a trigger operation (such as a click) is detected on the virtual background control, multiple virtual background images can be displayed on the page. The user can select a target virtual background image to replace the background area in the video image, as described in the previous embodiment. Optionally, the page can also provide a virtual background preview function. When the user selects a virtual background image, the video frame under that virtual background image can be previewed in the preview area of ​​the page.

[0164] It should be noted that the beauty filter control can be used to beautify the portraits in the video, the filter control can be used to add filters to the video, the headwear control can be used to add headwear to the portraits in the video, and the border control can be used to add various styles of borders to the video.

[0165] In addition, the software settings page can also include other modules such as general settings, video, audio, recording, subtitles and transcription, AI smart assistant, account and security, quality inspection, and shortcut keys, which users can select according to their actual needs.

[0166] The image processing method of this application can improve the foreground-background fusion effect in virtual background functions by changing the foreground illumination intensity. It can be applied to online video conferencing scenarios, live streaming scenarios, etc. For example, it can be used to replace the background image in a video conference with other virtual backgrounds. Specifically, the foreground region in the video frame can be segmented as the target object image, and then image synthesis is performed between the foreground region and the virtual background image (i.e., the reference image) using the image processing method provided in this application. Here, the foreground specifically refers to the image of the participants who are not in the surrounding environment.

[0167] Virtual backgrounds are an application of human face segmentation technology. During video conferences, participants can be automatically identified, and their surroundings can be replaced with a specified background image.

[0168] Furthermore, the brightness parameter perception model in this application takes downsampled brightness information as input and outputs brightness adjustment parameters. Both the input and output data volumes are relatively small, reducing computational load and enabling real-time image processing. This allows for applications in scenarios such as online video conferencing and live streaming. Additionally, the brightness adjustment parameters are determined by considering inter-frame transitions. To prevent frame jumps, the brightness adjustment parameters of the preceding frame are incorporated, making it better suited for continuous-frame scenarios such as online video conferencing.

[0169] Optionally, in this embodiment, before the step "extracting features from the target brightness information using the brightness parameter perception model to obtain the target brightness adjustment parameters", the image processing method may further include:

[0170] Acquire training data, which includes multiple sample images containing sample objects; and perform image segmentation on the sample images to obtain a first region image containing sample objects and a second region image not containing sample objects.

[0171] The brightness image of the first region image is subjected to brightness adjustment processing to obtain the sample first brightness image;

[0172] The brightness images corresponding to the first brightness image and the second region image of the sample are downsampled respectively to obtain the first brightness information and the second brightness information of the target sample; and the brightness information of the target sample is determined based on the first brightness information and the second brightness information of the target sample.

[0173] By using a brightness parameter perception model, feature extraction is performed on the brightness information of the target sample to obtain the sample brightness adjustment parameters.

[0174] According to the sample brightness adjustment parameters, the first sample brightness image is subjected to brightness adjustment processing to obtain the sample brightness image;

[0175] Based on the sample brightness image and the sample image, the parameters of the brightness parameter perception model are adjusted to obtain the trained brightness parameter perception model.

[0176] The first region image represents the foreground area of ​​the sample image, and the second region image represents the background area of ​​the sample image. The sample object can specifically be foreground elements such as a person in the sample image.

[0177] The target sample's first brightness information is the downsampled first brightness image, and the target sample's second brightness information is the downsampled brightness image corresponding to the second region image. The target sample's brightness information is obtained by fusing (e.g., stitching) the target sample's first and second brightness information. Specifically, the target sample's first and second brightness information can be stitched together to obtain stitched sample brightness information, and then normalized to obtain normalized target sample brightness information.

[0178] The sample brightness image is the brightness image of the first region image after brightness adjustment. In some embodiments, a loss value can be calculated between the sample brightness image and the brightness image corresponding to the first region image based on the pixel values ​​of the pixels in the sample brightness image and the brightness values ​​of the pixels in the corresponding first region image. The parameters of the brightness parameter perception model are then adjusted based on the loss value to obtain the trained brightness parameter perception model. This training process can be as follows: the parameters of the brightness parameter perception model are adjusted using a backpropagation algorithm, and the parameters are optimized based on the loss value so that the sample brightness image approaches the brightness image corresponding to the first region image, thus obtaining the trained brightness parameter perception model. Specifically, the calculated loss value can be made less than a preset loss value, which can be set according to the actual situation.

[0179] It should be noted that there is no need to smooth the brightness adjustment parameters during training, because the sample images are single-frame images.

[0180] Optionally, in this embodiment, the step "performing brightness adjustment processing on the brightness image of the first region image to obtain a sample first brightness image" may include:

[0181] The brightness image corresponding to the first region image is normalized to obtain a normalized brightness image.

[0182] Select a brightness adjustment parameter from the preset brightness adjustment parameter range;

[0183] Based on the brightness adjustment parameters, the normalized brightness image is subjected to brightness adjustment processing to obtain the adjusted brightness image;

[0184] The adjusted brightness image is subjected to inverse normalization to obtain the first brightness image of the sample.

[0185] The preset brightness adjustment parameter range can be set according to actual conditions; for example, it can be set to 0.8 to 2. In this embodiment, a number can be randomly selected from the preset brightness adjustment parameter range as the brightness adjustment parameter.

[0186] There are several methods for adjusting the brightness of a normalized brightness image. For example, a gamma transform can be performed on the normalized brightness image based on the brightness adjustment parameters to obtain the adjusted brightness image; brightness adjustment can also be achieved through methods such as LUT enhancement and linear enhancement.

[0187] Optionally, in this embodiment, the step "adjusting the parameters of the brightness parameter perception model based on the sample brightness image and the sample image to obtain the trained brightness parameter perception model" may include:

[0188] The sample brightness image, the channel images of the other channels in the first region image (excluding the brightness channel), and the second region image are fused to obtain a sample composite image.

[0189] Calculate the image loss information between the sample image and the synthesized sample image;

[0190] Based on the image loss information, the parameters of the brightness parameter perception model are adjusted to obtain the trained brightness parameter perception model.

[0191] Specifically, the channel images of the first region image, excluding the luminance channel, are the UV channels of the first region image, with the luminance channel being the Y channel. In this embodiment, the sample luminance image and the UV channels of the first region image are first fused to obtain a luminance-adjusted first region image; then, the luminance-adjusted first region image undergoes color space conversion from the YUV color space to the RGB color space; finally, the converted image is fused with the second region image to obtain a composite sample image. Specifically, this image fusion method can involve overlaying the converted image onto the second region image.

[0192] There are various ways to calculate image loss information, such as using loss functions like L1 (norm 1) loss, L2 (norm 2) loss, GAN (Generative Adversarial Networks) loss, or perceptual loss.

[0193] In a specific scenario, the training process of the brightness parameter perception model can be described as follows:

[0194] First, a large number of face images with different backgrounds can be collected as the original data for training this scheme. For an original image P from the training set (i.e., the sample image mentioned above), face segmentation is performed to obtain the foreground region A (specifically, the first region image mentioned above) and the background region B (specifically, the second region image mentioned above).

[0195] For the foreground region A, take the Y channel of the foreground region A, normalize it to the range of 0 to 1, and randomly select a number x from the interval of 0.8-2 as the brightness adjustment parameter. Use x to perform a gamma transform on the normalized Y channel, and then perform inverse normalization on the transformed Y channel. Combine the inverse normalized Y channel (i.e. the first brightness image of the above sample) with the original UV channel of the foreground region A to obtain the processed foreground region D. This step is to artificially change the brightness of the foreground region. Then, the Y channels of the foreground region D and the background region B are downsampled separately. The downsampled Y channels are then concatenated according to their channel dimensions, and the concatenated Y channels are normalized to obtain the target sample brightness information. The brightness information of the target sample is feature-extracted using a brightness parameter perception model, and a brightness adjustment parameter is output. Based on this brightness adjustment parameter, the brightness of the normalized first brightness image of the sample is adjusted, and then the adjusted first brightness image of the sample is inversely normalized to obtain the sample brightness image. Finally, the sample brightness image is merged with the original UV channels of the foreground region A to obtain the brightness-adjusted foreground region. The brightness-adjusted foreground region is then fused with the background region B to obtain the sample composite image R.

[0196] It is important to note that the brightness adjustment parameters do not need to undergo exponential moving average during training because it is a single-frame image.

[0197] The goal of model training is to make R approach P, which means restoring the artificially created inconsistencies in foreground and background brightness to their initial normal state. After training, given different foreground and background images, it can automatically adjust the foreground brightness. The L1 loss in equation (3) can be used to calculate the loss value during model training:

[0198] L1=|PR| (3)

[0199] Where L1 represents the image loss information between the sample image P and the synthesized sample image R.

[0200] This application proposes an automatic brightness adjustment method for virtual background and foreground. By analyzing the lighting conditions of the foreground and virtual background, it intelligently adjusts the brightness of foreground elements, such as faces and clothing, to match the brightness level of the virtual background image. This method is particularly suitable for users conducting video communication (such as video conferencing) in environments with uneven lighting. It effectively avoids visual disharmony caused by an overly dark foreground and an overly bright background, improves the realism of the synthesized image, enhances the user's professional image in video conferencing, and provides a more immersive and professional video conferencing experience. It has broad application prospects and market potential. Furthermore, the intelligent foreground brightness adjustment algorithm of this application is very lightweight. Its algorithm model takes downsampled foreground and background pixels as input and outputs brightness adjustment parameters. This algorithm can be computed in real time on most machines and introduces a smoothing mechanism for adjustment parameters between frames, enabling natural and smooth video transitions.

[0201] Specifically, such as Figure 1e As shown in the comparison images, image a is the original input (i.e., the real image); image b is the effect of a virtual background under current related technologies, where the foreground is too dark and the background is too bright, resulting in visual disharmony; image c shows the image effect after automatically adjusting the brightness of the virtual background foreground using the image processing method provided in this application. Figure 1e The comparison of the images shows that after introducing the image processing method of this application, the foreground figures and backgrounds are more integrated, and the virtual background function has a stronger sense of realism and immersion.

[0202] In one specific embodiment, the flowchart of the image processing method of this application can be as follows: Figure 1f As shown, the specific description is as follows:

[0203] 1. Receive video frames input from the camera and perform human image segmentation on the video frames;

[0204] 2. Convert the foreground region (i.e., the target object image above) obtained by portrait segmentation to RGB to YUV channels. The Y channel represents luminance information and the UV channel represents chrominance information. In this embodiment, only the luminance is adjusted, and only the Y channel is used to improve the calculation efficiency.

[0205] 3. Downsample the Y channel of the foreground region to obtain the downsampled Y channel (i.e., the first brightness information of the target mentioned above);

[0206] 4. At the same time, for the background area, that is, a virtual background image specified by the user (i.e., the reference image above), RGB to YUV channel is also converted, and the obtained Y channel is downsampled to obtain the downsampled background area Y channel (i.e. the target second brightness information).

[0207] Since photographers adjust colors based on the distribution of the brightness histogram, downsampling has a limited impact on the overall data distribution. Therefore, downsampling of the foreground and background can improve computational efficiency.

[0208] 5. The Y-channels of the downsampled foreground region and the downsampled background region are concatenated together according to the channel dimension. For example, if the data size of the downsampled foreground region Y-channel is 96*54 and the data size of the downsampled background region Y-channel is 96*54, then the size after concatenation according to the channel dimension is 2*96*54;

[0209] 6. Normalize the spliced ​​result to obtain the target brightness information; normalization can scale the data range to within 0-1, making it convenient as input for neural networks;

[0210] 7. Input the target brightness information into the brightness parameter perception model to obtain the initial brightness adjustment parameter x for the foreground area;

[0211] 8. Since video frames are continuous, but model predictions may contain abrupt changes, the initial brightness adjustment parameter x needs to be smoothed to eliminate jitter. Specifically, the smoothed brightness adjustment parameter corresponding to the previous video frame (i.e., the reference brightness adjustment parameter of the preceding image) can be used to perform an exponential moving average on the initial brightness adjustment parameter x to obtain the smoothed target brightness adjustment parameter s. t ;

[0212] 9. Adjust the target brightness parameter s based on the smoothed value obtained earlier. t The brightness of the normalized Y channel of the foreground region that has not been downsampled is adjusted using a brightness adjustment algorithm, such as gamma transform.

[0213] Gamma transform can be referenced Figure 1gThe horizontal axis represents the input brightness value, i.e., the brightness value of the Y channel of the foreground area before adjustment; the vertical axis represents the output brightness value, i.e., the brightness value of the Y channel of the foreground area after adjustment. It should be noted that the input brightness value ranges from 0 to 1. When s t Less than 1 (e.g., s) t When = 1 / 2.2), the foreground brightness is less than the virtual background brightness, so the overall brightness of the foreground image needs to be increased. This can be achieved through gamma transformation. When s t Greater than 1 (e.g., s) t When the brightness is 2.2), the brightness of the foreground is greater than that of the virtual background, so the brightness of the foreground image needs to be reduced as a whole. This can be achieved by using gamma transformation.

[0214] 10. Perform inverse normalization on the Y channel of the foreground area after brightness adjustment, expanding the value range of 0-1 to the original range. If the original range is represented as a limit range, the scaled range will be 16-235. If the original range is represented as a full range, the scaled range will be 0-255.

[0215] 11. Combine the UV channel of the foreground area with the Y channel processed in step 10, and convert YUV to RGB channel to obtain the foreground area after brightness adjustment.

[0216] 12. Finally, the brightness-adjusted foreground area is fused with the virtual background image to obtain the final result—the target composite image.

[0217] The fusion method is alpha compositing, and the process of alpha compositing is as follows:

[0218] Obtain the transparency channel parameters of the foreground region after brightness adjustment. Based on these transparency channel parameters, overlay the brightness-adjusted foreground region onto the virtual background image to obtain the target composite image.

[0219] The transparency channel, also known as the alpha channel (α channel or alpha channel), refers to the transparency and semi-transparency of an image. Alpha channel values ​​can be used to set the image's transparency. Using the alpha transparency channel, special image overlays and composites can be achieved, making certain parts of an image transparent based on different principles. This involves removing selected colors or brightness from the image, allowing the de-colored portion to show through the background, while the remaining portions retain their original image.

[0220] The larger the alpha channel parameter, the lower the transparency; conversely, the smaller the alpha channel parameter, the higher the transparency.

[0221] In this system, the pixel color of the foreground area after brightness adjustment is the foreground color, and the pixel color of the virtual background image is the background color. For a 256-level red, green, and blue color channel, when the transparency channel parameter of the foreground pixels is 255, they are opaque (all foreground color); when the transparency channel parameter is 0, they are fully transparent (all background color). For pixels in the foreground area that need to be displayed, their transparency channel parameter can be set larger; for pixels in the foreground area that do not need to be displayed, their transparency channel parameter can be set smaller.

[0222] As can be seen from the above, this embodiment can acquire the target object image to be synthesized, the reference image, and the brightness parameter perception model, and extract images in the brightness channel of the target object image and the reference image respectively to obtain a first brightness image corresponding to the target object image and a second brightness image corresponding to the reference image; perform downsampling processing on the first brightness image and the second brightness image respectively to obtain target first brightness information and target second brightness information; determine target brightness information based on the target first brightness information and the target second brightness information; extract features from the target brightness information through the brightness parameter perception model to obtain target brightness adjustment parameters; perform brightness adjustment processing on the first brightness image according to the target brightness adjustment parameters to obtain the target brightness image; perform image fusion processing on the target brightness image, the channel images in other channels of the target object image except for the brightness channel, and the reference image to obtain and output the target synthesized image.

[0223] This application can downsample the brightness images of both the target image and the reference image to obtain first and second brightness information of the target, thereby determining the target brightness information. This target brightness information is then input into a brightness parameter perception model to obtain target brightness adjustment parameters. Based on these parameters, the brightness of the first brightness image is adjusted, and then the image is synthesized with the reference image. This ensures brightness matching between the target image and the reference image, resolving the problem of brightness inconsistency in the synthesized image, improving the synthesis effect, and enhancing the visual harmony of the synthesized image. Furthermore, this brightness adjustment process only processes the brightness channel image and downsamples the brightness image, significantly reducing the amount of data processing and improving data processing efficiency. The model input is the downsampled target brightness information, and the output is the brightness adjustment parameters. This model algorithm is very lightweight, capable of real-time image processing, and has strong applicability to various scenarios.

[0224] Based on the method described in the preceding embodiments, the following will provide a more detailed description using the example of the image processing device being specifically integrated into a server.

[0225] This application provides an image processing method, such as... Figure 2 As shown, the specific process of this image processing method can be as follows:

[0226] 201. The server acquires the target object image to be synthesized, the reference image, and the brightness parameter perception model, and performs image extraction in the brightness channel on the target object image and the reference image respectively to obtain the first brightness image corresponding to the target object image and the second brightness image corresponding to the reference image.

[0227] This process involves image segmentation of the target object in the original image to obtain an image of the target object. The original image can be a video frame or a simple image; this embodiment does not impose any limitations on this. The segmented target object regions constitute the target object image.

[0228] In a specific scenario, if the background of a person in a live broadcast or video conference is to be replaced, the target image can be the person's image area in the live broadcast or video conference. Specifically, the live broadcast or video conference can be segmented to obtain the foreground area as the target image; while the reference image can be a virtual background image to be composited with the target image.

[0229] 202. The server performs downsampling processing on the first brightness image and the second brightness image respectively to obtain the first brightness information and the second brightness information of the target.

[0230] Downsampling can reduce the amount of subsequent data processing and significantly improve data processing efficiency. There are several ways to downsampling. For example, downsampling can be performed by proportionally reducing the width and height of the feature map, such as changing the brightness image from (W, H) to (W / 2, H / 2); another example is downsampling by randomly selecting a number of pixels from the brightness image as the downsampled brightness image.

[0231] Optionally, in this embodiment, the step "downsampling the first brightness image and the second brightness image respectively to obtain the first brightness information and the second brightness information of the target" may include:

[0232] Select a plurality of first target pixels from the pixels of the first brightness image;

[0233] Based on the plurality of first target pixels, construct the target first brightness information;

[0234] Select multiple second target pixels from the pixels of the second brightness image;

[0235] Based on the plurality of second target pixels, target second brightness information is constructed.

[0236] The selection method can be random or according to a preset method, which can be determined based on the actual situation. This embodiment does not impose any restrictions on this. For example, the preset method could be to select one pixel every N pixels.

[0237] 203. The server fuses the first brightness information and the second brightness information of the target to obtain the target brightness information.

[0238] There are several ways to achieve this fusion, such as splicing.

[0239] Optionally, in this embodiment, the step of "fusing the first target brightness information and the second target brightness information to obtain target brightness information" may include:

[0240] The first brightness information and the second brightness information of the target are fused to obtain the fused brightness information;

[0241] The fused brightness information is normalized to obtain the target brightness information.

[0242] By normalizing the data, the range can be scaled down to 0 to 1, which facilitates the processing of the brightness parameter perception model in the future.

[0243] 204. The server extracts features from the target brightness information using the brightness parameter perception model to obtain the target brightness adjustment parameters.

[0244] Specifically, the target brightness information includes the brightness data distribution information of the target image and the reference image. Through model training, the brightness parameter perception model can learn "how to transform the brightness of one image to be closer to the style of the other image based on the brightness distribution information of the two images". When the target brightness information is input into the brightness parameter perception model, the model can output parameters for adjusting the image brightness of the target image, i.e., output target brightness adjustment parameters.

[0245] The target brightness adjustment parameter is used to adjust the brightness of the first brightness image (or the target object image) so that the brightness of the target object image and the reference image are more matched, thereby improving the image synthesis effect.

[0246] Specifically, the target image can be an image from a video. In a specific scenario, image compositing processing needs to be performed on each frame of the video. The background region in each video frame is replaced with a reference image (specifically, a virtual background image), and the foreground region of each video frame is combined with the reference image to create a new video frame. To improve the realism of the composite video frame, the image processing method provided in this application can be used to predict the brightness adjustment parameters of the foreground region. The brightness of the foreground region can then be adjusted based on these parameters to match the brightness of the adjusted foreground region with that of the virtual background image. Since video frames are continuous, but model predictions may contain abrupt changes, the brightness of a particular frame may differ significantly from that of its neighboring frames, causing brightness jitter in the video. This application provides a method to eliminate brightness jitter by using the brightness adjustment parameters corresponding to adjacent video frames to smooth the brightness adjustment parameters output by the model, thus ensuring that the final brightness adjustment parameter value does not change significantly.

[0247] Optionally, in this embodiment, the step "extracting features from the target brightness information using the brightness parameter perception model to obtain target brightness adjustment parameters" may include:

[0248] The initial brightness adjustment parameters are obtained by extracting features from the target brightness information using the brightness parameter perception model.

[0249] Obtain the reference brightness adjustment parameters of the preceding image associated with the target object image;

[0250] Based on the reference brightness adjustment parameters, the initial brightness adjustment parameters are smoothed to obtain the target brightness adjustment parameters.

[0251] The preceding image is an image that has a temporal relationship with the target object image. Specifically, the preceding image is an adjacent video frame to the video frame to which the target object image belongs, and can be the previous frame of the video frame to which the target object image belongs. The reference brightness adjustment parameters of the preceding image are obtained in the process of "adjusting the brightness of the object image corresponding to the preceding image in order to perform image compositing with the virtual background image".

[0252] Optionally, in this embodiment, the step "smoothing the initial brightness adjustment parameters based on the reference brightness adjustment parameters to obtain the target brightness adjustment parameters" may include:

[0253] The parameter weights of the reference brightness adjustment parameter and the initial brightness adjustment parameter are determined respectively;

[0254] Based on the parameter weights, the reference brightness adjustment parameters and the initial brightness adjustment parameters are fused to obtain the target brightness adjustment parameters.

[0255] 205. The server performs brightness adjustment processing on the first brightness image according to the target brightness adjustment parameters to obtain the target brightness image.

[0256] In this embodiment, there are various methods for brightness adjustment, and this embodiment does not limit the methods used. For example, brightness adjustment can be performed through gamma transformation, LUT enhancement, linear enhancement, or other set nonlinear enhancement functions.

[0257] Here, LUT (Lookup Table) represents a mapping relationship. Through LUT, an input pixel array can be transformed into another pixel array. Specifically, different brightness adjustment parameters can correspond to different LUT tables. Each LUT table includes the mapping relationship between the input brightness value (i.e., the pixel value of a pixel in the first brightness image) and the output brightness value (i.e., the pixel value of a pixel in the target brightness image). In this embodiment, the LUT table corresponding to the target brightness adjustment parameter can be determined, and then the target brightness image can be determined based on the pixel values ​​of the pixels in the first brightness image and the LUT table.

[0258] 206. The server performs a fusion process on the target brightness image and the channel images of the target object image, excluding the brightness channel, to obtain the brightness-processed object image.

[0259] Among them, the channel images of the target object image, excluding the luminance channel, can include images under the UV channels.

[0260] 207. The server performs image fusion between the brightness-processed object image and the reference image to obtain and output the target composite image.

[0261] Specifically, the image fusion of the brightness-processed object image and the reference image can include: converting the brightness-processed object image from the YUV color space to the RGB color space, and then fusing the converted object image with the reference image. The image fusion method can be: overlaying the converted object image on the reference image.

[0262] This application proposes an automatic brightness adjustment method for virtual background and foreground. By analyzing the lighting conditions of the foreground and virtual background, it intelligently adjusts the brightness of foreground elements, such as faces and clothing, to match the brightness level of the virtual background image. This method is particularly suitable for users conducting video communication (such as video conferencing) in environments with uneven lighting. It effectively avoids visual disharmony caused by an overly dark foreground and an overly bright background, improves the realism of the synthesized image, enhances the user's professional image in video conferencing, and provides a more immersive and professional video conferencing experience. It has broad application prospects and market potential. Furthermore, the intelligent foreground brightness adjustment algorithm of this application is very lightweight. Its algorithm model takes downsampled foreground and background pixels as input and outputs brightness adjustment parameters. This algorithm can be computed in real time on most machines and introduces a smoothing mechanism for adjustment parameters between frames, enabling natural and smooth video transitions.

[0263] As can be seen from the above, this embodiment can obtain the target object image to be synthesized, the reference image, and the brightness parameter perception model through the server, and extract the image under the brightness channel of the target object image and the reference image respectively to obtain the first brightness image corresponding to the target object image and the second brightness image corresponding to the reference image; perform downsampling processing on the first brightness image and the second brightness image respectively to obtain the target first brightness information and the target second brightness information; perform fusion processing on the target first brightness information and the target second brightness information to obtain the target brightness information; perform feature extraction on the target brightness information through the brightness parameter perception model to obtain the target brightness adjustment parameters; perform brightness adjustment processing on the first brightness image according to the target brightness adjustment parameters to obtain the target brightness image; perform fusion processing on the target brightness image with the channel images under other channels in the target object image except for the brightness channel to obtain the brightness-processed object image; perform image fusion processing on the brightness-processed object image and the reference image to obtain and output the target synthesized image.

[0264] This application can downsample the brightness images of both the target image and the reference image to obtain first and second brightness information of the target, thereby determining the target brightness information. This target brightness information is then input into a brightness parameter perception model to obtain target brightness adjustment parameters. Based on these parameters, the brightness of the first brightness image is adjusted, and then the image is synthesized with the reference image. This ensures brightness matching between the target image and the reference image, resolving the problem of brightness inconsistency in the synthesized image, improving the synthesis effect, and enhancing the visual harmony of the synthesized image. Furthermore, this brightness adjustment process only processes the brightness channel image and downsamples the brightness image, significantly reducing the amount of data processing and improving data processing efficiency. The model input is the downsampled target brightness information, and the output is the brightness adjustment parameters. This model algorithm is very lightweight, capable of real-time image processing, and has strong applicability to various scenarios.

[0265] To better implement the above methods, embodiments of this application also provide an image processing apparatus, such as... Figure 3 As shown, the image processing device may include an acquisition unit 301, a downsampling unit 302, a determination unit 303, a feature extraction unit 304, a brightness adjustment unit 305, and a synthesis unit 306, as follows:

[0266] (1) Obtain unit 301;

[0267] The acquisition unit is used to acquire the target object image to be synthesized, the reference image, and the brightness parameter perception model, and to extract the image in the brightness channel of the target object image and the reference image respectively to obtain the first brightness image corresponding to the target object image and the second brightness image corresponding to the reference image.

[0268] (2) Downsampling unit 302;

[0269] The downsampling unit is used to perform downsampling processing on the first brightness image and the second brightness image respectively to obtain the first brightness information and the second brightness information of the target.

[0270] Optionally, in some embodiments of this application, the downsampling unit may include a first selection subunit, a first construction subunit, a second selection subunit, and a second construction subunit, as follows:

[0271] The first selection subunit is used to select a plurality of first target pixels from the pixels of the first brightness image;

[0272] The first construction subunit is used to construct target first brightness information based on the plurality of first target pixels;

[0273] The second selection subunit is used to select a plurality of second target pixels from the pixels of the second brightness image;

[0274] The second construction subunit is used to construct target second brightness information based on the plurality of second target pixels.

[0275] (3) Determine unit 303;

[0276] The determining unit is used to determine the target brightness information based on the first target brightness information and the second target brightness information.

[0277] Optionally, in some embodiments of this application, the determining unit may include a first fusion subunit and a normalization subunit, as follows:

[0278] The first fusion subunit is used to fuse the first brightness information of the target and the second brightness information of the target to obtain fused brightness information;

[0279] The normalization subunit is used to normalize the fused brightness information to obtain the target brightness information.

[0280] (4) Feature extraction unit 304;

[0281] The feature extraction unit is used to extract features from the target brightness information through the brightness parameter perception model to obtain the target brightness adjustment parameters.

[0282] Optionally, in some embodiments of this application, the feature extraction unit may include an extraction subunit, a parameter acquisition subunit, and a smoothing subunit, as follows:

[0283] The extraction subunit is used to extract features from the target brightness information through the brightness parameter perception model to obtain initial brightness adjustment parameters;

[0284] The parameter acquisition subunit is used to acquire the reference brightness adjustment parameters of the preceding image associated with the target object image;

[0285] The smoothing subunit is used to smooth the initial brightness adjustment parameters based on the reference brightness adjustment parameters to obtain the target brightness adjustment parameters.

[0286] Optionally, in some embodiments of this application, the extraction subunit may be specifically used to perform convolution processing on the target brightness information through the brightness parameter perception model to obtain convolutional feature information; perform pooling processing on the convolutional feature information to obtain pooled feature information; and perform fully connected processing on the pooled feature information to obtain initial brightness adjustment parameters.

[0287] Optionally, in some embodiments of this application, the smoothing subunit may be used to determine the parameter weights of the reference brightness adjustment parameter and the initial brightness adjustment parameter respectively; based on the parameter weights, the reference brightness adjustment parameter and the initial brightness adjustment parameter are fused to obtain the target brightness adjustment parameter.

[0288] (5) Brightness adjustment unit 305;

[0289] A brightness adjustment unit is used to perform brightness adjustment processing on the first brightness image according to the target brightness adjustment parameters to obtain a target brightness image.

[0290] (6) Synthesis unit 306;

[0291] The synthesis unit is used to perform image fusion processing on the target brightness image, the channel images of the target object image other than the brightness channel, and the reference image to obtain and output the target synthesized image.

[0292] Optionally, in some embodiments of this application, the synthesis unit may include a second fusion subunit and a third fusion subunit, as follows:

[0293] The second fusion subunit is used to fuse the target brightness image with the channel images of the target object image in other channels besides the brightness channel to obtain the brightness-processed object image;

[0294] The third fusion subunit is used to perform image fusion between the brightness-processed object image and the reference image to obtain and output the target composite image.

[0295] Optionally, in some embodiments of this application, the image processing apparatus may further include a training data acquisition unit, a first adjustment unit, a downsampling processing unit, an extraction unit, a second adjustment unit, and a parameter adjustment unit, as follows:

[0296] The training data acquisition unit is used to acquire training data, which includes multiple sample images, each containing a sample object; and to perform image segmentation on the sample images to obtain a first region image containing the sample object and a second region image not containing the sample object.

[0297] The first adjustment unit is used to perform brightness adjustment processing on the brightness image of the first region image to obtain the sample first brightness image;

[0298] The downsampling processing unit is used to perform downsampling processing on the brightness images corresponding to the first brightness image of the sample and the second region image, respectively, to obtain the first brightness information and the second brightness information of the target sample; and to determine the brightness information of the target sample based on the first brightness information and the second brightness information of the target sample.

[0299] The extraction unit is used to extract features from the brightness information of the target sample through the brightness parameter perception model to obtain the sample brightness adjustment parameters;

[0300] The second adjustment unit is used to perform brightness adjustment processing on the first brightness image of the sample according to the sample brightness adjustment parameters to obtain the sample brightness image;

[0301] The parameter adjustment unit is used to adjust the parameters of the brightness parameter perception model according to the sample brightness image and the sample image to obtain the trained brightness parameter perception model.

[0302] Optionally, in some embodiments of this application, the first adjustment unit may include a normalization processing subunit, a third selection subunit, an adjustment subunit, and an inverse normalization processing subunit, as follows:

[0303] The normalization processing subunit is used to normalize the brightness image corresponding to the first region image to obtain a normalized brightness image.

[0304] The third selection subunit is used to select brightness adjustment parameters from a preset brightness adjustment parameter range;

[0305] The adjustment subunit is used to perform brightness adjustment processing on the normalized brightness image based on the brightness adjustment parameters to obtain the adjusted brightness image;

[0306] The inverse normalization processing subunit is used to perform inverse normalization processing on the adjusted brightness image to obtain the sample first brightness image.

[0307] Optionally, in some embodiments of this application, the parameter adjustment unit may include an image fusion subunit, a loss calculation subunit, and a parameter adjustment subunit, as follows:

[0308] The image fusion subunit is used to fuse the sample brightness image, the channel images of the other channels in the first region image except the brightness channel, and the second region image to obtain a sample composite image.

[0309] A loss calculation subunit is used to calculate image loss information between the sample image and the synthesized sample image;

[0310] The parameter adjustment subunit is used to adjust the parameters of the brightness parameter perception model based on the image loss information to obtain the trained brightness parameter perception model.

[0311] As can be seen from the above, in this embodiment, the acquisition unit 301 acquires the target object image to be synthesized, the reference image, and the brightness parameter perception model, and extracts images in the brightness channel of the target object image and the reference image respectively to obtain a first brightness image corresponding to the target object image and a second brightness image corresponding to the reference image; the downsampling unit 302 performs downsampling processing on the first brightness image and the second brightness image respectively to obtain target first brightness information and target second brightness information; the determination unit 303 determines the target brightness information based on the target first brightness information and the target second brightness information; the feature extraction unit 304 performs feature extraction on the target brightness information through the brightness parameter perception model to obtain target brightness adjustment parameters; the brightness adjustment unit 305 performs brightness adjustment processing on the first brightness image according to the target brightness adjustment parameters to obtain the target brightness image; and the synthesis unit 306 performs image fusion processing on the target brightness image, the channel images in other channels of the target object image except for the brightness channel, and the reference image to obtain and output the target synthesized image.

[0312] This application can downsample the brightness images of both the target image and the reference image to obtain first and second brightness information of the target, thereby determining the target brightness information. This target brightness information is then input into a brightness parameter perception model to obtain target brightness adjustment parameters. Based on these parameters, the brightness of the first brightness image is adjusted, and then the image is synthesized with the reference image. This ensures brightness matching between the target image and the reference image, resolving the problem of brightness inconsistency in the synthesized image, improving the synthesis effect, and enhancing the visual harmony of the synthesized image. Furthermore, this brightness adjustment process only processes the brightness channel image and downsamples the brightness image, significantly reducing the amount of data processing and improving data processing efficiency. The model input is the downsampled target brightness information, and the output is the brightness adjustment parameters. This model algorithm is very lightweight, capable of real-time image processing, and has strong applicability to various scenarios.

[0313] This application also provides an electronic device, such as... Figure 4 The diagram shows a structural schematic of an electronic device involved in an embodiment of this application. This electronic device can be a terminal or a server, specifically:

[0314] The electronic device may include components such as a processor 401 with one or more processing cores, a memory 402 with one or more computer-readable storage media, a power supply 403, and an input unit 404. Those skilled in the art will understand that... Figure 4 The electronic device structure shown does not constitute a limitation on the electronic device and may include more or fewer components than shown, or combine certain components, or have different component arrangements. Wherein:

[0315] The processor 401 is the control center of the electronic device, connecting various parts of the device via various interfaces and lines. It executes software programs and / or modules stored in the memory 402, and calls data stored in the memory 402, to perform various functions and process data. Optionally, the processor 401 may include one or more processing cores; preferably, the processor 401 may integrate an application processor and a modem processor, wherein the application processor mainly handles the operating system, user interface, and applications, and the modem processor mainly handles wireless communication. It is understood that the modem processor may not be integrated into the processor 401.

[0316] The memory 402 can be used to store software programs and modules. The processor 401 executes various functional applications and data processing by running the software programs and modules stored in the memory 402. The memory 402 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, application programs required for at least one function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the electronic device, etc. In addition, the memory 402 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device. Accordingly, the memory 402 may also include a memory controller to provide the processor 401 with access to the memory 402.

[0317] The electronic device also includes a power supply 403 that supplies power to the various components. Preferably, the power supply 403 can be logically connected to the processor 401 through a power management system, thereby enabling functions such as charging, discharging, and power consumption management through the power management system. The power supply 403 may also include one or more DC or AC power supplies, recharging systems, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components.

[0318] The electronic device may also include an input unit 404, which can be used to receive input digital or character information, and generate keyboard, mouse, joystick, optical or trackball signal inputs related to user settings and function control.

[0319] Although not shown, the electronic device may also include a display unit, etc., which will not be described in detail here. Specifically, in this embodiment, the processor 401 in the electronic device loads the executable files corresponding to the processes of one or more applications into the memory 402 according to the following instructions, and the processor 401 runs the applications stored in the memory 402 to realize various functions, as follows:

[0320] The process involves acquiring a target image, a reference image, and a brightness parameter perception model. Image extraction is performed on both the target image and the reference image using their respective brightness channels to obtain a first brightness image corresponding to the target image and a second brightness image corresponding to the reference image. Downsampling is then applied to both the first and second brightness images to obtain first and second target brightness information. Based on this information, target brightness information is determined. Feature extraction is performed on the target brightness information using the brightness parameter perception model to obtain target brightness adjustment parameters. The first brightness image is then adjusted according to these parameters to obtain the target brightness image. Finally, the target brightness image, channel images from all channels of the target image except the brightness channel, and the reference image are fused together to obtain and output the synthesized target image.

[0321] For details on the implementation of each of the above operations, please refer to the previous examples, which will not be repeated here.

[0322] As can be seen from the above, this embodiment can acquire the target object image to be synthesized, the reference image, and the brightness parameter perception model, and extract images in the brightness channel of the target object image and the reference image respectively to obtain a first brightness image corresponding to the target object image and a second brightness image corresponding to the reference image; perform downsampling processing on the first brightness image and the second brightness image respectively to obtain target first brightness information and target second brightness information; determine target brightness information based on the target first brightness information and the target second brightness information; extract features from the target brightness information through the brightness parameter perception model to obtain target brightness adjustment parameters; perform brightness adjustment processing on the first brightness image according to the target brightness adjustment parameters to obtain the target brightness image; perform image fusion processing on the target brightness image, the channel images in other channels of the target object image except for the brightness channel, and the reference image to obtain and output the target synthesized image.

[0323] This application can downsample the brightness images of both the target image and the reference image to obtain first and second brightness information of the target, thereby determining the target brightness information. This target brightness information is then input into a brightness parameter perception model to obtain target brightness adjustment parameters. Based on these parameters, the brightness of the first brightness image is adjusted, and then the image is synthesized with the reference image. This ensures brightness matching between the target image and the reference image, resolving the problem of brightness inconsistency in the synthesized image, improving the synthesis effect, and enhancing the visual harmony of the synthesized image. Furthermore, this brightness adjustment process only processes the brightness channel image and downsamples the brightness image, significantly reducing the amount of data processing and improving data processing efficiency. The model input is the downsampled target brightness information, and the output is the brightness adjustment parameters. This model algorithm is very lightweight, capable of real-time image processing, and has strong applicability to various scenarios.

[0324] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be performed by instructions, or by instructions controlling related hardware. These instructions can be stored in a computer-readable storage medium and loaded and executed by a processor.

[0325] Therefore, embodiments of this application provide a computer-readable storage medium storing a plurality of instructions that can be loaded by a processor to execute steps in any of the image processing methods provided in embodiments of this application. For example, the instructions can execute the following steps:

[0326] The process involves acquiring a target image, a reference image, and a brightness parameter perception model. Image extraction is performed on both the target image and the reference image using their respective brightness channels to obtain a first brightness image corresponding to the target image and a second brightness image corresponding to the reference image. Downsampling is then applied to both the first and second brightness images to obtain first and second target brightness information. Based on this information, target brightness information is determined. Feature extraction is performed on the target brightness information using the brightness parameter perception model to obtain target brightness adjustment parameters. The first brightness image is then adjusted according to these parameters to obtain the target brightness image. Finally, the target brightness image, channel images from all channels of the target image except the brightness channel, and the reference image are fused together to obtain and output the synthesized target image.

[0327] For details on the implementation of each of the above operations, please refer to the previous examples, which will not be repeated here.

[0328] The computer-readable storage medium may include: read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0329] Since the instructions stored in the computer-readable storage medium can execute the steps of any of the image processing methods provided in the embodiments of this application, the beneficial effects that any of the image processing methods provided in the embodiments of this application can achieve can be realized, as detailed in the preceding embodiments, and will not be repeated here.

[0330] According to one aspect of this application, a computer program product or computer program is provided, comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the methods provided in the various alternative implementations of the image processing described above.

[0331] The above provides a detailed description of an image processing method and related equipment provided by the embodiments of this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. An image processing method, characterized in that, include: The target object image to be synthesized, the reference image, and the brightness parameter perception model are obtained. The target object image and the reference image are extracted in the brightness channel respectively to obtain the first brightness image corresponding to the target object image and the second brightness image corresponding to the reference image. The first brightness image and the second brightness image are downsampled respectively to obtain the first brightness information and the second brightness information of the target; Based on the first target brightness information and the second target brightness information, the target brightness information is determined; The target brightness adjustment parameters are obtained by extracting features from the target brightness information using the brightness parameter perception model. According to the target brightness adjustment parameters, the first brightness image is subjected to brightness adjustment processing to obtain the target brightness image; The target brightness image, the channel images of the target object image (excluding the brightness channel), and the reference image are subjected to image fusion processing to obtain and output the target composite image.

2. The method according to claim 1, characterized in that, The step of downsampling the first brightness image and the second brightness image respectively to obtain the first brightness information and the second brightness information of the target includes: Select a plurality of first target pixels from the pixels of the first brightness image; Based on the plurality of first target pixels, construct the target first brightness information; Select multiple second target pixels from the pixels of the second brightness image; Based on the plurality of second target pixels, target second brightness information is constructed.

3. The method according to claim 1, characterized in that, The step of extracting features from the target brightness information using the brightness parameter perception model to obtain target brightness adjustment parameters includes: The initial brightness adjustment parameters are obtained by extracting features from the target brightness information using the brightness parameter perception model. Obtain the reference brightness adjustment parameters of the preceding image associated with the target object image; Based on the reference brightness adjustment parameters, the initial brightness adjustment parameters are smoothed to obtain the target brightness adjustment parameters.

4. The method according to claim 3, characterized in that, The step of extracting features from the target brightness information using the brightness parameter perception model to obtain initial brightness adjustment parameters includes: The target brightness information is processed by convolution through the brightness parameter perception model to obtain convolution feature information; The convolutional feature information is pooled to obtain pooled feature information; The pooled feature information is processed by a fully connected layer to obtain the initial brightness adjustment parameters.

5. The method according to claim 3, characterized in that, The step of smoothing the initial brightness adjustment parameters based on the reference brightness adjustment parameters to obtain the target brightness adjustment parameters includes: The parameter weights of the reference brightness adjustment parameter and the initial brightness adjustment parameter are determined respectively; Based on the parameter weights, the reference brightness adjustment parameters and the initial brightness adjustment parameters are fused to obtain the target brightness adjustment parameters.

6. The method according to claim 1, characterized in that, Determining the target brightness information based on the first target brightness information and the second target brightness information includes: The first brightness information and the second brightness information of the target are fused to obtain the fused brightness information; The fused brightness information is normalized to obtain the target brightness information.

7. The method according to claim 1, characterized in that, The step of performing image fusion processing on the target brightness image, the channel images of the target object image (excluding the brightness channel), and the reference image to obtain and output the target composite image includes: The target brightness image is fused with the channel images of the target object image in other channels besides the brightness channel to obtain the object image after brightness processing. The brightness-processed object image is fused with the reference image to obtain and output the target composite image.

8. The method according to claim 1, characterized in that, Before extracting features from the target brightness information using the brightness parameter perception model to obtain the target brightness adjustment parameters, the method further includes: Acquire training data, which includes multiple sample images containing sample objects; and perform image segmentation on the sample images to obtain a first region image containing sample objects and a second region image not containing sample objects. The brightness image of the first region image is subjected to brightness adjustment processing to obtain the sample first brightness image; The brightness images corresponding to the first brightness image and the second region image of the sample are downsampled respectively to obtain the first brightness information and the second brightness information of the target sample; and the brightness information of the target sample is determined based on the first brightness information and the second brightness information of the target sample. By using a brightness parameter perception model, feature extraction is performed on the brightness information of the target sample to obtain the sample brightness adjustment parameters. According to the sample brightness adjustment parameters, the first sample brightness image is subjected to brightness adjustment processing to obtain the sample brightness image; Based on the sample brightness image and the sample image, the parameters of the brightness parameter perception model are adjusted to obtain the trained brightness parameter perception model.

9. The method according to claim 8, characterized in that, The step of performing brightness adjustment processing on the brightness image of the first region image to obtain the sample first brightness image includes: The brightness image corresponding to the first region image is normalized to obtain a normalized brightness image. Select a brightness adjustment parameter from the preset brightness adjustment parameter range; Based on the brightness adjustment parameters, the normalized brightness image is subjected to brightness adjustment processing to obtain the adjusted brightness image; The adjusted brightness image is subjected to inverse normalization to obtain the first brightness image of the sample.

10. The method according to claim 8, characterized in that, The step of adjusting the parameters of the brightness parameter perception model based on the sample brightness image and the sample image to obtain the trained brightness parameter perception model includes: The sample brightness image, the channel images of the other channels in the first region image (excluding the brightness channel), and the second region image are fused to obtain a sample composite image. Calculate the image loss information between the sample image and the synthesized sample image; Based on the image loss information, the parameters of the brightness parameter perception model are adjusted to obtain the trained brightness parameter perception model.

11. An image processing apparatus, characterized in that, include: The acquisition unit is used to acquire the target object image to be synthesized, the reference image, and the brightness parameter perception model, and to extract the image in the brightness channel of the target object image and the reference image respectively to obtain the first brightness image corresponding to the target object image and the second brightness image corresponding to the reference image. The downsampling unit is used to perform downsampling processing on the first brightness image and the second brightness image respectively to obtain the first brightness information and the second brightness information of the target; The determining unit is configured to determine target brightness information based on the first target brightness information and the second target brightness information; The feature extraction unit is used to extract features from the target brightness information through the brightness parameter perception model to obtain the target brightness adjustment parameters; A brightness adjustment unit is used to perform brightness adjustment processing on the first brightness image according to the target brightness adjustment parameters to obtain a target brightness image; The synthesis unit is used to perform image fusion processing on the target brightness image, the channel images of the target object image other than the brightness channel, and the reference image to obtain and output the target synthesized image.

12. An electronic device, characterized in that, It includes a memory and a processor; the memory stores an application program, and the processor runs the application program within the memory to perform the operations in the image processing method according to any one of claims 1 to 10.

13. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a plurality of instructions adapted for loading by a processor to perform the steps of the image processing method according to any one of claims 1 to 10.

14. A computer program product comprising a computer program or instructions, characterized in that, When the computer program or instructions are executed by a processor, they implement the steps of the image processing method according to any one of claims 1 to 10.

Images