An image processing method, apparatus, electronic device, and storage medium

Abstract

The application provides an image processing method, comprising: determining a region to be adjusted in an original image according to an irradiance corresponding to the original image and an average exposure parameter; adjusting the brightness of the region to be adjusted to obtain a local brightness adjustment image; and fusing the local brightness adjustment image into the original image to adjust the brightness of a local region of the original image. The application also provides an image processing device, an electronic device and a storage medium.

Description

TECHNICAL FIELD

[0001] The present application relates to the information processing technology field, in particular to an image processing method and device, electronic equipment and storage medium. BACKGROUND

[0002] In the inter-image rendering, in order to realize the realistic scene lighting effect, it is necessary to specify the global illumination in the renderer. The global illumination represents the comprehensive effect of direct lighting and indirect lighting. For example, the global illumination can be realized by light tracing, environment light shading, light probe and the like. When the light is emitted from the light source, after several reflections and refractions after hitting the obstacle, the object surface and the corner in the virtual scene will have light.

[0003] In the related art, the static lighting scene is processed, and the static lighting information of the object in the static lighting scene is pre-baked for direct use of the scene in the running. However, this method is not applicable to the dynamic lighting scene. In the dynamic lighting scene, the local image will change in brightness, which will cause poor lighting rendering effect in the dynamic lighting scene. SUMMARY

[0004] Therefore, the embodiment of the present application provides an image processing method and device, electronic equipment and storage medium, which can determine the to-be-adjusted region in the original image according to the irradiance corresponding to the original image in the virtual scene and the average exposure parameter, and adjust the brightness of the to-be-adjusted region, so as to realize the brightness adjustment of the local image. Therefore, not only the brightness adjustment can be accurately and efficiently realized, but also the image processing process reduces the occupation of hardware resources, reduces the increase of hardware equipment cost, reduces the processing delay, is more suitable for high code flow virtual scene, and improves the user experience.

[0005] The technical scheme of the embodiment of the present application is as follows:

[0006] The embodiment of the present application provides an image processing method, characterized in that the method comprises:

[0007] Obtaining an original image to be processed and irradiance corresponding to the original image;

[0008] Determining an average exposure parameter corresponding to the original image based on the original image;

[0009] Determining a to-be-adjusted region in the original image according to the irradiance corresponding to the original image and the average exposure parameter;

[0010] Adjusting the brightness of the to-be-adjusted region to obtain a local brightness adjustment image;

[0011] fuse the local brightness adjustment image into the original image to realize adjusting the brightness of the local region of the original image.

[0012] The embodiment of the present application also provides an image processing device, and the device comprises:

[0013] an information transmission module, which is used for acquiring an original image to be processed and an irradiance corresponding to the original image;

[0014] an information processing module, which is used for determining an average exposure parameter corresponding to the original image based on the original image;

[0015] The information processing module is used for determining a region to be adjusted in the original image according to the irradiance corresponding to the original image and the average exposure parameter.

[0016] The information processing module is used for adjusting the brightness of the region to be adjusted to obtain a local brightness adjustment image.

[0017] The information processing module is used for fusing the local brightness adjustment image into the original image to realize adjusting the brightness of the local region of the original image.

[0018] In the above scheme,

[0019] The information processing module is used for determining a virtual scene of the original image based on the acquired original image.

[0020] The information processing module is used for determining the type of the irradiance based on the virtual scene of the original image.

[0021] The information processing module is used for performing irradiance conversion processing on the original image according to the type of the irradiance to obtain the irradiance corresponding to the original image.

[0022] In the above scheme,

[0023] The information processing module is used for acquiring an over-bright region threshold difference parameter corresponding to the original image.

[0024] The information processing module is used for calculating an over-bright region threshold according to the average exposure parameter and the over-bright region threshold difference parameter.

[0025] The information processing module is used for determining the reference brightness of each pixel in the original image according to the irradiance corresponding to the original image.

[0026] The information processing module is used for comparing the reference brightness of each pixel with the over-bright region threshold to determine the pixel greater than or equal to the over-bright region threshold.

[0027] The information processing module is configured to determine, based on all pixels greater than or equal to the over-bright region threshold value, an over-bright region in the original image as a region to be adjusted.

[0028] In the above scheme,

[0029] The information processing module is configured to obtain an over-dark region threshold difference parameter corresponding to the original image;

[0030] The information processing module is configured to calculate an over-dark region threshold value according to the average exposure parameter and the over-dark region threshold difference parameter.

[0031] The information processing module is configured to determine, according to the irradiance corresponding to the original image, a reference brightness of each pixel in the original image.

[0032] The information processing module is configured to compare the reference brightness of each pixel with the over-dark region threshold value, and determine a pixel less than or equal to the over-dark region threshold value.

[0033] The information processing module is configured to determine, based on all pixels less than or equal to the over-dark region threshold value, an over-dark region in the original image as a region to be adjusted.

[0034] In the above scheme,

[0035] The information processing module is configured to determine, according to the type of the original image, a brightness adjustment function matched with the original image, wherein the slope of the brightness adjustment function is greater than or equal to 0 and less than 1.

[0036] The information processing module is configured to perform a decreasing adjustment on the brightness of the region to be adjusted according to the difference between the irradiance corresponding to the original image and the over-bright region threshold value based on the brightness adjustment function, to obtain a local brightness adjustment image.

[0037] The information processing module is configured to perform an increasing adjustment on the brightness of the region to be adjusted according to the difference between the irradiance corresponding to the original image and the over-dark region threshold value based on the brightness adjustment function, to obtain a local brightness adjustment image.

[0038] In the above scheme,

[0039] The information processing module is configured to determine an empirical threshold value matched with the original image to be processed.

[0040] The information processing module is configured to determine, based on the empirical threshold value matched with the original image to be processed and a boundary range of image processing matched with a color key parameter, an out-of-color pixel in the original image.

[0041] The information processing module is configured to adjust the gray value of the color bleeding pixel in the original image.

[0042] In the above solution,

[0043] The information processing module is configured to share the local brightness adjustment image between different use objects to achieve the same local area brightness when the original image corresponds to at least two use objects.

[0044] The information processing module is configured to determine the historical parameters of the target object according to the type of the virtual scene when the original image corresponds to one use object.

[0045] The information processing module is configured to obtain the corresponding over-bright area threshold and over-dark area threshold in the historical parameters of the target object.

[0046] The information processing module is configured to adjust any original image corresponding to the target object according to the obtained over-bright area threshold and over-dark area threshold.

[0047] The information processing module is configured to combine all the adjusted original images into video information and present in the virtual scene.

[0048] In the above solution,

[0049] The information processing module is configured to present an image template selection interface and present at least one brightness template in the image template selection interface.

[0050] The information processing module is configured to, in response to a selection operation of the brightness template triggered based on the brightness template selection interface, take the brightness template corresponding to the selection operation as the brightness of the local area corresponding to the original image.

[0051] In the above solution,

[0052] The information processing module is configured to, in response to a viewing operation of the image adjustment function item, present a content page including the original image to be processed and the local brightness adjustment image, and present at least one interactive function item in the content page, the interactive function item being used to realize interaction with the original image to be processed.

[0053] The information processing module is configured to receive a color picking instruction for the local brightness adjustment image triggered based on the interactive function item to execute the corresponding color picking instruction.

[0054] In the above solution,

[0055] The information processing module is configured to determine types of all irradiance in the virtual scene according to the virtual scene of the original image.

[0056] The information processing module is configured to calculate irradiance corresponding to each type of irradiance according to the type of irradiance.

[0057] The information processing module is configured to render different objects in the virtual scene according to the corresponding irradiance when the local brightness adjustment image is fused into the original image, so as to match the irradiance.

[0058] The embodiment of the present application further provides an electronic device, which comprises:

[0059] A memory is configured to store executable instructions.

[0060] A processor is configured to implement the image processing method in the preceding sequence when running the executable instructions stored in the memory.

[0061] The embodiment of the present application further provides a computer readable storage medium, which stores executable instructions, and the executable instructions are executed by a processor to implement the image processing method in the preceding sequence.

[0062] The embodiment of the present application has the following beneficial effects:

[0063] The embodiment of the present application acquires an original image to be processed and irradiance corresponding to the original image, determines an average exposure parameter corresponding to the original image based on the original image, determines a region to be adjusted in the original image according to the irradiance corresponding to the original image and the average exposure parameter, adjusts the brightness of the region to be adjusted to obtain a local brightness adjustment image, and fuses the local brightness adjustment image into the original image to adjust the brightness of the local region of the original image. Therefore, the brightness adjustment can be accurately and efficiently realized, the image processing process reduces the occupation of hardware resources, reduces the increase of hardware device cost, reduces the processing delay, is more suitable for high code flow virtual scenes, and improves the user experience. BRIEF DESCRIPTION OF DRAWINGS

[0064] Figure 1 A use scenario diagram of the image processing method provided by the embodiment of the present application is shown.

[0065] Figure 2 A composition structure diagram of the electronic device provided by the embodiment of the present application is shown.

[0066] Figure 3 An optional flow diagram of the image processing method provided by the embodiment of the present application is shown.

[0067] Figure 4 A schematic diagram of irradiance corresponding to the original image in the embodiment of the present application;

[0068] Figure 5 An optional flowchart of the image processing method provided by the embodiment of the present application;

[0069] Figure 6 A schematic diagram of the judgment process of the region to be adjusted in the embodiment of the present application;

[0070] Figure 7 A schematic diagram of the judgment result of the region to be adjusted in the embodiment of the present application;

[0071] Figure 8 A schematic diagram of the judgment result of the region to be adjusted in the embodiment of the present application;

[0072] Figure 9 A schematic diagram of the adjustment function curve;

[0073] Figure 10 A schematic diagram of the effect of the brightness adjustment function in the embodiment of the present application;

[0074] Figure 11 A schematic diagram of the effect of the brightness adjustment function in the embodiment of the present application

[0075] Figure 12 A schematic diagram of the effect of the brightness adjustment function in the embodiment of the present application. DETAILED DESCRIPTION

[0076] In order to make the objects, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the accompanying drawings, and the described embodiments should not be regarded as limiting the present application. All other embodiments obtained by those of ordinary skill in the art without making creative efforts fall within the scope of protection of the present application.

[0077] In the following description, “some embodiments” are described, which describe a subset of all possible embodiments, but it can be understood that “some embodiments” can be the same subset or different subsets of all possible embodiments, and can be combined with each other without conflict.

[0078] In the application of the related data collection and processing in the embodiments of the present application, the informed consent or separate consent of the personal information subject should be obtained in strict accordance with the requirements of the relevant laws and regulations, and the subsequent data use and processing should be carried out within the scope of authorization of the laws and regulations and the personal information subject.

[0079] Before the embodiments of the present application are further described in detail, the terms and terms involved in the embodiments of the present application are described, and the terms and terms involved in the embodiments of the present application are applicable to the following explanations.

[0080] 1) In response to a condition or state on which the operation performed depends, when the dependent condition or state is met, one or more operations performed can be real-time or have a set delay; in the absence of a specific description, there is no restriction on the execution order of multiple operations performed.

[0081] 2) Client, a carrier of specific functions in a terminal, for example, a mobile client (APP) is a carrier of specific functions in a mobile terminal, for example, a program that performs user gesture recognition.

[0082] 3) Component, a functional module of the view of the applet, also known as a front-end component, buttons, titles, tables, sidebars, content, and footers in a page, etc., components include modularized code to facilitate reuse in different pages of the applet.

[0083] 4) Mini Program, a program developed based on a front-end-oriented language (such as JavaScript) to implement services in a Hyper Text Markup Language (HTML) page, downloaded by a client (such as a browser or any client with an embedded browser core) via a network (such as the Internet), and interpreted and executed in the browser environment of the client, saving the step of installation in the client. For example, by voice instruction to wake up the applet in the terminal to achieve the function of downloading and running the applet for image editing, character background replacement, and various services in a social network client.

[0084] 5) RGB, a three-primary color encoding method, also known as the RGB color mode, is an industry standard for color, which obtains various colors by changing the three color channels of red (R), green (G), and blue (B) and their mutual superposition. RGB represents the three channels of red, green, and blue, which almost includes all colors that can be perceived by human vision, and is one of the most widely used color systems.

[0085] 6) Bleeding, when processing images using green screen technology, due to the reflection of the curtain, the edges of the person or object in front of the curtain are "stained" with the color of the curtain, which is called bleeding. In the process of handling bleeding, the color that needs to be removed is called the color key (Chroma key).

[0086] 7) Real-time rendering: Real-time rendering is a branch of computer graphics that deals with the production and analysis of real-time images. It covers everything from application graphical user interfaces (GUIs) to real-time image analysis, but usually refers to interactive three-dimensional computer graphics generated by a graphics processing unit (GPU). One example is video games.

[0087] Specifically includes: physically-based rendering, which means rendering images with a way of modeling light in the real world.

[0088] 8) Physically-based lighting: One of the sub-methods under the physically-based rendering method, which means using real-world light intensity units to guide the lighting of the game scene. Its advantage is that it can produce realistic images, and the lighting parameters can be directly found in the real world. For example, the light intensity of the sun reaching the ground at noon is about 110000 lux.

[0089] 9) Dynamic range: The ratio of the maximum and minimum values of a variable signal (such as sound or light). It can also be expressed as a logarithm with a base of 2. In the embodiments of the present invention, it refers to the range of variation of the image / light signal.

[0090] 10) Exposure Value (EV): EV is used as a unit to measure brightness. EV and brightness are exponentially related, and an increase of 1 in EV represents a doubling of brightness. EV(ISO100), or EV100, can also be used as an absolute brightness unit. For example, the EV100 of a full daylight outdoor scene is about 17, while in an indoor office environment, the EV100 is only about 7-8, and the full moon brightness at night has an EV100 of -3~-2.

[0091] 11) Global tonemapping: A method of mapping the dynamic range of the original image signal to a dynamic range suitable for display on a display. For example, adjusting the brightness of all video image frames in a game. The reason for this method is that after using the physically-based lighting rendering method, the original image obtained is the same as the physical world, and its dynamic range is very wide, while our commonly used display is not enough to display so many details.

[0092] 12) Local tonemapping: Global tonemapping processes the entire original image, while local tonemapping only processes part of the original image, such as local image brightness adjustment, which is a local tonemapping.

[0093] 13) Sky visibility: It refers to the degree of occlusion of the sky to a certain point in the scene. For example, the sky visibility in an underground parking lot is 0, while the sky visibility in a wide outdoor plaza is close to 1.

[0094] 14) Sky radiance: represents the intensity of incoming light from the sky received at a point in the scene, which is influenced by the degree of sky visibility.

[0095] 15) Scene average EV: average luminance (in EV100) of all pixels of the original image computed in the post-processing pass of the real-time rendering.

[0096] 16) Pixel reference luminance: used in the local tone mapping computation, whose value is related to the sky radiance.

[0097] Figure 1 The use scenario of the image processing method provided by the embodiment of the present application is shown in the following figure: Figure 1 The terminal (including the terminal 10-1 and the terminal 10-2) is provided with a client or an applet of the image processing software, and the image information is interacted with the server 200 through the network 300 to realize the background replacement processing. In order to support an example application, the image processing device can be a server, and the terminal running various clients can be used to display the image processing result of the image processing device. The two are connected through the network 40, which can be a wide area network or a local area network, or a combination of the two, and the data transmission is realized by using a wireless link. The terminal 10 submits the image to be processed, and the image processing device responds to the triggering operation of the image transformation function item to realize image processing, and the terminal 10 acquires and presents the target image, wherein the presented image includes a local brightness adjustment image to adapt to different brightness requirements in a virtual environment.

[0098] In some embodiments of the present application, a video client can be run in the graphical interface of the image processing device. The video client can adjust the brightness of the original image according to the user's control in the playing interface through various human-computer interaction modes (such as gestures, voice, etc.). For example, the brightness of the whole image or the local brightness of the image (such as a landscape image, a portrait image, a cartoon image, and a game image) can be adjusted. It can be understood that the above-mentioned image processing can also be migrated to the server, and the adjusted frame image is re-encoded by means of the hardware resources of the server to form a video with a background replacement effect.

[0099] Among them, the image processing method provided by the embodiment of the application can be implemented based on artificial intelligence. Artificial intelligence (AI) is the theory, method, technology and application system for using digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive environment, acquire knowledge and use knowledge to obtain optimal results. In other words, artificial intelligence is a comprehensive technology of computer science, which attempts to understand the essence of intelligence and produce a new intelligent machine that can react in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines, so that machines have the functions of perception, reasoning and decision-making.

[0100] Artificial intelligence technology is a comprehensive discipline, involving a wide range of fields, both hardware level technology and software level technology. Artificial intelligence basic technology generally includes technologies such as sensors, special artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction system, mechatronics, etc. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning, etc.

[0101] In the embodiment of the application, the artificial intelligence software technology mainly involved includes the above-mentioned speech processing technology and machine learning, etc. For example, it can involve the speech recognition technology (Automatic Speech Recognition, ASR) in the speech technology, which includes speech signal preprocessing, speech signal frequency analyzing, speech signal feature extraction, speech signal feature matching / recognition, speech training, etc.

[0102] For example, it can involve machine learning (ML), which is a multidisciplinary field involving probability theory, statistics, approximation theory, convex analysis, algorithmic complexity theory, etc. It is a specialized study of how computers simulate or implement human learning behavior to acquire new knowledge or skills, reorganize existing knowledge structure to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental approach to making computers intelligent, and its applications are widespread in various fields of artificial intelligence. Machine learning usually includes technologies such as deep learning, which includes artificial neural networks such as convolutional neural networks (CNN), recurrent neural networks (RNN), deep neural networks (DNN), etc.

[0103] It can be understood that the image processing method and voice processing provided by the present application can be applied to an intelligent device. The intelligent device can be any device with voice command recognition function, such as a smart terminal, a smart home device (such as a smart speaker, a smart washing machine, etc.), a smart wearable device (such as a smart watch), a vehicle-mounted intelligent central control system (which wakes up a small program in the terminal for executing different tasks through voice commands) or an AI intelligent medical device (which is awakened and triggered through voice commands).

[0104] The structure of the electronic device of the embodiment of the present application will be described in detail below. The electronic device can be implemented in various forms, such as a special terminal with image processing function, or a server provided with image processing function, for example, the server 200 in the foregoing Figure 1 . Figure 2 The schematic diagram of the component structure of the electronic device provided by the embodiment of the present application can be understood that Figure 2 only an exemplary structure of the electronic device is shown, not all structures, and part of the structure or all the structure shown can be implemented as needed. Figure 2

[0105] The electronic device provided by the embodiment of the present application includes at least one processor 201, a memory 202, a user interface 203 and at least one network interface 204. Each component in the electronic device is coupled together through a bus system 205. It can be understood that the bus system 205 is used to realize the connection and communication between the components. In addition to the data bus, the bus system 205 also includes power bus, control bus and status signal bus. However, for the purpose of clear illustration, only the data bus is shown in Figure 2 ​The various buses are labeled as bus system 205.

[0106] The user interface 203 can include a display, a keyboard, a mouse, a trackball, a click wheel, a key, a button, a touchpad, or a touchscreen, etc.

[0107] It can be understood that the memory 202 can be a volatile memory or a nonvolatile memory, and can also include both volatile and nonvolatile memories. The memory 202 in the embodiment of the present application can store data to support the operation of the terminal (such as 10-1). Examples of these data include any computer programs for operating on the terminal (such as 10-1), such as operating systems and application programs. The operating system includes various system programs, such as a framework layer, a core library layer, a driver layer, etc., for implementing various basic services and processing hardware-based tasks. The application program can include various application programs.

[0108] In some embodiments, the electronic device provided by the embodiment of the present application can be implemented in a combination of software and hardware. As an example, the electronic device provided by the embodiment of the present application can be a processor in the form of a hardware decoding processor programmed to execute the image processing method provided by the embodiment of the present application. For example, the processor in the form of a hardware decoding processor can use one or more application specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field programmable gate arrays (FPGAs), or other electronic elements.

[0109] As an example of the electronic device provided by the embodiment of the present application implemented in a combination of software and hardware, the electronic device provided by the embodiment of the present application can be directly embodied as a combination of software modules executed by the processor 201. The software modules can be located in the storage medium, and the storage medium is located in the memory 202. The processor 201 reads the executable instructions included in the software modules in the memory 202, and combines the necessary hardware (for example, including the processor 201 and other components connected to the bus 205) to complete the image processing method provided by the embodiment of the present application.

[0110] As an example, the processor 201 can be an integrated circuit chip having a processing capability of signals, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic device, discrete gate or transistor logic device, discrete hardware component, etc., wherein the general-purpose processor can be a microprocessor or any conventional processor.

[0111] As an example, the processor 201 can be an integrated circuit chip having a processing capability of signals, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic device, discrete gate or transistor logic device, discrete hardware component, etc., wherein the general-purpose processor can be a microprocessor or any conventional processor.

[0112] The memory 202 in the embodiment of the present application is used to store various types of data to support the operation of the electronic device. Examples of these data include any executable instructions for operating on the electronic device, such as executable instructions, programs implementing the image processing method of the embodiment of the present application can be included in the executable instructions.

[0113] In other embodiments, the electronic device provided by the embodiment of the present application can be implemented in software, Figure 2 The electronic device stored in the memory 202 is shown, which can be software in the form of programs and plug-ins, and includes a series of modules. As an example of the program stored in the memory 202, it can include an electronic device, and the image processing module of the electronic device includes the following software modules: information transmission module 2081, information processing module 2082. When the software modules in the electronic device are read into the RAM by the processor 201 and executed, the image processing method provided by the embodiment of the present application will be implemented. The functions of each software module in the electronic device in the embodiment of the present application are introduced below, wherein,

[0114] The information transmission module 2081 is configured to obtain an original image to be processed and an irradiance corresponding to the original image.

[0115] The information processing module 2082 is configured to determine an average exposure parameter corresponding to the original image based on the original image.

[0116] The information processing module 2082 is configured to determine a region to be adjusted in the original image according to the irradiance corresponding to the original image and the average exposure parameter.

[0117] The information processing module 2082 is configured to adjust the brightness of the region to be adjusted to obtain a local brightness adjustment image.

[0118] The information processing module 2082 is configured to fuse the local brightness adjustment image into the original image to adjust the brightness of the local region of the original image.

[0119] According to Figure 2 The electronic device shown in the figure, in one aspect of the present application, the present application also provides a computer program product or computer program, which includes computer instructions stored in a computer readable storage medium. The processor of the computer device reads the computer instructions from the computer readable storage medium, and the processor executes the computer instructions, so that the computer device executes different embodiments and combinations of embodiments provided in various optional implementation manners of the above image processing method.

[0120] In combination Figure 2 The image processing device shown illustrates the image processing method provided by the embodiments of the present application, referring to Figure 3 , Figure 3 An optional flowchart of the image processing method provided by the embodiments of the present application can be understood that Figure 3 The image processing method shown can be applied to the field of terminal image processing, and the local of the original image is locally tone mapped by the background image through the image processing process or the image processing applet, wherein Figure 3 The steps shown can be executed by various electronic devices running the image processing device, for example, can be a special terminal with image processing function, a server or a server cluster. The following will be described with respect to Figure 3 The steps shown.

[0121] Step 301: The image processing device acquires an original image to be processed and an irradiance corresponding to the original image.

[0122] In some embodiments of the present application, the original image to be processed and the irradiance corresponding to the original image can be acquired by the following way:

[0123] Based on the acquired original image, a virtual scene of the original image is determined; based on the virtual scene of the original image, the type of irradiance is determined; according to the type of irradiance, the original image undergoes irradiance transformation processing to obtain the irradiance corresponding to the original image. Here, irradiance describes the amount of light received by an object's surface, as referenced... Figure 4 , Figure 4 This is a schematic diagram of the irradiance corresponding to the original image in an embodiment of the present invention. Taking the sky irradiance as an example, when rendering a virtual environment that imitates the real world, the EV100 value is 17 in a full-sunlight outdoor scene in the real world, while the EV100 value is 7 in an indoor office environment. The original image is subjected to irradiance transformation processing to obtain the irradiance corresponding to the original image. The higher the brightness, the stronger the sky irradiance.

[0124] Furthermore, based on the virtual scene of the original image, the type of irradiance in all objects within the virtual scene can be determined; based on the type of irradiance, the irradiance corresponding to each irradiance type can be calculated; when the local brightness adjustment image is blended into the original image, different objects in the virtual scene are rendered according to the corresponding irradiance to achieve matching with the irradiance. Specifically, due to different game environment types, when the virtual environment is a dark scene such as underwater combat or cave combat, the type of irradiance can be underwater irradiance or irradiance illuminating virtual weapons. When calculating the irradiance for a point on the surface of an object, it is necessary to multiply by the cosine of the angle between the illumination direction and the normal to the object surface. As the game progresses and the virtual scene changes, the irradiance decreases, which makes the game user perceive a decrease in the brightness of the light in the image.

[0125] Step 302: The image processing device determines the average exposure parameter corresponding to the original image based on the original image.

[0126] Among them, with Figure 4 Taking the virtual scene in the game as an example, the reference brightness (unit: EV100) of each pixel in the image signal is compared with the average EV of the scene to determine whether there are overly bright or underly dark local images. For different game scenes, the average exposure parameters corresponding to the original image are also different. When EV100=10, the sky light intensity is 1024 cd / m2. For different intensities of light in the game virtual environment, different exposures can be used to obtain the same result. If physical lights are used, the appropriate light intensity value under the corresponding EV value can be obtained by directly querying the specification information. If physical lights are not used, the corresponding light and exposure value specifications can be formulated by using tables.

[0127] Step 303: The image processing device determines the region to be adjusted in the original image according to the irradiance corresponding to the original image and the average exposure parameter.

[0128] Reference Figure 5 , Figure 5 An optional flowchart of the image processing method provided by the embodiment of the present application is shown in the following steps:

[0129] Step 501: Obtain the over-bright region threshold difference parameter corresponding to the original image.

[0130] Step 502: Calculate the over-bright region threshold according to the average exposure parameter and the over-bright region threshold difference parameter.

[0131] Step 503: Determine the reference brightness of each pixel in the original image according to the irradiance corresponding to the original image.

[0132] Step 504: Compare the reference brightness of each pixel with the over-bright region threshold to determine the pixel greater than or equal to the over-bright region threshold.

[0133] Step 505: Determine the over-bright region in the original image as the region to be adjusted based on all the pixels greater than or equal to the over-bright region threshold.

[0134] Similarly, when determining the over-dark region, the over-dark region threshold difference parameter corresponding to the original image can be obtained; the over-dark region threshold can be calculated according to the average exposure parameter and the over-dark region threshold difference parameter; the reference brightness of each pixel in the original image can be determined according to the irradiance corresponding to the original image; the reference brightness of each pixel can be compared with the over-dark region threshold to determine the pixel less than or equal to the over-dark region threshold; and the over-dark region in the original image can be determined as the region to be adjusted based on all the pixels less than or equal to the over-dark region threshold. For further description of the determination process of the region to be adjusted, refer to Figure 6 , Figure 6 The flowchart of the determination process of the region to be adjusted in the embodiment of the present application is shown in the following steps: Figure 6 In the embodiment, the over-dark region threshold difference is 1. The over-dark region threshold difference is an external adjustable parameter with the unit of EV. Figure 6 In the embodiment, the over-bright region threshold difference is 2. The over-bright region threshold difference is an external adjustable parameter with the unit of EV. Figure 6 In the embodiment, the over-dark threshold = scene EV - over-dark region threshold difference. Figure 6The over-bright threshold value = scene EV + over-bright area threshold value difference. In this example, 1 + 2 = 3. When the reference brightness of the detected pixel is less than the over-dark threshold value, it is determined to be a pixel in the over-dark area. Figure 6 The over-bright threshold value = scene EV + over-bright area threshold value difference. In this example, 1 + 2 = 3. When the reference brightness of the detected pixel is less than the over-dark threshold value, it is determined to be a pixel in the over-dark area. Figure 6 The over-bright threshold value = scene EV + over-bright area threshold value difference. In this example, 1 + 2 = 3. When the reference brightness of the detected pixel is less than the over-dark threshold value, it is determined to be a pixel in the over-dark area.

[0135] By Figure 6 The reference Figure 7 And Figure 8 , Figure 7 Fig. 1 is a schematic diagram of the judgment result of the area to be adjusted in an embodiment of the present application, Figure 8 Fig. 1 is a schematic diagram of the judgment result of the area to be adjusted in an embodiment of the present application,

[0136] Step 304: The image processing device adjusts the brightness of the area to be adjusted to obtain a local brightness adjustment image.

[0137] In some embodiments of the present application, the adjustment of the brightness of the area to be adjusted to obtain a local brightness adjustment image can be realized by the following manner:

[0138] According to the type of the original image, a brightness adjustment function matched with the original image is determined, wherein the slope of the brightness adjustment function is greater than or equal to 0 and less than 1; based on the brightness adjustment function, the brightness of the area to be adjusted is reduced and adjusted according to the difference between the irradiance corresponding to the original image and the over-bright area threshold value to obtain a local brightness adjustment image; based on the brightness adjustment function, the brightness of the area to be adjusted is increased and adjusted according to the difference between the irradiance corresponding to the original image and the over-dark area threshold value to obtain a local brightness adjustment image.

[0139] The reference Figure 9 , Figure 9 Fig. 1 is a schematic diagram of the judgment result of the area to be adjusted in an embodiment of the present application,

[0140] Formula 1, wherein A, B, C, D and S are controllable parameters. The following is the function image when A = 0.74, B = 0.4, C = 0.24, D = 2.0 and S = 0.5, wherein the horizontal and vertical coordinate axes are both in EV. For example, Figure 7Taking the overbright area in the image as an example, the x-axis represents the difference between the pixel reference brightness and the overbrightness threshold. The y-axis represents how much brightness needs to be reduced (in EV). It should be noted that when adjusting the brightness of a local image using the brightness adjustment function shown in Formula 1, all controllable parameters in Formula 1 can be dynamically adjusted according to the lighting design requirements of the virtual environment. However, regardless of how the controllable parameters in Formula 1 are adjusted, the curve of Formula 1 after adjustment must satisfy that the slope of all values ​​is greater than or equal to 1, and any value is acceptable. To more clearly describe the controllable parameters in Formula 1, the controllable parameters in Formula 1 are defined as follows: Domain of ABCDF: A is greater than or equal to 0, D is greater than or equal to 0, S is greater than or equal to 0, B and C are unrestricted. The value range of ABCD can also be restricted to [-10, 10], and S is between [0, 2]. For mini-games provided by instant messaging clients, in order to reduce the computational load on the terminal when adjusting local images and alleviate the burden on hardware devices, the controllable parameters in Formula 1 can be set with default values, namely: A=1.5, B=0.4, C=0.8, D=5.7, S=1.0. By setting the above default controllable parameters, the mini-games provided by the messaging client can flexibly adjust the brightness of local images of different types of mini-games to meet the needs of users.

[0141] by Figure 8 Taking an overly dark area as an example, when adjusting an overly dark area, the x-axis represents the difference between the pixel reference brightness and the overly dark threshold, and the y-axis represents how much brightness needs to be increased (in EV). Figure 9 The brightness adjustment function shown has a slope of 0 at x=0, ensuring that the initial brightness change in the adjusted original image is smooth and natural, avoiding abrupt adjustments. As x increases, y flattens out, but the slope of the curve is always less than or equal to 1, ensuring that the relative brightness of two pixels remains constant before and after adjustment. This constraint is controlled empirically by adjusting the range of values ​​for parameters A, B, C, D, and S to ensure the accuracy of the adjustment.

[0142] Therefore, by converting the adjustment result from units of EV to the corresponding multiple through the brightness adjustment function, any frame of the game image in the game video is multiplied by the corresponding multiple in all lighting calculations in the rendering process. Ultimately, the dynamic range of the original image information is reduced, resulting in a better user experience.

[0143] Step 305: The image processing device merges the local brightness adjustment image into the original image to adjust the brightness of a local area of ​​the original image.

[0144] To better illustrate Figure 9 The adjustment function curve shown is for reference. Figure 10 ,Figure 10 The effect schematic diagram of the luminance adjustment function in the embodiment of the present application, if the slope is not 0 at x=0 as shown in the left curve, it may cause Figure 10 The boundary line in the middle floor forms a relatively obvious demarcation line, which is caused by the too fast change of the value near the boundary line, therefore, the slope is 0 at x=0, which ensures that the place where the luminance of the adjusted original image starts to change is smooth and natural, avoiding too abrupt luminance adjustment.

[0145] Figure 11 The effect schematic diagram of the luminance adjustment function in the embodiment of the present application, if the slope of the curve can be greater than 1, so that Figure 11 The relative luminance relationship of the original adjusted region and the non-adjusted region has been completely destroyed, affecting the user's use, but the slope of the curve is less than or equal to 1, which ensures that the relative luminance of the two pixels remains unchanged before and after adjustment.

[0146] Figure 12 The effect schematic diagram of the luminance adjustment function in the embodiment of the present application, limiting the slope to be 0 at x=0, limiting the slope to be less than or equal to 1, which ensures that the place where the luminance of the adjusted original image starts to change is smooth and natural, avoiding too abrupt luminance adjustment.

[0147] In some embodiments of the present application, an empirical threshold value matching the original image to be processed can also be determined; based on the empirical threshold value matching the original image to be processed and the boundary range of image processing matching the color key parameter, the overflow color pixels in the original image are determined; and the gray value of the overflow color pixels in the original image is adjusted. Wherein, by adjusting the empirical threshold value P (P≥R), the pixels in the original image with a color key distance in the closed interval [R, P] are screened out, and these pixels are defined as overflow color pixels, that is, due to the reflection of the curtain, some of them appear overflow color, affecting the perception and need to be corrected. The value of the overflow color pixel can be set as a gray value, wherein as an example, it can be corrected according to (R+G+B) / 3, in some embodiments of the present application, because the perception of RGB color is not the same in the process of color observation by different users in different use scenarios, the weight of the overflow color pixel can be flexibly adjusted according to the use habits of different users, for example: Grey = 0.2*R + 0.6 *G + 0.3 *B, to adapt to the use habits of different users, so that users obtain more comfortable use experience.

[0148] In some embodiments of the present application, when the use object corresponding to the original image is at least two, the local brightness adjustment image is shared between different use objects to achieve the same local area brightness; when the use object corresponding to the original image is one, the historical parameters of the target object are determined according to the type of the virtual scene; the corresponding over-bright area threshold and over-dark area threshold are obtained in the historical parameters of the target object; any original image corresponding to the target object is adjusted according to the obtained over-bright area threshold and over-dark area threshold; all adjusted original images are combined into video information, which is presented in the virtual scene, wherein an image template selection interface is presented, and at least one brightness template is presented in the image template selection interface; in response to a selection operation of the brightness template triggered based on the brightness template selection interface, the brightness template corresponding to the selection operation is used as the brightness of the local area corresponding to the original image.

[0149] In some embodiments of the present application, in response to a viewing operation on an image adjustment function item, a content page including the original image to be processed and the local brightness adjustment image is presented, and at least one interactive function item is presented in the content page, which is used to realize interaction with the original image to be processed; a color picking instruction based on the interactive function item is received to execute the corresponding color picking instruction. Wherein, when the user performs a background replacement operation on the image to be processed through the APP of the image processing function, the image processing APP can provide different image templates to the user in the image template selection interface, and these image templates have different display styles. In some embodiments, the selection operation of the background image template can be triggered by triggering the background image template selection interface, such as obtaining the mouse click operation or the touch operation of the screen of the user for the live video content. In some examples, the selection operation of the background image template can also be triggered by the selection shortcut key set on the terminal, such as obtaining the click operation of the user for the Print Screen key on the keyboard; or by obtaining the click operation of the user for the selection operation shortcut key of the background image template on the mobile smart device such as mobile phone and tablet computer. Since the color may be distorted after brightness adjustment, the user can select the K_nearest_cluster closest to the user input color key K_input as needed, and then weight the two to obtain the final color key K (refer to formula 2):

[0150] Formula 2

[0151] Due to the light and shadow change in the actual scene, in order to automatically adapt to the light and shadow change, a color tracking strategy is used, after a user sets a color key through a color picking instruction, the color key is still automatically updated to conform to the local image brightness adjustment.

[0152] The present application has the following beneficial technical effects:

[0153] The embodiment of the present application acquires an original image to be processed and irradiance corresponding to the original image; determines an average exposure parameter corresponding to the original image based on the original image; determines a region to be adjusted in the original image according to the irradiance corresponding to the original image and the average exposure parameter; adjusts the brightness of the region to be adjusted to obtain a local brightness adjustment image; and fuses the local brightness adjustment image into the original image to adjust the brightness of the local region of the original image. Thus, the brightness adjustment can be accurately and efficiently realized, the image processing process reduces the occupation of hardware resources, reduces the increase of hardware device cost, reduces the processing delay, is more suitable for high code flow virtual scene, and improves the user experience.

[0154] The above is only an embodiment of the present application, and is not used to limit the protection scope of the present application, any modification, equivalent replacement and improvement within the spirit and principle of the present application should be included in the protection scope of the present application.

Claims

1. An image processing method, characterized in that, The method includes: Obtain the original image to be processed and the corresponding irradiance of the original image; Based on the original image, determine the average exposure parameter corresponding to the original image; The overbright area threshold is calculated based on the average exposure parameter and the overbright area threshold difference parameter corresponding to the original image; the reference brightness of each pixel in the original image is determined based on the irradiance corresponding to the original image. The reference brightness of each pixel is compared with the overbright area threshold to determine the pixels that are greater than or equal to the overbright area threshold. Based on all pixels greater than or equal to the overbright area threshold, the overbright areas in the original image are determined as areas to be adjusted; The brightness of the area to be adjusted is adjusted to obtain a local brightness adjustment image; The local brightness adjustment image is fused into the original image to adjust the brightness of a local area of ​​the original image.

2. The method according to claim 1, characterized in that, The process of acquiring the original image to be processed and the irradiance corresponding to the original image includes: Based on the acquired original image, determine the virtual scene of the original image; Based on the virtual scene of the original image, determine the type of irradiance; Based on the type of irradiance, the original image is subjected to irradiance transformation processing to obtain the irradiance corresponding to the original image.

3. The method according to claim 1, characterized in that, The method further includes: Obtain the threshold difference parameter of the dark area corresponding to the original image; The threshold for the dark area is calculated based on the average exposure parameter and the threshold difference parameter for the dark area. Based on the irradiance corresponding to the original image, determine the reference brightness of each pixel in the original image; The reference brightness of each pixel is compared with the dark area threshold to determine the pixels that are less than or equal to the dark area threshold. Based on all pixels less than or equal to the dark area threshold, the dark areas in the original image are identified as areas to be adjusted.

4. The method according to claim 1, characterized in that, The step of adjusting the brightness of the area to be adjusted to obtain a local brightness adjustment image includes: Based on the type of the original image, a brightness adjustment function matching the original image is determined, wherein the slope of the brightness adjustment function is greater than or equal to 0 and less than 1; Based on the brightness adjustment function, the brightness of the area to be adjusted is reduced according to the difference between the irradiance corresponding to the original image and the threshold of the overbright area, to obtain a local brightness adjustment image; Based on the brightness adjustment function, the brightness of the area to be adjusted is increased according to the difference between the irradiance corresponding to the original image and the threshold of the dark area, to obtain a local brightness adjustment image.

5. The method according to claim 1, characterized in that, The method further includes: Determine an empirical threshold that matches the original image to be processed; Based on an empirical threshold matching the original image to be processed and a boundary range of image processing matching the color key parameters, the overflow pixels in the original image are determined; The grayscale values ​​of the overflow pixels in the original image are adjusted.

6. The method according to claim 1, characterized in that, The method further includes: When the original image corresponds to at least two users, the local brightness adjustment image is shared between different users to achieve the same local area brightness. When the original image corresponds to a single user object, the historical parameters of the target object are determined based on the type of the virtual scene. Obtain the corresponding overly bright area threshold and overly dark area threshold from the historical parameters of the target object; Based on the obtained overly bright area threshold and overly dark area threshold, adjust any original image corresponding to the target object; All the original images, after being adjusted, are combined into video information and presented in the virtual scene.

7. The method according to claim 6, characterized in that, The method further includes: An image template selection interface is presented, and at least one brightness template is presented in the image template selection interface; In response to a brightness template selection operation triggered by the brightness template selection interface, the brightness template corresponding to the selection operation is used as the brightness of the local region corresponding to the original image.

8. The method according to claim 1, characterized in that, The method further includes: In response to a viewing operation of an image adjustment function, a content page is presented, including the original image to be processed and the local brightness adjustment image, and at least one interactive function is presented on the content page, the interactive function being used to enable interaction with the original image to be processed; Receive a color sampling instruction for the local brightness adjustment image triggered by the interactive function item, and execute the corresponding color sampling instruction.

9. The method according to claim 1, characterized in that, The method further includes: Based on the virtual scene of the original image, determine the type of all irradiance in the virtual scene; Calculate the irradiance corresponding to each type of irradiance based on the type of irradiance; When the local brightness adjustment image is fused into the original image, different objects in the virtual scene are rendered according to the corresponding irradiance to achieve matching with the irradiance.

10. An image processing apparatus, characterized in that, The device includes: The information transmission module is used to acquire the original image to be processed and the irradiance corresponding to the original image; The information processing module is used to determine the average exposure parameter corresponding to the original image based on the original image. The information processing module is configured to: calculate an overbright area threshold based on the average exposure parameter and the overbright area threshold difference parameter corresponding to the original image; determine the reference brightness of each pixel in the original image based on the irradiance corresponding to the original image; compare the reference brightness of each pixel with the overbright area threshold to determine pixels that are greater than or equal to the overbright area threshold; and determine the overbright areas in the original image as areas to be adjusted based on all pixels that are greater than or equal to the overbright area threshold. The information processing module is used to adjust the brightness of the area to be adjusted to obtain a local brightness adjustment image; The information processing module is used to fuse the local brightness adjustment image into the original image, so as to adjust the brightness of a local area of ​​the original image.

11. The apparatus according to claim 10, characterized in that, The information processing module is also used to determine the virtual scene of the original image based on the acquired original image; Based on the virtual scene of the original image, determine the type of irradiance; Based on the type of irradiance, the original image is subjected to irradiance transformation processing to obtain the irradiance corresponding to the original image.

12. The apparatus according to claim 10, characterized in that, The information processing module is also used to obtain the threshold difference parameter of the dark area corresponding to the original image; The threshold for the dark area is calculated based on the average exposure parameter and the threshold difference parameter for the dark area. Based on the irradiance corresponding to the original image, determine the reference brightness of each pixel in the original image; The reference brightness of each pixel is compared with the dark area threshold to determine the pixels that are less than or equal to the dark area threshold. Based on all pixels less than or equal to the dark area threshold, the dark areas in the original image are identified as areas to be adjusted.

13. The apparatus according to claim 10, characterized in that, The information processing module is further configured to determine a brightness adjustment function that matches the original image based on the type of the original image, wherein the slope of the brightness adjustment function is greater than or equal to 0 and less than 1; Based on the brightness adjustment function, the brightness of the area to be adjusted is reduced according to the difference between the irradiance corresponding to the original image and the threshold of the overbright area, to obtain a local brightness adjustment image; Based on the brightness adjustment function, the brightness of the area to be adjusted is increased according to the difference between the irradiance corresponding to the original image and the threshold of the dark area, to obtain a local brightness adjustment image.

14. The apparatus according to claim 10, characterized in that, The information processing module is also used to determine an empirical threshold that matches the original image to be processed; Based on an empirical threshold matching the original image to be processed and a boundary range of image processing matching the color key parameters, the overflow pixels in the original image are determined; The grayscale values ​​of the overflow pixels in the original image are adjusted.

15. The apparatus according to claim 10, characterized in that, The information processing module is used to share the local brightness adjustment image among different users when there are at least two users corresponding to the original image, so as to achieve the same local area brightness. The information processing module is used to determine the historical parameters of the target object based on the type of the virtual scene when the original image corresponds to a single user object. The information processing module is used to obtain the corresponding overly bright area threshold and overly dark area threshold from the historical parameters of the target object. The information processing module is used to adjust any original image corresponding to the target object based on the acquired overly bright area threshold and overly dark area threshold. The information processing module is used to combine all the adjusted original images into video information and present it in the virtual scene.

16. A computer program product comprising a computer program or instructions, characterized in that, When the computer program or instructions are executed by the processor, they implement the image processing method according to any one of claims 1 to 9.

17. An electronic device, characterized in that, The electronic device includes: Memory, used to store executable instructions; A processor, when executing executable instructions stored in the memory, implements the image processing method according to any one of claims 1 to 9.

18. A computer-readable storage medium storing executable instructions, characterized in that, When the executable instructions are executed by the processor, they implement the image processing method according to any one of claims 1 to 9.

Images