Image processing method, device, electronic device and storage medium

Abstract

The disclosed embodiments provide an image processing method, device, electronic device and storage medium, the method comprising: obtaining a video frame to be processed, and determining a target normal map of the video frame to be processed; determining target illumination intensity information of at least one pixel in the video frame to be processed based on the target normal map and pre-set light source attribute information; determining display information of the corresponding pixel based on at least one target illumination intensity information, so as to determine the target video frame corresponding to the video frame to be processed based on the display information. The technical solution of the disclosed embodiments can use a normal estimation algorithm to determine the normal map of the video frame to be processed, and then can determine the target illumination intensity information of each pixel based on the relationship between the light source corresponding to the special effect and each pixel in the normal map, so as to display the corresponding pixel based on the target illumination intensity information, which not only improves the efficiency of normal map determination, but also improves the accuracy and universality of special effect addition.

Description

Technical Field

[0001] The embodiments of the present disclosure relate to the field of image processing technology, and in particular, to an image processing method, device, electronic device, and storage medium. Background Art

[0002] With the development of network technology, more and more applications have entered the lives of users, especially a series of software that can shoot short videos, which are deeply loved by users.

[0003] When shooting corresponding videos or images based on short video shooting software, corresponding special effects are often added for the users in the video frames, which can not only enrich the content displayed in the video frames, but also provide technical effects for the user experience.

[0004] Currently, when adding special effects to a target object, it is necessary to first determine the target normal map corresponding to it. The existing method of determining the normal map mainly relies on end-to-end training with deep learning, that is, using paired normal data to train a model, and then using the model to determine the normal map of each video frame.

[0005] The problem with this method is that paired normal data is very difficult to obtain, and it is necessary to use a camera with depth information to collect information about objects in the actual scene. However, it is also limited by the errors and precision of the equipment itself, and the data quality obtained is poor. Correspondingly, the trained model is also inaccurate. Furthermore, even if a large amount of data is obtained, the normal results obtained by each device are not consistent due to differences in the collection equipment. Finally, even if there is no problem with data collection, the deployment of deep learning models is limited by the hardware environment, which will bring a certain amount of model inference time. In order to speed up the inference, the input resolution and model size of the model are usually low, which has a certain impact on the quality of the output results, especially for special effects algorithms in mobile scenarios. There are problems of long time consumption and poor special effects adding effect. Summary of the invention

[0006] The present disclosure provides an image processing method, device, electronic device and storage medium to achieve determination of a normal map based on a mobile terminal, thereby improving the convenience of normal map determination and achieving the technical effect of adding universal special effects.

[0007] In a first aspect, an embodiment of the present disclosure provides an image processing method, the method comprising:

[0008] Acquire a video frame to be processed, and determine a target normal map of the video frame to be processed;

[0009] Determining target light intensity information of at least one pixel in the video frame to be processed according to the target normal map and preset light source attribute information;

[0010] Based on at least one target light intensity information, display information of a corresponding pixel point is determined, so as to determine a target video frame corresponding to the video frame to be processed based on the display information. In a second aspect, an embodiment of the present disclosure further provides an image processing device, the device comprising:

[0011] A normal map determination module, used to obtain a video frame to be processed and determine a target normal map of the video frame to be processed;

[0012] A light intensity determination module, used to determine target light intensity information of at least one pixel in a to-be-processed video frame according to the target normal map and preset light source attribute information;

[0013] The target video frame display module is used to determine display information of corresponding pixel points based on at least one target light intensity information, so as to determine a target video frame corresponding to the video frame to be processed based on the display information.

[0014] In a third aspect, an embodiment of the present disclosure further provides an electronic device, the electronic device comprising:

[0015] one or more processors;

[0016] a storage device for storing one or more programs,

[0017] When the one or more programs are executed by the one or more processors, the one or more processors implement the image processing method as described in any of the embodiments of the present disclosure.

[0018] In a fourth aspect, the embodiments of the present disclosure further provide a storage medium comprising computer executable instructions, which, when executed by a computer processor, are used to execute the image processing method as described in any one of the embodiments of the present disclosure.

[0019] The technical solution of the disclosed embodiment can determine the target illumination intensity information of each pixel in the video frame to be processed according to the target normal map and the preset light source attribute information after obtaining the video frame to be processed and determining the target normal map of the video frame to be processed, and then determine the display information of the corresponding pixel based on the target illumination intensity information of each pixel, so as to determine the target video frame corresponding to the video frame to be processed based on the display information, thereby solving the problem in the prior art that the quality of the learning model obtained through training is poor and inconsistent due to the poor quality of the training samples, thereby the normal map determined is inaccurate, and when the model is applied to the terminal device, the performance requirements of the terminal device are high, and there are problems such as low efficiency in determining the normal map and poor universality. However, the technical solution can use a normal estimation algorithm to determine the normal map of the video frame to be processed, and then determine the target illumination intensity information of each pixel according to the relationship between the light source corresponding to the special effect and each pixel in the normal map, so as to display the corresponding pixel based on the target illumination intensity information, thereby improving the efficiency of normal map determination and the accuracy and universality of special effect addition. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and the originals and elements are not necessarily drawn to scale.

[0021] Figure 1 A schematic diagram of a flow chart of an image processing method provided in the first embodiment of the present disclosure;

[0022] Figure 2 A schematic diagram of a flow chart of an image processing method provided in the first embodiment of the present disclosure;

[0023] Figure 3 A schematic diagram of determining a target normal graph provided in the first embodiment of the present disclosure;

[0024] Figure 4 A schematic diagram of the structure of an image processing device provided in Embodiment 2 of the present disclosure;

[0025] Figure 5 This is a schematic diagram of the structure of an electronic device provided in Embodiment 3 of the present disclosure. DETAILED DESCRIPTION

[0026] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments described herein, which are instead provided for a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0027] It should be understood that the various steps described in the method embodiments of the present disclosure may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.

[0028] The term "including" and its variations used herein are open inclusions, i.e., "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". The relevant definitions of other terms will be given in the following description.

[0029] It should be noted that the concepts such as "first" and "second" mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0030] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0031] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0032] Before introducing the present technical solution, an example description of the application scenario can be given first. The disclosed technical solution can be applied to any screen that requires special effects display, such as in the video shooting process. After the video shooting is completed, corresponding special effects can be added to each video frame in the video; or, each time a video frame is shot, it can be uploaded to the server so that the server processes the video frame to achieve the addition of corresponding special effects. In the present technical solution, the added special effects can be lighting effects, which are information that can be displayed when the light illuminates the target object in the video frame to be processed. Lighting effects can be special effects scenes created based on the light emitted by a virtual light source.

[0033] It should be noted that the implementation of this technical solution can be performed by the server, or by the client, or by the configuration of the client and the server. For example, the client shoots the corresponding video frame, and the client processes the video frame to add corresponding special effects to the video frame; or the shot video frame is uploaded to the server, and after the server completes the processing, it is sent to the client, so that the client displays the video frame with the special effects added.

[0034] Embodiment 1

[0035] Figure 1 This is a flowchart of an image processing method provided in the first embodiment of the present disclosure. The present embodiment is applicable to any image display scenario supported by the Internet, and is used to add corresponding special effects to video frames. The method can be executed by an image processing device, which can be implemented in the form of software and / or hardware. Optionally, it can be implemented by an electronic device, which can be a mobile terminal, a PC, or a server. Any image display scenario is usually implemented by the cooperation of a client and a server. The method provided in this embodiment can be executed by the server, the client, or the cooperation of the client and the server.

[0036] S110, obtaining a video frame to be processed, and determining a target normal map of the video frame to be processed.

[0037] It should also be noted that the device for executing the video processing method provided in the embodiment of the present disclosure can be integrated into an application software having the function of processing each video frame in the video, and the software can be installed in an electronic device, and optionally, the electronic device can be a mobile terminal or a PC, etc. The application software can be a type of software for image / video processing, and its specific application software will not be described one by one here, as long as it can realize image / video processing.

[0038] It should be further explained that the implementation of the technical solution can be implemented by the client or by the server; it can be a situation where each video frame in the video is processed after the video shooting is completed and sent to the client for display, or it can be a situation where each received video frame is processed sequentially during the video shooting process.

[0039] The video frame currently received by the client or server can be used as the current video frame, and after receiving the target video, the client or server can process the video frames in the target video in sequence and use the video frame currently being processed as the current video frame. The normal map corresponding to the video frame to be processed is used as the target normal map.

[0040] In order to achieve the universality of determining the target normal map, that is, to make it applicable to mobile terminals, the normal map determination method adopted by the present technical solution can be used to process it.

[0041] In this embodiment, the target normal map of the entire image can be determined, and the added special effects are usually added to the entire target object, or the target special effects are added to a part of the target object. At this time, the entire normal map corresponding to the target object or the normal map of the local target object can be determined.

[0042] A first implementation method may be to sequentially obtain each to-be-processed video frame in the target video; determine the gradient information of at least one pixel point in the to-be-processed video frame in a first direction and a second direction to obtain the normal information of the at least one pixel point; and obtain the target normal map based on the normal information of the at least one pixel point.

[0043] Among them, the target video can be a video that the mobile terminal needs to process. Each video frame in the target video can be used as a video frame to be processed. Each pixel point has a normal information corresponding to it, and the normal information may include gradient information in a first direction and a second direction. The first direction may be a horizontal direction, and the second direction may be a vertical direction. According to the normal information of each pixel point, the target normal map corresponding to the video frame to be processed can be determined. At this time, each pixel point in the target normal map has gradient information in two directions.

[0044] Specifically, a normal map determination algorithm may be used to determine the gradient information of each pixel in the video frame to be processed, thereby obtaining a target normal map of the video frame to be processed.

[0045] A second implementation method may be: obtaining a video frame to be processed, and determining a target segmentation area corresponding to the video frame to be processed based on a pre-trained image segmentation model; determining gradient information of at least one pixel point in the target segmentation area in a first direction and a second direction, obtaining normal information of at least one pixel point, and determining a target normal map of the video frame to be processed based on the normal information.

[0046] Among them, the image segmentation model is a pre-trained neural network model. The input of the image segmentation model can be the current video frame, and the output of the model can be the portrait segmentation result corresponding to the current video frame, that is, the segmented sub-image to be processed. The image segmentation model is a neural network, and the structure of the network can be VGG, ResNet, GoogleNet, MobileNet, ShuffleNet, etc. For different network structures, the amount of calculation of each network structure is different, which can be understood as not all models are lightweight. That is, some models have a large amount of calculation and are not suitable for deployment on mobile terminals, while models with small amount of calculation, efficient calculation, and simple are easier to deploy on mobile terminals. If the implementation of this technical solution is based on a mobile terminal, then the MobileNet and ShuffleNet model structures can be used. The principle of the above model structure is to transform the traditional convolution into separable convolution, namely depthwise convolution and point-wise convolution, in order to reduce the amount of calculation; Inverted Residuals is used to improve the feature extraction ability of depthwise convolution; At the same time, the simple operation of shuffle channel is also used to improve the expression ability of the model. The above is the basic module design of the model. The model is basically composed of the above modules stacked together. The advantage of this type of model is that the inference time is less and can be applied to terminals with high time requirements. If it is implemented by a server, then any of the above neural networks can be used, as long as it can achieve the portrait segmentation of the video frame, and then obtain the segmented sub-image to be processed of the portrait segmentation result. It should be noted that the above is only a description of the image segmentation model and does not limit it. The area to add special effects to the video frame to be processed can be predetermined, or it can be predetermined that special effects are to be added to the target object in the video frame to be processed, then the area corresponding to the target object is the target segmentation area. After determining the target segmentation area, the gradient information of each pixel in the target segmentation area can be determined. According to the normal information of each pixel, the target normal map of the video frame to be processed is determined.

[0047] Exemplarily, the image segmentation model is used to segment the user image in the video frame. Figure 2 After receiving the video frame to be processed, the video frame to be processed is segmented based on the image segmentation model set in the terminal device to obtain the target user in the video frame to be processed. The normal map corresponding to the target user in the video frame to be processed can be determined based on the fast normal estimation of the image gradient, and this normal map is used as the target normal map. The effect of the target normal map can be seen in Figure 3 .

[0048] In this embodiment, the gradient information of at least one pixel point in the first direction and the second direction is determined to obtain the normal information of at least one pixel point, including: filtering the video frame to be processed by joint bilateral filtering to obtain the video frame to be used; using the Sobel operator to determine the gradient information of each pixel point in the first direction and the second direction in the video frame to be used, and determining the normal information of at least one pixel point.

[0049] It should be noted that the advantage of determining the target normal map is that related special effects can be performed based on the normal information.

[0050] Exemplarily, when shooting the corresponding scene based on a mobile terminal, a camera can usually be used. The sensor set in the camera will be affected by the environment, resulting in the presence of corresponding background noise in the video frame to be processed. However, this type of noise does not belong to the image content itself and can be filtered out to accurately estimate the normal information of the image. The filtering method for filtering out image noise is mainly joint bilateral filtering. This method not only filters out the noise in the image, but also retains the edge information in the image. The edge information is relatively important for the normal estimation of the video frame to be processed. That is, the advantage of using joint bilateral filtering is that it can filter out the noise while also better maintaining the edge information. After the filtering is completed, the sobel operator can be used to determine the normal information of each pixel in the video frame to be processed. Simply put, this algorithm can calculate two gradient information in the horizontal and vertical directions of the image. The gradient information is positive and negative. The physical meaning of the gradient information is the probability of representing the edge of the pixel. The positive and negative of the gradient information can characterize the direction of the pixel. In this case, the direction of the corresponding pixel in the horizontal and vertical directions, that is, the normal information, can be obtained. The effect diagram can be seen in Figure 3 .

[0051] S120. Determine target light intensity information of at least one pixel in the video frame to be processed according to the target normal map and preset light source attribute information.

[0052] Among them, light source special effects can be added to the video frame to be processed, that is, after the light source is hit at the position of the target normal map determined in the video frame to be processed, the corresponding special effects can be displayed. The special effects displayed correspond to the specific position information of the light source and the normal information of the pixel point. The light source attribute information can be the light source position information. When the light source is irradiated on the target normal map, the illumination intensity of the corresponding pixel point can be determined according to the normal information of each pixel point, and the illumination intensity determined at this time can be used as the target illumination intensity information.

[0053] Specifically, the light source position information of the target light source may be acquired, and the target light intensity information of each pixel may be determined according to the light source position information and the normal information of each pixel in the target normal map.

[0054] In this embodiment, only the pixel points whose normal information has been determined in the to-be-processed video frame may be processed to determine the target light intensity information of the pixel points.

[0055] Optionally, the target light intensity information of at least one pixel in the video frame to be processed is determined based on the target normal map and pre-set light source attribute information, including: for each pixel, determining the target normal information corresponding to the current pixel in the target normal map, and determining the target light intensity information of the current pixel based on the target normal information, light source attribute information and shooting angle information of the video frame to which the current pixel belongs.

[0056] It should also be noted that the processing method for each pixel in the processed video frame is the same, and the determination of the target light intensity information of one pixel can be used as an example to introduce it, and the pixel currently introduced is used as the current pixel.

[0057] The normal information corresponding to the current pixel is used as the target normal information. The light source attribute information includes the light source position information and / or the illumination angle information. The shooting angle information can be understood as the relative shooting angle between the camera device and the current pixel when shooting the video frame to be processed.

[0058] Specifically, the target normal information of the current pixel point can be determined. The target light intensity information of the current pixel point can be determined by the target normal information, light intensity information, light source position information and shooting angle information.

[0059] In this embodiment, the light source attribute information includes light source position information, and the target light intensity information of the current pixel is determined according to the target normal information, the light source attribute information and the shooting angle information of the video frame to which the current pixel belongs, including: determining the lighting direction information of the current pixel according to the light source position information; determining the diffuse reflection value of the current pixel according to the lighting direction information, the target normal information and a preset diffuse reflection coefficient value; determining the target reflection angle according to the lighting direction information and the target normal information, and determining the reflection intensity value of the current pixel according to the target reflection angle, the shooting angle information and the preset reflection coefficient value; determining the target light intensity information according to the diffuse reflection value, the reflection intensity value and the ambient light intensity value corresponding to the light source attribute information.

[0060] It should also be noted that the lighting special effects method adopted by the present technical solution can be realized through the Feng lighting model. The Feng lighting model is mainly characterized by three components, namely: ambient light, diffuse reflection value and specular lighting value. Ambient light (Ambient Lighting) can be understood as: even in the dark, there is usually still some light in the world (moon, distant light), so objects are almost never completely dark. In order to simulate this effect, an ambient lighting constant can be used, which will always give the object some light information. Diffuse reflection (Diffuse Lighting) can be understood as: simulating the directional impact of the light source on the object (Directional Impact). It is the most visually significant component in the Feng lighting model. The more a part of an object is facing the light source, the brighter it will be. Another dimension of specular lighting (Specular Lighting) can be understood as: simulating the bright spots that appear on shiny objects. The color of specular lighting will be more inclined to the color of light than the color of the object. The target light intensity information of each pixel can be determined based on the above information.

[0061] Among them, the light source position information can be represented by world coordinates. The lighting direction information can be represented by a relative angle. By calculating the value between the world coordinates of the current pixel and the light source position coordinates, the lighting direction information corresponding to the current pixel can be determined. The diffuse reflection coefficient value, the reflection coefficient value, and the ambient light information are pre-set. The target reflection angle can be understood as the angle obtained after the light source irradiates the current pixel and is reflected by the normal information of the current pixel.

[0062] Specifically, the pixel position of the current pixel is pos, the light source position information is light_pos, and the shooting angle of the shooting device is viewpos. The lighting direction information lightDir is determined based on the light source position information light_pos and the pixel position pos of the current pixel. Optionally, the cosine calculation method can be used to determine the lighting direction information of the light source corresponding to the current pixel. Then calculate the degree of similarity between the normal information norm of the current pixel and the lighting direction information lightDir (dot product of norm and lightDir) to determine the intermediate value. According to the intermediate value and the diffuse reflection intensity coefficient a1, that is, the diffuse reflection coefficient value, the final diffuse reflection value of the current pixel is obtained. By calculating the corresponding reflection angle reflectDir when the target normal information norm of the current pixel is the central axis and the light direction light_pos is the incident line. According to the approximate value of the calculated reflection angle reflectDir and the shooting angle viewDir (viewPos-pos), the closer the reflection angle is, the stronger the mirror reflection is. The approximate value is multiplied by the mirror illumination coefficient a2 to obtain the final reflection intensity value, that is, the mirror reflection intensity value. The target illumination intensity information of the current pixel is determined according to the maximum value of the ambient light, diffuse reflection value, and reflection intensity value.

[0063] Optionally, determining the target light intensity information according to the diffuse reflection value, the reflection intensity value and the ambient light intensity value corresponding to the light source attribute information includes: taking the maximum value among the diffuse reflection value, the reflection intensity value and the ambient light intensity value as the target light intensity information.

[0064] Specifically, the maximum value among the three values ​​determined above may be used as the target light intensity information of the current pixel.

[0065] S130. Determine display information of a corresponding pixel point based on at least one target light intensity information, so as to determine a target video frame corresponding to the to-be-processed video frame based on the display information.

[0066] Optionally, determining the display information of the corresponding pixel point based on at least one target light intensity information includes: updating the display information of the corresponding pixel point according to the target light intensity information of at least one pixel point and the corresponding pixel value information.

[0067] Specifically, each pixel in the video frame to be processed has a corresponding color value. The brightness value of the corresponding pixel can be updated based on the target light intensity information, and then the corresponding pixel in the video frame is updated based on the updated brightness value and the color value of the corresponding pixel.

[0068] The technical solution of the disclosed embodiment can use OpenGL to draw from the segmentation result to the final effect output, which is very fast. In addition, the drawing area is limited to the segmented area, and other areas do not participate in the calculation, which greatly reduces the time consumption, is conducive to the development and deployment of mobile special effects, and further improves the image processing efficiency.

[0069] The technical solution of the disclosed embodiment can determine the target illumination intensity information of each pixel in the video frame to be processed according to the target normal map and the preset light source attribute information after obtaining the video frame to be processed and determining the target normal map of the video frame to be processed, and then determine the display information of the corresponding pixel based on the target illumination intensity information of each pixel, so as to determine the target video frame corresponding to the video frame to be processed based on the display information, thereby solving the problem in the prior art that the quality of the learning model obtained through training is poor and inconsistent due to the poor quality of the training samples, thereby the normal map determined is inaccurate, and when the model is applied to the terminal device, the performance requirements of the terminal device are high, and there are problems such as low efficiency in determining the normal map and poor universality. However, the technical solution can use a normal estimation algorithm to determine the normal map of the video frame to be processed, and then determine the target illumination intensity information of each pixel according to the relationship between the light source corresponding to the special effect and each pixel in the normal map, so as to display the corresponding pixel based on the target illumination intensity information, thereby improving the efficiency of normal map determination and the accuracy and universality of special effect addition.

[0070] Embodiment 2

[0071] Figure 4 This is a schematic diagram of the structure of an image processing device provided in the second embodiment of the present disclosure, such as Figure 4 As shown, the device includes: a normal map determination module 210, a light intensity determination module 220 and a target video frame display module 230.

[0072] Among them, the normal map determination module 210 is used to obtain the video frame to be processed and determine the target normal map of the video frame to be processed; the light intensity determination module 220 is used to determine the target light intensity information of at least one pixel point in the video frame to be processed according to the target normal map and pre-set light source attribute information; the target video frame display module 230 is used to determine the display information of the corresponding pixel point based on at least one target light intensity information, so as to determine the target video frame corresponding to the video frame to be processed based on at least one display information.

[0073] On the basis of the above technical solution, the normal map determination module includes:

[0074] A video frame acquisition unit is used to sequentially acquire at least one to-be-processed video frame in a target video; a normal information determination unit is used to determine the gradient information of at least one pixel point in the to-be-processed video frame in a first direction and a second direction to obtain the normal information of at least one pixel point; and a normal map determination unit is used to obtain the target normal map based on the normal information of at least one pixel point.

[0075] On the basis of the above technical solution, the normal map determination module includes:

[0076] A segmentation region determination unit is used to obtain a video frame to be processed, and determine a target segmentation region corresponding to the video frame to be processed based on a pre-trained image segmentation model; a normal map determination unit is used to determine the gradient information of at least one pixel point in the target segmentation region in a first direction and a second direction, obtain the normal information of at least one pixel point, and determine the target normal map of the video frame to be processed based on the normal information.

[0077] On the basis of the above technical solution, the normal information determination unit includes:

[0078] The subunit for determining the video frame to be used is used to filter the video frame to be processed by joint bilateral filtering to obtain the video frame to be used; the subunit for determining the normal information is used to use the Sobel operator to determine the gradient information of each pixel point in the video frame to be used in the first direction and the second direction, and determine the normal information of at least one pixel point.

[0079] Based on the above technical solution, the light intensity determination module is also used to determine the target normal information corresponding to the current pixel point in the target normal map for at least one pixel point, and determine the target light intensity information of the current pixel point based on the target normal information, light source attribute information and the shooting angle information of the video frame to which the current pixel point belongs.

[0080] On the basis of the above technical solution, the light source attribute information includes light source position information, and the light intensity determination module further includes:

[0081] A lighting direction determining unit, used to determine lighting direction information of a current pixel point according to the light source position information;

[0082] A diffuse reflection value determination unit, used to determine the diffuse reflection value of the current pixel point according to the illumination direction information, the target normal information and a preset diffuse reflection coefficient value;

[0083] a reflection intensity value determining unit, configured to determine a target reflection angle according to the illumination direction information and the target normal information, and to determine a reflection intensity value of the current pixel point according to the target reflection angle, the shooting angle information and a preset reflection coefficient value;

[0084] The target illumination intensity determination unit is used to determine the target illumination intensity information according to the diffuse reflection value, the reflection intensity value and the ambient light intensity value corresponding to the light source attribute information.

[0085] On the basis of the above technical solution, the target illumination intensity determination unit is further used to use the maximum value among the diffuse reflection value, the reflection intensity value and the ambient light intensity value as the target illumination intensity information.

[0086] On the basis of the above technical solution, the target video frame display module is further used to determine display information of at least one pixel point according to target illumination intensity information of at least one pixel point and corresponding pixel value information.

[0087] The technical solution of the disclosed embodiment can determine the target illumination intensity information of at least one pixel in the video frame to be processed according to the target normal map and the preset light source attribute information after obtaining the video frame to be processed and determining the target normal map of the video frame to be processed, and then determine the display information of the corresponding pixel based on the target illumination intensity information of the at least one pixel, so as to determine the target video frame corresponding to the video frame to be processed based on the at least one display information, thereby solving the problem in the prior art that the quality of the learning model obtained through training is poor and inconsistent due to the poor quality of the training samples, thereby the normal map determined is inaccurate, and when the model is applied to the terminal device, the performance requirements of the terminal device are high, and there are problems such as low efficiency in determining the normal map and poor universality. However, the technical solution can use a normal estimation algorithm to determine the normal map of the video frame to be processed, and then determine the target illumination intensity information of at least one pixel according to the relationship between the light source corresponding to the special effect and the at least one pixel in the normal map, so as to display the corresponding pixel based on the target illumination intensity information, thereby improving not only the efficiency of normal map determination but also the accuracy and universality of special effect addition.

[0088] The image processing device provided in the embodiments of the present disclosure can execute the image processing method provided in any embodiment of the present disclosure, and has the corresponding functional modules and beneficial effects of the execution method.

[0089] It is worth noting that the various units and modules included in the above-mentioned device are only divided according to functional logic, but are not limited to the above-mentioned division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the protection scope of the embodiments of the present disclosure.

[0090] Embodiment 3

[0091] Figure 5 This is a schematic diagram of the structure of an electronic device provided by the third embodiment of the present disclosure. Figure 5 , which shows an electronic device (eg, Figure 5 The terminal device in the embodiment of the present disclosure may include but is not limited to mobile terminals such as mobile phones, notebook computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 5 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0092] like Figure 5 As shown, the electronic device 300 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 301, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 302 or a program loaded from a storage device 306 into a random access memory (RAM) 303. In the RAM 303, various programs and data required for the operation of the electronic device 300 are also stored. The processing device 301, the ROM 302, and the RAM 303 are connected to each other via a bus 304. An edit / output (I / O) interface 305 is also connected to the bus 304.

[0093] Typically, the following devices may be connected to the I / O interface 305: input devices 306 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; output devices 307 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 308 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 309. The communication devices 309 may allow the electronic device 300 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 5 The electronic device 300 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0094] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication device 309, or installed from the storage device 308, or installed from the ROM 302. When the computer program is executed by the processing device 301, the above-mentioned functions defined in the method of the embodiment of the present disclosure are executed.

[0095] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0096] The electronic device provided by the embodiment of the present disclosure and the image processing method provided by the above embodiment belong to the same inventive concept. The technical details not fully described in this embodiment can be referred to the above embodiment, and this embodiment has the same beneficial effects as the above embodiment.

[0097] Embodiment 4

[0098] The embodiments of the present disclosure provide a computer storage medium on which a computer program is stored. When the program is executed by a processor, the image processing method provided by the above embodiments is implemented.

[0099] It should be noted that the computer-readable medium disclosed above may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0100] In some embodiments, the client and the server may communicate using any currently known or future developed network protocol such as HTTP (HyperText Transfer Protocol), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

[0101] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0102] The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device:

[0103] Acquire a video frame to be processed, and determine a target normal map of the video frame to be processed;

[0104] Determining target light intensity information of at least one pixel in the video frame to be processed according to the target normal map and preset light source attribute information;

[0105] Based on at least one target light intensity information, display information of a corresponding pixel point is determined, so as to determine a target video frame corresponding to the to-be-processed video frame based on the display information.

[0106] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including, but not limited to, object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0107] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0108] The units involved in the embodiments described in the present disclosure may be implemented by software or hardware. The name of a unit does not limit the unit itself in some cases. For example, a normal map determination module may also be described as an "image determination module".

[0109] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0110] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0111] According to one or more embodiments of the present disclosure, [Example 1] provides an image processing method, the method comprising:

[0112] Acquire a video frame to be processed, and determine a target normal map of the video frame to be processed;

[0113] Determining target light intensity information of at least one pixel in the video frame to be processed according to the target normal map and preset light source attribute information;

[0114] Based on at least one target light intensity information, display information of a corresponding pixel point is determined, so as to determine a target video frame corresponding to the to-be-processed video frame based on the at least one display information.

[0115] According to one or more embodiments of the present disclosure, [Example 2] provides an image processing method, the method further comprising:

[0116] Optionally, the obtaining of a to-be-processed video frame and determining a target normal map of the to-be-processed video frame includes:

[0117] Sequentially acquiring at least one to-be-processed video frame in the target video;

[0118] Determine gradient information of at least one pixel point in the to-be-processed video frame in a first direction and a second direction to obtain normal information of at least one pixel point;

[0119] The target normal map is obtained based on normal information of at least one pixel point.

[0120] According to one or more embodiments of the present disclosure, [Example 3] provides an image processing method, the method further comprising:

[0121] Optionally, the obtaining of a to-be-processed video frame and determining a target normal map of the to-be-processed video frame includes:

[0122] Acquire a video frame to be processed, and determine a target segmentation area corresponding to the video frame to be processed based on a pre-trained image segmentation model;

[0123] Determine gradient information of at least one pixel point in the target segmentation area in a first direction and a second direction, obtain normal information of at least one pixel point, and determine a target normal map of the video frame to be processed based on the normal information.

[0124] According to one or more embodiments of the present disclosure, [Example 4] provides an image processing method, the method further comprising:

[0125] Optionally, determining gradient information of at least one pixel point in a first direction and a second direction to obtain normal information of at least one pixel point includes:

[0126] Performing filtering processing on the video frame to be processed by joint bilateral filtering to obtain a video frame to be used;

[0127] The Sobel operator is used to determine the gradient information of each pixel point in the video frame to be used in the first direction and the second direction, and the normal information of at least one pixel point is determined.

[0128] According to one or more embodiments of the present disclosure, [Example 5] provides an image processing method, the method further comprising:

[0129] Optionally, determining target illumination intensity information of at least one pixel in a to-be-processed video frame according to the target normal map and preset light source attribute information includes:

[0130] For at least one pixel point, determine the target normal information corresponding to the current pixel point in the target normal map, and determine the target light intensity information of the current pixel point based on the target normal information, light source attribute information and shooting angle information of the video frame to which the current pixel point belongs.

[0131] According to one or more embodiments of the present disclosure, [Example 6] provides an image processing method, the method further comprising:

[0132] Optionally, the light source attribute information includes light source position information, and determining the target light intensity information of the current pixel point according to the target normal information, the light source attribute information, and the shooting angle information of the video frame to which the current pixel point belongs includes:

[0133] Determining the illumination direction information of the current pixel point according to the light source position information;

[0134] Determine the diffuse reflection value of the current pixel point according to the illumination direction information, the target normal information and a preset diffuse reflection coefficient value;

[0135] Determine a target reflection angle according to the illumination direction information and the target normal information, and determine a reflection intensity value of the current pixel point according to the target reflection angle, the shooting angle information and a preset reflection coefficient value;

[0136] The target illumination intensity information is determined according to the diffuse reflection value, the reflection intensity value, and the ambient light intensity value corresponding to the light source attribute information.

[0137] According to one or more embodiments of the present disclosure, [Example 7] provides an image processing method, the method further comprising:

[0138] Optionally, determining the target illumination intensity information according to the diffuse reflection value, the reflection intensity value, and the ambient light intensity value corresponding to the light source attribute information includes:

[0139] The maximum value among the diffuse reflection value, the reflection intensity value and the ambient light intensity value is used as the target light intensity information.

[0140] According to one or more embodiments of the present disclosure, [Example 8] provides an image processing method, the method further comprising:

[0141] Optionally, determining display information of a corresponding pixel point based on at least one target light intensity information includes:

[0142] At least one display information of a display class is determined according to the target illumination intensity information of at least one pixel point and the corresponding pixel value information.

[0143] According to one or more embodiments of the present disclosure, [Example 9] provides an image processing device, the device comprising:

[0144] A normal map determination module, used to obtain a video frame to be processed and determine a target normal map of the video frame to be processed;

[0145] A light intensity determination module, used to determine target light intensity information of at least one pixel in a to-be-processed video frame according to the target normal map and preset light source attribute information;

[0146] The target video frame display module is used to determine the display information of the corresponding pixel point based on at least one target light intensity information, so as to determine the target video frame corresponding to the video frame to be processed based on the at least one display information.

[0147] The above description is only a preferred embodiment of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present disclosure (but not limited to) by each other to form a technical solution.

[0148] In addition, although each operation is described in a specific order, this should not be understood as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.

[0149] Although the subject matter has been described in language specific to structural features and / or methodological logical actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are merely example forms of implementing the claims.

Claims

1. An image processing method, characterized in that: include: Acquire a video frame to be processed, and determine a target normal map of the video frame to be processed; Determining target illumination intensity information of at least one pixel in a to-be-processed video frame according to the target normal map and pre-set light source attribute information, specifically comprising: determining, for the pixel, target normal information corresponding to the current pixel in the target normal map, and determining target illumination intensity information of the current pixel according to the target normal information, light source attribute information, and shooting angle information of the video frame to which the current pixel belongs; wherein the light source attribute information includes light source position information; Based on at least one target light intensity information, display information of a corresponding pixel point is determined, so as to determine a target video frame corresponding to the to-be-processed video frame based on the display information.

2. The method according to claim 1, characterized in that The obtaining of the to-be-processed video frame and determining the target normal map of the to-be-processed video frame includes: Obtain each to-be-processed video frame in the target video in sequence; Determine gradient information of at least one pixel point in the to-be-processed video frame in a first direction and a second direction to obtain normal information of the at least one pixel point; The target normal map is obtained based on the normal information of the at least one pixel point.

3. The method according to claim 1, characterized in that The obtaining of the to-be-processed video frame and determining the target normal map of the to-be-processed video frame includes: Acquire a video frame to be processed, and determine a target segmentation area corresponding to the video frame to be processed based on a pre-trained image segmentation model; Determine gradient information of at least one pixel point in the target segmentation area in a first direction and a second direction, obtain normal information of at least one pixel point, and determine a target normal map of the video frame to be processed based on the normal information.

4. The method according to claim 2 or 3, characterized in that: Determining gradient information of at least one pixel point in a first direction and a second direction to obtain normal information of at least one pixel point includes: Performing filtering processing on the video frame to be processed by joint bilateral filtering to obtain a video frame to be used; The Sobel operator is used to determine the gradient information of each pixel point in the video frame to be used in the first direction and the second direction, and the normal information of at least one pixel point is determined.

5. The method according to claim 1, characterized in that The determining the target illumination intensity information of the current pixel point according to the target normal information, the light source attribute information, and the shooting angle information of the video frame to which the current pixel point belongs includes: Determining the illumination direction information of the current pixel point according to the light source position information; Determine the diffuse reflection value of the current pixel point according to the illumination direction information, the target normal information and a preset diffuse reflection coefficient value; Determine a target reflection angle according to the illumination direction information and the target normal information, and determine a reflection intensity value of the current pixel point according to the target reflection angle, the shooting angle information and a preset reflection coefficient value; The target illumination intensity information is determined according to the diffuse reflection value, the reflection intensity value, and the ambient light intensity value corresponding to the light source attribute information.

6. The method according to claim 5, characterized in that The determining the target illumination intensity information according to the diffuse reflection value, the reflection intensity value, and the ambient light intensity value corresponding to the light source attribute information includes: The maximum value among the diffuse reflection value, the reflection intensity value and the ambient light intensity value is used as the target light intensity information.

7. The method according to claim 1, characterized in that The determining of display information of corresponding pixel points based on at least one target light intensity information includes: According to the target illumination intensity information of at least one pixel and the corresponding pixel value information, the display information of the corresponding pixel is updated.

8. An image processing device, characterized in that: include: A normal map determination module, used to obtain a video frame to be processed and determine a target normal map of the video frame to be processed; A light intensity determination module, used to determine target light intensity information of at least one pixel in a to-be-processed video frame according to the target normal map and preset light source attribute information; A target video frame display module, used to determine display information of corresponding pixel points based on at least one target light intensity information, so as to determine a target video frame corresponding to the video frame to be processed based on the display information; The light intensity determination module is specifically used to: determine, for the pixel point, the target normal information corresponding to the current pixel point in the target normal map, and determine the target light intensity information of the current pixel point based on the target normal information, light source attribute information and the shooting angle information of the video frame to which the current pixel point belongs; wherein the light source attribute information includes light source position information.

9. An electronic device, characterized in that: The electronic device comprises: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the image processing method as described in any one of claims 1 to 7.

10. A storage medium comprising computer executable instructions, wherein the computer executable instructions are used to perform the image processing method according to any one of claims 1 to 7 when executed by a computer processor.

Images