Image material determination method and related device

Abstract

The present application provides an image material determination method and related device, firstly, determining the target feature data of the target image; then, determining the highlight feature data and the standard feature data according to the target feature data, the highlight feature data is used to indicate the image feature that eliminates the influence of saturated pixels, and the standard feature data is used to indicate the image feature that does not eliminate the influence of saturated pixels; then, determining the fusion feature data according to the highlight feature data and the standard feature data; finally, determining the material data of the target image according to the fusion feature data, the material data includes at least one of a diffuse reflection material map, a roughness material map, a specular material map, and a normal material map. There is no need to collect too much input target image, only a single target image is needed to obtain accurate material data, which greatly improves the efficiency and accuracy of material estimation.

Description

Technical Field

[0001] The present application relates to the field of image analysis technology, and in particular to an image material determination method and related device. Background Art

[0002] With the development of technology, in many fields it is necessary to determine the true material information of an object based on its image. However, due to the inherent complexity of real-world materials, additional cameras, video cameras and other acquisition equipment are required to perform detailed spatial and angular sampling of the object to obtain the input image, which greatly affects the efficiency of material analysis of the image. Summary of the invention

[0003] In view of this, the present application provides a method for determining accurate material data based on a single image without requiring other parameters, which can greatly improve the efficiency and accuracy of material estimation.

[0004] In a first aspect, an embodiment of the present application provides a method for determining image material, the method comprising:

[0005] determining target feature data of the target image;

[0006] Determine highlight feature data and standard feature data according to the target feature data, wherein the highlight feature data is used to indicate image features where the influence of saturated pixels is eliminated, and the standard feature data is used to indicate image features where the influence of saturated pixels is not eliminated;

[0007] Determine fusion feature data according to the highlight feature data and the standard feature data;

[0008] Material data of the target image is determined according to the fusion feature data, wherein the material data includes at least one of a diffuse reflection material map, a roughness material map, a specular material map, and a normal material map.

[0009] In a second aspect, an embodiment of the present application provides an image material determination device, the device comprising:

[0010] A feature determination unit, for determining target feature data of a target image;

[0011] a branch extraction unit, configured to determine highlight feature data and standard feature data according to the target feature data, wherein the highlight feature data is used to indicate image features that eliminate the influence of saturated pixels, and the standard feature data is used to indicate image features that do not eliminate the influence of saturated pixels;

[0012] A fusion extraction unit, used to determine fusion feature data according to the highlight feature data and the standard feature data;

[0013] A material estimation unit is used to determine material data of the target image according to the fused feature data, wherein the material data includes at least one of a diffuse reflection material map, a roughness material map, a specular material map and a normal material map.

[0014] In a third aspect, an embodiment of the present application provides an electronic device, comprising a processor, a memory, a communication interface, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the processor, and the program includes instructions for executing the steps of any method in the first aspect of the embodiment of the present application.

[0015] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program for electronic data exchange, wherein the computer program enables a computer to execute part or all of the steps described in any method of the first aspect of the embodiment of the present application.

[0016] In a fifth aspect, an embodiment of the present application provides a computer program product, wherein the computer program product includes a non-transitory computer-readable storage medium storing a computer program, and the computer program is operable to cause a computer to execute some or all of the steps described in any method of the first aspect of the embodiment of the present application. The computer program product may be a software installation package.

[0017] It can be seen that through the above-mentioned image material determination method and related devices, first, the target feature data of the target image is determined; then, the highlight feature data and the standard feature data are determined according to the target feature data, the highlight feature data is used to indicate the image features that eliminate the influence of saturated pixels, and the standard feature data is used to indicate the image features that do not eliminate the influence of saturated pixels; then, the fusion feature data is determined according to the highlight feature data and the standard feature data; finally, the material data of the target image is determined according to the fusion feature data, and the material data includes at least one of a diffuse reflection material map, a roughness material map, a specular material map, and a normal material map. There is no need to collect too much of the input target image, and only a single target image is needed to obtain accurate material data, which greatly improves the efficiency and accuracy of material estimation. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying any creative work.

[0019] Figure 1A system architecture diagram of an image material determination method provided in an embodiment of the present application;

[0020] Figure 2 A schematic diagram of a flow chart of a method for determining image material provided in an embodiment of the present application;

[0021] Figure 3A A schematic diagram of the architecture of a material estimation model provided in an embodiment of the present application;

[0022] Figure 3B A schematic diagram of the architecture of another material estimation model provided in an embodiment of the present application;

[0023] Figure 3C A schematic diagram of the architecture of another material estimation model provided in an embodiment of the present application;

[0024] Figure 4 A schematic diagram of a flow chart of another method for determining image material provided in an embodiment of the present application;

[0025] Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application;

[0026] Figure 6 A block diagram of the functional units of an image material determination device provided in an embodiment of the present application;

[0027] Figure 7 A block diagram of the functional units of another image material determination device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0028] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0029] The terms "first", "second", etc. in the specification and claims of this application and the above-mentioned drawings are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but optionally includes steps or units that are not listed, or optionally includes other steps or units inherent to these processes, methods, products or devices.

[0030] It should be understood that the term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article indicates that the associated objects before and after are in an "or" relationship. The "plurality" appearing in the embodiments of the present application refers to two or more.

[0031] The "connection" that appears in the embodiments of the present application refers to various connection methods such as direct connection or indirect connection to achieve communication between devices, and the embodiments of the present application do not impose any limitations on this.

[0032] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0033] The background technology and related terms of this application are explained below.

[0034] Background technology related:

[0035] Basic concepts:

[0036] Physically Based Rendering (PBR) refers to a technology that uses rendering equations that simulate real phenomena to render real images based on basic real physical laws and mathematical derivations. Compared with the traditional basic simulation rendering, PBR begins to comply with the laws of physics, making the rendering more realistic. However, due to the current limitations of hardware level, it is not completely calculated according to the laws of the real world. It is a rendering technology between pure empirical algorithm simulation rendering and real physical rendering. PBR rendering takes into account physical laws such as energy conservation, Fresnel reflection law, and light absorption to better express the details and roughness of the surface of objects, anisotropy, and distinguish between metal and non-metal materials, translucent materials and other complex material properties.

[0037] It can be seen that in the field of graphics, the analysis of object materials is becoming more and more important, and rendering technology is becoming more and more sophisticated. The existing analysis method can use multiple cameras to perform detailed spatial and angular sampling of objects, and perform material analysis of objects based on the multiple images obtained by sampling. This greatly affects the efficiency of material analysis of objects. In addition, the multiple images collected may have some noise themselves. The existing methods cannot remove the impact of noise and cannot guarantee the accuracy of the final material information.

[0038] To solve the above problems, the present application provides an image material determination method and related devices, which can determine the material data of an object in an image based on a single image, without the need for an additional camera to capture multiple images and without requiring other parameters as additional input, and using a dual-branch neural network model to eliminate the influence of noise, which can greatly improve the efficiency and accuracy of determining material data.

[0039] First, combine Figure 1 The system architecture of a method for determining image material provided in an embodiment of the present application is described. Figure 1 A system architecture diagram of an image material determination method provided in an embodiment of the present application, the system architecture 100 includes a collection device 110 and a material determination device 120, wherein the collection device 110 may be a device for collecting images, such as a camera, a laser device, etc., for collecting images of objects, and the collection device 110 may be connected to the material determination device 120 by wire or wirelessly, and send the collected image to the material determination device 120, or the collection device 110 may upload the image to the network without being connected to the material determination device 120. The material determination device 120 may include various handheld devices with wireless communication functions, vehicle-mounted devices, wearable devices, computing devices, or other processing devices connected to a wireless modem, as well as various forms of user equipment (UE), mobile stations (MS), terminal devices, servers, and the like.

[0040] In a possible embodiment, the material determination device 120 may be equipped with a material estimation model or a related material estimation algorithm to obtain images from the acquisition device 110 or other channels (such as a network platform, etc.), and perform corresponding processing through the material estimation model to obtain material data.

[0041] It is understandable that no matter how many images are captured by the acquisition device 110 , the material determination device 120 will perform the process of the material determination method based on a single image.

[0042] It can be seen that through the above system architecture, there is no need for detailed sampling through a large number of cameras. Only a single image is needed to obtain its material data, which greatly improves the efficiency and accuracy of material estimation.

[0043] After understanding the system architecture of this application, Figure 2 A method for determining image material in the implementation of this application is described. Figure 2 A schematic diagram of a process for determining image material provided in an embodiment of the present application specifically includes the following steps:

[0044] Step 201, determining target feature data of a target image.

[0045] The target image may be an image containing a target object, and the target feature data may be a feature vector reflecting the target image.

[0046] In a possible embodiment, the target image may first be pre-processed before feature extraction, and the pre-processing steps may include grayscale conversion, color space standardization, etc., and then preliminary feature extraction processing may be performed, and the preliminary feature extraction processing steps may include determining the gradient of each pixel, dividing the image into multiple unit areas, counting the gradient histogram of each unit area, obtaining target feature data based on the histogram, etc. Different pre-processing and feature extraction processing may be adopted according to different needs, and no specific limitation is made here.

[0047] In one possible embodiment, the target image can be input into the feature extraction module of the material estimation model, and the target feature data can be obtained according to the output of the feature extraction module. It can be understood that the above-mentioned feature extraction module can first regularize the target image. For example, the color image can be parsed into three channels of R (red), G (green) and B (blue), each of which has a value between 0 and 255. After the target image is regularized, it can be convolved through the convolution module to obtain the target feature data. The number of convolution layers in the convolution module is proportional to the complexity of the target feature data. For example, the target feature data can be a 128-dimensional feature vector or a 64-dimensional feature vector. No specific limitation is made here. The convolution module in the feature extraction module can be flexibly adjusted according to the needs to adjust the dimension of the target feature data finally determined.

[0048] It can be seen that by determining the target feature data of the target image, the target feature number can be extracted based on the document target image to provide data reference for subsequent material estimation.

[0049] Step 202: determining highlight feature data and standard feature data according to the target feature data.

[0050] The highlight feature data is used to indicate image features after the influence of saturated pixels is eliminated, and the standard feature data is used to indicate image features after the influence of saturated pixels is not eliminated.

[0051] Among them, the target feature data can be input into the highlight branch module of the material estimation model, and the highlight feature data can be determined according to the output of the highlight branch module; and the target feature data can be input into the standard branch module of the material estimation model, and the standard feature data can be determined according to the output of the standard branch module.

[0052] In a possible embodiment, the highlight branch module includes a first highlight convolution module, a first hole convolution module and a first standard convolution module. The target feature data can be input into the first highlight convolution module, and the first highlight convolution data, the second highlight convolution data and the third highlight convolution data are obtained according to the output of the first highlight convolution module; the second highlight convolution data and the third highlight convolution data are input into the first hole convolution module, and the fourth highlight convolution data is obtained according to the output of the first hole convolution module; the first highlight convolution data and the fourth highlight convolution data are input into the first standard convolution module, and the highlight feature data is obtained according to the output of the first standard convolution module.

[0053] In a possible embodiment, the standard branch module includes a second standard convolution module, a second hole convolution module and a third standard convolution module. The target feature data can be input into the second standard convolution module, and the first standard convolution data, the second standard convolution data and the third standard convolution data are obtained according to the output of the second standard convolution module; the second standard convolution data and the third standard convolution data are input into the second hole convolution module, and the fourth standard convolution data is obtained according to the output of the second hole convolution module; the first standard convolution data and the fourth standard convolution data are input into the third standard convolution module, and the standard feature data is obtained according to the output of the third standard convolution module.

[0054] It can be seen that by determining the highlight feature data and the standard feature data according to the target feature data, the influence of saturated pixels in the image can be eliminated, unnecessary features can be avoided from being extracted, and the accuracy of material estimation can be improved.

[0055] Step 203: determining fusion feature data according to the highlight feature data and the standard feature data.

[0056] The highlight feature data and the standard feature data may be input into a feature fusion module of a material estimation model, and the fused feature data may be obtained according to an output of the feature fusion module.

[0057] In a possible embodiment, the feature fusion module includes a feature connection module, a self-attention feature selection module and a fusion convolution module. The highlight feature data and the standard feature data can be input into the feature connection module, and the feature connection data can be obtained according to the output of the feature connection module; the feature connection data can be input into the self-attention feature selection module, and the feature weight data can be obtained according to the output of the self-attention feature selection module; the feature weight data can be input into the fusion convolution module, and the fusion feature data can be obtained according to the output of the fusion convolution module.

[0058] It can be seen that by determining the fused feature data based on the highlight feature data and the standard feature data, the highlight feature data and the standard feature data can be fused, so that the features describing the target image are increased, while the information under each feature is not increased, thereby eliminating the interference of saturated pixels and improving the accuracy of material estimation.

[0059] Step 204: determine the material data of the target image according to the fused feature data.

[0060] The material data includes at least one of a diffuse reflection material map, a roughness material map, a specular material map and a normal material map.

[0061] It can be seen that through the above-mentioned image material determination method, first, the target feature data of the target image is determined; then, the highlight feature data and the standard feature data are determined according to the target feature data, the highlight feature data is used to indicate the image features that eliminate the influence of saturated pixels, and the standard feature data is used to indicate the image features that do not eliminate the influence of saturated pixels; then, the fusion feature data is determined according to the highlight feature data and the standard feature data; finally, the material data of the target image is determined according to the fusion feature data, and the material data includes at least one of a diffuse reflection material map, a roughness material map, a specular material map, and a normal material map. There is no need to collect too much of the input target image, and accurate material data can be obtained with only a single target image, which greatly improves the efficiency and accuracy of material estimation.

[0062] For ease of understanding, the material estimation model in the embodiment of the present application is described below. Figure 3A As shown, Figure 3A A schematic diagram of the architecture of a material estimation model provided for an embodiment of the present application, the material estimation model 300 includes a feature extraction module 310, a highlight branch module 320, a standard branch module 330 and a feature fusion module 340, wherein a target image can be used as input data and first input into the feature extraction module 310, then the target feature data output by the feature extraction module 310 are respectively input into the highlight branch module 320 and the standard branch module 330, the highlight feature data output by the highlight branch module 320 and the standard feature data output by the standard branch module 330 are input into the feature fusion module 340 together, the material data of the target image is determined according to the output of the feature fusion module 340, and finally the diffuse material map, roughness material map, mirror material map and normal material map can be respectively determined from four independent prediction branches.

[0063] Specifically, Figure 3B As shown, Figure 3BA schematic diagram of the architecture of another material estimation model provided for an embodiment of the present application, wherein the feature extraction module 310 may include at least one convolution module 311 for feature extraction, the highlight branch module 320 may include a first highlight convolution module 321, a first hole convolution module 322 and a first standard convolution module 323, the standard branch module 330 may include a second standard convolution module 331, a second hole convolution module 332 and a third standard convolution module 333, and the feature fusion module 340 may include a feature connection module 341, a self-attention feature selection module 342 and a fusion convolution module 343. The above-mentioned first highlight convolution module 321 may include at least one convolution unit for highlight feature extraction, the above-mentioned first hole convolution module 322 may include at least one hole convolution unit, the above-mentioned first standard convolution module 323 may include at least one convolution unit for standard feature extraction, the above-mentioned second standard convolution module 331 may include at least one convolution unit for standard feature extraction, the above-mentioned second hole convolution module 332 may include at least one hole convolution unit, and the above-mentioned third standard convolution module 333 may include at least one convolution unit for standard feature extraction.

[0064] Furthermore, in a possible embodiment, the material estimation model can be as follows Figure 3C As shown, Figure 3C A schematic diagram of the architecture of another material estimation model provided for an embodiment of the present application, including a convolution unit a for extracting target feature data, highlight convolution units h1, h2, h3, h4, h5, h6, hole convolution units h7, standard convolution units h8, and h9 for extracting highlight feature data, standard convolution units s1, s2, s3, s4, s5, s6, hole convolution units s7, s8, and s9 for extracting standard feature data, a feature connection unit for determining fused feature data, an Attention-based Feature Selection Module (AFS), a fused convolution unit f1, a fused convolution unit f2, a fused convolution unit f3, and a fused convolution unit f4.

[0065] Among them, the highlight convolution unit h1 to the highlight convolution unit h6 can be understood as the first highlight convolution module, the hole convolution unit h7 can be understood as the first hole convolution module, the standard convolution unit h8 and the standard convolution unit h9 can be understood as the first standard convolution module; the standard convolution unit s1 to the standard convolution unit s6 can be understood as the second standard convolution module, the hole convolution unit s7 can be understood as the second hole convolution module, the standard convolution unit s8 and the standard convolution unit s9 can be understood as the second standard convolution module; the fused convolution unit f1 to the fused convolution unit f4 can be understood as the fused convolution module.

[0066] Specifically, the target image is input into the convolution layer a, the convolution layer a outputs the target feature data, and the target feature data is respectively transmitted to the highlight convolution unit h1 and the standard convolution unit s1.

[0067] Among them, highlight convolution unit h1 to highlight convolution unit h4 perform convolution operations in sequence, highlight convolution unit h4 can pass the output data from highlight convolution unit h3 to standard convolution unit h8. At the same time, highlight convolution unit h4 to highlight convolution unit h6 perform convolution operations in sequence, highlight convolution unit h6 can pass the output data from highlight convolution unit h5 and its own output data to hole convolution unit h7, hole convolution unit h7 can connect the output data from highlight convolution unit h5 and the output data from highlight convolution unit h6, standard convolution unit h8 can connect the output data from highlight convolution unit h3 and the output data from hole convolution unit h7, standard convolution unit h9 can perform convolution operations and output highlight feature data.

[0068] Among them, the standard convolution unit s1 to the standard convolution unit s4 perform convolution operations in sequence, the standard convolution unit s4 can pass the output data of the AIDS standard convolution unit s3 to the standard convolution unit s8, and at the same time, the standard convolution unit s4 to the standard convolution unit s4 perform convolution operations in sequence, the standard convolution unit s6 can pass the output data from the standard convolution unit s5 and its own output data to the void convolution unit s7, the void convolution unit s7 can connect the output data from the standard convolution unit s5 and the output data from the standard convolution unit s6, the standard convolution unit s8 can connect the output data from the standard convolution unit s3 and the output data of the void convolution unit s7, and the standard convolution unit s9 can perform convolution operations and output standard feature data.

[0069] Among them, the highlight feature data and the standard feature data are input into the feature connection unit for merging, and the feature connection unit outputs the merged data and passes the merged data to the self-attention feature selection AFS module, and then passes through the fusion convolution unit f1, the fusion convolution unit f2, the fusion convolution unit f3, and the fusion convolution unit f4 in sequence to obtain the fusion feature data. Finally, based on the fusion feature data, the diffuse reflection material map, the roughness material map, the specular material map, and the normal material map are output from the four prediction branches.

[0070] It should be noted that each highlight convolution unit can include multiple convolution branch operations, including normalization (IN, etc.) processing, activation function (sigmod, etc.) processing, advanced activation function (Leaky ReLU, etc.) processing, inception branch operations for retaining global and local information, etc. The highlight branch module can repair the highlight area in the target image, but it will produce transitional blurred normals and biased mirror components, and it is necessary to combine the standard branch module to obtain more complete fusion feature data, and each standard convolution unit can use the existing convolution operation method, which will not be repeated here. The self-attention feature selection module can include a pooling layer and multiple convolution layers. The self-attention feature selection module can assign high weights to information-rich features and suppress the expression of useless features.

[0071] It is understandable that the feature connection unit is often used to combine features, fuse features extracted by multiple convolutional feature extraction frameworks, or fuse information from the output layer. In other words, the features describing the target image itself are increased, but the information under each feature is not increased.

[0072] It can be seen that the material estimation model with the above dual-stream architecture can eliminate the influence of saturated pixels. There is no need to collect too much input target images. Only a single target image is needed to obtain accurate material data, which greatly improves the efficiency and accuracy of material estimation.

[0073] Combine the following Figure 4 Another method for determining image material provided in an embodiment of the present application is described. Figure 4 A schematic flow chart of another method for determining image material provided in an embodiment of the present application specifically includes the following steps:

[0074] Step 401: input the training image into a preset model, and obtain estimated material data according to the output of the preset model.

[0075] The estimated material data includes an estimated diffuse reflection material map, an estimated roughness material map, an estimated mirror material map, and an estimated normal material map. The above training images can come from a training database and are annotated images. The above preset model can be a model that has not completed training, and its architecture is consistent with the material model architecture in the embodiment of the present application.

[0076] Step 402: Process the estimated diffuse reflection material map, the estimated roughness material map, the estimated specular material map, and the estimated normal material map using a first loss function to obtain first difference data.

[0077] Among them, the first loss function can use pixel loss to penalize the differences in pixel space. For example, the estimated roughness material map, the estimated mirror material map and the estimated normal material map can be processed using map loss and render loss to obtain first difference data, which will not be elaborated here.

[0078] Step 403: Process the estimated diffuse reflection material map and the estimated normal material map using a second loss function to obtain second difference data.

[0079] Among them, the second loss function may include global D and local D, and global D and local D are added to the estimated diffuse material map and the estimated normal material map respectively to form an adversarial loss function to obtain second difference data.

[0080] It can be seen that since the first loss function usually produces blurred textures and lacks high-frequency details after processing, especially the final generated estimated diffuse material map and estimated normal material map will be blurry. In order to prevent this situation, the estimated diffuse material map and estimated normal material map can be specially processed by the second loss function to improve the accuracy of the diffuse material map and normal material map finally determined by the material estimation model.

[0081] Step 404: adjust the preset model according to the first difference data and the second difference data to obtain a material estimation model.

[0082] It can be seen that the material estimation model obtained through the above training method can eliminate the influence of saturated pixels. There is no need to collect too much input target images. Only a single target image is needed to obtain accurate material data, which greatly improves the efficiency and accuracy of material estimation.

[0083] Step 405: determine target feature data of the target image.

[0084] Step 406: Determine highlight feature data and standard feature data according to the target feature data.

[0085] Step 407: Determine fusion feature data according to the highlight feature data and the standard feature data.

[0086] Step 408: Determine the material data of the target image according to the fused feature data.

[0087] It can be seen that through the above-mentioned image material determination method, first, the target feature data of the target image is determined; then, the highlight feature data and the standard feature data are determined according to the target feature data, the highlight feature data is used to indicate the image features that eliminate the influence of saturated pixels, and the standard feature data is used to indicate the image features that do not eliminate the influence of saturated pixels; then, the fusion feature data is determined according to the highlight feature data and the standard feature data; finally, the material data of the target image is determined according to the fusion feature data, and the material data includes at least one of a diffuse reflection material map, a roughness material map, a specular material map, and a normal material map. There is no need to collect too much of the input target image, and accurate material data can be obtained with only a single target image, which greatly improves the efficiency and accuracy of material estimation.

[0088] Combine the following Figure 5 An electronic device in an embodiment of the present application is described. Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application is shown in FIG. Figure 5 As shown, the electronic device 500 includes a processor 501, a communication interface 502 and a memory 503, wherein the processor, the communication interface and the memory are interconnected, wherein the electronic device 500 may also include a bus 504, through which the processor 501, the communication interface 502 and the memory 503 may be interconnected, and the bus 504 may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The bus 504 may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 5 The memory 503 is used to store computer programs, and the computer programs include program instructions. The processor is configured to call the program instructions to execute the above Figure 2 , Figure 4 All or part of the methods described in .

[0089] The above mainly introduces the scheme of the embodiment of the present application from the perspective of the execution process on the method side. It is understandable that, in order to realize the above functions, the electronic device includes a hardware structure and / or software module corresponding to the execution of each function. It should be easily appreciated by those skilled in the art that, in combination with the units and algorithm steps of each example described in the embodiments provided herein, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present application.

[0090] The embodiment of the present application can divide the electronic device into functional units according to the above method example. For example, each functional unit can be divided according to each function, or two or more functions can be integrated into one processing unit. The above integrated unit can be implemented in the form of hardware or in the form of software functional units. It should be noted that the division of units in the embodiment of the present application is schematic and is only a logical function division. There may be other division methods in actual implementation.

[0091] In the case of dividing each functional module according to each function, the following is combined Figure 6 A device for determining image material in an embodiment of the present application is described in detail. Figure 6 This is a block diagram of the functional units of an image material determination device provided in an embodiment of the present application. The image material determination device 600 includes:

[0092] A feature determination unit 610 determines target feature data of a target image in English;

[0093] A branch extraction unit 620, configured to determine highlight feature data and standard feature data according to the target feature data, wherein the highlight feature data is used to indicate image features from which the influence of saturated pixels is eliminated, and the standard feature data is used to indicate image features from which the influence of saturated pixels is not eliminated;

[0094] A fusion extraction unit 630, configured to determine fusion feature data according to the highlight feature data and the standard feature data;

[0095] The material estimation unit 640 is used to determine the material data of the target image according to the fused feature data, wherein the material data includes at least one of a diffuse reflection material map, a roughness material map, a specular material map and a normal material map.

[0096] It can be seen that through the above-mentioned image material determination method and related devices, first, the target feature data of the target image is determined; then, the highlight feature data and the standard feature data are determined according to the target feature data, the highlight feature data is used to indicate the image features that eliminate the influence of saturated pixels, and the standard feature data is used to indicate the image features that do not eliminate the influence of saturated pixels; then, the fusion feature data is determined according to the highlight feature data and the standard feature data; finally, the material data of the target image is determined according to the fusion feature data, and the material data includes at least one of a diffuse reflection material map, a roughness material map, a specular material map, and a normal material map. There is no need to collect too much of the input target image, and only a single target image is needed to obtain accurate material data, which greatly improves the efficiency and accuracy of material estimation.

[0097] In the case of an integrated unit, the following combination Figure 7 Another image material determination device 700 in an embodiment of the present application is described in detail, and the image material determination device 700 includes a processing unit 701 and a communication unit 702, wherein the processing unit 701 is used to execute any step in the above method embodiment, and when performing data transmission such as sending, the communication unit 702 can be selectively called to complete the corresponding operation.

[0098] The image material determination device 700 may further include a storage unit 703 for storing program codes and data. The processing unit 701 may be a processor, the communication unit 702 may be a wireless communication module, and the storage unit 703 may be a memory.

[0099] The processing unit 701 is specifically used for:

[0100] determining target feature data of the target image;

[0101] Determine highlight feature data and standard feature data according to the target feature data, wherein the highlight feature data is used to indicate image features where the influence of saturated pixels is eliminated, and the standard feature data is used to indicate image features where the influence of saturated pixels is not eliminated;

[0102] Determine fusion feature data according to the highlight feature data and the standard feature data;

[0103] Material data of the target image is determined according to the fusion feature data, wherein the material data includes at least one of a diffuse reflection material map, a roughness material map, a specular material map, and a normal material map.

[0104] It can be seen that through the above-mentioned image material determination method and related devices, first, the target feature data of the target image is determined; then, the highlight feature data and the standard feature data are determined according to the target feature data, the highlight feature data is used to indicate the image features that eliminate the influence of saturated pixels, and the standard feature data is used to indicate the image features that do not eliminate the influence of saturated pixels; then, the fusion feature data is determined according to the highlight feature data and the standard feature data; finally, the material data of the target image is determined according to the fusion feature data, and the material data includes at least one of a diffuse reflection material map, a roughness material map, a specular material map, and a normal material map. There is no need to collect too much of the input target image, and only a single target image is needed to obtain accurate material data, which greatly improves the efficiency and accuracy of material estimation.

[0105] An embodiment of the present application also provides a computer storage medium, wherein the computer storage medium stores a computer program for electronic data exchange, and the computer program enables a computer to execute part or all of the steps of any method recorded in the above method embodiments.

[0106] The embodiment of the present application also provides a computer program product, the computer program product includes a non-transitory computer-readable storage medium storing a computer program, the computer program is operable to cause a computer to execute some or all of the steps of any method described in the method embodiment. The computer program product may be a software installation package, and the computer includes an electronic device.

[0107] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

[0108] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0109] In the several embodiments provided in the present application, it should be understood that the disclosed device can be implemented in other ways. For example, the device embodiments described above are only schematic, such as the division of the above-mentioned units, which is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical or other forms.

[0110] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0111] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0112] If the above-mentioned integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable memory. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a memory, including a number of instructions to enable a computer device (which can be a personal computer, server or network device, etc.) to perform all or part of the steps of the above-mentioned methods of each embodiment of the present application. The aforementioned memory includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, disk or CD-ROM and other media that can store program codes.

[0113] A person skilled in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable memory, and the memory can include: a flash drive, a read-only memory (English: Read-Only Memory, abbreviated as: ROM), a random access memory (English: Random Access Memory, abbreviated as: RAM), a magnetic disk or an optical disk, etc.

[0114] The embodiments of the present application are introduced in detail above. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method and core idea of ​​the present application. At the same time, for general technical personnel in this field, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A method for determining image material, characterized in that: The method comprises: determining target feature data of the target image; Determine highlight feature data and standard feature data according to the target feature data, specifically, input the target feature data into a highlight branch module of a material estimation model, and determine the highlight feature data according to an output of the highlight branch module; and input the target feature data into a standard branch module of the material estimation model, and determine the standard feature data according to an output of the standard branch module, the highlight feature data being used to indicate image features that eliminate the influence of saturated pixels, and the standard feature data being used to indicate image features that do not eliminate the influence of saturated pixels; Determine fusion feature data according to the highlight feature data and the standard feature data; Material data of the target image is determined according to the fusion feature data, wherein the material data includes at least one of a diffuse reflection material map, a roughness material map, a specular material map, and a normal material map.

2. The method according to claim 1, characterized in that The highlight branch module includes a first highlight convolution module, a first hole convolution module and a first standard convolution module; the highlight branch module inputting the target feature data into the material estimation model and determining the highlight feature data according to the output of the highlight branch module includes: Inputting the target feature data into the first highlight convolution module, and obtaining first highlight convolution data, second highlight convolution data, and third highlight convolution data according to the output of the first highlight convolution module; Inputting the second highlight convolution data and the third highlight convolution data into the first hole convolution module, and obtaining fourth highlight convolution data according to the output of the first hole convolution module; The first highlight convolution data and the fourth highlight convolution data are input into the first standard convolution module, and the highlight feature data is obtained according to the output of the first standard convolution module.

3. The method according to claim 1, characterized in that The standard branch module includes a second standard convolution module, a second hole convolution module and a third standard convolution module; the inputting the target feature data into the standard branch module of the material estimation model and determining the standard feature data according to the output of the standard branch module include: Inputting the target feature data into the second standard convolution module, and obtaining first standard convolution data, second standard convolution data and third standard convolution data according to the output of the second standard convolution module; Inputting the second standard convolution data and the third standard convolution data into the second dilated convolution module, and obtaining fourth standard convolution data according to the output of the second dilated convolution module; The first standard convolution data and the fourth standard convolution data are input into the third standard convolution module, and the standard feature data is obtained according to the output of the third standard convolution module.

4. The method according to claim 1, characterized in that: The determining of fusion feature data according to the highlight feature data and the standard feature data includes: The highlight feature data and the standard feature data are input into a feature fusion module of a material estimation model, and the fused feature data is obtained according to an output of the feature fusion module.

5. The method according to claim 4, characterized in that The feature fusion module includes a feature connection module, a self-attention feature selection module and a fusion convolution module; the feature fusion module inputs the highlight feature data and the standard feature data into the material estimation model, and obtains the fused feature data according to the output of the feature fusion module, including: Inputting the highlight feature data and the standard feature data into the feature connection module, and obtaining feature connection data according to the output of the feature connection module; Inputting the feature connection data into the self-attention feature selection module, and obtaining feature weight data according to the output of the self-attention feature selection module; The feature weight data is input into the fused convolution module, and the fused feature data is obtained according to the output of the fused convolution module.

6. The method according to claim 1, characterized in that Before determining the target feature data of the target image, the method further includes: Input the training image into a preset model, and obtain estimated material data according to the output of the preset model, wherein the estimated material data includes an estimated diffuse reflection material map, an estimated roughness material map, an estimated specular material map, and an estimated normal material map; The estimated diffuse reflection material map, the estimated roughness material map, the estimated specular material map and the estimated normal material map are processed by a first loss function to obtain first difference data; Processing the estimated diffuse reflection material map and the estimated normal material map by a second loss function to obtain second difference data; The preset model is adjusted according to the first difference data and the second difference data to obtain a material estimation model.

7. An image material determination device, characterized in that: The device comprises: A feature determination unit, used to determine target feature data of a target image; A branch extraction unit, configured to determine highlight feature data and standard feature data according to the target feature data, specifically, inputting the target feature data into a highlight branch module of a material estimation model, and determining the highlight feature data according to an output of the highlight branch module; and inputting the target feature data into a standard branch module of the material estimation model, and determining the standard feature data according to an output of the standard branch module, wherein the highlight feature data is used to indicate image features that eliminate the influence of saturated pixels, and the standard feature data is used to indicate image features that do not eliminate the influence of saturated pixels; A fusion extraction unit, used to determine fusion feature data according to the highlight feature data and the standard feature data; A material estimation unit is used to determine material data of the target image according to the fused feature data, wherein the material data includes at least one of a diffuse reflection material map, a roughness material map, a specular material map and a normal material map.

8. An electronic device, characterized in that: The method comprises a processor, a memory, a communication interface and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the processor, and the programs include instructions for executing the steps in the method according to any one of claims 1 to 6.

9. A computer storage medium, characterized in that The computer storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a processor, the processor is caused to perform the method according to any one of claims 1 to 6.

Images