Method and device for analyzing surface material of object

Abstract

The embodiment of the present application discloses a method and device for analyzing the surface material of an object. The method includes: acquiring a visual image of an object in a scene, where the visual image may be a two-dimensional image acquired by a common acquisition device; performing a visual three-dimensional reconstruction based on the visual image, and determining the coding information of multiple pixels in the visual image. The material feature extraction network can be used to extract and analyze the coding information of multiple pixels in the visual image acquired by the common acquisition device, and the features of each pixel in the coding information can be converted into features that are interconnected with the material, thereby determining the material category of each pixel. According to the material category of each pixel, the material categories of different pixels belonging to the same object are merged to obtain the surface material category of the object.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a method and device for analyzing the surface material of an object. Background Art

[0002] Currently, various objects are made of a variety of materials, and people's ability to identify materials is limited. The analysis technology of the surface material of objects can help people identify the surface material of objects. This technology can be applied to computer vision, computer graphics, virtual reality, augmented reality and other fields.

[0003] In the prior art, 4D (dimension) light field images are collected by light field cameras. The 4D light field images have rich image information, so that the bidirectional reflectance distribution function (BRDF) of the object surface can be analyzed from the 4D light field images through a deep neural network method, and then the material type can be classified.

[0004] However, the object surface material type recognition method in the prior art mainly relies on 4D light field images, and compared with 4D light field images, the visual images collected by ordinary cameras have less image information. Visual images can also be understood as two-dimensional images. The two-dimensional image is a two-dimensional slice of the 4D light field image. It only records the intensity and direction information of the light emitted by the physical point projected to the current pixel position. The 4D light field image is the radiance value of the light along a certain direction at any point in the free space. The collection of all directional light rays in the space constitutes a data set of the light field. Therefore, the object surface material type recognition method in the prior art cannot accurately identify the visual image, thereby reducing the recognition accuracy of the object surface material. Summary of the invention

[0005] The embodiments of the present application hope to provide a method and device for analyzing the surface material of an object, which improves the recognition accuracy of the surface material of the object through the process of visual three-dimensional reconstruction, material feature extraction and analysis, and material category merging for visual images.

[0006] The technical solution of the embodiment of the present application is implemented as follows:

[0007] In a first aspect, an embodiment of the present application provides a method for analyzing the surface material of an object, the method comprising: acquiring a visual image of an object in a scene; performing visual three-dimensional reconstruction based on the visual image, and determining coding information of multiple pixels in the visual image, wherein the coding information represents the relationship between features of the pixels in the visual image; performing material feature extraction and analysis on the coding information of the multiple pixels through a material feature extraction network to obtain a material category of each pixel; and merging material categories of different pixels belonging to the same object according to the material category of each pixel to obtain a surface material category of the object.

[0008] In a second aspect, an embodiment of the present application provides an analysis device for the surface material of an object, the device comprising: an acquisition module for acquiring a visual image of an object in a scene; a visual three-dimensional reconstruction module for performing visual three-dimensional reconstruction based on the visual image, and determining coding information of multiple pixels in the visual image, wherein the coding information characterizes the relationship between features of the pixels in the visual image; a material feature extraction module for extracting and parsing material features of the coding information of the multiple pixels through a material feature extraction network to obtain a material category of each pixel; and a classification module for merging material categories of different pixels belonging to the same object according to the material category of each pixel to obtain a surface material category of the object.

[0009] In a third aspect, an embodiment of the present application provides an analysis device for the surface material of an object, the device comprising a memory for storing executable instructions, and a processor for implementing the above-mentioned method for analyzing the surface material of an object when executing the executable instructions stored in the memory.

[0010] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having executable instructions stored thereon, which are used to implement the above-mentioned method for analyzing the surface material of an object when executed by a processor.

[0011] The embodiment of the present application provides a method and device for analyzing the surface material of an object. According to the scheme provided in the embodiment of the present application, a visual image of an object in a scene is obtained, and the visual image can be a two-dimensional image collected by an ordinary acquisition device; a visual three-dimensional reconstruction is performed based on the visual image, and the coding information of multiple pixels in the visual image is determined, and the coding information represents the relationship between the features of the pixels in the visual image. Since the features of the pixels of different objects are different, and the features of the pixels of the same object are interconnected, the material features of the coding information of multiple pixels are extracted and analyzed through a material feature extraction network, and the features of each pixel in the coding information can be converted into features that are interconnected with the material, thereby determining the material category of each pixel. According to the material category of each pixel, the material categories of different pixels belonging to the same object are merged to obtain the surface material category of the object. The embodiment of the present application can extract and analyze the material features through the material feature extraction network for the visual image collected by the ordinary acquisition device, avoiding the complicated data collection process, thereby improving the recognition accuracy of the surface material of the object. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 A flowchart of optional steps of a method for analyzing the surface material of an object provided in an embodiment of the present application;

[0013] Figure 2 A schematic structural diagram of a material feature extraction network provided in an embodiment of the present application;

[0014] Figure 3 A flowchart of optional steps of another method for analyzing the surface material of an object provided in an embodiment of the present application;

[0015] Figure 4 An optional step flow chart of visual image encoding provided in an embodiment of the present application;

[0016] Figure 5 An optional step flow chart of material analysis based on NeRF network provided in an embodiment of the present application;

[0017] Figure 6 A schematic diagram of the structure of a device for analyzing the surface material of an object provided in an embodiment of the present application;

[0018] Figure 7 A schematic diagram of the structure of an analysis device for the surface material of an object provided in an embodiment of the present application. DETAILED DESCRIPTION

[0019] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. It should be understood that some embodiments described here are only used to explain the technical solutions of the present application and are not used to limit the technical scope of the present application.

[0020] This application actually provides a method for analyzing the surface material of an object, such as Figure 1 As shown, Figure 1 An optional step flow chart of a method for analyzing the surface material of an object provided in an embodiment of the present application, the method for analyzing the surface material of an object comprises the following steps:

[0021] S101. Acquire a visual image of an object in a scene.

[0022] The visual image in the embodiment of the present application may be a two-dimensional image acquired by a common acquisition device. Compared with a 4D light field image, the acquisition method of the visual image is easier, which reduces the requirements for image acquisition.

[0023] In the embodiment of the present application, a common optical camera can be used to collect visual images, that is, the method for analyzing the surface material of an object provided in the embodiment of the present application can be applied to a terminal. For example, the terminal can be a mobile phone, a mobile phone, a notebook, a tablet computer, a POS (Point of Sales) machine, and a car computer, etc., which is not limited in the embodiment of the present application.

[0024] In the embodiments of the present application, the scene can be a scene of any content form, which can be understood as the background of the captured object. The number of objects in the scene can be one or more. The object can be a single material object or an object composed of multiple materials, and the material category can include but is not limited to metal, ceramic, glass, plastic, fiber, rubber, cotton and cloth. The visual image can be a single image or multiple images, and the multiple images can be images of the object at multiple different viewing angles. The images of the object at multiple different viewing angles all correspond to the same scene, so that the light field of the scene can be constructed later using the material feature extraction network.

[0025] S102, performing visual three-dimensional reconstruction based on the visual image, and determining coding information of a plurality of pixels in the visual image; the coding information represents the relationship between the features of the pixels in the visual image.

[0026] In an embodiment of the present application, the device for analyzing the surface material of an object can perform three-dimensional reconstruction processing on the collected visual image, thereby obtaining coding information of multiple pixel points in the visual image.

[0027] It should be noted that the multiple pixel points may be all the pixel points in the visual image, which improves the integrity of the encoded information. The multiple pixel points may also be some of the pixel points in the visual image. Exemplarily, taking the visual image including N viewing angle images as an example, a visual three-dimensional reconstruction is performed based on the visual image to determine the encoding information of the pixel points that exist in the N viewing angle images. The encoding information of the pixel points can characterize the relationship between the features of the pixel points in the N viewing angle images, thereby improving the accuracy of the encoding information.

[0028] In the embodiments of the present application, visual 3D reconstruction is a 3D reconstruction technology based on vision. Through processes such as deep data acquisition, preprocessing, point cloud registration and fusion, and object surface generation, the real scene is portrayed as a mathematical model that conforms to computer logic expression. Visual 3D reconstruction can determine the coding information of multiple pixels in the visual image through steps such as positioning, dense reconstruction, surface reconstruction, and texture mapping. The specific implementation methods of visual 3D reconstruction include, but are not limited to: traditional methods (COLMAP and COLMAP+OpenMVS), deep learning methods (COLMAP+MVSNet and COLMAP+R-MVSNet).

[0029] Exemplarily, the visual 3D reconstruction is explained by taking the COLMAP method as an example. The COLMAP method performs visual 3D reconstruction through the steps of data acquisition, sparse reconstruction, depth map estimation and dense reconstruction. The COLMAP method can be understood as a 3D reconstruction (Pipeline) combining motion structure recovery SfM (Structure-from-Motion) and MVS (Multi-View Stereo). The SfM method is based on the principle of multi-visual geometry and is used to realize 3D reconstruction from motion, that is, to infer 3D information from 2D images without time series. The SfM method may include but is not limited to incremental (incremental / sequential SfM), global (global SfM) and hybrid (hybrid SfM) and hierarchical (hierarchicaSfM). Among them, the incremental SfM is to triangulate and estimate the camera pose (pnp) while performing local bundle adjustment (Bundle Adjustment, BA), that is, first match the two perspective images, then reconstruct the entire scene, and then add the new perspective image to adjust the accuracy of the reconstructed scene. The global SfM first calculates the matching relationship for all images, triangulates them to generate 3D points, estimates the pose through camera pose estimation (pnp), and then uses global bundle adjustment (Bundle Adjustment, BA) for an overall optimization, that is, directly matches all viewpoint images and then reconstructs the entire scene.

[0030] In the embodiment of the present application, the encoded information may represent information obtained by combining feature vectors such as image features, spatial position information, and image color information, as well as the relationship between them.

[0031] In an embodiment of the present application, a device for analyzing the surface material of an object performs visual three-dimensional reconstruction based on a visual image to obtain spatial position information of pixel points, and combines feature vectors representing the spatial position relationship of multiple pixel points, image features, and image color information, as well as the relationship between feature vectors to determine the coding information of the pixel points, thereby improving the richness and accuracy of the coding information.

[0032] S103, extracting and parsing material features of the coded information of multiple pixels through a material feature extraction network to obtain a material category of each pixel.

[0033] In an embodiment of the present application, the material feature extraction network can be any form of fully connected network. There is no restriction on the structure of the fully connected network adopted, as long as it can extract and analyze the material features of the encoded information, including but not limited to a fully connected deep network, a Neural Radiance Fields (NeRf) network, and a PixelNeRF network.

[0034] For example, the material feature extraction network is a NeRF network. Figure 2 As shown, Figure 2 A schematic structural diagram of a material feature extraction network provided in an embodiment of the present application. Figure 2 The layer in it represents the layer, ReLU represents the linear correction function, image feature represents the image feature, channel represents the channel, concat represents the merging of the features obtained from different fully connected layers, and material represents the material. The NeRf network includes 8 fully connected layers, which are activated by ReLU and each layer has 256 channels. The NeRf network also includes 1 additional fully connected layer, which is activated by ReLU and has 128 channels. The input of the NeRF network is the spatial coordinate position (x, y, z) of the pixel point and the viewing direction d, where the viewing direction d can be understood as the position direction d of the light emitted at a certain viewing angle.

[0035] In an embodiment of the present application, the encoded information may include feature vectors such as image features, spatial position information of multiple pixels, and image color information. The encoded information is input into a material feature extraction network to perform material feature extraction and analysis to obtain the material category of each pixel.

[0036] S104 . According to the material category of each pixel point, the material categories of different pixel points belonging to the same object are merged to obtain the surface material category of the object.

[0037] In an embodiment of the present application, the material feature extraction network outputs multiple material categories of multiple pixels. The visual image may include one or more objects. If there is one object, it is necessary to merge the material categories of multiple pixels of the object to obtain the surface material category of the object. If there are multiple objects, it is necessary to first distinguish the pixels of different objects, and merge the material categories of multiple pixels belonging to the same object to obtain the surface material category of the object.

[0038] In an embodiment of the present application, when the material categories of multiple pixels are merged to obtain the surface material category of the object, it can be achieved in the following way. The material categories of multiple pixels are merged, and the probability of each material category is calculated. In one way, the material category corresponding to the maximum probability is used as the surface material category of the object. In this case, only one material category is output. For example, the probability that the material category of an object is metal is 70%, the probability that it is plastic is 20%, and the probability that it is cloth is 10%, then the surface material category of the object is metal. In another way, the material category corresponding to the probability exceeding the preset value is used as the surface material category of the object. In this case, multiple material categories can be output. For example, the probability that the material category of an object is metal is 60%, and the probability that it is ceramic is 40%. The preset value is set to 20%, then the surface material category of the object is a combination of metal and ceramic, that is, the object includes multiple material categories. The embodiment of the present application does not limit the output form of the surface material category of the object.

[0039] In an embodiment of the present application, an object surface material analysis device obtains a visual image of an object in a scene, wherein the visual image may be a two-dimensional image acquired by an ordinary acquisition device; a visual three-dimensional reconstruction is performed based on the visual image to determine the coding information of multiple pixels in the visual image, and the coding information represents the relationship between the features of the pixels in the visual image. Since the features of the pixels of different objects are different, the features of the pixels of the same object are interconnected. Therefore, through a material feature extraction network, the material features of the coding information of multiple pixels are extracted and analyzed, and the features of each pixel in the coding information can be converted into features that are interconnected with the material, thereby determining the material category of each pixel. The object surface material analysis device merges the material categories of different pixels belonging to the same object according to the material category of each pixel, and obtains the surface material category of the object. The embodiment of the present application can extract and analyze the material features through a material feature extraction network for the visual image acquired by the ordinary acquisition device, thereby avoiding the complicated data acquisition process, thereby improving the recognition accuracy of the surface material of the object.

[0040] In some embodiments, the above S102 can be implemented in the following way: an analysis device for the surface material of an object performs feature extraction on a visual image to obtain image features of the visual image; obtains image color information of the visual image; performs visual three-dimensional reconstruction based on the image features to determine spatial position information of multiple pixels in the reconstructed image; and determines encoding information of each pixel among the multiple pixels based on the image features, the spatial position information of the multiple pixels, and the image color information.

[0041] In the embodiment of the present application, the image features may include the color features, texture features, shape features and spatial relationship features of the image. The image features represent the features of multiple dimensions in the image and the relationships between different dimensions. The object surface material analysis device may perform visual three-dimensional reconstruction based on the image features, thereby determining the spatial position information of multiple pixels in the reconstructed image. The image features, spatial position information and image color information of the pixel points may be feature vectors, and these feature vectors are combined into one feature vector, and the combined feature vector is determined as the encoding information of the pixel points.

[0042] In the embodiment of the present application, image features are used to represent the relationship between the color, texture, shape and space of an image. Exemplarily, image features may include color features, texture features, shape features and spatial relationship features of an image. Among them, color features are a global feature that characterizes the surface properties of the scene corresponding to an image or an image region; texture features are also a global feature that characterizes the surface properties of the scene corresponding to an image or an image region; shape features have two types of representation methods, one is contour features, and the other is regional features. The contour features of an image are mainly for the outer boundaries of an object, while the regional features of an image are related to the entire shape region; spatial relationship features refer to the spatial position or relative direction relationship between multiple targets segmented from an image. These relationships can also be divided into connection or adjacency relationships, overlap or overlapping relationships, and inclusion or containment relationships.

[0043] Exemplarily, the image feature extraction network can be any form of convolutional neural network (CNN) network. There is no restriction on the structure of the convolutional neural network used, as long as it can extract image features from visual images, including but not limited to LeNet, AlexNet, CaffeNet, ZFNet, VGG, NIN, GoogLeNet, Inception, ResNet, and Inception V4.

[0044] In the embodiment of the present application, the visual image carries image color information, and the analysis device of the surface material of the object can obtain the image color information of the visual image.

[0045] In the embodiment of the present application, the image color information can represent the optical three primary colors information of the pixel points of the visual image, and the optical three primary colors include red (Red), green (Green) and blue (Blue). Exemplarily, the image color information can be expressed as the RGB value of the pixel point or the pixel value.

[0046] In the embodiment of the present application, the spatial position information may include the spatial coordinates of the pixel point and the coordinate depth information corresponding to the spatial coordinates, and the coordinate depth information represents the distance between the acquisition device and the pixel point. For example, the spatial coordinates of the pixel point are three-dimensional coordinates (x, y, z). The spatial coordinates of each pixel point are (x, y, z), and the coordinate depth information is

[0047] Exemplarily, the spatial coordinates of the pixel point are three-dimensional coordinates (x, y, z). For each pixel point, the spatial position information includes three-dimensional coordinates (3 dimensions), coordinate depth information corresponding to the spatial coordinates (1 dimension), and the image color information is RGB value (3 dimensions). The image features can be different dimensions such as 64 dimensions, 128 dimensions or 256 dimensions. The specific dimensions can be set by technical personnel in this field according to actual needs. The embodiments of the present application are not limited to this. Correspondingly, the dimension of the encoding information of each pixel point can be 71, 135 or 263, etc.

[0048] In the embodiment of the present application, the object surface material analysis device obtains the image color information of the visual image; the object surface material analysis device extracts the features of the visual image to obtain the image features of the visual image; based on the image features, the visual three-dimensional reconstruction is performed to determine the spatial position information of multiple pixels in the reconstructed image; based on the image features, the spatial position information of the multiple pixels and the image color information, the coding information of each pixel in the multiple pixels is determined. The coding information describes the feature vector and the relationship between the feature vectors from multiple dimensions, thereby improving the accuracy of the coding information.

[0049] In some embodiments, when extracting features from a visual image to obtain image features of the visual image, it can be achieved in the following manner: the object surface material analysis device performs at least one of filtering denoising and deblurring on the visual image to obtain a preprocessed image; and image feature extraction is performed on the preprocessed image through an image feature extraction network to obtain image features.

[0050] In the embodiment of the present application, the device for analyzing the surface material of an object can filter and denoise the visual image by using adaptive Wiener filter, median filter, morphological noise filter and wavelet denoising. The visual image can be deblurred by using image enhancement, image restoration and super-resolution reconstruction.

[0051] In the embodiment of the present application, the device for analyzing the surface material of an object performs at least one preprocessing operation such as filtering, denoising and deblurring on the visual image, and then performs feature extraction, thereby improving the accuracy of the image features.

[0052] In some embodiments, the visual image includes multiple perspective images, and may also include a single image. Two examples are listed here to illustrate multiple perspective images and a single image respectively.

[0053] Example 1: When a visual image includes multiple perspective images, visual three-dimensional reconstruction is performed based on image features to determine the spatial position information of multiple pixels in the reconstructed image, which can be achieved in the following manner. The object surface material analysis device performs pixel matching on multiple perspective images according to the image features of each perspective image in the multiple perspective images to obtain multiple pixel matching pairs, where a pixel matching pair represents the same pixel existing between different perspective images; based on the multiple pixel matching pairs, a three-dimensional image reconstruction is performed on the scene corresponding to the reference image in the multiple perspective images to obtain the spatial position information of multiple pixels corresponding to the multiple pixel matching pairs; wherein the reference image is any one of the multiple perspective images.

[0054] In the embodiment of the present application, since multiple perspective images are images of objects in the same scene, the pixels in the different perspective images are the same. The same pixel points in the different perspective images are grouped into a pixel matching pair, and multiple pixel matching pairs can be obtained.

[0055] In some embodiments of the present application, when pixel matching is performed on multiple perspective images to obtain multiple pixel matching pairs, it can be achieved in the following way: arbitrarily select one image from the multiple perspective images as a reference image; align the remaining images from the multiple perspective images except the reference image with the reference image so that the positions of the acquisition devices in the multiple perspective images are on the same horizontal line; then pixel matching is performed based on the aligned multiple perspective images to obtain multiple pixel matching pairs.

[0056] It is understandable that, since each viewing angle image includes multiple pixels, when forming a pixel matching pair, some pixels may not be matched. Taking N (N is an integer greater than 1) viewing angle images as an example, two methods are listed for explanation. In the first method, if pixel a exists in all N viewing angle images, pixel a can form a pixel matching pair. In this case, if pixel b only exists in M ​​(M is an integer less than N) viewing angle images, the pixel cannot form a pixel matching pair, and pixel b is discarded. Since each pixel matching pair includes image features of N viewing angle images, which come from N viewing angle images, the accuracy of the pixel matching pair is improved. In the second method, if pixel c exists in a preset ratio of viewing angle images, pixel c can form a pixel matching pair. The preset ratio can be set by a person skilled in the art according to actual needs, for example, 70%, 80%, 90%, and this embodiment of the present application is not limited. In other words, pixel c can exist in N viewing angle images, or it can exist in a preset ratio × N viewing angle images. The first method of forming pixel matching pairs improves the accuracy of pixel matching pairs. The second method of forming pixel matching pairs improves the integrity of pixel matching pairs. The embodiment of the present application can obtain multiple pixel matching pairs according to the first or second method, and a pixel matching pair represents the same pixel existing between different perspective images.

[0057] It should be noted that, whether it is the first or second method to obtain multiple pixel matching pairs, there will be situations where pixels are not matched. The spatial position information of multiple pixel points in the embodiment of the present application belongs to the pixel points in the multiple pixel matching pairs. In other words, the pixel points that cannot form a pixel matching pair are discarded, which improves the calculation efficiency compared to the method of obtaining the spatial coordinate position information of all pixel points. In addition, since the spatial position information of multiple pixel points belongs to the pixel points in the multiple pixel matching pairs, when the encoding information of each pixel point is obtained later, its image features come from multiple perspective images, thereby improving the accuracy of the encoding information.

[0058] In the embodiment of the present application, each viewing angle image includes multiple pixels. Since the multiple viewing angle images are images of objects in the same scene, the pixels in different viewing angle images are the same, and therefore, the spatial position information of the multiple pixels is shared. In the embodiment of the present application, one image can be arbitrarily selected from the multiple viewing angle images as a reference image, and a three-dimensional image reconstruction is performed on the scene corresponding to the reference image in the multiple viewing angle images according to the multiple pixel matching pairs, and the spatial position information of the multiple pixels corresponding to the multiple pixel matching pairs is obtained, and the spatial position information of the multiple pixels in the multiple viewing angle images can be obtained.

[0059] In some embodiments, the analysis device of the surface material of the object can perform sparse reconstruction on the scene corresponding to the reference image in the multiple perspective images according to the multiple pixel matching pairs, and obtain the spatial position information of the multiple pixel points corresponding to the multiple pixel matching pairs. Among them, sparse reconstruction can be understood as Structure from Motion (SfM) reconstruction. The SfM method may include but is not limited to incremental / sequential SfM, global SfM, hybrid SfM and hierarchical SfM. The specific description of the SfM method is consistent with that described in S102 and will not be repeated here.

[0060] It can be understood that the embodiments of the present application can also densely reconstruct the scene corresponding to the reference image in multiple perspective images based on multiple pixel matching pairs, and obtain the spatial position information of multiple pixels corresponding to the multiple pixel matching pairs, thereby improving the integrity and comprehensiveness of the multiple pixels.

[0061] In some embodiments, when the visual image includes multiple perspective images, based on the pixel matching pairs in Example 1, when determining the encoding information of each pixel in the multiple pixels based on the image features, the spatial position information of the multiple pixels and the image color information, it can be achieved in the following manner. The object surface material analysis device obtains single encoding information of each pixel of the multiple pixels in each perspective image based on the image features of each perspective image, the image color information of each perspective image and the spatial position information of the multiple pixels in each perspective image; according to the pixel matching pairs, the single encoding information belonging to the same pixel in each perspective image is combined to obtain the encoding information of each pixel in the multiple pixels.

[0062] It should be noted that when a visual image includes multiple viewing angle images, when the feature extraction is performed on the visual image, the feature extraction is performed on each viewing angle image in the multiple viewing angle images to obtain the sub-image features of each viewing angle image, and the image features include the sub-image features of the multiple viewing angle images. For the convenience of description, the sub-image features and image features are no longer distinguished, and the sub-image features of each viewing angle image are referred to as image features.

[0063] In the embodiment of the present application, for each viewing angle image, the image features, image color information and spatial position information of multiple pixels are combined to obtain single coding information of each pixel in the viewing angle image. For the pixel in the pixel matching pair, the pixel matching pair represents the same pixel existing between different viewing angle images, and the single coding information of the same pixel from different viewing angle images is combined to obtain the coding information of the pixel. In this way, the coding information of each pixel can be obtained.

[0064] In an embodiment of the present application, the coding information of a pixel point includes multiple single coding information, and the single coding information contains image features of a perspective image, image color information, and spatial position information of multiple pixels, thereby improving the richness of the coding information.

[0065] Example 2: When the visual image includes a single image, the visual 3D reconstruction is performed based on the image features to determine the spatial position information of multiple pixels in the reconstructed image, which can be achieved in the following manner: The object surface material analysis device performs monocular reconstruction of the scene corresponding to the single image based on the image features to determine the spatial position information of multiple pixels in the single image.

[0066] In an embodiment of the present application, when the visual image is a single image, and the single image includes multiple pixels, the scene corresponding to the single image is monocularly reconstructed based on the image features to obtain the spatial position information of the multiple pixels in the single image. Exemplarily, monocular reconstruction can be understood as performing three-dimensional reconstruction of a single image based on a monocular depth estimation network to obtain the three-dimensional coordinate information of the pixel points. Three-dimensional reconstruction is to restore the three-dimensional coordinates of the pixel points of the target object in the camera coordinate system based on the position of the pixel points in the image coordinate system and the distance of the target object from the origin of the camera coordinate system. The distance of the above-mentioned target object from the origin of the camera coordinate system, that is, the focal length, can be obtained by a depth camera or binocular vision.

[0067] In an embodiment of the present application, by performing monocular reconstruction of the scene corresponding to a single image based on image features, the spatial position information of multiple pixels in a single image is determined without the need for pixel matching, and a three-dimensional image reconstruction is performed on the scene corresponding to a reference image in multiple perspective images. This reduces the amount of data processing, increases the processing speed, and improves the efficiency of image visual three-dimensional reconstruction.

[0068] In the embodiment of the present application, the device for analyzing the surface material of an object also determines the coding information of each pixel of multiple pixels in a single image based on the image features of a single image, the image color information of a single image, and the spatial position information of multiple pixels in a single image. Compared with the situation in the related art where visual images cannot be used to classify material types, the embodiment of the present application determines the coding information of each pixel by performing monocular reconstruction on a single image, and then extracts and analyzes the material features through a material feature extraction network to obtain the material category of each pixel, and obtains the surface material category of the object after statistical merging, thereby improving the recognition accuracy of the surface material category of the object.

[0069] In some embodiments, the above-mentioned S103 can be implemented in the following manner: the analysis device of the surface material of the object extracts the illumination features of the encoded information of multiple pixels through the initial layer of the material feature extraction network to obtain the illumination features of each pixel; and analyzes the illumination features of each pixel through the decision layer of the material feature extraction network to obtain the material category of each pixel.

[0070] In the embodiment of the present application, the coded information of multiple pixels is input into the material feature extraction network, and the material feature extraction network extracts material features from the coded information of multiple pixels and outputs the material category of each pixel. To facilitate the description of the material feature extraction process, the embodiment of the present application divides the material feature extraction network into an initial layer and a decision layer. The initial layer is used to extract illumination features, and the decision layer is used to parse the illumination features to obtain the material category of each pixel.

[0071] Exemplarily, the material feature extraction network is a NeRF network. The coding information of multiple pixels is input into the NeRF network to extract the illumination features. This example is performed in the first half of the NeRF network, and the first half is represented by the NeRF initial layer. Since the initial layer extracts and analyzes the illumination features of the coded information of multiple pixels respectively, the output is a series of feature vector groups, that is, the illumination features of each pixel. This part can be regarded as the construction of the light field. The coded information is converted into illumination features through the NeRF initial layer. The illumination features obtained by different objects at different angles are different, and the illumination features obtained by the same object at different angles are interrelated.

[0072] In some embodiments, the material feature extraction network includes a preset number of neurons. Through the initial layer of the material feature extraction network, the illumination feature extraction is performed on the coded information of multiple pixels to obtain the illumination feature of each pixel, which can be achieved in the following way. The coded information of a preset number of pixels is selected from the coded information of multiple pixels; according to the initial layer of the material feature extraction network, the illumination feature extraction is performed on the coded information of a preset number of pixels to obtain the illumination feature of each pixel.

[0073] In an embodiment of the present application, the NeRF network includes a series of fully connected networks, and the preset number of neurons can be set by a person skilled in the art according to actual needs. For example, the preset number can be determined according to the order of magnitude of the encoded information, which is not limited by the embodiment of the present application.

[0074] For example, if the number of neurons is set to 1000, 1000 coded information is randomly selected from the coded information of all pixels and input into the NeRF network. It can be understood that the more neurons there are, the more coded information can be used for material feature extraction, and the more comprehensive the illumination features obtained, thereby improving the accuracy of material classification.

[0075] In the embodiment of the present application, there are a lot of coded information of pixels in the visual image, and the analysis device of the surface material of the object extracts material features from a part of the coded information, thereby reducing the amount of data processing and improving the efficiency of material feature extraction.

[0076] In the embodiment of the present application, the illumination features of each pixel point mentioned above can be understood as a series of feature vector groups. In this example, the illumination features of multiple pixels are aggregated and analyzed using the second half of the NeRF network, which is represented by the NeRF decision layer. The NeRF decision layer is used to merge and analyze the illumination features of the same pixel point at different angles into the material category of the pixel point, that is, the material category output by the NeRF network is at the pixel level.

[0077] In the embodiment of the present application, since the light field information of different objects at different viewing angles is different, and the light field information of the same object at different viewing angles is interconnected, the features of each pixel in the coded information can be converted into features interconnected with the light field information through the material feature extraction network, so that the features after the light field is constructed can be analyzed and the material category of each pixel in the visual image can be determined. Compared with the method of classifying material types using 4D light field images, the embodiment of the present application analyzes the surface material of the object through the light field, avoiding the complicated data collection process, thereby improving the data processing efficiency.

[0078] In some embodiments, before the encoding information of multiple pixels is input into the material feature extraction network, the present application also includes a process of training the material feature extraction network, using training visual image samples to train the material feature extraction network. The above-mentioned material feature extraction network is obtained in the following manner: obtaining training visual image samples, wherein the training visual image samples include multiple perspective image samples of the object sample, and surface material category samples of the object sample; inputting the training visual image samples into the initial material feature extraction network to obtain the predicted surface material category of the object sample; obtaining the loss value according to the predicted surface material category of the object sample and the preset loss function; training the initial material feature extraction network according to the loss value to obtain the material feature extraction network.

[0079] In an embodiment of the present application, a preset loss function is used to train and optimize the predicted surface material category of the object sample, and the loss value is calculated. Then, the initial material feature extraction network is trained according to the loss value to obtain the material feature extraction network until the training termination condition is met, for example, the number of training times reaches a preset number of times, or the loss value reaches a preset threshold, etc.

[0080] In the embodiments of the present application, the preset loss function can be set by those skilled in the art according to actual needs, including but not limited to the intersection-and-union ratio (DiceLoss), smooth L1 loss function and cross entropy loss function.

[0081] In some embodiments, the above-mentioned acquisition of training visual image samples can be implemented in the following manner: the acquisition device acquires images of the object samples to obtain multiple viewpoint image samples of the object samples; the surface material category samples of the object samples are acquired after the surface material categories of the object samples are labeled; or, multiple viewpoint image sample sets are acquired; the multiple viewpoint image sample sets are analyzed according to a preset surface material analysis method to obtain surface material category samples of the object samples.

[0082] In the embodiment of the present application, the training visual image samples in the training process include multiple view image samples of the object sample and surface material category samples of the object sample. The collection of multiple view image samples can be obtained by general collection equipment, such as RGB camera, or directly use the existing multiple view image sample set.

[0083] In an embodiment of the present application, surface material category samples of object samples can be collected separately according to the sources of multiple perspective image samples. For example, the surface material category of the object in the multiple perspective image samples collected by an RGB camera can be manually calibrated; the surface material category of the object obtained using the existing multiple perspective image sample set can be obtained by analyzing using a preset high-precision surface material analysis method.

[0084] In the embodiment of the present application, the preset surface material analysis method can be selected by technical personnel in this field according to actual needs, including but not limited to the surface material analysis method in the related technology, as long as it can analyze multiple perspective image sample sets to determine the surface material category of the object.

[0085] In an embodiment of the present application, the device for analyzing the surface material of an object can obtain training visual image samples in different ways, thereby improving the richness of the training sample set.

[0086] In some embodiments, the above S104 can be implemented in the following manner. The object surface material analysis device performs instance segmentation on multiple pixel points in the visual image to obtain multiple pixel sets belonging to different objects; wherein the pixel sets correspond to the objects one by one; for each pixel set, the material characteristics of each pixel point in the visual image are combined to obtain the material category of each pixel point in each pixel set; the material category of each pixel point in each pixel set is merged to obtain the surface material category belonging to the same object, until the multiple pixel sets are merged separately, and the surface material categories of different objects in the visual image are obtained.

[0087] In an embodiment of the present application, the number of objects in the visual image can be one or more. When the number of objects is multiple, multiple pixels in the visual image belong to multiple objects. Therefore, it is necessary to perform instance segmentation on the multiple pixels. An instance can be understood as an object, so that the pixels belonging to different instances (i.e., objects) are segmented to obtain multiple pixel sets belonging to different objects.

[0088] In the embodiment of the present application, instance segmentation is for pixels in an image. When the visual image includes a single image, multiple pixels in the visual image are segmented to obtain multiple pixel sets belonging to each object. Instance segmentation refers to automatically using the object detection method to frame different instances from the image, and then using the semantic segmentation method to perform pixel-by-pixel labeling in different instance areas. Instance segmentation is used to obtain pixels belonging to the same object. For example, an image includes two cats, and instance segmentation is used to distinguish which pixels belong to the first cat and which pixels belong to the second cat.

[0089] In the embodiment of the present application, the above-mentioned object detection needs to provide not only the category of the object in the image, but also the location (bounding box) of the object. Semantic segmentation needs to predict the label of which category each pixel of the input image belongs to. Instance segmentation is based on semantic segmentation and also needs to distinguish different individuals of the same category.

[0090] In some embodiments, multiple pixels in a visual image can be segmented by an instance segmentation network to obtain multiple pixel sets belonging to different objects. The instance segmentation network can be any form of classification network. The embodiment of the present application does not limit the structure of the classification network used, as long as it can perform instance segmentation on pixels, including but not limited to Fully Convolution Networks (FCN), AlexNet, VGG-Net, GoogLeNet, U-Net, SegNet, DeepLab, PSPNet.

[0091] In the embodiment of the present application, the device for analyzing the surface material of an object performs instance segmentation on multiple pixel points in a visual image to obtain multiple pixel sets belonging to different objects; one object corresponds to one pixel set; for each pixel set, the material characteristics of each pixel point in the visual image are combined to obtain the material category of each pixel point in each pixel set; the material category of each pixel point in each pixel set is merged to obtain the surface material category belonging to the same object. The method of classifying the material categories of multiple pixel points through pixel sets improves the accuracy of classification.

[0092] In some embodiments, when the visual image includes multiple perspective images, instance segmentation is performed on multiple pixel points in the visual image to obtain multiple pixel sets belonging to different objects, which can be achieved in the following manner: The object surface material analysis device performs instance segmentation on multiple pixel points in a reference image of the multiple perspective images to obtain multiple pixel sets belonging to different objects; wherein the reference image is any one of the multiple perspective images.

[0093] In the embodiment of the present application, when the visual image includes multiple viewing angle images, since the multiple viewing angle images are all images of objects in the same scene, a reference image is determined in the multiple viewing angle images, and the object contained in the reference image is the same as the object contained in the multiple viewing angle images. The embodiment of the present application performs instance segmentation on multiple pixel points in the reference image to obtain multiple pixel sets belonging to each object, thereby improving the efficiency of instance segmentation.

[0094] It is understandable that the embodiment of the present application can also perform instance segmentation on multiple pixel points in each perspective image separately to obtain multiple pixel sets belonging to multiple objects in each perspective image, and then perform de-overlapping and merging on multiple pixel sets belonging to multiple objects in each perspective image to obtain multiple pixel sets belonging to different objects. By performing instance segmentation on multiple pixel points in each perspective image separately, objects that are blocked due to perspective are avoided from being missed, thereby improving the comprehensiveness and accuracy of the instance segmentation results.

[0095] It should be noted that the reference image used in the above three-dimensional image reconstruction based on multiple pixel point matching pairs is consistent with the reference image in this example. The three-dimensional image reconstruction is based on the reference image, and the instance segmentation is subsequently performed based on the reference image. The images used before and after are consistent, thereby ensuring the consistency of data processing.

[0096] In the embodiment of the present application, since each pixel carries features such as spatial position information (such as three-dimensional coordinates) and image color information (such as RGB values), the material features of the pixel also carry features such as spatial position information and image color information. The pixel set includes multiple pixels, each of which carries features such as spatial position information and image color information. These features can be used to determine the corresponding pixel in the material feature set, thereby obtaining the material features of the pixel. By traversing the pixels in the entire pixel set, the material features of each pixel in the pixel set can be obtained.

[0097] In the embodiment of the present application, the material category output by the NeRF network is at the pixel level, and the material categories of these pixels need to be merged. Since a pixel set includes multiple pixels belonging to the same object, the material category of each pixel in each pixel set is merged to obtain the surface material category belonging to the same object.

[0098] In the embodiment of the present application, the device for analyzing the surface material of an object obtains multiple pixel sets belonging to different objects by performing instance segmentation on multiple pixel points in a visual image; for each pixel set, the material characteristics of each pixel point in the visual image are combined to obtain the material category of each pixel point in each pixel set; the material category of each pixel point in each pixel set is merged to obtain the surface material category belonging to the same object, until multiple pixel sets are merged separately, and the surface material categories of different objects in the visual image are obtained. This avoids the complicated data collection process and improves the recognition accuracy of the surface material of an object.

[0099] In some embodiments, merging the material category of each pixel point in each pixel set to obtain the surface material category belonging to the same object can be achieved in the following manner: The object surface material analysis device performs statistics on the material category of each pixel point in each pixel set for the same material category to obtain a statistical value of each material category; based on the statistical value and a preset statistical threshold, the surface material category belonging to the same object is determined.

[0100] In an embodiment of the present application, since the pixel set includes multiple pixel points belonging to the same object, the material categories of the multiple pixel points may be different. Therefore, it is also necessary to count the material categories in each pixel set and determine the surface material category of the object from the multiple material categories.

[0101] In the embodiment of the present application, the preset statistical threshold can be set by those skilled in the art according to actual needs. Exemplarily, the preset statistical threshold can be related to the number of multiple pixel points. For example, the preset statistical threshold can be set to 90%, 80%, or 70% of the number of multiple pixel points. Assuming that the preset statistical threshold is 80% of the number of multiple pixel points, the statistical values ​​of multiple material categories include: the statistical value of metal is 82% of the number of multiple pixel points, and the statistical value of ceramic is 18% of the number of multiple pixel points, and metal is used as the surface material category of the object. This example determines the material category corresponding to the statistical value of multiple material categories that is greater than the preset statistical threshold as the surface material category of the object corresponding to the pixel set, thereby improving the accuracy of the surface material category.

[0102] When determining the surface material category belonging to the same object based on the statistical value and the preset statistical threshold, the above can also be implemented in the following way: Calculate the ratio of the statistical value of each material category to the number of multiple pixel points to obtain the statistical probability of each material category, and determine the material category corresponding to the maximum value of the statistical probabilities of multiple material categories as the surface material category of the object corresponding to the pixel set.

[0103] In some embodiments, the above S102 can be implemented in the following manner: the object surface material analysis device performs instance segmentation on multiple pixel points in the visual image to obtain multiple pixel sets belonging to different objects; wherein the pixel sets correspond to the objects one by one; based on the multiple pixel sets belonging to different objects in the visual image, visual three-dimensional reconstruction is performed to determine the encoding information of multiple pixel points belonging to different objects in the visual image.

[0104] In the embodiment of the present application, instance segmentation is for pixels in an image. The method for analyzing the surface material of an object provided in the embodiment of the present application can also first perform instance segmentation on multiple pixels in a visual image to distinguish sets of pixels belonging to different objects. For a set of pixels of the same object, a visual three-dimensional reconstruction is performed on the scene corresponding to the object to obtain the encoding information of multiple pixels of the object. Then, through a material feature extraction network, the material feature extraction and analysis of the encoding information of multiple pixels of the object can be performed to obtain the material category of each pixel of the object.

[0105] In an embodiment of the present application, the analysis device for the surface material of an object may first perform instance segmentation on the pixels of the visual image, and then perform the steps of visual three-dimensional reconstruction and material feature extraction based on the visual image. It is also possible to first perform the steps of visual three-dimensional reconstruction and material feature extraction based on the visual image, and then perform the step of instance segmentation on the pixels of the visual image. It is understandable that if the visual image is an image of a single object in a single scene, the step of instance segmentation is not required. The embodiment of the present application does not limit the order of executing the steps of instance segmentation.

[0106] The following is an explanation of an exemplary application of the embodiments of the present application in a practical application scenario.

[0107] Before analyzing the surface material category of an object in a visual image, the embodiment of the present application also includes a training phase for a NeRF network. Exemplarily, multiple perspective image samples and surface material category samples of a scene are collected; the multiple perspective image samples and surface material category samples are used to construct a training data set. The collection of multiple perspective image samples can be obtained using a general RGB camera, or the existing multiple perspective image sample sets can be directly used. The surface material category samples can be obtained separately according to the sources of the multiple perspective image samples. For example, the surface material category of an object in the multiple perspective image samples collected using an RGB camera can be manually calibrated; the surface material category of an object obtained using the existing multiple perspective image sample sets can be obtained by analyzing using a preset high-precision surface material analysis method. The NeRF network is trained using the constructed training data set and the trained NeRF network is saved.

[0108] The embodiment of the present application is described by taking the NeRf network as a material feature extraction network, the visual image including N (N is an integer greater than 1) perspective images, the camera as an acquisition device, the sparse reconstruction as a three-dimensional image reconstruction, the spatial coordinates and coordinate depth as spatial position information, and the RGB value as an example. In the prediction stage of the surface material category of an object in the embodiment of the present application, multiple images of different perspectives in a single scene are input into the trained network model to obtain the surface material category of the object in the scene. Figure 3 As shown, Figure 3 A flowchart of optional steps of another method for analyzing the surface material of an object provided in an embodiment of the present application.

[0109] S301, obtaining N perspective images.

[0110] S302: Perform visual 3D reconstruction based on N viewing angle images to obtain coding information of multiple pixel points.

[0111] S303: Input the coded information of multiple pixels into the NeRF network, extract and analyze the material features, and obtain the material category of each pixel.

[0112] S304: Determine the surface material category of the object according to the material category of each pixel point.

[0113] The method for analyzing the material category of an object surface consists of two parts: the encoding process based on visual images and the material category analysis process based on the NeRF network. They are described below.

[0114] This example illustrates the encoding process of visual images, namely Figure 3 S301 and S302. Based on Figure 3 , the embodiment of the present application provides an optional step flow chart of visual image encoding, such as Figure 4 The visual image encoding process includes S401-S406.

[0115] S401, obtaining N perspective images.

[0116] S402 : Preprocess the N viewing angle images to obtain preprocessed N viewing angle images.

[0117] Preprocessing includes filtering, denoising, and deblurring.

[0118] S403 , extracting image features from the preprocessed N viewing angle images respectively to obtain image features of the N viewing angle images.

[0119] The convolutional network can be used to extract image features from N perspective images respectively.

[0120] S404: align and match pixels of the multiple perspective images according to the image features of the N perspective images to obtain multiple pixel point matching pairs.

[0121] Select any one of the N perspective images as the reference image, and align the remaining images of the N perspective images except the reference image with the reference image, so that the camera positions in the N perspective images are on the same horizontal line. Then, pixel matching is performed based on the aligned N perspective images to obtain multiple pixel matching pairs. A pixel matching pair contains image features of N perspective images, which are derived from the N perspective images. Finally, when performing instance segmentation, instance segmentation is also performed on the reference image, and the surface material category of the object is output by combining the material categories of multiple pixels.

[0122] S405 . Perform sparse reconstruction on the scene according to the multiple pixel point matching pairs, and obtain spatial coordinates of multiple pixel points corresponding to the multiple pixel point matching pairs.

[0123] The purpose of 3D image reconstruction in this step is to obtain the spatial coordinates of the pixel points in the pixel matching pair, so there is no need to consider the density of the reconstructed pixel points. Sparse reconstruction includes but is not limited to incremental reconstruction, global reconstruction, hybrid reconstruction and hierarchical reconstruction.

[0124] S406. For each viewing angle image, the image features, RGB values, spatial coordinates, and coordinate depths corresponding to the spatial coordinates of the viewing angle image are merged into a feature vector; the feature vectors on different viewing angle images are combined into a feature vector group according to pixel point matching pairs to obtain a feature vector group for each pixel point.

[0125] The feature vector is a single encoding information of a pixel, and the feature vector group is the encoding information.

[0126] This example illustrates the material category analysis process of the NeRF network, namely Figure 3 S303 and S304. Based on Figure 3 and Figure 4 , the embodiment of the present application provides an optional step flow chart of material analysis based on NeRF network, such as Figure 5 As shown in the figure, the material category analysis process of NeRF network includes S407-SS409.

[0127] S407, inputting the coded information groups of the multiple pixel points into the NeRF network respectively, performing feature extraction and analysis on the coded information groups of the multiple pixel points in the NeRF network, and obtaining the illumination features of each pixel point.

[0128] In the embodiment of the present application, the NeRF network is composed of a series of fully connected networks, and the number of neurons input can be freely set, generally determined by the order of magnitude of pixel matching pairs. For example, if the number of neurons is set to 1000, 1000 encoding information is randomly selected from the encoding information of multiple pixels and input into the NeRF network.

[0129] This example is performed in the first half of the NeRF network, called the NeRF initial layer. Since the coded information of multiple pixels is extracted and analyzed separately in the initial layer, the output is still a series of feature vector groups, that is, the illumination features of each pixel. This part can be regarded as the construction of the light field. The coded information is converted into illumination features through the NeRF initial layer. The illumination features obtained by different objects at different angles are different, and the illumination features obtained by the same object at different angles are interconnected.

[0130] S408. Perform feature aggregation and feature analysis on the illumination features of each pixel in the NeRF network to obtain the material category of each pixel.

[0131] After a series of illumination features are obtained in S407, the illumination features are aggregated and analyzed using the second half of NeRF, which is called the NeRF decision layer. The function of this part of the network is to merge the illumination features of the same pixel from different perspectives and analyze them into material categories.

[0132] S409, merging the material categories of the same object according to the material categories of each pixel point, and outputting the surface material category of each object in the reference image of the N perspective images, where the reference image is any one of the N perspective images.

[0133] The material categories output by NeRF are at the pixel level, so the material categories of these pixels need to be merged. This example can perform instance segmentation on the reference image through the instance segmentation network, merge the categories of the objects corresponding to each pixel set, and output the surface material category of each object.

[0134] The object surface material analysis method provided in the embodiment of the present application uses NeRF to construct a light field of a scene corresponding to multiple perspective images, and uses the light field to analyze the object surface material, avoiding the complicated data collection process. Only a set of multiple perspective images need to be input to analyze the object surface material and obtain the object surface material category. Multiple perspective images can be obtained using ordinary acquisition equipment, so the object surface material analysis method provided in the embodiment of the present application can be applied to mobile terminals, such as mobile phones.

[0135] In some embodiments, the embodiments of the present application can also analyze the surface material of an object in a single image to obtain the surface material category of the object. Exemplarily, the surface material analysis device of an object extracts features from an input single image to obtain image features; based on the image features, the scene corresponding to the single image is monocularly reconstructed to determine the spatial position information of multiple pixels in the single image; the image features, the spatial position information of the multiple pixels and the image color information are merged to obtain the encoding information of each pixel in the multiple pixels. The encoding information of the multiple pixels is input into the NeRF network. The NeRF network constructs a light field through different pixels in a single image, that is, the image features, RGB values, spatial coordinates and coordinate depth of a single image are used to construct a light field in combination with the positional relationship of different pixels. An instance segmentation network is used to perform instance segmentation on multiple pixels of a single image, and the material category of the object surface is obtained in combination with the material category of each pixel output by the NeRF network.

[0136] Based on the method for analyzing the surface material of an object in the embodiment of the present application, the embodiment of the present application also provides a device for analyzing the surface material of an object, such as Figure 6 As shown, Figure 6The schematic diagram of the structure of an object surface material analysis device provided in an embodiment of the present application, the object surface material analysis device 60 comprises:

[0137] An acquisition module 601 is used to acquire a visual image of an object in a scene;

[0138] A visual 3D reconstruction module 602 is used to perform visual 3D reconstruction based on the visual image, and determine coding information of a plurality of pixels in the visual image, wherein the coding information represents a relationship between features of the pixels in the visual image;

[0139] The material feature extraction module 603 is used to extract and analyze the material features of the coded information of the plurality of pixels through a material feature extraction network to obtain the material category of each pixel;

[0140] The classification module 604 is used to merge the material categories of different pixel points belonging to the same object according to the material categories of the respective pixel points, so as to obtain the surface material category of the object.

[0141] In some embodiments, the visual 3D reconstruction module 602 is further used to extract features from the visual image to obtain image features of the visual image;

[0142] The acquisition module 601 is also used to acquire the image color information of the visual image;

[0143] The visual three-dimensional reconstruction module 602 is also used to perform visual three-dimensional reconstruction based on the image features, determine the spatial position information of multiple pixels in the reconstructed image; based on the image features, the spatial position information of the multiple pixels and the image color information, determine the coding information of each pixel in the multiple pixels.

[0144] In some embodiments, the visual image includes: multiple perspective images;

[0145] The visual three-dimensional reconstruction module 602 is also used to perform pixel matching on the multiple perspective images according to the image features of each perspective image in the multiple perspective images, so as to obtain multiple pixel matching pairs, where a pixel matching pair represents the same pixel existing between different perspective images; based on the multiple pixel matching pairs, perform three-dimensional image reconstruction on the scene corresponding to the reference image in the multiple perspective images, so as to obtain spatial position information of multiple pixels corresponding to the multiple pixel matching pairs; wherein the reference image is any one of the multiple perspective images.

[0146] In some embodiments, the visual three-dimensional reconstruction module 602 is also used to obtain single coding information of each pixel of the multiple pixels of each perspective image based on the image features of each perspective image, the image color information of each perspective image, and the spatial position information of the multiple pixels of each perspective image; according to the pixel matching pairs, the single coding information belonging to the same pixel in each perspective image is combined to obtain the coding information of each pixel among the multiple pixels.

[0147] In some embodiments, the material feature extraction module 603 is also used to extract illumination features from the coded information of the multiple pixels through the initial layer of the material feature extraction network to obtain illumination features of each pixel; and to analyze the illumination features of each pixel through the decision layer of the material feature extraction network to obtain the material category of each pixel.

[0148] In some embodiments, the material feature extraction network includes a preset number of neurons;

[0149] The material feature extraction module 603 is also used to select the coding information of the preset number of pixel points from the coding information of the multiple pixel points; according to the initial layer of the material feature extraction network, the illumination feature extraction is performed on the coding information of the preset number of pixel points to obtain the illumination feature of each pixel point.

[0150] In some embodiments, the object surface material analysis device 60 further includes a training module;

[0151] The training module is used to obtain training visual image samples, wherein the training visual image samples include multiple perspective image samples of object samples and surface material category samples of the object samples; the training visual image samples are input into an initial material feature extraction network to obtain a predicted surface material category of the object sample; a loss value is obtained based on the predicted surface material category of the object sample and a preset loss function; the initial material feature extraction network is trained based on the loss value to obtain the material feature extraction network.

[0152] In some embodiments, the training module is used to perform image acquisition on object samples by an acquisition device to obtain multiple perspective image samples of the object samples; obtain surface material category samples of the object samples after labeling the surface material categories of the object samples; or, obtain multiple perspective image sample sets; analyze the multiple perspective image sample sets according to a preset surface material analysis method to obtain surface material category samples of the object samples.

[0153] In some embodiments, the visual image comprises: a single image;

[0154] The visual 3D reconstruction module 602 is further used to perform monocular reconstruction of the scene corresponding to the single image according to the image features, and determine the spatial position information of multiple pixel points in the single image.

[0155] In some embodiments, the visual three-dimensional reconstruction module 602 is also used to preprocess the visual image by filtering, denoising and deblurring to obtain a preprocessed image; and to extract image features from the preprocessed image through an image feature extraction network to obtain the image features.

[0156] In some embodiments, the classification module 604 is also used to perform instance segmentation on multiple pixel points in the visual image to obtain multiple pixel sets belonging to different objects; wherein the pixel sets correspond to objects one by one; for each pixel set, the material category of each pixel point in each pixel set is obtained in combination with the material characteristics of each pixel point in the visual image; the material category of each pixel point in each pixel set is merged to obtain the surface material category belonging to the same object, until the multiple pixel sets are merged separately, and the surface material categories of different objects in the visual image are obtained.

[0157] In some embodiments, the classification module 604 is also used to count the material category of each pixel point in each pixel set as the same material category to obtain the statistical value of each material category; based on the statistical value and the preset statistical threshold, determine the surface material category belonging to the same object.

[0158] In some embodiments, the visual image includes: multiple perspective images;

[0159] The classification module 604 is also used to perform instance segmentation on multiple pixel points in the reference image of the multiple perspective images to obtain multiple pixel sets belonging to different objects; wherein the reference image is any one of the multiple perspective images.

[0160] In some embodiments, the classification module 604 is further used to perform instance segmentation on a plurality of pixels in the visual image to obtain a plurality of pixel sets belonging to different objects; wherein the pixel sets correspond one to one to the objects;

[0161] The visual 3D reconstruction module 602 is further used to perform visual 3D reconstruction based on multiple pixel sets belonging to different objects in the visual image, and determine the coding information of multiple pixel points belonging to different objects in the visual image.

[0162] It should be noted that the device for analyzing the surface material of an object provided in the above-mentioned embodiment only uses the division of the above-mentioned program modules as an example when analyzing the surface material of an object. In practical applications, the above-mentioned processing can be assigned to different program modules as needed, that is, the internal structure of the device is divided into different program modules to complete all or part of the processing described above. In addition, the device for analyzing the surface material of an object provided in the above-mentioned embodiment and the embodiment of the method for analyzing the surface material of an object belong to the same concept. The specific implementation process and beneficial effects thereof are detailed in the method embodiment, which will not be repeated here. For technical details not disclosed in the embodiment of this device, please refer to the description of the method embodiment of this application for understanding.

[0163] In the embodiments of the present application, Figure 7 This is a schematic diagram of the structure of the analysis device for the surface material of an object proposed in the embodiment of the present application, such as Figure 7 As shown, the device 70 proposed in the embodiment of the present application may also include a processor 701 and a memory 702 storing executable instructions of the processor 701. In some embodiments, the object surface material analysis device 70 may also include a communication interface 703 and a bus 704 for connecting the processor 701, the memory 702 and the communication interface 703.

[0164] In the embodiment of the present application, the processor 701 may be at least one of an application-specific integrated circuit (ASIC), a digital signal processor (DSP), a digital signal processing device (DSPD), a programmable logic device (PLD), a field programmable gate array (FPGA), a central processing unit (CPU), a controller, a microcontroller, and a microprocessor. It is understandable that for different devices, the electronic device used to implement the above-mentioned processor function may also be other, and the embodiment of the present application does not specifically limit it.

[0165] In the embodiment of the present application, the bus 704 is used to connect the communication interface 703, the processor 701 and the memory 702, as well as the mutual communication between these devices.

[0166] In an embodiment of the present application, the processor 701 is used to obtain a visual image of an object in a scene; perform visual three-dimensional reconstruction based on the visual image to determine coding information of a plurality of pixels in the visual image, wherein the coding information represents the relationship between features of the pixels in the visual image; perform material feature extraction and analysis on the coding information of the plurality of pixels through a material feature extraction network to obtain a material category of each pixel; and, based on the material category of each pixel, merge the material categories of different pixels belonging to the same object to obtain a surface material category of the object.

[0167] The memory 702 in the object surface material analysis device 70 can be connected to the processor 701, and the memory 702 is used to store executable program code and data, the program code includes computer operation instructions, and the memory 702 may include a high-speed RAM memory, and may also include a non-volatile memory, for example, at least two disk memories. In practical applications, the above-mentioned memory 702 can be a volatile memory (volatile memory), such as a random access memory (Random-Access Memory, RAM); or a non-volatile memory (non-volatile memory), such as a read-only memory (Read-Only Memory, ROM), a flash memory (flash memory), a hard disk (Hard Disk Drive, HDD) or a solid-state drive (Solid-State Drive, SSD); or a combination of the above-mentioned types of memory, and provide instructions and data to the processor 701.

[0168] In addition, each functional module in the embodiment of the present application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above integrated unit can be implemented in the form of hardware or in the form of software functional modules.

[0169] If the integrated unit is implemented in the form of a software function module and is not sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this embodiment is essentially or the part that contributes to the prior art or the whole or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server, or network device, etc.) or a processor to perform all or part of the steps of the method of this embodiment. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc., various media that can store program codes.

[0170] An embodiment of the present application provides a computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements the method for analyzing the surface material of an object as described in any of the above embodiments.

[0171] Exemplarily, the program instructions corresponding to a method for analyzing the surface material of an object in the present embodiment can be stored on a storage medium such as a CD, a hard disk, or a USB flash drive. When the program instructions corresponding to a method for analyzing the surface material of an object in the storage medium are read or executed by an electronic device, the method for analyzing the surface material of an object as described in any of the above embodiments can be implemented.

[0172] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of hardware embodiments, software embodiments, or embodiments in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage and optical storage, etc.) that contain computer-usable program code.

[0173] The present application is described with reference to implementation flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the process in the flowchart. Figure 1 A process or multiple processes and / or boxes Figure 1A device that provides the functions specified in a block or multiple blocks.

[0174] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which is implemented in the implementation flow diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0175] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operation steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for executing the steps in the flowchart. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0176] The above description is only a preferred embodiment of the present application and is not intended to limit the protection scope of the present application.

Claims

1. A method for analyzing the surface material of an object, characterized in that: The method comprises: Obtaining visual images of objects in a scene; Performing visual three-dimensional reconstruction based on the visual image, determining encoding information of a plurality of pixels in the visual image, wherein the encoding information represents a relationship between features of the pixels in the visual image; Through a material feature extraction network, material features are extracted and analyzed from the coded information of the plurality of pixels to obtain the material category of each pixel; the material feature extraction network is a fully connected network; According to the material categories of the respective pixel points, the material categories of different pixel points belonging to the same object are merged to obtain the surface material category of the object; The performing of visual three-dimensional reconstruction based on the visual image and determining the coding information of a plurality of pixels in the visual image includes: Performing feature extraction on the visual image to obtain image features of the visual image; Acquiring image color information of the visual image; Performing visual three-dimensional reconstruction based on the image features to determine spatial position information of multiple pixel points in the reconstructed image; Based on the image features, the spatial position information of the multiple pixels and the image color information, the encoding information of each pixel in the multiple pixels is determined.

2. The method according to claim 1, characterized in that The visual image includes: a plurality of perspective images; Performing visual three-dimensional reconstruction based on the image features to determine spatial position information of multiple pixel points in the reconstructed image includes: According to the image feature of each of the multiple perspective images, pixel matching is performed on the multiple perspective images to obtain multiple pixel matching pairs, where a pixel matching pair represents a same pixel existing between different perspective images; Based on the multiple pixel point matching pairs, a three-dimensional image reconstruction is performed on the scene corresponding to the reference image in the multiple perspective images to obtain spatial position information of multiple pixel points corresponding to the multiple pixel point matching pairs; wherein the reference image is any one of the multiple perspective images.

3. The method according to claim 2, characterized in that The determining, based on the image feature, the spatial position information of the plurality of pixels and the image color information, encoding information of each pixel among the plurality of pixels comprises: Based on the image features of each viewing angle image, the image color information of each viewing angle image, and the spatial position information of the multiple pixels of each viewing angle image, single encoding information of each pixel of the multiple pixels of each viewing angle image is obtained; According to the pixel matching pairs, the single encoding information belonging to the same pixel in each of the perspective images is combined to obtain encoding information of each pixel in the multiple pixels.

4. The method according to any one of claims 1 to 3, characterized in that: The material feature extraction network is divided into an initial layer and a decision layer, wherein the initial layer is used to extract illumination features, and the decision layer is used to parse the illumination features; the material feature extraction network is used to extract and parse the encoded information of the plurality of pixels to obtain the material category of each pixel, including: Selecting encoding information of a preset number of pixel points from the encoding information of the plurality of pixel points; According to the initial layer of the material feature extraction network, the illumination feature is extracted from the coding information of the preset number of pixels to obtain the illumination feature of each pixel; the illumination feature is the coding information of each pixel; The illumination features of each pixel are analyzed through the decision layer of the material feature extraction network to obtain the material category of each pixel.

5. The method according to claim 4, characterized in that The material feature extraction network includes a preset number of neurons.

6. The method according to claim 4, characterized in that The material feature extraction network is obtained in the following way: Acquire training visual image samples, wherein the training visual image samples include multiple perspective image samples of object samples and surface material category samples of the object samples; Inputting the training visual image samples into an initial material feature extraction network to obtain a predicted surface material category of the object sample; Obtaining a loss value according to the predicted surface material category of the object sample and a preset loss function; The initial material feature extraction network is trained according to the loss value to obtain the material feature extraction network.

7. The method according to claim 6, characterized in that The obtaining of training visual image samples comprises: The acquisition device acquires images of the object sample to obtain multiple viewing angle image samples of the object sample; Obtaining a surface material category sample of the object sample obtained by marking the surface material category of the object sample; Alternatively, obtaining multiple viewpoint image sample sets; The plurality of perspective image sample sets are analyzed according to a preset surface material analysis method to obtain surface material category samples of the object samples.

8. The method according to any one of claims 1 to 3, characterized in that: The visual images include: a single image; Performing visual three-dimensional reconstruction based on the image features to determine spatial position information of multiple pixel points in the reconstructed image includes: According to the image features, a monocular reconstruction is performed on the scene corresponding to the single image to determine the spatial position information of a plurality of pixels in the single image.

9. The method according to any one of claims 1 to 3, characterized in that: The extracting image features from the visual image to obtain the image features of the visual image includes: Performing at least one of filtering, denoising and deblurring on the visual image to obtain a preprocessed image; The image features are extracted from the preprocessed image through an image feature extraction network to obtain the image features.

10. The method according to any one of claims 1 to 3, characterized in that: The method of merging the material categories of different pixels belonging to the same object according to the material categories of the respective pixels to obtain the surface material category of the object includes: Performing instance segmentation on a plurality of pixel points in the visual image to obtain a plurality of pixel sets belonging to different objects; wherein the pixel sets correspond to the objects one by one; For each pixel set, combining the material features of each pixel in the visual image to obtain the material category of each pixel in each pixel set; The material category of each pixel point in each pixel set is merged to obtain the surface material category belonging to the same object, until multiple pixel sets are merged respectively, to obtain the surface material categories of different objects in the visual image.

11. The method according to claim 10, characterized in that The merging of the material categories of each pixel point in each pixel set to obtain the surface material category belonging to the same object includes: The material category of each pixel point in each pixel set is counted as the same material category to obtain the statistical value of each material category; Based on the statistical value and the preset statistical threshold, the surface material category belonging to the same object is determined.

12. The method according to claim 10, characterized in that The visual image includes: a plurality of perspective images; Performing instance segmentation on multiple pixel points in the visual image to obtain multiple pixel sets belonging to different objects, including: Perform instance segmentation on multiple pixel points in a reference image of the multiple perspective images to obtain multiple pixel sets belonging to different objects; wherein the reference image is any one of the multiple perspective images.

13. The method according to any one of claims 1 to 3, characterized in that: The performing visual three-dimensional reconstruction based on the visual image and determining the coding information of a plurality of pixel points in the visual image includes: Performing instance segmentation on a plurality of pixel points in the visual image to obtain a plurality of pixel sets belonging to different objects; wherein the pixel sets correspond to the objects one by one; Based on multiple pixel sets belonging to different objects in the visual image, visual three-dimensional reconstruction is performed to determine encoding information of multiple pixel points belonging to different objects in the visual image.

14. An analysis device for the surface material of an object, characterized in that: The device comprises: An acquisition module, used to acquire visual images of objects in the scene; A visual 3D reconstruction module, used for extracting features from the visual image to obtain image features of the visual image; The acquisition module is further used to acquire image color information of the visual image; The visual 3D reconstruction module is further used to perform visual 3D reconstruction based on the image features, and determine spatial position information of multiple pixels in the reconstructed image; based on the image features, the spatial position information of the multiple pixels, and the image color information, determine encoding information of each of the multiple pixels, wherein the encoding information represents the relationship between the features of the pixels in the visual image; A material feature extraction module, used to extract and analyze the material features of the coded information of the plurality of pixels through a material feature extraction network to obtain the material category of each pixel; the material feature extraction network is a fully connected network; The classification module is used to merge the material categories of different pixel points belonging to the same object according to the material categories of the respective pixel points, so as to obtain the surface material category of the object.

Images