Material extraction method and apparatus, device and storage medium

Abstract

The present application discloses a material extraction method and apparatus, a device and a storage medium. The method comprises: performing segmentation processing on an image to be processed, to obtain an object segmentation area, and dividing the object segmentation area into N image blocks to be processed, wherein the object segmentation area is used for representing a non-background area in the image to be processed, and N is an integer greater than 1; respectively calculating feature mutual information between every two image blocks to be processed, and determining, on the basis of the feature mutual information, an object image block among the N image blocks to be processed; and performing material extraction on the basis of the object image block to obtain materials of objects included in the object image block. According to the present application, manual experience screening is not needed, thereby avoiding increasing the interference to subsequent material extraction and affecting the efficiency and accuracy of material extraction due to the influence of manual experience, and achieving high-precision reduction.

Description

Material extraction method, device, equipment and storage medium

[0001] This application claims priority to the Chinese patent application filed with the China Patent Office on June 14, 2023, application number 202310708769.8, and application name “A material extraction method, device, equipment and storage medium”, all contents of which are incorporated by reference into this application. Technical Field

[0002] The embodiments of the present application relate to the field of artificial intelligence technology, and in particular to a material extraction method, device, equipment, and storage medium. Background Art

[0003] With the rapid development of artificial intelligence technology, it can be applied to the construction of various scenarios such as virtual scenes or physical scenes more and more effectively. In building construction, the main structure of the building, namely the building facade, usually includes structures such as walls, windows, balconies and decorations. It can be used to determine the main style and purpose of a building and is the most critical part of architectural design. Therefore, the analysis of building facades can play an important role in the construction of virtual scenes or physical scenes, and has therefore always received widespread attention.

[0004] Walls are the most common elements in buildings. Different wall materials determine the overall style of the wall and have a great impact on the building's lighting, heat dissipation and other uses. Therefore, how to efficiently and quickly extract wall materials from architectural images is an important issue that needs to be solved urgently.

[0005] Currently, common material classification methods typically rely on manual experience to extract representative wall area blocks from the target image. These blocks are then fed into a trained neural network classifier to output the wall material classification results. The best matching wall material is then determined based on the classification results. However, this method relies too much on manual experience to extract representative wall area blocks from the target image, making it susceptible to the influence of manual experience, which increases interference with subsequent material extraction, affecting the efficiency and accuracy of material extraction. This makes it difficult to accurately restore building walls based on the extracted wall materials in downstream use cases.

[0006] Summary of the Invention

[0007] The embodiments of the present application provide a material extraction method, apparatus, device and storage medium for quickly and accurately determining object image blocks based on feature mutual information, without relying on manual experience screening, avoiding the influence of manual experience, increasing subsequent interference with material extraction, thereby affecting the efficiency and accuracy of material extraction, and thus achieving high-precision restoration.

[0008] In one aspect, an embodiment of the present application provides a material extraction method, which is executed by a computer device and includes:

[0009] Segmenting the image to be processed to obtain an object segmentation region, and dividing the object segmentation region into N image blocks to be processed, wherein the object segmentation region is used to represent a non-background region in the image to be processed, and N is an integer greater than 1;

[0010] Calculating feature mutual information between every two image blocks to be processed in the N image blocks to be processed, and determining the object image block from the N image blocks to be processed based on the feature mutual information;

[0011] Material extraction is performed based on the object image block to obtain the material of the object included in the object image block.

[0012] Another aspect of the present application provides a material extraction device, which is deployed on a computer device and includes:

[0013] an acquisition unit, configured to segment the image to be processed to obtain an object segmentation region, and divide the object segmentation region into N image blocks to be processed, wherein the object segmentation region is used to represent a non-background region in the image to be processed, and N is an integer greater than 1;

[0014] a determining unit, configured to respectively calculate feature mutual information between every two image blocks to be processed in the N image blocks to be processed, and determine the object image block from the N image blocks to be processed based on the feature mutual information;

[0015] The processing unit is used to extract material based on the object image block to obtain the material of the object included in the object image block.

[0016] Another aspect of the present application provides a computer device, comprising: a memory, a processor, and a bus system;

[0017] Wherein, the memory is used to store programs;

[0018] The processor is used to implement the above-mentioned methods when executing the program in the memory;

[0019] The bus system is used to connect the memory and the processor so that the memory and the processor can communicate with each other.

[0020] Another aspect of the present application provides a computer-readable storage medium, wherein instructions are stored in the computer-readable storage medium. When the computer-readable storage medium is run on a computer, the computer is enabled to execute the above-mentioned methods.

[0021] Another aspect of the present application provides a computer program product, including a computer program, which implements the methods described in the above aspects when executed by a processor.

[0022] It can be seen from the above technical solutions that the embodiments of the present application have the following beneficial effects:

[0023] The image to be processed is segmented to obtain the non-background region of the image to be processed, namely the object segmentation region. The object segmentation region is then divided into N image blocks to be processed, and the feature mutual information between each two image blocks to be processed is calculated. The feature mutual information can reflect the similarity between any image block and other image blocks, and thus whether the image block can represent an object. In this way, the object image block can be determined from the N image blocks to be processed based on the feature mutual information. The object image block is an image block that can represent an object, such as a wall. Then, material extraction is performed based on the object image block to obtain the material of the object included in the object image block. Through the above method, the feature mutual information between each two image blocks to be processed in the N image blocks to be processed can be calculated. The object image block can be quickly and accurately determined based on the feature mutual information, without relying on manual experience screening, avoiding the influence of manual experience and increasing interference with subsequent material extraction, thereby affecting the efficiency and accuracy of material extraction, thereby achieving high-precision restoration. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] FIG1 is a schematic diagram of an architecture of an image data processing system according to an embodiment of the present application;

[0025] FIG2 is a flow chart of an embodiment of a material extraction method in an embodiment of the present application;

[0026] FIG3 is a flow chart of another embodiment of the material extraction method in the embodiment of the present application;

[0027] FIG4 is a flow chart of another embodiment of the material extraction method in the embodiment of the present application;

[0028] FIG5 is a flow chart of another embodiment of the material extraction method in the embodiment of the present application;

[0029] FIG6 is a flow chart of another embodiment of the material extraction method in the embodiment of the present application;

[0030] FIG7 is a flow chart of another embodiment of the material extraction method in the embodiment of the present application;

[0031] FIG8 is a flow chart of another embodiment of the material extraction method in the embodiment of the present application;

[0032] FIG9 is a flow chart of another embodiment of the material extraction method in the embodiment of the present application;

[0033] FIG10 is a flow chart of another embodiment of the material extraction method in the embodiment of the present application;

[0034] FIG11 is a flow chart of another embodiment of the material extraction method in the embodiment of the present application;

[0035] FIG12 is a flow chart of another embodiment of the material extraction method in the embodiment of the present application;

[0036] FIG13 is a flow chart of another embodiment of the material extraction method in the embodiment of the present application;

[0037] FIG14 is a schematic diagram of a principle flow chart of a material extraction method according to an embodiment of the present application;

[0038] FIG15 is another schematic diagram of the principle flow of the material extraction method according to an embodiment of the present application;

[0039] FIG16 is a schematic diagram of an embodiment of a material extraction device according to an embodiment of the present application;

[0040] FIG17 is a schematic diagram of an embodiment of a computer device in an embodiment of the present application. DETAILED DESCRIPTION

[0041] The embodiments of the present application provide a material extraction method, apparatus, and device storage medium for quickly and accurately determining object image blocks based on feature mutual information, without relying on manual experience screening, avoiding the influence of manual experience, increasing subsequent interference with material extraction, thereby affecting the efficiency and accuracy of material extraction, and thus achieving high-precision restoration.

[0042] The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present application and the drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can, for example, be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "corresponding to" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0043] It is understandable that in the specific implementation of this application, related data such as images to be processed and feature mutual information are involved. When the above embodiments of this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions.

[0044] To facilitate understanding, some terms or concepts involved in the embodiments of this application are first explained.

[0045] 1. Building Facade: Buildings in most scenarios (such as gaming or home design) can be roughly divided into three parts: the roof, the facade, and the ground floor. The facade is the main structure of the building, typically including walls, windows, balconies, and decorative elements. The facade determines the style and purpose of a building and is the most critical aspect of architectural design. Therefore, analyzing the facade plays a vital role in urban planning, energy efficiency, and the construction of virtual or physical scenarios.

[0046] 2. Wall material: Walls are the most common elements in buildings, and wall material refers to the building materials used for walls. Common wall materials include marble (mostly seen in European buildings), concrete (modern new buildings), red bricks (ordinary houses), wooden boards, etc. Different wall materials determine the overall style of the wall, and also have a great impact on the lighting and heat dissipation of the building.

[0047] 3. Unreal Engine (UE): Unreal Engine UE is a frequently used engine in game development, film and television rendering, and graphics effects. Unreal Engine UE can represent various objects through materials, which are UE materials. Different materials are composed of different parameters such as basic color, roughness, metallicity, normals, and textures.

[0048] It is understandable that the material extraction method disclosed in this application also involves artificial intelligence (AI) technology, and material extraction is automatically achieved through artificial intelligence technology.

[0049] With the research and advancement of artificial intelligence technology, artificial intelligence technology has been studied and applied in many fields, such as common smart homes, smart wearable devices, virtual assistants, smart speakers, smart marketing, unmanned driving, autonomous driving, drones, robots, smart medical care, smart customer service, etc. It is believed that with the development of technology, artificial intelligence technology will be applied in more fields and play an increasingly important role.

[0050] It should be understood that the material extraction method provided in the present application can be applied to various scenarios, including but not limited to artificial intelligence, cloud technology, maps, smart transportation, etc., and is used to determine the object image block through the feature mutual information between each two of the N image blocks to be processed in the object segmentation area, to quickly screen out the object image blocks representing the object, and then perform material extraction based on the object image block to obtain the material of the object included in the object image block, so that the virtual model can be restored with high precision based on the material of the object, for application in scenarios such as game virtual construction scenes, smart home construction scenes, smart city construction scenes, etc.

[0051] In order to solve the above problems, the present application proposes a material extraction method, which is applied to the data object control system shown in Figure 1. Please refer to Figure 1. Figure 1 is a schematic diagram of the architecture of the data object control system in an embodiment of the present application. As shown in Figure 1, the server performs segmentation processing on the image to be processed provided by the terminal device to obtain the non-background area in the image to be processed, that is, the object segmentation area, and then divides the object segmentation area into N image blocks to be processed, and calculates the feature mutual information between each two image blocks to be processed. The feature mutual information can reflect the similarity between any image block and other image blocks, and then reflect whether the image block can represent an object. In this way, the object image block can be determined from the N image blocks to be processed based on the feature mutual information. The object image block is an image block that can represent an object, such as a wall. Then, material extraction is performed based on the object image block to obtain the material of the object included in the object image block. Through the above method, the feature mutual information between every two image blocks to be processed in the N image blocks to be processed in the object segmentation area can be calculated, and the object image blocks can be quickly and accurately determined based on the feature mutual information without relying on manual experience screening, avoiding the influence of manual experience, increasing the interference of subsequent material extraction, thereby affecting the efficiency and accuracy of material extraction, and thus achieving high-precision restoration.

[0052] It is understandable that Figure 1 shows a terminal device. In actual scenarios, more types of terminal devices can participate in the data processing process. Terminal devices also include but are not limited to mobile phones, computers, intelligent voice interaction devices, smart home appliances, car terminals, etc. The specific number and type depend on the actual scenario and are not limited here. In addition, Figure 1 shows a server, but in actual scenarios, there can also be multiple servers involved, especially in the scenario of multi-model training interaction. The number of servers depends on the actual scenario and is not limited here.

[0053] It should be noted that in this embodiment, the server can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. Terminal devices and servers can be directly or indirectly connected via wired or wireless communication. Terminal devices and servers can be connected to form a blockchain network, which is not limited in this application.

[0054] In combination with the above introduction, the material extraction method in this application will be introduced below. Please refer to Figure 2. An embodiment of the material extraction method in the embodiment of this application includes:

[0055] In S101, the image to be processed is segmented to obtain an object segmentation region, and the object segmentation region is divided into N image blocks to be processed, wherein the object segmentation region is used to represent the non-background area in the image to be processed, and N is an integer greater than 1;

[0056] The image to be processed is an image that needs to be subjected to material extraction. The image to be processed includes an object, and material extraction is to obtain the material of the object. The embodiment of the present application does not limit the type of the object. For example, the object can be a wall. In the embodiment of the present application, when it is necessary to perform material extraction on the image to be processed, the image to be processed can be segmented first to obtain the non-background area in the image to be processed, that is, the object segmentation area. Then, the object segmentation area can be divided into N image blocks to be processed, so that the object image block that can represent the image to be processed can be extracted from the N image blocks to be processed.

[0057] As shown in Figure 14, for example, assuming that the image to be processed is a building picture in a virtual building scene in a game, and the object is a wall, the image to be processed (for example, a building picture of an ordinary building as shown in Figure 14, including walls, windows, etc.) can be first input into a segmentation model (for example, a fully convolutional neural network U-net for image segmentation, etc.), and the background area (the window area that is removed as shown in Figure 14) and the non-background area (for example, the wall area with the windows removed) are segmented to obtain the object segmentation area (for example, the wall area with the background areas such as windows removed in the wall extraction module as shown in Figure 14).

[0058] It can be understood that this embodiment sets the pixel values ​​of the background area (i.e., non-wall area) in the processed image to zero (i.e., mask) so that the object image block (i.e., image block that can represent the wall) can be better extracted later.

[0059] It is understandable that, since the building walls in actual scenes are basically composed of one material, it can be considered that the object segmentation area (i.e., the wall area) in the image to be processed is uniform, so material extraction can be performed on a piece of area therein, i.e., the object image block (i.e., the image block that can represent the wall), and the material corresponding to the image to be processed can be extracted. Therefore, after obtaining the object segmentation area, this embodiment can sample image blocks of a certain size (e.g., 32x32) from the object segmentation area (i.e., the segmented wall area) in order from top to bottom and from left to right, denoted as Pi, where i = 1, 2, ..., N, N is the total number of image blocks, so that the object image block (i.e., the image block that can represent the wall) can be filtered out from these image blocks later.

[0060] In S102, feature mutual information between every two image blocks to be processed in the N image blocks to be processed is calculated respectively, and based on the feature mutual information, the target image block is determined from the N image blocks to be processed;

[0061] It is understandable that, due to the influence of lighting conditions, camera shooting angle, image resolution and other factors in actual scenes, the object segmentation area (i.e., wall area) of the image to be processed is prone to noise interference caused by the superposition of multiple factors. Therefore, in order to better extract the object image block (i.e., the image block that can represent the wall) from the object segmentation area (i.e., wall area), so as to facilitate the subsequent better material extraction of the object image block (i.e., the image block that can represent the wall), after obtaining N image blocks to be processed, the feature mutual information between each two image blocks to be processed can be calculated separately. The feature mutual information can reflect the similarity between any image block and other image blocks, and then reflect whether the image block can represent the object. Then, based on the feature mutual information, the object image block can be better screened out from the N image blocks to be processed.

[0062] After obtaining N image blocks to be processed, the feature mutual information between each two image blocks to be processed can be calculated. In one possible implementation, the feature mutual information may include texture feature similarity. In this case, a local binary pattern (LBP) feature algorithm may be first used to calculate the texture features of each image block to be processed. Then, based on the texture features of each image block to be processed, the texture feature similarity between each two image blocks to be processed (for example, cosine similarity and structural similarity algorithms) may be calculated as the feature mutual information between the image blocks.

[0063] When the feature mutual information includes texture feature similarity, S102 can be implemented by first calculating the texture feature of each image block to be processed based on the pixel value of each pixel in each image block to be processed. Since the image block representing the image to be processed (such as a wall) can be defined as an image block with the highest similarity to all other image blocks (such as a wall block), the sum of the texture feature similarities between each image block to be processed and the other N-1 image blocks to be processed can be calculated respectively, so that the image block to be processed corresponding to the largest sum value can be used as the object image block. For example, if the object included in the image to be processed is a wall, then the image block to be processed can be a wall block, so the sum of the similarities between any wall block and all other blocks can be calculated, and the largest one among them, that is, the image block to be processed corresponding to the sum of the texture feature similarities with the largest value, can be taken as the object image block.

[0064] It can be understood that the algorithm for extracting texture features of each image block to be processed is not limited to the LBP image feature algorithm, and other image feature descriptors such as the scale-invariant feature transform (SIFT) algorithm, the fast feature point extraction and description (Oriented FAST and Rotated BRIEF, ORB) algorithm, and image features extracted based on neural networks can also be used. No specific restrictions are made here.

[0065] It can be understood that the calculation method of the feature mutual information between the image blocks to be processed is not limited to the single-scale cosine similarity and structural similarity algorithms, and methods such as mean square error, peak signal-to-noise ratio, and multi-scale similarity metrics can also be used to measure the mutual information of the blocks. No specific restrictions are made here.

[0066] In S103 , material extraction is performed based on the object image block to obtain the material of the object included in the object image block.

[0067] After obtaining the object image block that can represent the object, material extraction can be directly performed on the object image block to obtain the material of the object included in the object image block.

[0068] The image to be processed is segmented to obtain the non-background region of the image to be processed, namely the object segmentation region. The object segmentation region is then divided into N image blocks to be processed, and the feature mutual information between each two image blocks to be processed is calculated. The feature mutual information can reflect the similarity between any image block and other image blocks, and thus whether the image block can represent an object. In this way, the object image block can be determined from the N image blocks to be processed based on the feature mutual information. The object image block is an image block that can represent an object, such as a wall. Then, material extraction is performed based on the object image block to obtain the material of the object included in the object image block. Through the above method, the feature mutual information between each two image blocks to be processed in the N image blocks to be processed can be calculated. The object image block can be quickly and accurately determined based on the feature mutual information, without relying on manual experience screening, avoiding the influence of manual experience and increasing interference with subsequent material extraction, thereby affecting the efficiency and accuracy of material extraction, thereby achieving high-precision restoration.

[0069] The embodiment of the present application does not limit the method of material extraction. In one possible implementation method, the material of the object can be obtained by performing material classification based on the object image block through a neural network classifier.

[0070] In one possible implementation, in order to avoid training a neural network classifier for material classification, the material extraction is performed based on the object image block in S103, and the method for obtaining the material of the object included in the object image block may be (not shown in the drawings):

[0071] In S201, texture matching is performed on the object image block and each candidate material image in the candidate material image set to obtain a texture matching result corresponding to each candidate material image;

[0072] In S202 , based on the texture matching result corresponding to each candidate material image, an object material image is determined from the candidate material image set, where the object material image is used to represent the material of the object included in the object image block.

[0073] In an embodiment of the present application, after obtaining the object image block, texture matching can be performed on the object image block and each candidate material image in the candidate material image set respectively to obtain a texture matching result corresponding to each candidate material image. The texture matching result can reflect whether the texture of the candidate material image is similar to that of the material of the object image block, so that the candidate material image most similar to the object image block can be found based on the texture matching result.

[0074] The candidate material image set is a set of materials created based on the Unreal Engine that can be used to represent various objects (for example, various walls in a virtual building scene in a game). Different materials are composed of different basic color, roughness, metallicity, normals, texture and other parameters. UE materials can be understood as a type of material. UE materials can be assigned to virtual models so that the color and texture of the virtual model are consistent with the material, thereby achieving high-fidelity restoration of the model. In addition, UE materials can also be assigned to a plane, so the generated plane is the material image, i.e., the candidate material image.

[0075] It is understandable that management objects or design objects (for example, game scene architects in game virtual construction scenes) can also add or subtract UE materials in the candidate material image set according to actual application needs, without having to readjust the calculation parameters of each module, and quickly adapt to the needs of various scene businesses.

[0076] As shown in Figure 14, for example, assuming that the image to be processed is a building picture in a virtual building scene in a game, and assuming that the extracted object image block is a wall image block as shown in Figure 14, a set of candidate material images related to the wall material can be searched from the virtual engine.

[0077] The texture matching module shown in Figure 14 can perform texture matching on the acquired object image block with each candidate material image in the candidate material image set. Specifically, the material texture features of each candidate material image can be calculated using the LBP feature algorithm based on the pixel value of each pixel point in each candidate material image. Then, based on the texture features of the object image block and the material texture features of each candidate material image, the texture feature similarity (for example, cosine similarity and structural similarity) between the object image block and each candidate material image can be calculated respectively, and the texture feature similarity can be used as the texture matching result corresponding to each candidate material image.

[0078] It can be understood that the algorithm for extracting texture features of each candidate material image is not limited to the LBP feature algorithm, and other image feature descriptors such as SIFT algorithm, ORB algorithm, and image features extracted based on neural networks can also be used. No specific restrictions are made here.

[0079] In an embodiment of the present application, after obtaining the texture matching result corresponding to each candidate material image, the most suitable material image, that is, the object material image, can be screened out from the candidate material image set based on the texture matching result corresponding to each candidate material image.

[0080] When the texture matching result is texture feature similarity, after obtaining the texture matching result corresponding to each candidate material image, the candidate material image corresponding to the texture feature similarity with the largest value can be screened out as the object material image based on the texture matching result corresponding to each candidate material image.

[0081] It is understandable that in actual application, this embodiment will also return the top three texture matching results (for example, as shown in Figure 14, the texture matching module returns the top three texture matching results, such as three candidate material images of different materials), so that the management object or design object (for example, the game scene architect in the virtual construction scene of the game) can make corrections to the top three texture matching results obtained or make better matching choices based on the actual application scenario to obtain the most suitable object material image.

[0082] The above method directly filters the object material image through texture matching, without the need to train a model capable of material classification processing for the target image block. This not only effectively reduces the cost of material extraction, but also simplifies deployment and reduces computational complexity, thereby improving the efficiency of material acquisition to a certain extent. At the same time, by texture matching the object image block with each candidate material image in the candidate material image set, not only can the object material image be accurately extracted, but also important information such as the material's texture properties can be obtained based on the texture assignment. This enables the subsequent high-fidelity restoration of the virtual model based on the texture properties of the object material image in downstream usage scenarios.

[0083] Based on the embodiment corresponding to FIG. 2 above, in another optional embodiment of the material extraction method provided in the embodiment of the present application, as shown in FIG. 3 , after determining the object material image from the set of candidate material images based on the texture matching results corresponding to each candidate material image, the method further includes:

[0084] In S301, the object material image is recolored based on the color value of the object image block to obtain a recolored material image;

[0085] In S302 , based on the recolored material image, the virtual model corresponding to the image to be processed is rendered to obtain a final virtual model.

[0086] It can be understood that since the matched object material image (i.e., UE material) is consistent with the object image block in texture, but there may still be large differences in color, this embodiment can recolor the object material image based on the color value of the object image block to obtain a recolored material image, that is, a recolored material image that is consistent with the object image block in texture and color, so that the virtual model corresponding to the processed image can be rendered based on the recolored material image to obtain the final virtual model, thereby achieving high-precision and high-fidelity restoration of the final virtual model.

[0087] As shown in FIG14 , for example, the texture matching module returns a texture matching result. By comparing the colors of the object image block and the object material image (i.e., UE material), the object material image can be recolored based on the color value of the object image block. Specifically, the object material image (i.e., UE material) can be recolored by calculating the color adjustment grayscale, that is, the object image block is smoothed to obtain the color value corresponding to each pixel in the object image block, and the color average value of the object image block is calculated based on the color value corresponding to each pixel in the object image block. Then, based on the color average value of the object image block, as shown in FIG15 , the color grayscale particles to be used can be determined from the M color grayscale particles randomly initialized, and the color grayscale particles to be used can be recolored. The color grayscale value of the color grayscale particle is multiplied by the color value of the object material image. For example, the red-green-blue (RGB) color channel value of the object material image m is (R, G, B). The UE engine recolors by multiplying an RGB color grayscale triplet P = (a, b, c) of the color grayscale particle to be used, where 0≤a, b, c≤1, with the color of the object material image, and the coloring result is f = p*m = (R*a, G*b, B*c). Then, the recoloring value f is assigned to the object material image to obtain the recolored material image, so that the final recolored material image has a consistent visual effect (i.e., consistent texture and color) with the object image block.

[0088] The embodiment of the present application can render the virtual model corresponding to the processed image based on the recolored material image. Specifically, the virtual engine UE can assign the recolored material image to the pre-modeled building model (that is, it can be understood as coloring the building model) to obtain the final virtual model, thereby achieving high-fidelity restoration of the virtual model.

[0089] Based on the embodiment corresponding to FIG. 2 , in another optional embodiment of the material extraction method provided by the embodiment of the present application, as shown in FIG. 4 , the feature mutual information includes texture feature similarity; S102 calculates the feature mutual information between every two image blocks to be processed in N image blocks to be processed, and determines the object image block from the N image blocks to be processed based on the feature mutual information, including:

[0090] In S401, the texture feature of each image block to be processed is calculated based on the pixel value of each pixel in each image block to be processed;

[0091] In S402, based on the texture features of each image block to be processed, the sum of the texture feature similarities between each image block to be processed and the other N-1 image blocks to be processed is calculated respectively;

[0092] In S403 , the image block to be processed corresponding to the maximum sum value is taken as the target image block.

[0093] In an embodiment of the present application, after acquiring N image blocks to be processed, the texture features of each image block to be processed can be calculated based on the pixel value of each pixel in each image block to be processed. Furthermore, based on the texture features of each image block to be processed, the sum of the texture feature similarities between each image block to be processed and the other N-1 image blocks to be processed can be calculated. The image block to be processed corresponding to the largest sum is then selected as the target image block. This allows the most representative image block to be processed to be selected as the target image block from the N image blocks to be processed based on important information such as texture properties, thereby improving the accuracy of material extraction.

[0094] In one possible implementation, based on the pixel value of each pixel in each image block to be processed, the texture features of each image block to be processed can be calculated by taking each pixel (for example, pixel K) of each processed image block as the center of a circle and a radius threshold (for example, radius R) as the radius, first traversing the first associated pixel of each pixel within the radius range in the image block to be processed, and then, based on the pixel value of each pixel and the pixel value of the first associated pixel of each pixel, a local binary pattern feature algorithm can be used to calculate the local feature value corresponding to each pixel, and then, based on the local feature value of each pixel, the texture features of each image block to be processed can be obtained by sorting out.

[0095] After obtaining the texture features of each image block to be processed, the texture feature similarities (for example, cosine similarity and structural similarity (SSIM)) between each image block to be processed and the other N-1 image blocks to be processed can be calculated based on the texture features of each image block to be processed, and the sum of the texture feature similarities between each image block to be processed and the other N-1 image blocks to be processed can be obtained. Then, the image block to be processed corresponding to the sum of the texture feature similarities with the largest value, that is, the image block with the highest similarity to all other wall blocks, can be used as the object image block.

[0096] Based on the embodiment corresponding to FIG. 3 , in another optional embodiment of the material extraction method provided in the embodiment of the present application, as shown in FIG. 5 , S301 recolors the object material image based on the color of the object image block to obtain a recolored material image, including:

[0097] In S501, a smoothing process is performed on the object image block to obtain a color value corresponding to each pixel in the object image block, and a color average value of the object image block is calculated based on the color value corresponding to each pixel in the object image block;

[0098] In S502, based on the color average value of the target image block, a color grayscale particle to be used is determined from M color grayscale particles that are randomly initialized, where M is an integer greater than or equal to 1;

[0099] In S503 , the color grayscale value of the color grayscale particle to be used is multiplied by the color value of the object material image to obtain a recolored material image.

[0100] It can be understood that since the matched object material image (i.e., UE material) is consistent with the object image block in texture, there may still be large differences in color. Therefore, this embodiment can first smooth the object image block to obtain the color value corresponding to each pixel in the object image block, and calculate the color average of the object image block based on the color value corresponding to each pixel in the object image block. Then, based on the color average of the object image block, the color grayscale particles to be used are determined from the M randomly initialized color grayscale particles, and the color grayscale values ​​of the color grayscale particles to be used are multiplied by the color value of the object material image to obtain the recolored material image.

[0101] By performing recoloring processing in the above manner, a recolored material image that is consistent with the object image block in texture and color can be obtained, so that the virtual model corresponding to the processed image can be rendered based on the recolored material image to obtain the final virtual model, thereby achieving high-precision and high-fidelity restoration of the final virtual model.

[0102] In one possible implementation, the color average value of the object image block can be calculated by first performing Gaussian denoising on the object image block (i.e., image block r) using a conventional Gaussian filtering algorithm to obtain a blurred image corresponding to the denoised pixel value. Then, the color value corresponding to each pixel in the blurred image can be directly obtained based on the blurred image. For example, after denoising the object image block, the color value of pixel i in the blurred image is (ri, gi, bi). Then, based on the color value corresponding to each pixel in the object image block, the color average value of the object image block is calculated. Specifically, all the pixels can be added up and then divided by the total number to obtain the color average value average(sum(ri), sum(gi), sum(bi)).

[0103] As shown in Figure 15, M color grayscale particles can be randomly initialized first, and then the square error between the color average value of the object image block and the current color grayscale value of each color grayscale particle can be calculated respectively to obtain the error of each color grayscale particle at the corresponding current moment. Then, the current moment error can be compared with the error threshold. If the current moment error with the smallest value is less than the error threshold, the color grayscale particle corresponding to the current moment error with the smallest value can be determined as the color grayscale particle to be used.

[0104] Next, the color grayscale value of the color grayscale particle to be used may be multiplied by the color value of the object material image to obtain a recoloring result, and then the object material image may be colored based on the recoloring result to obtain a recolored material image.

[0105] Based on the embodiment corresponding to FIG. 5 , in another optional embodiment of the material extraction method provided in the embodiment of the present application, as shown in FIG. 6 , S502 determines a color grayscale particle to be used from M randomly initialized color grayscale particles based on the color average value of the object image block, including:

[0106] In S601, the current color grayscale value of each color grayscale particle at the current moment is obtained;

[0107] In S602, the square error between the color average value of the object image block and the current color grayscale value of each color grayscale particle is calculated to obtain the current moment error of each color grayscale particle at the current moment;

[0108] In S603 , if the error at the current moment with the smallest value is smaller than the error threshold, the color grayscale particle corresponding to the error at the current moment with the smallest value is determined as the color grayscale particle to be used.

[0109] In an embodiment of the present application, M color grayscale particles can be randomly initialized first, and the current color grayscale value of each color grayscale particle at the current moment can be obtained. Then, the square error between the color average value of the object image block and the current color grayscale value of each color grayscale particle can be calculated to obtain the current moment error of each color grayscale particle at the current moment. Then, the current moment error can be compared with the error threshold. If the current moment error with the smallest value is less than the error threshold, the color grayscale particle corresponding to the current moment error with the smallest value can be determined as the color grayscale particle to be used. In this way, a more accurate color grayscale particle to be used can be determined from the M color grayscale particles, and then the color grayscale particle to be used can be used for recoloring processing to obtain a recolored material image consistent with the color of the object image block.

[0110] Since the similarity of two colors in human vision can be expressed by the similarity of the two colors in the Hue-Saturation-Value (HSV) color space, the two most similar colors in the HSV color space can be expressed by the minimum square error as shown in the following formula (1):

[0111] Where v represents the RGB color value of the object image block r, which can be obtained by Gaussian smoothing the object image block and then calculating the color mean. p represents the RGB color triplet of the grayscale value of the grayscale particle, such as (a, b, c). h(·) is a color conversion function used to convert the input RGB color triplet into an HSV color triplet.

[0112] In order to better find the color grayscale particles that are most similar or consistent with the color of the object image block, so as to better recolor the object material image, this embodiment is shown in Figure 15. The particle swarm optimization algorithm (PS0) is used to randomly initialize M RGB color grayscale particles. For example, M (for example, M>100) particles are randomly initialized. Any color grayscale particle i at the current moment corresponds to another set of RGB color grayscale values. Wherein, the superscript t=0 represents the current initialization time 0. Then, based on the above formula (2), it can be converted into calculating the square error between each color grayscale particle and the object image block in HSV color: Then, the score of each particle i at the current moment can be evaluated based on the square error, that is, It is used to judge the similarity between the color of each color grayscale particle and the color of the object image block in the HSV color space. Further, the color grayscale particle with the largest score can be found as the current global optimal estimation particle. Alternatively, the color grayscale particle with the smallest current moment error (i.e., the minimum square error at the current moment) can be found as the current global optimal estimation particle, i.e.

[0113] As shown in Figure 15, after finding the current global optimal estimated particle, the minimum current moment error can be compared with the error threshold. If the minimum current moment error is less than the error threshold, it can be understood that the current global optimal estimated particle is the color grayscale particle that is most similar or consistent with the color of the object image block, that is, the color grayscale particle to be used.

[0114] Based on the embodiment corresponding to FIG6 above, in another optional embodiment of the material extraction method provided by the embodiment of the present application, as shown in FIG7 , after S602 respectively calculates the square error between the color average value of the object image block and the initial color grayscale value of each color grayscale particle to obtain the error of each color grayscale particle at the corresponding current moment, the method further includes:

[0115] In S701, if the error at the current moment with the smallest value is greater than the error threshold, the color grayscale value of each color grayscale particle at the next moment is updated based on the error at the current moment with the smallest value, to obtain an updated color grayscale value of each color grayscale particle;

[0116] In S702, the square error between the color average value of the object image block and the updated color grayscale value of each color grayscale particle is calculated respectively to obtain the next moment error of each color grayscale particle at the next moment;

[0117] In S703 , if the error at the next moment with the smallest value is smaller than the error threshold, the color grayscale particle corresponding to the error at the next moment with the smallest value is determined as the color grayscale particle to be used.

[0118] In an embodiment of the present application, after obtaining the current moment error with the smallest value, the current moment error with the smallest value can be compared with the error threshold. If the current moment error with the smallest value is greater than the error threshold, the color grayscale value of each color grayscale particle at the next moment can be updated based on the current moment error with the smallest value to obtain the updated color grayscale value of each color grayscale particle, and then the square error between the color average value of the object image block and the updated color grayscale value of each color grayscale particle is calculated respectively to obtain the next moment error of each color grayscale particle at the next moment. Then, the next moment error is compared with the error threshold. If the next moment error with the smallest value is less than the error threshold, the color grayscale particle corresponding to the next moment error with the smallest value can be determined as the color grayscale particle to be used.

[0119] As shown in FIG15 , after the current global optimal estimation particle is found, the error at the current moment with the minimum value can be compared with the error threshold. If the error at the current moment with the minimum value is greater than the error threshold, it can be understood that the current global optimal estimation particle is the color grayscale particle that is least similar to the color of the object image block. Therefore, as shown in FIG15 , the color grayscale value of each color grayscale particle at the next moment can be continued to be updated based on the error at the current moment with the minimum value corresponding to the current global optimal estimation particle. Specifically, the grayscale value can be updated using the following formulas (2) and (3) for the error at the current moment with the minimum value corresponding to the current global optimal estimation particle and the color grayscale value of each color grayscale particle to obtain the updated color grayscale value at the next moment.

[0120] Among them, c1 and c2 are learning factors, is the optimal value of the color grayscale particle i from time t = 0 to the current time t, that is, g t is the global optimal value at the current time t, that is is the updated momentum of the color grayscale particle i at the current time t.

[0121] After obtaining the updated color grayscale value of each color grayscale particle at the next moment, you can use The square error between the color average value of the object image block and the updated color grayscale value of each color grayscale particle is calculated respectively to obtain the next moment error of each color grayscale particle at the next moment.

[0122] Then, the color grayscale particle with the smallest next moment error (i.e., the minimum square error at the next moment) can be found as the current global optimal estimation particle, i.e.

[0123] As shown in Figure 15, after finding the global optimal estimation particle at the next moment, the minimum next moment error can be compared with the error threshold. If the minimum next moment error is less than the error threshold, it can be understood that the global optimal estimation particle at the next moment is the color grayscale particle that is most similar or consistent with the color of the object image block, that is, the color grayscale particle to be used. Conversely, if the minimum next moment error is greater than the error threshold, it can be understood that, as shown in 15, by repeatedly executing the steps of updating the color grayscale value of each color grayscale particle at the next moment, respectively calculating the square error between the color average value of the object image block and the color grayscale value of each color grayscale particle updated at the next moment, and comparing the minimum next moment error with the error threshold, a global optimal estimation particle that is most similar or consistent with the color of the object image block, that is, the color grayscale particle to be used, can be found, so that the color grayscale value of the color grayscale particle to be used can be multiplied with the color value of the object material image to obtain the recoloring result, and the object material image can be recolored based on the recoloring result to obtain the recolored material image.

[0124] Based on the embodiment corresponding to FIG. 4 , in another optional embodiment of the material extraction method provided in the embodiment of the present application, as shown in FIG. 8 , S401 calculates the texture features of each image block to be processed based on the pixel value of each pixel in each image block to be processed, including:

[0125] In S801, each pixel point of each image block to be processed is used as the center of a circle, and a radius threshold is used as a radius, and the first associated pixel point of each pixel point within the radius range in the image block to be processed is traversed;

[0126] In S802, a local feature value of each pixel is calculated based on the pixel value of each pixel and the pixel value of the first associated pixel of each pixel;

[0127] In S803, the texture features of each image block to be processed are obtained based on the local feature values ​​of each pixel point.

[0128] In an embodiment of the present application, the texture features of each image block to be processed can be calculated by taking each pixel point of each image block to be processed as the center of a circle and a radius threshold as the radius, traversing the first associated pixel point of each pixel point within the radius range in the image block to be processed, and calculating the local feature value corresponding to each pixel point based on the pixel value of each pixel point and the pixel value of the first associated pixel point of each pixel point. Then, the texture features of each image block to be processed can be obtained based on the local feature value of each pixel point.

[0129] In one possible implementation, the texture features of each image block to be processed can be calculated by taking each pixel point (for example, any pixel point K) of each processed image block as the center of a circle and a radius threshold (for example, a threshold R) as the radius, and traversing the pixel points in the image block to be processed that appear within a range with K as the center and R as the radius as the first associated pixel points.

[0130] For a certain pixel point K in the image to be processed, the local binary pattern feature algorithm shown in the following formula (4) can be used to simply and efficiently calculate the texture feature corresponding to each pixel point, that is, the local eigenvalue, so as to better characterize the texture characteristics in the image block to be processed through the texture feature, thereby better describing the characteristics of the building wall:

[0131] Where δ is the activation pulse function, K is the pixel in the image block to be processed, Q is the number of pixels within a circle with a radius of R centered at pixel K, and f(·) is the pixel value of the pixel in the image block to be processed.

[0132] After obtaining the local feature value of each pixel, the local feature value of each pixel can be sorted according to the position of each pixel on the image to be processed to obtain the texture feature of each image block to be processed.

[0133] In an embodiment of the present application, for each pixel point in the image block to be processed, the local eigenvalue of the pixel point can be determined by combining the pixel value of the pixel point and the pixel value of the first associated pixel point of the pixel point, taking into account the influence of the surrounding pixels of the pixel point on the local eigenvalue of the pixel point, thereby better characterizing the texture characteristics in the image block to be processed and better describing the characteristics of the object.

[0134] Based on the embodiment corresponding to FIG4 above, in another optional embodiment of the material extraction method provided by the embodiment of the present application, as shown in FIG9 , the texture feature similarity includes texture cosine similarity and texture structure similarity; S402 calculates the sum of the texture feature similarities between each image block to be processed and the other N-1 image blocks to be processed based on the texture features of each image block to be processed, including:

[0135] In S901, based on the texture features of each image block to be processed, the texture cosine similarity between each image block to be processed and the other N-1 image blocks to be processed is calculated respectively;

[0136] In S902, based on the texture features of each image block to be processed, the texture structure similarities between each image block to be processed and the other N-1 image blocks to be processed are calculated respectively;

[0137] In S903 , the texture cosine similarities and texture structure similarities between each image block to be processed and the other N−1 image blocks to be processed are summed to obtain a sum of texture feature similarities.

[0138] In an embodiment of the present application, since cosine similarity can measure the statistical similarity of two feature vectors, and structural similarity can measure the visual similarity of two image features, in this embodiment, texture similarity can be expressed as texture cosine similarity and texture structure similarity. Therefore, after obtaining the texture features of each image block to be processed, the texture cosine similarity between each image block to be processed and the other N-1 image blocks to be processed can be calculated respectively. At the same time, the texture structure similarity between each image block to be processed and the other N-1 image blocks to be processed can be calculated respectively. Then, the texture cosine similarity and texture structure similarity between each image block to be processed and the other N-1 image blocks to be processed are added to obtain the sum of the texture feature similarities.

[0139] In a possible implementation, after obtaining the texture features of each image block to be processed, the feature mutual information between the image blocks to be processed can be calculated. Specifically, it can be assumed that the LBP feature expressions of the image block to be processed i and the image block to be processed j are Q i and Q j , so the cosine similarity algorithm can be used to calculate the texture cosine similarity between the image block i to be processed and the image block j to be processed, that is, cos(Q i ,Q j ), similarly, the texture cosine similarity between each image block to be processed and other image blocks to be processed can be calculated.

[0140] Then, the structural similarity algorithm can be used to calculate the texture structure similarity between the image block i to be processed and the image block j to be processed, that is, SSIM (Q i ,Q j ), similarly, the texture structure similarity between each image block to be processed and other image blocks to be processed can be calculated.

[0141] Then, the correlation between the image blocks i and j can be described by the sum of the texture cosine similarity and texture structure similarity corresponding to the image blocks i and j, that is, SIM(i,j)=cos(Q i ,Q j )+SSIM(Q i ,Q j), similarly, the texture cosine similarity and texture structure similarity between each image block to be processed and the other N-1 image blocks to be processed can be summed to obtain the sum of the texture feature similarities corresponding to each image block to be processed, so that the most suitable image block can be quickly and accurately screened out as the object image block based on the sum of the texture feature similarities corresponding to each image block to be processed.

[0142] Based on the embodiment corresponding to FIG. 2 above, in another optional embodiment of the material extraction method provided in the embodiment of the present application, as shown in FIG. 10 , texture matching is performed on the object image block with each candidate material image in the candidate material image set to obtain a texture matching result corresponding to each candidate material image, including:

[0143] In S1001, the material texture features of each candidate material image are calculated based on the pixel value of each pixel in each candidate material image;

[0144] In S1002, based on the texture features of the object image block and the material texture features of each candidate material image, the texture feature similarity between the object image block and each candidate material image is calculated respectively, and the texture feature similarity is used as the texture matching result corresponding to each candidate material image.

[0145] In an embodiment of the present application, when texture matching is performed on the object image block with each candidate material image in the candidate material image set, the material texture features of each candidate material image can be calculated based on the pixel value of each pixel point in each candidate material image. Then, based on the texture features of the object image block and the material texture features of each candidate material image, the texture feature similarity between the object image block and each candidate material image can be calculated respectively, and the texture feature similarity can be used as the texture matching result corresponding to each candidate material image.

[0146] In one possible implementation, when texture matching is performed on the object image block with each candidate material image in the candidate material image set, the material texture features of each candidate material image can be calculated based on the pixel value of each pixel in each candidate material image. Specifically, each pixel of each candidate material image (for example, pixel K) can be used as the center of the circle, and the radius threshold (for example, radius R) can be used as the radius. The second associated pixel points corresponding to each pixel within the radius range in each candidate material image can be traversed first. Then, based on the pixel value of each pixel and the pixel value of the second associated pixel point corresponding to each pixel, a local binary pattern feature algorithm can be used to calculate the local feature value corresponding to each pixel. Then, the material texture features of each candidate material image can be obtained based on the local feature value of each pixel.

[0147] After obtaining the material texture features of each candidate material image, the texture feature similarity between the object image block and each candidate material image is calculated based on the texture features of the object image block and the material texture features of each candidate material image. Specifically, it can be achieved by assuming that the LBP feature expressions of the object image block x and each candidate material image y are Q x and Q y Therefore, the cosine similarity algorithm can be used to calculate the texture cosine similarity between the object image block x and each candidate material image y, that is, cos(Q x ,Q y ).

[0148] Then, the structural similarity algorithm can be used to calculate the texture structure similarity between the object image block x and each candidate material image y, namely SSIM (Q x ,Q y ).

[0149] Then, the sum of the texture cosine similarity and texture structure similarity corresponding to the object image block x and each candidate material image y can be used as the texture feature similarity to describe the correlation between the object image block x and each candidate material image y, that is, SIM(x,y)=cos(Q x ,Q y )+SSIM(Q x ,Q y ), that is, each candidate material image corresponds to the texture matching result.

[0150] Based on the embodiment corresponding to FIG. 10 , in another optional embodiment of the material extraction method provided in the embodiment of the present application, as shown in FIG. 11 , S1001 calculates the material texture features of each candidate material image based on the pixel value of each pixel in each candidate material image, including:

[0151] In S1101, each pixel point of each candidate material image is used as the center of a circle, and the radius threshold is used as the radius, and the second associated pixel point of each pixel point within the radius range in each candidate material image is traversed;

[0152] In S1102, a local feature value of each pixel is calculated based on the pixel value of each pixel and the pixel value of a second associated pixel of each pixel;

[0153] In S1103 , the texture features of each candidate material image are obtained based on the local feature value of each pixel point.

[0154] In an embodiment of the present application, the material texture features of each candidate material image can be calculated by taking each pixel point of each candidate material image as the center of a circle and the radius threshold as the radius, traversing the second associated pixel points corresponding to each pixel point within the radius range in each candidate material image, and calculating the local feature value corresponding to each pixel point based on the pixel value of each pixel point and the pixel value of the second associated pixel point corresponding to each pixel point. Then, the material texture features of each candidate material image can be obtained based on the local feature value of each pixel point.

[0155] In one possible implementation, the material texture features of each candidate material image can be calculated by taking each pixel point of each candidate material image (for example, any pixel point K) as the center of a circle and a radius threshold (for example, a threshold R) as the radius, and traversing the pixel points in each candidate material image that appear within the range of K as the center and R as the radius as the second associated pixel points.

[0156] Next, for a certain pixel point K in each candidate material image, the local binary pattern feature algorithm shown in the above formula (4) can also be used to simply and efficiently calculate the texture feature corresponding to each pixel point, that is, the local eigenvalue.

[0157] It can be understood that when the above formula (4) is currently used, δ is the activation pulse function, the current K is the pixel point in each candidate material image, the current Q is the number of pixels within the circle with a radius of R with the current pixel point K as the center, and the current f(·) is the pixel value of the pixel point on each candidate material image.

[0158] After obtaining the local feature value of each pixel on each candidate material image, the local feature value of each pixel can be sorted according to the position of each pixel on each candidate material image to obtain the material texture feature of each candidate material image.

[0159] In an embodiment of the present application, for each pixel point in a candidate material image, the local eigenvalue of the pixel point can be determined by combining the pixel value of the pixel point and the pixel value of the second associated pixel point of the pixel point, taking into account the influence of the surrounding pixels of the pixel point on the local eigenvalue of the pixel point, thereby better characterizing the texture characteristics in the image block to be processed and better describing the characteristics of the object.

[0160] Based on the embodiment corresponding to FIG2 above, in another optional embodiment of the material extraction method provided by the embodiment of the present application, as shown in FIG12 , before S101 performs segmentation processing on the image to be processed to obtain an object segmentation region and divides the object segmentation region into N image blocks to be processed, the method further includes: S1202 in S1201; S101 includes: S1203;

[0161] In S1201, data enhancement processing is performed on the sample images in the training sample image set to obtain an enhanced extended sample image set, wherein each sample image in the extended sample image set has a corresponding segmentation annotation label;

[0162] In S1202, each sample image in the expanded sample image set is used as input data of the segmentation model, and the segmentation annotation label is used as result data of the segmentation model to update the model parameters of the segmentation model;

[0163] In S1203 , the image to be processed is input into the trained segmentation model, the trained segmentation model outputs an object segmentation region, and the object segmentation region is divided into N image blocks to be processed.

[0164] In this embodiment, in order to better obtain the object segmentation area, this embodiment can perform data enhancement processing on the sample images in the training sample image set to obtain an extended sample image set after enhancement processing. Then, each sample image in the extended sample image set can be used as input data of the segmentation model, and the segmentation annotation label can be used as the result data of the segmentation model to update the model parameters of the segmentation model to obtain a trained segmentation model, so that the image to be processed can be subsequently input into the trained segmentation model, and the object segmentation area can be output through the trained segmentation model, and the object segmentation area can be divided into N image blocks to be processed.

[0165] Among them, the wall extraction module shown in Figure 14 can be used to segment the wall area from the input image to be processed, that is, the object segmentation area, wherein the segmentation model can be an AI neural network-based segmentation model as shown in Figure 14, for example, a fully convolutional neural network U-Net model for image segmentation, a fully convolutional network (Fully Convolutional Networks, FCN), etc., and no specific restrictions are made here.

[0166] Among them, the segmentation annotation label is the annotation label corresponding to each sample image in the extended sample image set. In a series of image-annotation pairs, the image in each image-annotation pair is the sample image (for example, a building image), and the annotation is a segmentation mask image of the same size as the image. The corresponding value can be expressed as 0 or 1, where 0 indicates that the sample image is the background and 1 indicates that the sample image is the building wall.

[0167] It is understandable that the training sample image set can be derived from the labeled data of crispy chicken (for example, pixel-level labeled data of the wall), which can be obtained from public data sources such as the Creative Management Platform (CMP), the distributed database RueMonge, the Enterprise Cloud Platform (ECP) or the Enterprise Trust Intranet Manager Solutions (eTRIMS), etc., that is, these public data can be aggregated to construct a wall segmentation dataset with a rich series of image-segmentation annotation label pairs, that is, the training sample image set.

[0168] In one possible implementation, in order to make the training sample image set more diverse, to help the segmentation model better learn the ability to distinguish between the background and the target object, to enhance the generalization of the segmentation model, and to enable the segmentation model to better cope with the diverse and complex shooting conditions in actual application scenarios, this embodiment can perform data enhancement of multiple strategies on each training image in the training sample image set, including random color transformation, image contrast transformation, exposure transformation, and random rotation, etc., to obtain an extended sample image set after enhancement, and match the corresponding segmentation annotation label for each sample image in the extended sample image set in an image-annotation pair manner.

[0169] In an embodiment of the present application, each sample image in the extended sample image set can be used as input data of the segmentation model, and the predicted segmentation result corresponding to each sample image is output by the segmentation model. Then, the segmentation annotation label can be used as the result data of the segmentation model, that is, the loss calculation (for example, cross entropy or information entropy, etc.) can be performed based on the predicted segmentation result corresponding to each sample image and the segmentation annotation label corresponding to each sample image, and the model parameters of the segmentation model can be updated based on the calculated loss value to obtain a trained segmentation model.

[0170] After obtaining the trained segmentation model, the image to be processed can be input into the trained segmentation model, the trained segmentation model outputs the object segmentation region, and the object segmentation region is divided into N image blocks to be processed.

[0171] The embodiment of the present application performs segmentation processing through a trained segmentation model, and can obtain N image blocks to be processed more quickly and accurately.

[0172] Based on the embodiment corresponding to FIG. 2 above, in another optional embodiment of the material extraction method provided by the embodiment of the present application, as shown in FIG. 13 , in step S103, texture matching is performed on the object image block with each candidate material image in the candidate material image set. Before obtaining the texture matching result corresponding to each candidate material image, the method further includes:

[0173] In S1301, based on the material properties of each original material, each original material is arranged, reorganized, and angle-converted to generate an adjusted material image;

[0174] In S1302 , each adjusted material image is resized based on the size of the target image block to obtain a candidate material image, thereby acquiring a candidate material image set.

[0175] In an embodiment of the present application, in order to better perform texture matching between the object image block and each candidate material image in the candidate material image set, before performing texture matching between the object image block and each candidate material image in the candidate material image set, this embodiment can first perform material adjustment on the original material, that is, based on the material properties of each original material, each original material is arranged, reorganized, angle converted, and other processing is performed to generate an adjusted material image, and then based on the size of the object image block, each adjusted material image is resized to obtain the candidate material image corresponding to each adjusted material image, so as to obtain a candidate material image set.

[0176] Taking the object as an example, since the size and direction of the wall units in the building are usually different in actual application scenarios (for example, two walls are both brick walls, but the size and arrangement direction of the bricks may be different), this embodiment can first use the material adjustment function in the Unreal Engine UE to arrange and reorganize each original material based on the material properties of each original material, and convert each original material at different angles (that is, transform the material attribute parameters) to generate several material images with different UV sizes and different rotation angles, that is, adjust the material image.

[0177] In order to better perform texture matching between the object image block and each candidate material image in the candidate material image set, the size of each adjusted material image can be adjusted to an image with the same size as the object image block based on the size of the object image block, that is, the candidate material image, so that the candidate material image set can be summarized.

[0178] By the above method, a candidate material image that is adapted to different arrangements and angles and has the same size as the object image block can be obtained, so that the object image block can be better matched with the candidate material image, thereby improving matching accuracy and matching efficiency.

[0179] The material extraction device in the present application is described in detail below. Please refer to FIG16 , which is a schematic diagram of an embodiment of the material extraction device in the present application. The material extraction device 20 includes:

[0180] An acquisition unit 201 is configured to segment the image to be processed to obtain an object segmentation region, and divide the object segmentation region into N image blocks to be processed, wherein the object segmentation region is used to represent a non-background region in the image to be processed, and N is an integer greater than 1;

[0181] a determining unit 202 configured to calculate feature mutual information between every two image blocks to be processed in the N image blocks to be processed, and determine the target image block from the N image blocks to be processed based on the feature mutual information;

[0182] The processing unit 203 is configured to extract material based on the object image block to obtain the material of the object included in the object image block.

[0183] In a possible implementation, the processing unit 203 is configured to perform texture matching on the object image block and each candidate material image in the candidate material image set, respectively, to obtain a texture matching result corresponding to each candidate material image;

[0184] Based on the texture matching result corresponding to each candidate material image, an object material image is determined from the candidate material image set, where the object material image is used to represent the material of the object included in the object image block.

[0185] In a possible implementation, based on the embodiment corresponding to FIG. 16 above, in another embodiment of the material extraction device provided in the embodiment of the present application,

[0186] The processing unit 203 is further configured to perform recoloring processing on the object material image based on the color value of the object image block to obtain a recolored material image;

[0187] The processing unit 203 is further configured to render the virtual model corresponding to the image to be processed based on the recolored material image to obtain a final virtual model.

[0188] In one possible implementation, based on the embodiment corresponding to FIG. 16 , in another embodiment of the material extraction device provided in the embodiment of the present application, the determining unit 202 may be specifically configured to:

[0189] Calculating the texture features of each image block to be processed based on the pixel value of each pixel point in each image block to be processed;

[0190] Based on the texture features of each image block to be processed, respectively calculate the sum of the texture feature similarities between each image block to be processed and the other N-1 image blocks to be processed;

[0191] The image block to be processed corresponding to the maximum sum value is taken as the object image block.

[0192] In one possible implementation, based on the embodiment corresponding to FIG. 16 , in another embodiment of the material extraction device provided in the embodiment of the present application, the determining unit 202 may be specifically configured to:

[0193] Smoothing the object image block to obtain a color value corresponding to each pixel in the object image block, and calculating a color average of the object image block based on the color value corresponding to each pixel in the object image block;

[0194] Based on the color average value of the object image block, determine the color grayscale particle to be used from M randomly initialized color grayscale particles, where M is an integer greater than or equal to 1;

[0195] Multiply the color grayscale value of the color grayscale particle to be used by the color value of the object material image to obtain a recolored material image.

[0196] In one possible implementation, based on the embodiment corresponding to FIG. 16 , in another embodiment of the material extraction device provided in the embodiment of the present application, the determining unit 202 may be specifically configured to:

[0197] Get the current color grayscale value of each color grayscale particle at the current moment;

[0198] Calculate the square error between the color average value of the object image block and the current color grayscale value of each color grayscale particle respectively, and obtain the current moment error of each color grayscale particle at the current moment;

[0199] If the error at the current moment with the smallest value is smaller than the error threshold, the color grayscale particle corresponding to the error at the current moment with the smallest value is determined as the color grayscale particle to be used.

[0200] In a possible implementation, based on the embodiment corresponding to FIG. 16 above, in another embodiment of the material extraction device provided in the embodiment of the present application,

[0201] The processing unit 203 is further configured to update the color grayscale value of each color grayscale particle at the next moment based on the error at the current moment with the smallest value, if the error at the current moment with the smallest value is greater than the error threshold, to obtain an updated color grayscale value of each color grayscale particle;

[0202] The processing unit 203 is further configured to calculate the square error between the color average value of the object image block and the updated color grayscale value of each color grayscale particle, and obtain the next moment error of each color grayscale particle at the next moment;

[0203] The determining unit 202 is further configured to determine the color grayscale particle corresponding to the next moment error with the smallest value as the color grayscale particle to be used if the next moment error with the smallest value is smaller than the error threshold.

[0204] In one possible implementation, based on the embodiment corresponding to FIG. 16 , in another embodiment of the material extraction device provided in the embodiment of the present application, the processing unit 203 may be specifically configured to:

[0205] Taking each pixel point of each image block to be processed as the center of a circle and a radius threshold as the radius, traversing the first associated pixel point of each pixel point within the radius range in the image block to be processed;

[0206] Calculating a local feature value of each pixel based on the pixel value of each pixel and the pixel value of a first associated pixel of each pixel;

[0207] The texture features of each image block to be processed are obtained based on the local eigenvalues ​​of each pixel.

[0208] In one possible implementation, based on the embodiment corresponding to FIG. 16 , in another embodiment of the material extraction device provided in the embodiment of the present application, the processing unit 203 may be specifically configured to:

[0209] Based on the texture features of each image block to be processed, the texture cosine similarity between each image block to be processed and the other N-1 image blocks to be processed is calculated respectively;

[0210] Based on the texture features of each image block to be processed, the texture structure similarity between each image block to be processed and the other N-1 image blocks to be processed is calculated respectively;

[0211] The texture cosine similarity and texture structure similarity between each image block to be processed and the other N-1 image blocks to be processed are summed to obtain the sum of texture feature similarities.

[0212] In one possible implementation, based on the embodiment corresponding to FIG. 16 , in another embodiment of the material extraction device provided in the embodiment of the present application, the processing unit 203 may be specifically configured to:

[0213] Calculate the material texture features of each candidate material image based on the pixel value of each pixel in each candidate material image;

[0214] Based on the texture features of the object image block and the material texture features of each candidate material image, the texture feature similarity between the object image block and each candidate material image is calculated respectively, and the texture feature similarity is used as the texture matching result corresponding to each candidate material image.

[0215] In one possible implementation, based on the embodiment corresponding to FIG. 16 , in another embodiment of the material extraction device provided in the embodiment of the present application, the processing unit 203 may be specifically configured to:

[0216] Taking each pixel point of each candidate material image as the center of the circle and the radius threshold as the radius, traversing the second associated pixel point of each pixel point within the radius range in each candidate material image;

[0217] Calculating a local feature value of each pixel based on a pixel value of each pixel and a pixel value of a second associated pixel of each pixel;

[0218] The texture features of each candidate material image are obtained based on the local eigenvalue of each pixel.

[0219] In a possible implementation, based on the embodiment corresponding to FIG. 16 above, in another embodiment of the material extraction device provided in the embodiment of the present application,

[0220] The processing unit 203 is further configured to perform data enhancement processing on the sample images in the training sample image set to obtain an extended sample image set after the enhancement processing, wherein each sample image in the extended sample image set has a corresponding segmentation annotation label;

[0221] The processing unit 203 is further configured to use each sample image in the expanded sample image set as input data of the segmentation model, and use the segmentation annotation label as result data of the segmentation model to update the model parameters of the segmentation model;

[0222] The acquisition unit 201 may be specifically configured to:

[0223] The image to be processed is input into the trained segmentation model, the trained segmentation model outputs the object segmentation area, and the object segmentation area is divided into N image blocks to be processed.

[0224] In a possible implementation, based on the embodiment corresponding to FIG. 16 above, in another embodiment of the material extraction device provided in the embodiment of the present application,

[0225] The acquisition unit 201 is further configured to perform permutation and reorganization, and angle conversion processing on each original material based on the material properties of each original material, to generate an adjusted material image;

[0226] The processing unit 203 is further configured to resize each adjusted material image based on the size of the object image block to obtain a candidate material image, thereby obtaining a candidate material image set.

[0227] On the other hand, the present application provides another schematic diagram of a computer device, as shown in Figure 17. Figure 17 is a schematic diagram of the structure of a computer device provided in an embodiment of the present application. The computer device 300 may have relatively large differences due to different configurations or performances, and may include one or more central processing units (CPU) 310 (for example, one or more processors) and memory 320, and one or more storage media 330 (for example, one or more mass storage devices) storing application programs 331 or data 332. Among them, the memory 320 and the storage medium 330 can be temporary storage or permanent storage. The program stored in the storage medium 330 may include one or more modules (not shown in the figure), each module may include a series of instruction operations in the computer device 300. Furthermore, the central processing unit 310 can be configured to communicate with the storage medium 330 to execute a series of instruction operations in the storage medium 330 on the computer device 300.

[0228] The computer device 300 may also include one or more power supplies 340, one or more wired or wireless network interfaces 350, one or more input and output interfaces 360, and / or one or more operating systems 333, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, etc.

[0229] The computer device 300 is also used to execute the steps in the embodiments corresponding to FIG. 2 to FIG. 13 .

[0230] Another aspect of the present application provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps in the embodiments corresponding to FIG. 2 to FIG. 13 are implemented.

[0231] Another aspect of the present application provides a computer program product comprising a computer program, which, when executed by a processor, implements the steps in the embodiments corresponding to FIG. 2 to FIG. 13 .

[0232] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0233] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0234] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0235] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0236] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

Claims

1. A material extraction method, the method being executed by a computer device, comprising: Segmenting the image to be processed to obtain an object segmentation region, and dividing the object segmentation region into N image blocks to be processed, wherein the object segmentation region is used to represent a non-background region in the image to be processed, and N is an integer greater than 1; respectively calculating feature mutual information between every two of the N image blocks to be processed, and determining a target image block from the N image blocks to be processed based on the feature mutual information; Material extraction is performed based on the object image block to obtain the material of the object included in the object image block.

2. The method according to claim 1, wherein the step of extracting material based on the object image block to obtain the material of the object included in the object image block comprises: Performing texture matching on the object image block and each candidate material image in the candidate material image set, respectively, to obtain a texture matching result corresponding to each candidate material image; Based on the texture matching result corresponding to each candidate material image, an object material image is determined from the candidate material image set, and the object material image is used to represent the material of the object included in the object image block.

3. The method according to claim 2, after determining the object material image from the candidate material image set based on the texture matching result corresponding to each candidate material image, the method further comprises: Based on the color value of the object image block, recoloring the object material image to obtain a recolored material image; Based on the recolored material image, the virtual model corresponding to the image to be processed is rendered to obtain a final virtual model.

4. The method according to claim 1, wherein the feature mutual information comprises texture feature similarity; and the step of respectively calculating the feature mutual information between every two of the N image blocks to be processed, and determining the object image block from the N image blocks to be processed based on the feature mutual information, comprises: Calculating the texture features of each of the image blocks to be processed based on the pixel value of each pixel in each of the image blocks to be processed; Based on the texture features of each of the image blocks to be processed, respectively calculating the sum of the texture feature similarities between each of the image blocks to be processed and other N-1 image blocks to be processed; The image block to be processed corresponding to the maximum sum value is taken as the object image block.

5. The method according to claim 3, wherein the recoloring of the object material image based on the color of the object image block to obtain the recolored material image comprises: Performing a smoothing process on the object image block to obtain a color value corresponding to each pixel in the object image block, and calculating a color average value of the object image block based on the color value corresponding to each pixel in the object image block; Based on the color average value of the object image block, determine the color grayscale particle to be used from M color grayscale particles that are randomly initialized, where M is an integer greater than or equal to 1; The color grayscale value of the color grayscale particle to be used is multiplied by the color value of the object material image to obtain the recolored material image.

6. The method according to claim 5, wherein the step of determining the color grayscale particle to be used from the M color grayscale particles randomly initialized based on the color average value of the object image block comprises: Get the current color grayscale value of each color grayscale particle at the current moment; Calculate the square error between the color average value of the object image block and the current color grayscale value of each color grayscale particle respectively, and obtain the current moment error of each color grayscale particle at the current moment; If the current moment error with the smallest value is smaller than the error threshold, the color grayscale particle corresponding to the current moment error with the smallest value is determined as the color grayscale particle to be used.

7. The method according to claim 6, wherein after respectively calculating the square error between the color average value of the object image block and the initial color grayscale value of each color grayscale particle to obtain the current moment error of each color grayscale particle at the current moment, the method further comprises: If the error at the current moment with the smallest value is greater than the error threshold, then based on the error at the current moment with the smallest value, the color grayscale value of each color grayscale particle at the next moment is updated to obtain an updated color grayscale value of each color grayscale particle; Calculating the square error between the color average value of the object image block and the updated color grayscale value of each color grayscale particle respectively, to obtain the error of each color grayscale particle at the next moment; If the error at the next moment with the smallest value is smaller than the error threshold, the color grayscale particle corresponding to the error at the next moment with the smallest value is determined as the color grayscale particle to be used.

8. The method according to claim 4, wherein the calculating the texture feature of each of the image blocks to be processed based on the pixel value of each pixel in each of the image blocks to be processed comprises: Taking each pixel point of each of the to-be-processed image blocks as the center of a circle and a radius threshold as a radius, traversing the first associated pixel point of each pixel point within the radius range in the to-be-processed image block; Calculating a local feature value of each pixel based on a pixel value of each pixel and a pixel value of a first associated pixel of each pixel; The texture features of each image block to be processed are obtained based on the local feature value of each pixel point.

9. The method according to claim 4, wherein the texture feature similarity comprises texture cosine similarity and texture structure similarity; and the step of calculating the sum of texture feature similarities between each image block to be processed and other N-1 image blocks to be processed based on the texture features of each image block to be processed comprises: Based on the texture features of each of the image blocks to be processed, respectively calculating the texture cosine similarity between each of the image blocks to be processed and other N-1 image blocks to be processed; Based on the texture features of each of the image blocks to be processed, respectively calculating the texture structure similarity between each of the image blocks to be processed and other N-1 image blocks to be processed; The texture cosine similarity and texture structure similarity between each of the to-be-processed image blocks and other N-1 to-be-processed image blocks are summed to obtain the sum of the texture feature similarities.

10. The method according to claim 2, wherein the step of performing texture matching on the object image block and each candidate material image in the candidate material image set to obtain a texture matching result corresponding to each candidate material image comprises: Calculating the material texture features of each candidate material image based on the pixel value of each pixel in each candidate material image; Based on the texture features of the object image block and the material texture features of each candidate material image, the texture feature similarity between the object image block and each candidate material image is calculated respectively, and the texture feature similarity is used as the texture matching result corresponding to each candidate material image.

11. The method according to claim 10, wherein the calculating the material texture feature of each candidate material image based on the pixel value of each pixel in each candidate material image comprises: Taking each pixel point of each candidate material image as the center of a circle and a radius threshold as the radius, traversing the second associated pixel point of each pixel point within the radius range in each candidate material image; Calculating a local feature value of each pixel based on a pixel value of each pixel and a pixel value of a second associated pixel of each pixel; The texture features of each candidate material image are obtained based on the local feature value of each pixel.

12. The method according to claim 1, before segmenting the image to be processed to obtain an object segmentation region and dividing the object segmentation region into N image blocks to be processed, the method further comprises: Performing data enhancement processing on sample images in the training sample image set to obtain an extended sample image set after the enhancement processing, wherein each sample image in the extended sample image set has a corresponding segmentation annotation label; Using each sample image in the extended sample image set as input data of a segmentation model, and using the segmentation annotation label as result data of the segmentation model, to update model parameters of the segmentation model; The segmenting process of the image to be processed to obtain an object segmentation region, and dividing the object segmentation region into N image blocks to be processed includes: The image to be processed is input into a trained segmentation model, the object segmentation region is output through the trained segmentation model, and the object segmentation region is divided into N image blocks to be processed.

13. The method according to claim 2, before performing texture matching on the object image block and each candidate material image in the candidate material image set to obtain a texture matching result corresponding to each candidate material image, the method further comprises: Based on the material properties of each original material, each original material is arranged, reorganized, and angle-converted to generate an adjusted material image; Based on the size of the object image block, each of the adjusted material images is resized to obtain the candidate material image, so as to obtain the candidate material image set.

14. A material extraction device, the device being deployed on a computer device, comprising: An acquisition unit, configured to segment the image to be processed to obtain an object segmentation region, and divide the object segmentation region into N image blocks to be processed, wherein the object segmentation region is used to represent a non-background region in the image to be processed, and N is an integer greater than 1; a determining unit, configured to respectively calculate feature mutual information between every two of the N image blocks to be processed, and determine a target image block from the N image blocks to be processed based on the feature mutual information; The processing unit is used to extract material based on the object image block to obtain the material of the object included in the object image block.

15. A computer device, comprising a memory, a processor and a bus system, wherein the memory stores a computer program, and the processor implements the steps of any one of the methods of claims 1 to 13 when executing the computer program; The bus system is used to connect the memory and the processor so that the memory and the processor can communicate with each other.

16. A computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the method according to any one of claims 1 to 13.

17. A computer program product, comprising a computer program, which, when executed by a processor, implements the steps of the method according to any one of claims 1 to 13.

Images