Method, device and equipment for generating material parameter identification model and storage medium

Abstract

The present disclosure provides a method, device and equipment for generating a material parameter identification model. The method comprises the following steps: obtaining a target image pair, a target text and a material parameter label of a first object, wherein the target image pair comprises a first image and a second image, the second image is an image obtained by projecting the first image on a surface of the first object, the target text comprises an environmental parameter, and the material parameter label of the first object is used to represent parameters of a material on the surface of the first object; and training the target image pair, the target text and the material parameter label of the first object to obtain the material parameter identification model. The method can realize multi-modal model design based on a feature map of the target image pair and the target text, enrich training samples, improve the recognition accuracy of the material parameter identification model, and accurately analyze material parameters based on the material parameter identification model.

Description

Technical Field

[0001] This disclosure relates to the field of computer science, specifically to the fields of computer vision and material recognition technology, and particularly to a method, apparatus, device, and storage medium for generating a material parameter recognition model. Background Technology

[0002] Material parameters are used to characterize various properties of materials, and their applications are wide-ranging. If accurate material parameters are obtained through analysis, the material's properties can be maximized.

[0003] Currently, material analysis primarily focuses on classifying materials. While material parameter analysis exists, research using computer vision for this purpose is limited, making it difficult to guarantee accurate results. Furthermore, different material categories, and even materials within the same category, can possess different parameters; therefore, a method for accurately identifying material parameters is urgently needed. Summary of the Invention

[0004] This disclosure provides a method, apparatus, device, and storage medium for generating a material parameter identification model.

[0005] In a first aspect, embodiments of this disclosure propose a method for generating a material parameter recognition model, comprising: acquiring a target image pair, target text, and material parameter labels of a first object, wherein the target image pair includes a first image and a second image, the second image is an image obtained by projecting the first image onto the surface of the first object, the target text includes environmental parameters, and the material parameter labels of the first object are used to characterize the parameters of the material to which the surface of the first object belongs; and training based on the target image pair, the target text, and the material parameter labels of the first object to obtain a material parameter recognition model.

[0006] In some embodiments, a material parameter recognition model is trained based on a target image pair, target text, and material parameter labels of a first object, including: encoding the target image pair and target text respectively to obtain feature maps of the target image pair and the target text; fusing the feature maps of the target image pair and the target text to obtain a fused feature map; and training based on the fused feature map and the material parameter labels of the first object to obtain the material parameter recognition model.

[0007] In some embodiments, fusing the feature maps of the target image pair and the target text to obtain a fused feature map includes: fusing the feature maps of the target image pair and the target text using an attention mechanism to obtain a fused feature map.

[0008] In some embodiments, the target text further includes: the material category of the first object.

[0009] In some embodiments, an attention mechanism is used to fuse the feature maps of the target image pair and the target text to obtain a fused feature map, including:

[0010] The feature maps of the first image and the second image are fused to obtain the first feature map; and the feature maps of the environmental parameters and the material category of the first object are fused to obtain the second feature map.

[0011] A cross-attention mechanism is used to perform feature crossing on the first feature map and the second feature map to obtain a first cross-feature map and a second cross-feature map.

[0012] The first cross feature map and the second cross feature map are merged to obtain the fused feature map.

[0013] In some embodiments, a first feature map is obtained by fusing feature maps of a first image and a second image; and a second feature map is obtained by fusing feature maps of environmental parameters and feature maps of the material category of a first object, including:

[0014] The feature maps of the first image and the second image are fused to obtain a first fused feature map; the first fused feature map is processed using a self-attention mechanism to obtain a first feature map; and the feature maps of environmental parameters and the material category of the first object are fused to obtain a second fused feature map; the second fused feature map is processed using a self-attention mechanism to obtain a second feature map.

[0015] And / or,

[0016] The first cross feature map and the second cross feature map are fused to obtain a fused feature map, including: fusing the first cross feature map and the second cross feature map to obtain a third fused feature map; the third fused feature map is processed using a self-attention mechanism to obtain the fused feature map.

[0017] In some embodiments, the material category of the first object is determined based on the following steps: projecting a preset light onto the surface of the first object using a projection device and acquiring a third image using a photographing device; inputting the third image into a preset material classification model to obtain the material category of the first object.

[0018] In some embodiments, inputting a third image into a preset material classification model to obtain the material category of a first object includes: obtaining a fourth image from the third image according to a preset masking region, wherein the fourth image includes an image of the first object; and inputting the fourth image into the preset material classification model to obtain the material category of the first object.

[0019] In some embodiments, the masking region is determined based on the following steps: projecting a first image onto the surface of a first object using a projection device and obtaining a second image using a shooting device; segmenting the second image to obtain a segmented image, wherein the segmented image is an image including the first object; comparing the segmented image with the first image to determine the projection region; and generating a masking region based on the projection region.

[0020] In some embodiments, environmental parameters are determined based on the following steps: acquiring environmental parameters via sensors.

[0021] Secondly, embodiments of this disclosure propose a method for identifying material parameters, comprising: acquiring a first image pair, environmental parameters of a second object in the first image pair, and the material category of the second object, wherein the first image pair includes a third image and a fourth image, and the fourth image is an image obtained by projecting the third image onto the surface of the second object; and inputting the first image pair, the environmental parameters of the second object, and the material category of the second object into a pre-trained material parameter identification model to obtain the material parameters of the second object.

[0022] In some embodiments, the method for identifying the material parameters further includes: determining the material loss of the second object based on the material parameters of the second object.

[0023] In some embodiments, the method for identifying the material parameters further includes: determining display parameters based on the material parameters of the second object;

[0024] Based on the display parameters, the image to be enhanced is optimized for display.

[0025] In some embodiments, the image to be enhanced includes: an image projected onto the surface of a second object; or, a virtual reality image in virtual reality.

[0026] Thirdly, embodiments of this disclosure propose an apparatus for generating a material parameter recognition model, comprising: an image acquisition module configured to acquire a target image pair, target text, and a material parameter label of a first object, wherein the target image pair includes a first image and a second image, the second image is an image obtained by projecting the first image onto the surface of the first object, the target text includes environmental parameters, and the material parameter label of the first object is used to characterize the parameters of the material to which the surface of the first object belongs; and a model training module configured to train the model based on the target image pair, the target text, and the material parameter label of the first object to obtain a material parameter recognition model.

[0027] In some embodiments, the model training module includes:

[0028] The encoding unit is configured to encode the target image pair and the target text respectively, to obtain the feature map of the target image pair and the feature map of the target text;

[0029] The fusion unit is configured to fuse the feature maps of the target image pair and the feature maps of the target text to obtain a fused feature map.

[0030] The training unit is configured to be trained based on the fused feature map and the material parameter labels of the first object to obtain the material parameter recognition model.

[0031] In some embodiments, the fusion unit is further configured to: fuse the feature maps of the target image pair and the feature maps of the target text using an attention mechanism to obtain a fused feature map.

[0032] In some embodiments, the target text further includes: the material category of the first object.

[0033] In some embodiments, the fusion unit includes:

[0034] The first fusion subunit is configured to fuse the feature map of the first image and the feature map of the second image to obtain a first feature map; and to fuse the feature map of the environmental parameters and the feature map of the material category of the first object to obtain a second feature map;

[0035] The feature crossing subunit is configured to perform feature crossing on the first feature map and the second feature map using a cross attention mechanism to obtain the first cross feature map and the second cross feature map.

[0036] The second fusion subunit is configured to fuse the first cross feature map and the second cross feature map to obtain a fused feature map.

[0037] In some embodiments, the first fusion subunit is further configured to:

[0038] The feature maps of the first image and the second image are fused to obtain a first fused feature map; the first fused feature map is processed using a self-attention mechanism to obtain a first feature map; and the feature maps of environmental parameters and the material category of the first object are fused to obtain a second fused feature map; the second fused feature map is processed using a self-attention mechanism to obtain a second feature map.

[0039] And / or,

[0040] The second fusion subunit is further configured to: fuse the first cross feature map and the second cross feature map to obtain a fused feature map, including: fusing the first cross feature map and the second cross feature map to obtain a third fused feature map; and using a self-attention mechanism to process the third fused feature map to obtain a fused feature map.

[0041] In some embodiments, the apparatus for generating a material parameter identification model further includes:

[0042] The image acquisition module is configured to project a preset light onto the surface of a first object using a projection device, and to acquire a third image using a shooting device.

[0043] The category recognition module is configured to input the third image into a preset material classification model to obtain the material category of the first object.

[0044] In some embodiments, the category recognition module is further configured to: obtain a fourth image from a third image based on a preset masking area, wherein the fourth image includes an image of the first object; input the fourth image into a preset material classification model to obtain the material category of the first object.

[0045] In some embodiments, the apparatus for generating a material parameter identification model further includes:

[0046] The image acquisition module is configured to project a first image onto the surface of a first object using a projection device, and to acquire a second image using a shooting device.

[0047] The image segmentation module is configured to segment the second image to obtain a segmented image, wherein the segmented image is an image including the first object;

[0048] The image comparison module is configured to compare the segmented image with the first image to determine the projection area;

[0049] The region generation module is configured to generate a masking region based on the projection region.

[0050] In some embodiments, the apparatus for generating a material parameter identification model further includes a parameter acquisition module configured to acquire environmental parameters via sensors.

[0051] Fourthly, embodiments of this disclosure propose a material parameter identification device, comprising: an image acquisition module configured to acquire a first image pair, environmental parameters of a second object in the first image pair, and the material category of the second object, the first image pair including a third image and a fourth image, the fourth image being an image obtained by projecting the third image onto the surface of the second object; and a parameter identification module configured to input the first image pair, the environmental parameters of the second object, and the material category of the second object into a pre-trained material parameter identification model to obtain the material parameters of the second object.

[0052] In some embodiments, the material parameter identification device further includes a loss determination module configured to determine the material loss of the second object based on the material parameters of the second object.

[0053] In some embodiments, the material parameter identification device further includes: a display parameter determination module configured to determine display parameters based on the material parameters of the second object; and an optimization adjustment module configured to optimize the display of the image to be enhanced based on the display parameters.

[0054] In some embodiments, the image to be enhanced includes: an image projected onto the surface of a second object; or, a virtual reality image in virtual reality.

[0055] Fifthly, embodiments of this disclosure provide an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the methods described in the first or second aspect.

[0056] In a sixth aspect, embodiments of this disclosure provide a non-transitory computer-readable storage medium storing computer instructions for causing a computer to perform the methods described in the first or second aspect.

[0057] The method, apparatus, device, and storage medium for generating a material parameter recognition model provided in this disclosure first acquire a target image pair, target text, and material parameter labels of a first object. The target image pair includes a first image and a second image, the second image being an image obtained by projecting the first image onto the surface of the first object. The target text includes environmental parameters, and the material parameter labels of the first object characterize the parameters of the material to which the surface of the first object belongs. Then, the model is trained based on the target image pair, target text, and material parameter labels of the first object to obtain a material parameter recognition model. This allows for multimodal model design based on feature maps of the target image pair and target text, enriching the training samples and improving the recognition accuracy of the material parameter recognition model, thereby enabling accurate analysis of material parameters based on the model.

[0058] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0059] Other features, objects, and advantages of this disclosure will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings. The drawings are provided for a better understanding of the invention and are not intended to limit the scope of this disclosure. Wherein:

[0060] Figure 1 This is an exemplary system architecture diagram to which this disclosure can be applied;

[0061] Figure 2 This is a flowchart of an embodiment of a method for generating a material parameter identification model according to the present disclosure;

[0062] Figure 3 This is a flowchart of an embodiment of a method for generating a material parameter identification model according to the present disclosure;

[0063] Figure 4 This is a flowchart of an embodiment of a method for generating a material parameter identification model according to the present disclosure;

[0064] Figure 5 This is a schematic diagram for identifying the material category of the first object;

[0065] Figure 6 This is a schematic diagram of the generated projection area;

[0066] Figure 7 This is a schematic diagram of acquiring the projected content;

[0067] Figure 8 This is a flowchart of an embodiment of a method for generating a material parameter identification model according to the present disclosure;

[0068] Figure 9 This is a schematic diagram of the material parameter identification model;

[0069] Figure 10 This is a flowchart of an embodiment of the material parameter identification method according to the present disclosure;

[0070] Figure 11 This is an application scenario diagram of the material parameter identification method according to this disclosure;

[0071] Figure 12 This is an application scenario diagram of the material parameter identification method according to this disclosure;

[0072] Figure 13 This is an application scenario diagram of the material parameter identification method according to this disclosure;

[0073] Figure 14 This is an application scenario diagram of the material parameter identification method according to this disclosure;

[0074] Figure 15 This is a schematic diagram of one embodiment of the apparatus for generating a material parameter identification model according to the present disclosure;

[0075] Figure 16 This is a schematic diagram of a structure of an embodiment of a material parameter identification device according to the present disclosure;

[0076] Figure 17This is a block diagram of an electronic device used to implement the method for generating a material parameter identification model or a method for identifying material parameters in accordance with the embodiments of this disclosure. Detailed Implementation

[0077] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0078] It should be noted that, unless otherwise specified, the embodiments and features described in this disclosure can be combined with each other. This disclosure will now be described in detail with reference to the accompanying drawings and embodiments.

[0079] Figure 1 An exemplary system architecture 100 is shown, illustrating an embodiment of a method and apparatus for generating a material parameter identification model or a method or apparatus for identifying material parameters that can be applied according to the present disclosure.

[0080] like Figure 1 As shown, system architecture 100 may include terminal devices 101 and 102, network 103, and server 104. Network 103 serves as the medium for providing communication links between terminal devices 101 and 102 and server 104. Network 103 may include various connection types, such as wired or wireless communication links or fiber optic cables, etc.

[0081] Users can use terminal devices 101 and 102 to interact with server 104 via network 103 to receive or send target image pairs, material parameters of the first object, etc. Various client applications, such as camera software, can be installed on terminal devices 101 and 102.

[0082] Terminal devices 101 and 102 can be either hardware or software. When terminal devices 101 and 102 are hardware, they can be various electronic devices, including but not limited to smartphones, tablets, laptops, and desktop computers. When terminal devices 101 and 102 are software, they can be installed in the aforementioned electronic devices. They can be implemented as multiple software programs or software modules, or as a single software program or software module. No specific limitations are made here.

[0083] Server 104 can provide various services. For example, server 104 can obtain target image pairs, target text, and material parameter labels of the first object from terminal devices 101 and 102, and train a material parameter recognition model.

[0084] It should be noted that server 104 can be either hardware or software. When server 104 is hardware, it can be implemented as a distributed server cluster consisting of multiple servers, or as a single server. When server 104 is software, it can be implemented as multiple software programs or software modules (e.g., used to provide distributed services), or as a single software program or software module. No specific limitations are made here.

[0085] It should be noted that the method for generating a material parameter identification model or the method for identifying material parameters provided in this embodiment of the disclosure is generally executed by the server 104, and correspondingly, the material parameter identification device or the material parameter identification device is generally set in the server 104.

[0086] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0087] Continue to refer to Figure 2 The diagram illustrates a flow 200 of an embodiment of a method for generating a material parameter identification model according to the present disclosure. This method for generating a material parameter identification model may include the following steps:

[0088] Step 201: Obtain the target image pair, target text, and material parameter label of the first object. The target image pair includes a first image and a second image. The second image is an image obtained by projecting the first image onto the surface of the first object. The target text includes environmental parameters. The material parameter label of the first object is used to characterize the parameters of the material to which the surface of the first object belongs.

[0089] In this embodiment, the execution entity of the method for generating a material parameter identification model (e.g.) Figure 1 The server 104 shown can be accessed from a terminal device (e.g., Figure 1 The terminal devices 101 and 102 shown acquire the target image pair, the target text, and the material parameter label of the first object; or, the aforementioned execution entity (e.g., Figure 1 The terminal devices 101 and 102 shown can obtain the target image pair, target text, and material parameter label of the first object locally.

[0090] Here, the material parameter label of the first object can be used to characterize the parameters of the material to which the surface of the first object belongs.

[0091] In this embodiment, the identification of material parameters can be used to determine material loss; after determining the material loss, corresponding maintenance measures can be taken according to the material loss to slow down the rate of material loss; the identification of material parameters can also be used to optimize the images displayed on the material and the images in virtual reality to improve the user experience.

[0092] In one example, the first image can be the image corresponding to the projected image (i.e., the image of the projection source).

[0093] In one example, the second image may be an image obtained by projecting the first image onto the surface of the first object using a projection device.

[0094] Correspondingly, in this example, the first image is projected onto the surface of the first object to obtain the image presented on the surface of the first object, i.e., the second image. For example, the image on the surface of the first object is obtained through an acquisition module in the first object, which can be a module with functions such as image acquisition, scanning, or recording.

[0095] Here, the projection device can be a device with projection function, such as a projector or projector. The surface of the first object can be a plane used to project and present the first image.

[0096] It should be noted that the acquisition module in the first object can be the image acquisition module in the display.

[0097] Correspondingly, in this example, after projecting the first image onto the surface of the first object, a second image can be acquired using an imaging device. This imaging device can be a device with imaging capabilities, such as a camera, video camera, or video recorder. During actual imaging, the device captures both the image presented on the surface of the first object and the background where the first object is located. In subsequent processes, to improve the model's recognition accuracy, the first object can be segmented from the second image.

[0098] It should be noted that the first object can be an object used to present the first image, such as a projection screen, a screen, a monitor, a touch screen, the display of an electronic device, an electronic whiteboard, a wall, etc.

[0099] Here, the target text can be used to characterize all features of the target text, which may include environmental parameters, such as the environmental parameters of the first object, like light parameters, temperature parameters, humidity parameters, etc.

[0100] Step 202: Train the material parameter recognition model based on the target image pair, target text, and material parameter labels of the first object.

[0101] In this embodiment, the execution entity can use the target image pair, target text, and material parameter labels of the first object as training samples to train the initial material parameter recognition model, thereby obtaining the trained material parameter recognition model. This material parameter recognition model can be used to identify material parameters, such as color parameters for various colors, reflectivity, refractive index, gain, and viewing angle.

[0102] It should be noted that the initial parameter identification model can be an untrained model or a model from any stage before the end of training. The end of training can be the time corresponding to the time when the trained material parameter identification model is obtained in step 202.

[0103] The method for generating a material parameter recognition model provided in this disclosure first obtains a target image pair, target text, and material parameter labels for a first object. The target image pair includes a first image and a second image, the second image being an image obtained by projecting the first image onto the surface of the first object. The target text includes environmental parameters, and the material parameter labels for the first object are used to characterize the parameters of the material to which the surface of the first object belongs. Then, the model is trained based on the target image pair, target text, and material parameter labels for the first object to obtain a material parameter recognition model. This method enables multimodal model design based on feature maps of the target image pair and target text, enriches the training samples, and improves the recognition accuracy of the material parameter recognition model, thereby enabling accurate analysis of material parameters based on the material parameter recognition model.

[0104] Continue to refer to Figure 3 The diagram illustrates a flow 300 of an embodiment of a method for generating a material parameter identification model according to the present disclosure. This method for generating a material parameter identification model may include the following steps:

[0105] Step 301: Obtain the target image pair, target text, and material parameter label of the first object. The target image pair includes a first image and a second image. The second image is an image obtained by projecting the first image onto the surface of the first object. The target text includes environmental parameters. The material parameter label of the first object is used to characterize the parameters of the material to which the surface of the first object belongs.

[0106] In this embodiment, the execution entity of the method for generating a material parameter identification model (e.g.) Figure 1 The server 104 shown can be accessed from a terminal device (e.g., Figure 1 The terminal devices 101 and 102 shown acquire the target image pair, the target text, and the material parameter label of the first object; or, the aforementioned execution entity (e.g., Figure 1 The terminal devices 101 and 102 shown can obtain the target image pair, target text, and material parameter label of the first object locally.

[0107] It should be noted that step 301 corresponds to step 201 in the aforementioned embodiment. The specific implementation method can be referred to the aforementioned description of step 201, which will not be repeated here.

[0108] Step 302: Encode the target image pair and the target text respectively to obtain the feature map of the target image pair and the feature map of the target text.

[0109] In this embodiment, the aforementioned execution entity may use image encoding corresponding to the target image pair to extract features from the target image pair in order to obtain a feature map of the target image pair; and use text encoding corresponding to the target text to extract features from the target text in order to obtain a feature map of the target text.

[0110] Here, the feature maps of the target image pair may include the feature maps of the first image and the second image.

[0111] Here, the feature map of the target text can be used to characterize all the features of the target text, which may include environmental parameters, such as the environmental parameters of the first object, like light parameters, temperature parameters, humidity parameters, etc.

[0112] Step 303: Fuse the feature maps of the target image pair and the target text to obtain a fused feature map.

[0113] In this embodiment, the aforementioned execution entity can concatenate the feature maps of the target image pair and the target text to obtain a fused feature map. This fused feature map can be the feature map obtained by fusing the feature maps of the target image pair and the target text.

[0114] In one example, feature fusion can be achieved by fusing features between the feature maps of the target image pair and the feature maps of the target text through a fully connected layer in a pre-defined neural network; or by fusing the feature maps of the target image pair and the feature maps of the target text through an attention mechanism.

[0115] It should be noted that, in this embodiment, any method of fusing feature maps of target image pairs and feature maps of target text can be included within the scope of protection of this disclosure.

[0116] Step 304: Train the material parameter recognition model based on the fused feature map and the material parameter labels of the first object.

[0117] In this embodiment, the aforementioned execution entity can use the fused feature map and the material parameter labels of the first object as training samples to train the initial material parameter recognition model, thereby obtaining the trained material parameter recognition model.

[0118] It should be noted that the initial parameter identification model can be an untrained model or a model from any stage before the end of training. The end of training can be the time corresponding to the time when the trained material parameter identification model is obtained in step 304.

[0119] The method for generating a material parameter recognition model provided in this disclosure first obtains a target image pair, target text, and material parameter labels for a first object. The target image pair includes a first image and a second image, the second image being an image obtained by projecting the first image onto the surface of the first object. The target text includes environmental parameters, and the material parameter labels for the first object characterize the parameters of the material to which the surface of the first object belongs. Next, the target image pair and the target text are encoded separately to obtain feature maps of the target image pair and the target text. Then, the feature maps of the target image pair and the target text are fused to obtain a fused feature map. Finally, the model is trained based on the fused feature map and the material parameter labels for the first object to obtain a material parameter recognition model. This method enables multimodal model design based on the feature maps of the target image pair and the target text, and by fusing features to share features between the feature maps of the target image pair and the target text, it enhances the fitting ability of the material parameter recognition model and improves its recognition accuracy, thereby enabling accurate analysis of material parameters based on the material parameter recognition model.

[0120] In some optional implementations of this embodiment, the environmental parameters are determined based on the following steps: acquiring environmental parameters through sensors.

[0121] In this implementation, the environment parameter can be the environment parameter of the first object, such as the first object placed in darkness or the first object placed under light.

[0122] In one example, environmental parameters can be obtained through various sensors, such as light sensors, humidity sensors, and temperature sensors.

[0123] In this implementation, environmental parameters are acquired through sensors to further analyze the correlation between environmental parameters and the material to which the surface of the first object belongs, so that the accuracy of the data input into the initial material parameter recognition model is very high, thereby improving the recognition accuracy of the material parameter recognition model obtained by training based on the initial material parameter recognition model.

[0124] Continue to refer to Figure 4 , Figure 4 A flow 400 of an embodiment of a method for generating a material parameter identification model according to the present disclosure is shown. The method for generating a material parameter identification model may include the following steps:

[0125] Step 401: Obtain the target image pair, target text, and material parameter label of the first object. The target image pair includes a first image and a second image. The second image is an image obtained by projecting the first image onto the surface of the first object. The target text includes environmental parameters. The material parameter label of the first object is used to characterize the parameters of the material to which the surface of the first object belongs.

[0126] In this embodiment, the execution entity of the method for generating a material parameter identification model (e.g.) Figure 1 The server 104 shown can be accessed from a terminal device (e.g., Figure 1 The terminal devices 101 and 102 shown acquire the target image pair, the target text, and the material parameter label of the first object; or, the aforementioned execution entity (e.g., Figure 1 The terminal devices 101 and 102 shown can obtain the target image pair, target text, and material parameter label of the first object locally.

[0127] It should be noted that step 401 corresponds to step 301 in the aforementioned embodiment. The specific implementation method can be referred to the aforementioned description of step 301, which will not be repeated here.

[0128] Step 402: Encode the target image pair and the target text respectively to obtain the feature map of the target image pair and the feature map of the target text.

[0129] In this embodiment, the aforementioned execution entity may use image encoding corresponding to the target image pair to extract features from the target image pair in order to obtain a feature map of the target image pair; and use text encoding corresponding to the target text to extract features from the target text in order to obtain a feature map of the target text.

[0130] It should be noted that step 402 corresponds to step 302 in the aforementioned embodiment. The specific implementation method can be referred to the aforementioned description of step 302, which will not be repeated here.

[0131] Step 403: The feature maps of the target image pair and the target text are fused using an attention mechanism to obtain a fused feature map.

[0132] In this embodiment, the aforementioned execution entity can employ an attention mechanism to concatenate the feature maps of the target image pair and the target text to obtain a fused feature map. This fused feature map integrates features from both the target image pair and the target text feature maps.

[0133] Here, the aforementioned attention mechanism can enhance the features that better characterize the material parameters of the first object during the feature fusion process. Thus, the fused feature map obtained by fusing features can accurately characterize the material parameters of the first object, further improving the recognition accuracy of the material parameter recognition model trained based on the fused feature map.

[0134] Step 404: Train the material parameter recognition model based on the fused feature map and the material parameter labels of the first object.

[0135] In this embodiment, the aforementioned execution entity can use the fused feature map and the material parameter labels of the first object as training samples to train the initial material parameter recognition model, thereby obtaining the trained material parameter recognition model.

[0136] It should be noted that step 404 corresponds to step 304 in the aforementioned embodiment. The specific implementation method can be found in the aforementioned description of step 304, and will not be repeated here.

[0137] The method for generating a material parameter recognition model provided in this disclosure first obtains a target image pair, target text, and material parameter labels for a first object. The target image pair includes a first image and a second image, the second image being an image obtained by projecting the first image onto the surface of the first object. The target text includes environmental parameters, and the material parameter labels for the first object are used to characterize the parameters of the material to which the surface of the first object belongs. Next, the target image pair and target text are encoded separately to obtain feature maps of the target image pair and the target text. Then, an attention mechanism is used to fuse the feature maps of the target image pair and the target text to obtain a fused feature map. Finally, the model is trained based on the fused feature map and the material parameter labels for the first object to obtain a material parameter recognition model. This method enables multimodal model design based on the feature maps of the target image pair and the target text. By fusing features through an attention mechanism to share features between the feature maps of the target image pair and the target text, the model enhances the material parameter features that better characterize the first object during feature sharing. Therefore, the fused feature map obtained from the fused features accurately characterizes the material parameters of the first object, further improving the fitting ability and recognition accuracy of the material parameter recognition model. This allows for accurate analysis of material parameters based on the material parameter recognition model.

[0138] In some optional implementations of this embodiment, the target text further includes: the material category of the first object.

[0139] In this implementation, the target text may further include the material category of the first object. This material category can be used to characterize the type of material to which the first object belongs, such as paint, wallpaper, marble, fiberglass curtain, metal curtain, wood panel, etc.

[0140] It should be noted that the material category of the first object mentioned above can be the result obtained by identification through a preset material classification model; or the result obtained by manual annotation by the user.

[0141] In one example, a pre-defined material classification model can be obtained by training the model using an image of the first object and the material label of the first object.

[0142] In this implementation, by introducing the material category of the first object into the target text, the features in the feature map of the target text are enriched. Thus, during the fusion process of the feature map of the target image pair and the feature map of the target text, the features in the fused feature map are more closely aligned with the material of the first object, further improving the recognition accuracy of the material parameter recognition model.

[0143] In some optional implementations of this embodiment, the material category of the first object is determined based on the following steps: projecting a preset light onto the surface of the first object using a projection device, and acquiring a third image using a shooting device; inputting the third image into a preset material classification model to obtain the material category of the first object.

[0144] In this implementation, the execution entity can project a preset light onto the surface of the first object using a projection device, and acquire the image presented on the surface of the first object using a shooting device to obtain a third image; then, the third image is input into a preset material classification model to obtain the material category of the first object.

[0145] Correspondingly, in this example, a third image is obtained by taking pictures, previewing, recording, etc., of the image presented on the surface of the first object using an imaging device.

[0146] It should be noted that the preset light can be a different color than the surface of the first object, so as not to interfere with the effect presented on the surface of the first object when the preset light is projected. The preset light can be various types of white light.

[0147] In one example, a pre-defined material classification model can be obtained by training using the material labels of a third image and a first object.

[0148] In this implementation, the material category of the first object is first accurately identified through a material classification model. Then, the material category of the first object is introduced into the target text, which enriches the features in the feature map of the target text. As a result, during the fusion of the feature map of the target image pair and the feature map of the target text, the features in the fused feature map are more closely aligned with the material to which the surface of the first object belongs, thereby further improving the recognition accuracy of the material parameter recognition model. This enables accurate analysis of material parameters based on the material parameter recognition model.

[0149] In some optional implementations of this embodiment, inputting the third image into a preset material classification model to obtain the material category of the first object may include:

[0150] According to the preset masking area, a fourth image is obtained from the third image, wherein the fourth image may be an image including the first object;

[0151] The fourth image is input into a preset material classification model to obtain the material category of the first object.

[0152] In this implementation, the aforementioned execution entity can extract a fourth image from the third image based on a preset masking area; then, the fourth image is input into a preset material classification model to obtain the material category of the first object. The masking area can be an area already obscured by other layers, for example, by layers with different transparency or textures.

[0153] In one example, Figure 5 In the process, white light is projected onto the surface of a first object using a projection device, and then content is captured from the content presented on the surface of the first object using a shooting device to obtain a third image; then, by combining the masked area, content is extracted from the third image to obtain a fourth image, which is an image including the first object; then, the fourth image is input into a preset material classification model, and the material is classified through the material classification model to obtain the material category of the first object.

[0154] It should be noted that the masked area mentioned above can be an automatically labeled and already masked area; or a masked area generated in the future or in an existing manner.

[0155] In this implementation, a fourth image is extracted from the third image by masking the region. The fourth image is then input into a preset material classification model to obtain the material category of the first object. This avoids interference from areas in the third image other than the region where the first object is located (i.e., the fourth image under white light), further improving the accuracy of identifying the material category of the first object. Next, the material category of the first object is introduced into the target text, enriching the features in the feature map of the target text. Thus, during the fusion process of the feature maps of the target image pair and the target text, the features in the fused feature map are more closely aligned with the material of the first object, further improving the recognition accuracy of the material parameter recognition model. This enables accurate analysis of material parameters based on the material parameter recognition model.

[0156] In some optional implementations of this embodiment, the masking area is determined based on the following steps:

[0157] A first image is projected onto the surface of a first object using a projection device, and a second image is acquired using a shooting device. The second image is then segmented to obtain a segmented image, which includes the first object. The segmented image is compared with the first image to determine the projection area. A masking area is generated based on the projection area.

[0158] In this implementation, the execution entity can project a first image onto the surface of a first object using a projection device, and obtain a second image from the image presented on the surface of the first object using the projection device; then, the second image is segmented to obtain a segmented image; then, the segmented image is compared with the first image to determine the projection area; then, a masking area is generated based on the projection area.

[0159] It should be noted that image segmentation generally includes the regions in the image that need to be preserved, such as the region where the first object is located.

[0160] In one example, Figure 6 In this process, a first image is projected onto the surface of a first object using a projection device, and the content of the image projected onto the surface of the first object is acquired by a camera to obtain a second image; then, the second image is segmented to obtain a segmented image; then, the segmented image is compared with the first image to obtain a projection area; finally, a masking area is generated based on the projection area.

[0161] Correspondingly, in this example, performing image segmentation on the second image to obtain a segmented image may include: segmenting the second image using an image segmentation algorithm or a preset segmentation model to obtain a segmented image.

[0162] Correspondingly, in this example, comparing the segmented image with the first image to determine the projection region may include:

[0163] The segmented image is compared with the first image by analyzing the image parameters to determine the projection area, which can be the area on the surface of the first object where the image is presented.

[0164] In this implementation, a fourth image is extracted from the third image by masking the area, and then the fourth image is input into a preset material classification model to obtain the material category of the first object. This avoids interference from areas other than the area where the first object is located in the third image (i.e., the fourth image under white light), and further improves the accuracy of identifying the material category of the first object.

[0165] In one example, generating a masking region based on the projection region can include:

[0166] A masking area can be generated by obscuring the projection area; or, a masking area can be formed by changing the transparency of the projection area to obscure the area displayed by the projection area.

[0167] Here, the masking area can be used to obscure the content displayed in the projection area, for example, by using a different transparency or texture than the projection area to obscure the projection area.

[0168] In one example, Figure 7 In this process, a first image (i.e., the image corresponding to the screen of the projection source) is projected onto the surface of a first object using a projection device; then, the surface of the first object is photographed using a shooting device to obtain a second image; and then, based on the masking area, the projection content corresponding to the area where the first object is located is extracted from the second image.

[0169] In this implementation, a first image is projected onto the surface of a first object using a projection device, and a second image is acquired using a capturing device. The second image is then segmented to obtain a segmented image. The segmented image is then compared with the first image to determine the projection area. A masking area is generated based on the projection area. Next, a fourth image is extracted from a third image based on this masking area. This fourth image is then input into a preset material classification model to obtain the material category of the first object. This avoids interference from areas in the third image other than the region where the first object is located, further improving the accuracy of identifying the material category of the first object. Furthermore, the accurate material category of the first object is introduced into the target text, enriching the features in the feature map of the target text. This makes the features in the fused feature map more closely match the material of the first object during the fusion process of the feature maps of the target image pair and the target text, further improving the recognition accuracy of the material parameter recognition model. This enables accurate analysis of material parameters based on the material parameter recognition model.

[0170] Continue to refer to Figure 8 , Figure 8 A flow 800 of an embodiment of a method for generating a material parameter identification model according to the present disclosure is shown. The method for generating a material parameter identification model may include the following steps:

[0171] Step 801: Obtain the target image pair, target text, and material parameter label of the first object. The target image pair includes a first image and a second image. The second image is an image obtained by projecting the first image onto the surface of the first object. The target text includes environmental parameters. The material parameter label of the first object is used to characterize the parameters of the material to which the surface of the first object belongs.

[0172] In this embodiment, the execution entity of the method for generating a material parameter identification model (e.g.) Figure 1 The server 104 shown can be accessed from a terminal device (e.g., Figure 1 The terminal devices 101 and 102 shown acquire the target image pair, the target text, and the material parameter label of the first object; or, the aforementioned execution entity (e.g., Figure 1 The terminal devices 101 and 102 shown can obtain the target image pair, target text, and material parameter label of the first object locally.

[0173] It should be noted that step 801 corresponds to step 301 in the aforementioned embodiment. The specific implementation method can be referred to the aforementioned description of step 301, which will not be repeated here.

[0174] Step 802: Encode the target image pair and the target text respectively to obtain the feature map of the target image pair and the feature map of the target text.

[0175] In this embodiment, the aforementioned execution entity may use image encoding corresponding to the target image pair to extract features from the target image pair in order to obtain a feature map of the target image pair; and use text encoding corresponding to the target text to extract features from the target text in order to obtain a feature map of the target text.

[0176] It should be noted that step 802 corresponds to step 302 in the aforementioned embodiment. The specific implementation method can be referred to the aforementioned description of step 302, which will not be repeated here.

[0177] Step 803: Fuse the feature maps of the first image and the second image to obtain a first feature map; and fuse the feature maps of environmental parameters and the material category of the first object to obtain a second feature map.

[0178] In this embodiment, the aforementioned execution entity can perform feature stitching on the feature maps of the first image and the second image to obtain a first feature map; and stitch the feature map of the environmental parameters and the feature map of the material category of the first object to obtain a second feature map.

[0179] Step 804: Use a cross-attention mechanism to perform feature crossing on the first feature map and the second feature map to obtain a first cross-feature map and a second cross-feature map.

[0180] In this embodiment, the aforementioned execution entity can perform feature crossing on the first feature map and the second feature map using a cross-attention mechanism to obtain a first cross-feature map and a second cross-feature map. The first cross-feature map and the second cross-feature map can be feature maps obtained by performing feature crossing on the first feature map and the second feature map using a cross-attention mechanism.

[0181] Specifically, the first feature map can be used as a query vector, and the second feature map can be used as a key vector and a value vector to input into the cross-attention module to obtain the first cross-feature map; and the second feature map can be used as a query vector, and the first feature map can be used as a key vector and a value vector to input into the cross-attention module to obtain the second cross-feature map.

[0182] Step 805: Fuse the first cross feature map and the second cross feature map to obtain the fused feature map.

[0183] In this embodiment, the aforementioned execution entity can splice the first cross-fusion feature map and the second cross-feature map to obtain a fused feature map.

[0184] Step 806: Train the material parameter recognition model based on the fused feature map and the material parameter labels of the first object.

[0185] In this embodiment, the aforementioned execution entity can use the fused feature map and the material parameter labels of the first object as training samples to train the initial material parameter recognition model, thereby obtaining the trained material parameter recognition model.

[0186] It should be noted that step 806 corresponds to step 304 in the aforementioned embodiment. The specific implementation method can be found in the aforementioned description of step 304, and will not be repeated here.

[0187] The method for generating a material parameter recognition model provided in this application embodiment obtains a target image pair, target text, and a material parameter label for a first object. The target image pair includes a first image and a second image, the second image being an image obtained by projecting the first image onto the surface of the first object. The target text includes environmental parameters, and the material parameter label for the first object is used to characterize the parameters of the material to which the surface of the first object belongs. Next, the target image pair and the target text are encoded separately to obtain feature maps of the target image pair and the target text. Then, the feature maps of the first image and the second image are fused to obtain a first feature map, and the feature map of the environmental parameters and the feature map of the material category of the first object are fused to obtain a second feature map. Finally, a cross-attention mechanism is used to perform feature cross-interaction on the first and second feature maps to obtain... The first and second cross-feature maps are obtained; then, the first and second cross-feature maps are fused to obtain a fused feature map; then, the model is trained based on the fused feature map and the material parameter labels of the first object to obtain a material parameter recognition model; the model can achieve multimodal design based on the feature maps of the target image and the target text, and through the cross-attention mechanism, it shares the features between the feature maps of the target image and the target text during the feature fusion process, which enhances the material parameter features that can better represent the first object. Thus, the material parameters of the first object can be accurately represented by the fused feature map obtained by fusing features, which further improves the fitting ability and recognition accuracy of the material parameter recognition model, and thus enables accurate analysis of material parameters based on the material parameter recognition model.

[0188] In some optional implementations of this embodiment, a first feature map is obtained by fusing feature maps of the first image and the second image; and a second feature map is obtained by fusing feature maps of environmental parameters and feature maps of the material category of the first object, including:

[0189] The feature maps of the first image and the second image are fused to obtain a first fused feature map; the first fused feature map is processed using a self-attention mechanism to obtain a first feature map; and the feature maps of environmental parameters and the material category of the first object are fused to obtain a second fused feature map; the second fused feature map is processed using a self-attention mechanism to obtain a second feature map.

[0190] And / or,

[0191] The first cross feature map and the second cross feature map are fused to obtain a fused feature map, including: fusing the first cross feature map and the second cross feature map to obtain a third fused feature map; the third fused feature map is processed using a self-attention mechanism to obtain the fused feature map.

[0192] It should be noted that the first feature map mentioned above can be a feature map after feature enhancement by applying a self-attention mechanism to the first fused feature map. The second feature map mentioned above can be a feature map after feature enhancement by applying a self-attention mechanism to the second fused feature map.

[0193] In one example, an attention mechanism is used to fuse the feature maps of a target image pair and the target text to obtain a fused feature map. This can include: fusing the feature maps of a first image and a second image to obtain a first fused feature map; processing the first fused feature map using a self-attention mechanism to obtain a first feature map; fusing the feature maps of environmental parameters and the material category of the first object to obtain a second fused feature map; processing the second fused feature map using a self-attention mechanism to obtain a second feature map; performing feature crossing on the first and second feature maps using a cross-attention mechanism to obtain a first cross-feature map and a second cross-feature map; and fusing the first cross-feature map and the second cross-feature map to obtain the fused feature map.

[0194] In one example, an attention mechanism is used to fuse the feature maps of the target image pair and the target text to obtain a fused feature map. This can include: fusing the feature maps of the first image and the second image to obtain a first feature map; fusing the feature maps of environmental parameters and the feature map of the material category of the first object to obtain a second feature map; using a cross-attention mechanism to perform feature crossing on the first and second feature maps to obtain a first cross-feature map and a second cross-feature map; fusing the first cross-feature map and the second cross-feature map to obtain a third fused feature map; and using a self-attention mechanism to process the third fused feature map to obtain the fused feature map.

[0195] In one example, an attention mechanism is used to fuse the feature maps of a target image pair and the feature map of the target text to obtain a fused feature map. This can include: fusing the feature maps of a first image and a second image to obtain a first fused feature map; processing the first fused feature map using a self-attention mechanism to obtain a first feature map; fusing the feature maps of environmental parameters and the material category of the first object to obtain a second fused feature map; processing the second fused feature map using a self-attention mechanism to obtain a second feature map; performing feature crossing on the first and second feature maps using a cross-attention mechanism to obtain a first cross feature map and a second cross feature map; fusing the first cross feature map and the second cross feature map to obtain a third fused feature map; and processing the third fused feature map using a self-attention mechanism to obtain a fused feature map.

[0196] Correspondingly, in this example, such as Figure 9 As shown, the first image (i.e., the source image) and the second image (i.e., the projected content image) are encoded to obtain feature maps of the first image and features of the second image; environmental parameters (e.g., lighting parameters) and the material category of the first object (i.e., material classification) are encoded to obtain feature maps of environmental parameters and features of the material category of the first object; then, the feature maps of the first image and the second image are fused to obtain a first fused feature map; then, a self-attention mechanism is applied to the first fused feature map to obtain a first feature map; and then, the feature maps of environmental parameters and the material category of the first object are fused to obtain a second fused feature map; then, a self-attention mechanism is applied to the second fused feature map to obtain a second feature map. Finally, the first feature map (i.e., the feature map of the source image) is encoded to obtain a feature map of the first image and features of the second image. Figure 1 ) as the query vector (Q1), the second feature map (i.e., the feature) Figure 2The first cross-feature map is obtained by inputting the key vector (K2) and value vector (V2) into the cross-attention module. The second cross-feature map is then used as the query vector (Q2), and the first cross-feature map is input as the key vector (K1) and value vector (V1) into the cross-attention module to obtain the second cross-feature map. The first and second cross-feature maps are then fused to obtain the third fused feature map. A self-attention mechanism is then used to process the third fused feature map to obtain the fused feature map. Finally, the material parameters (i.e., material properties) of the first object are obtained through this material parameter recognition model.

[0197] In this implementation, the features between the feature maps of the target image pair and the target text are shared during the feature fusion process through self-attention and cross-attention mechanisms. This enhances the material parameter features that better characterize the first object, thereby accurately representing the material parameters of the first object through the fused feature map obtained by fusion. This further improves the fitting ability and recognition accuracy of the material parameter recognition model, enabling accurate analysis of material parameters based on the material parameter recognition model.

[0198] Continue to refer to Figure 10 , Figure 10 A flow 1000 of an embodiment of a method for identifying material parameters according to the present disclosure is shown. The method for identifying material parameters may include the following steps:

[0199] Step 1001: Obtain the first image pair, the environmental parameters of the second object in the first image pair, and the material category of the second object in the first image pair.

[0200] In this embodiment, the entity executing the material parameter identification method (e.g.) Figure 1 The terminal devices 101, 102 or server 104 shown can acquire a first image pair through a shooting device, acquire environmental parameters of the second object in the first image pair through a sensor, and identify the material category of the second object through a preset material classification model. The first image pair includes a third image and a fourth image, and the fourth image is the image obtained by projecting the third image onto the surface of the second object.

[0201] It should be noted that the preset material classification model can be the model used to identify material categories in the above embodiments.

[0202] Step 1002: Input the first image pair, the environmental parameters of the second object, and the material category of the second object in the first image pair into the pre-trained material parameter recognition model to obtain the material parameters of the second object in the first image pair.

[0203] In this embodiment, the execution entity can input the first image pair, the environmental parameters of the second object, and the material category of the second object in the first image pair into a pre-trained material parameter recognition model to obtain the material parameters of the second object in the first image pair. Based on this material parameter recognition model, the entity can accurately analyze the material parameters. The second object and the first object can be the same or different objects.

[0204] Here, the pre-trained material parameter recognition model can be the model generated by the method described above for generating material parameter recognition models. The material parameters of the second object are material-related parameters, such as reflectivity, refractive index, gain, and viewing angle.

[0205] The material parameter identification method provided in this disclosure identifies the material parameters of the second object in a first image pair through a pre-trained material parameter identification model, thereby enabling accurate analysis of the material parameters based on the material parameter identification model.

[0206] Continue to refer to Figure 11 , Figure 11 This is an application scenario diagram of a method for generating material parameter identification models. For example... Figure 11 As shown, the method for generating a material parameter identification model may include the following steps:

[0207] The first step is to obtain the first image and lighting parameters.

[0208] The second step is to project the first image onto the surface of the first object; then extract the content from the image presented on the surface of the first object to obtain the second image.

[0209] The third step is to segment the second image to obtain a segmented image; then, compare and analyze the segmented image with the first image to determine the projection region; and then, generate a masking region based on the projection region.

[0210] The fourth step involves projecting a preset light onto the surface of the first object and extracting content from the acquired third image using a shooting device to obtain a fourth image. Then, the fourth image is input into a preset material classification model to classify the materials and obtain the material category of the first object.

[0211] The fifth step is to encode the first image, the second image, the lighting parameters, and the material category of the first object to obtain feature maps of the first image, the second image, the lighting parameters, and the material category of the first object.

[0212] The sixth step is to fuse the feature maps of the first image, the second image, the illumination parameters, and the material category of the first object to obtain a fused feature map.

[0213] The seventh step is to train the material parameter recognition model based on the fused feature map and the material parameter labels of the first object.

[0214] Step 8: Obtain the first image pair, the environmental information of the second object in the first image pair, and the material category of the second object.

[0215] In the ninth step, the first image pair, the environmental information of the second object in the first image pair, and the material category of the second object are input into the material parameter recognition model in the sixth step to obtain the material parameters of the second object.

[0216] It should be noted that the specific implementation methods for steps one through seven can refer to the embodiments corresponding to the above-described method for generating material parameter identification models. The specific implementation methods for steps eight and nine can refer to the embodiments corresponding to the above-described method for material parameters.

[0217] In this embodiment, the following technical effects are achieved:

[0218] (1) A second image is obtained by projecting a first image onto the surface of a first object using a projection device; then, a segmented image obtained by segmenting based on the second image is compared with the first image. Since projection is used, the image is projected onto the surface of the object, and the difference between the second and first images is calculated. The computational cost is low.

[0219] (2) Images are combined with text to achieve multimodal model training. The inherent features of the images and text are shared, and the semantic information is richer, so that the material parameters can be accurately identified.

[0220] (3) Multimodal models are end-to-end training and prediction processes, which are convenient to use and have high real-time performance.

[0221] (4) Multimodal models can analyze material parameters in real time. Based on projection, they are low-cost, easy to train, and provide accurate and rapid predictions.

[0222] In some optional implementations of this embodiment, the method for identifying the material parameter further includes:

[0223] Based on the material parameters of the second object, determine the material loss of the second object.

[0224] In this implementation, the material loss of the second object can be determined using its material parameters. These material parameters can be used to assess the degree of material loss and estimate the material's lifespan.

[0225] Lifetime loss can be used to characterize the degree of wear of the material to which the surface of a second object belongs.

[0226] In one example, a correspondence between material parameters and material wear life can be pre-established. After obtaining the material parameters of the second object, the material wear of the second object can be determined based on the aforementioned correspondence.

[0227] In one example, the material loss of the second object can be input into a preset material loss recognition model to obtain the material loss of the second object.

[0228] It should be noted that this material loss model can be trained using material parameters and material loss parameters.

[0229] exist Figure 12 In this process, the first image pair, the environmental parameters of the second object in the first image pair, and the material category (i.e., material classification) of the second object in the first image pair are input into a pre-trained material parameter recognition model to obtain the material parameters (i.e., material parameters) of the second object; then, based on the material parameters, the loss of the material parameters is estimated to determine the material loss.

[0230] It should be noted that the material category of the second object can be determined by referring to the description of the material category of the first object in the above embodiments.

[0231] In this implementation, the identified material parameters can be applied to material testing. By using the material parameters of the second object, the material wear and service life can be determined, thus enabling more reasonable use and protection of the material.

[0232] In some optional implementations of this embodiment, the method for identifying the material parameter further includes:

[0233] The display parameters are determined based on the material parameters of the second object;

[0234] Based on the display parameters, the image to be enhanced is optimized for display.

[0235] In this implementation, the display parameters corresponding to the material parameters of the second object can be determined first; then, the display parameters of the image to be enhanced can be optimized and adjusted according to the display parameters to enhance the display.

[0236] Here, the display parameter can be used to optimize the display of the image to be enhanced. The image to be enhanced can be one whose actual display effect does not meet expectations. This expectation can be set according to the user's needs (or preferences).

[0237] In one example, determining the display parameters based on the material parameters of the second object may include: pre-establishing a correspondence between the material parameters of the second object and the display parameters; and then determining the display parameters based on the correspondence and the material parameters of the second object.

[0238] In one example, determining the display parameters based on the material parameters of the second object may include: obtaining the user's requirement information; establishing a correspondence between the material parameters of the second object, the user's requirement information, and the display information; and determining the corresponding display information based on the correspondence, the material parameters of the second object, and the user's requirement information.

[0239] In this implementation, the identified material parameters can be applied to the application scenario of enhanced display. By using the material parameters of the second object, the display parameters can be determined to optimize and adjust the display parameters of the image to be enhanced, thereby enhancing the display.

[0240] In some optional implementations of this embodiment, the image to be enhanced may include: an image obtained by projecting it onto the surface of a second object using a projection device; or, a virtual reality image in virtual reality.

[0241] In this implementation, enhanced display effects can be achieved in different scenarios based on different images to be enhanced.

[0242] In one example, the image obtained by projecting it onto the surface of a second object through a projection device can be optimized, that is, the projection parameters of the projection device can be adjusted to achieve an enhanced display effect in the projection scene.

[0243] exist Figure 13 In this process, the first image pair, the environmental parameters of the second object in the first image pair, and the material category of the second object in the first image pair are input into a pre-trained material parameter recognition model to obtain the material parameters of the second object; then, the projection parameters of the projector are optimized and adjusted in real time based on the material parameters to achieve the best projection effect.

[0244] It should be noted that many projectors nowadays are equipped with cameras. These cameras capture a first image pair projected onto the surface of a second object, and then the material type of the second object is determined from this first image pair. Ambient light parameters are acquired using a light sensor. This data is then input into a material parameter recognition model to obtain the material parameters.

[0245] In one example, enhanced display effects in a virtual reality scene can be achieved by optimizing virtual reality images within the virtual reality environment.

[0246] exist Figure 14In the first image pair, the environmental parameters of the second object in the first image pair, and the material category of the second object in the first image pair are input into a pre-trained material parameter recognition model to obtain the material parameters of the second object; then, these material parameters are used for rendering in the virtual world to make the virtual world more realistic (i.e., Figure 14 (The rendered image in the image).

[0247] In this implementation, the image displayed in the virtual world is optimized by calculating the material parameters for actual display, thereby improving the rendering effect.

[0248] In this implementation, after obtaining accurate material parameters through analysis, the projection device can achieve the best projection effect, the virtual world can be rendered more realistically, and the use of materials can be more standardized.

[0249] Further reference Figure 15 As an implementation of the methods shown in the above figures, this disclosure provides an embodiment of an apparatus for generating a material parameter identification model, which is similar to... Figure 2 Corresponding to the method embodiments shown, this device can be specifically applied to various electronic devices.

[0250] like Figure 15 As shown, the apparatus 1500 for generating a material parameter recognition model in this embodiment may include an image acquisition module 1501 and a model training module 1502. The image acquisition module 1501 is configured to acquire a target image pair, target text, and a material parameter label for a first object. The target image pair includes a first image and a second image, the second image being an image obtained by projecting the first image onto the surface of the first object. The target text includes environmental parameters, and the material parameter label for the first object is used to characterize the parameters of the material to which the surface of the first object belongs. The model training module 1502 is configured to train the model based on the target image pair, the target text, and the material parameter label for the first object to obtain a material parameter recognition model.

[0251] In this embodiment, the specific processing of the image acquisition module 1501 and the model training module 1502 in the apparatus 1500 for generating material parameter recognition models, and the resulting technical effects, can be found in the following references: Figure 2 The relevant descriptions of steps 201-202 in the corresponding embodiments will not be repeated here.

[0252] In some optional implementations of this embodiment, the model training module 1502 includes:

[0253] The encoding unit is configured to encode the target image pair and the target text respectively, to obtain the feature map of the target image pair and the feature map of the target text;

[0254] The fusion unit is configured to fuse the feature maps of the target image pair and the feature maps of the target text to obtain a fused feature map.

[0255] The training unit is configured to be trained based on the fused feature map and the material parameter labels of the first object to obtain the material parameter recognition model.

[0256] In some optional implementations of this embodiment, the fusion unit is further configured to: fuse the feature maps of the target image pair and the feature maps of the target text using an attention mechanism to obtain a fused feature map.

[0257] In some optional implementations of this embodiment, the target text may further include: the material category of the first object.

[0258] In some optional implementations of this embodiment, the fusion unit includes:

[0259] The first fusion subunit is configured to fuse the feature map of the first image and the feature map of the second image to obtain a first feature map; and to fuse the feature map of the environmental parameters and the feature map of the material category of the first object to obtain a second feature map;

[0260] The feature crossing subunit is configured to perform feature crossing on the first feature map and the second feature map using a cross attention mechanism to obtain the first cross feature map and the second cross feature map.

[0261] The second fusion subunit is configured to fuse the first cross feature map and the second cross feature map to obtain a fused feature map.

[0262] In some optional implementations of this embodiment, the first fusion subunit is further configured as follows:

[0263] The feature maps of the first image and the second image are fused to obtain a first fused feature map; the first fused feature map is processed using a self-attention mechanism to obtain a first feature map; and the feature maps of environmental parameters and the material category of the first object are fused to obtain a second fused feature map; the second fused feature map is processed using a self-attention mechanism to obtain a second feature map.

[0264] And / or,

[0265] The second fusion subunit is further configured to: fuse the first cross feature map and the second cross feature map to obtain a fused feature map, including: fusing the first cross feature map and the second cross feature map to obtain a third fused feature map; and using a self-attention mechanism to process the third fused feature map to obtain a fused feature map.

[0266] In some optional implementations of this embodiment, the apparatus 1500 for generating the material parameter identification model further includes:

[0267] The image acquisition module 1501 is configured to project a preset light onto the surface of a first object using a projection device, and to acquire a third image using a shooting device.

[0268] The category recognition module is configured to input the third image into a preset material classification model to obtain the material category of the first object.

[0269] In some optional implementations of this embodiment, the category recognition module is further configured to: obtain a fourth image from the third image according to a preset masking area, wherein the fourth image includes an image of the first object; input the fourth image into a preset material classification model to obtain the material category of the first object.

[0270] In some optional implementations of this embodiment, the apparatus 1500 for generating the material parameter identification model further includes:

[0271] The image acquisition module 1501 is configured to project a first image onto the surface of a first object using a projection device, and to acquire a second image using a shooting device.

[0272] The image segmentation module is configured to segment the second image to obtain a segmented image, wherein the segmented image is an image including the first object;

[0273] The image comparison module is configured to compare the segmented image with the first image to determine the projection area;

[0274] The region generation module is configured to generate a masking region based on the projection region.

[0275] In some optional implementations of this embodiment, the apparatus 1500 for generating material parameter identification models further includes a parameter acquisition module configured to acquire environmental parameters through sensors.

[0276] Further reference Figure 16 As an implementation of the methods shown in the above figures, this disclosure provides an embodiment of a material parameter identification device, which is similar to... Figure 10 Corresponding to the method embodiments shown, this device can be specifically applied to various electronic devices.

[0277] like Figure 16As shown, the material parameter identification device 1600 of this embodiment may include an image acquisition module 1601 and a parameter identification module 1602. The image acquisition module 1601 is configured to acquire a first image pair, environmental parameters of the second object in the first image pair, and the material category of the second object. The parameter identification module 1602 is configured to input the first image pair, the environmental parameters of the second object, and the material category of the second object into a pre-trained material parameter identification model to obtain the material parameters of the second object.

[0278] In this embodiment, the specific processing of the image acquisition module 1601 and the parameter recognition module 1602 in the material parameter recognition device 1600 and the resulting technical effects can be referred to respectively. Figure 10 The relevant descriptions of steps 1001-1002 in the corresponding embodiments will not be repeated here.

[0279] In some embodiments, the material parameter identification device further includes a loss determination module configured to determine the material loss of the second object based on the material parameters of the second object.

[0280] In some embodiments, the material parameter identification device further includes: a display parameter determination module configured to determine display parameters based on the material parameters of the second object; and an optimization adjustment module configured to optimize the display of the image to be enhanced based on the display parameters.

[0281] In some embodiments, the image to be enhanced includes: an image projected onto the surface of a second object; or, a virtual reality image in virtual reality.

[0282] According to embodiments of this disclosure, this disclosure also provides an electronic device and a readable storage medium.

[0283] Figure 17 A schematic block diagram of an example electronic device 1700 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0284] like Figure 17As shown, device 1700 includes a computing unit 1701, which can perform various appropriate actions and processes according to a computer program stored in read-only memory (ROM) 1702 or a computer program loaded into random access memory (RAM) 1703 from storage unit 1708. The RAM 1703 may also store various programs and data required for the operation of device 1700. The computing unit 1701, ROM 1702, and RAM 1703 are interconnected via bus 1704. Input / output (I / O) interface 1705 is also connected to bus 1704.

[0285] Multiple components in device 1700 are connected to I / O interface 1705, including: input unit 1706, such as a keyboard, mouse, etc.; output unit 1707, such as various types of displays, speakers, etc.; storage unit 1708, such as a disk, optical disk, etc.; and communication unit 1709, such as a network card, modem, wireless transceiver, etc. Communication unit 1709 allows device 1700 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0286] The computing unit 1701 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 1701 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 1701 performs the various methods and processes described above, such as methods for generating material parameter identification models or methods for identifying material parameters. For example, in some embodiments, the methods for generating material parameter identification models or methods for identifying material parameters may be implemented as computer software programs tangibly contained in a machine-readable medium, such as storage unit 1708. In some embodiments, part or all of the computer program may be loaded and / or installed on device 1700 via ROM 1702 and / or communication unit 1709. When the computer program is loaded into RAM 1703 and executed by the computing unit 1701, one or more steps of the methods for generating material parameter identification models or methods for identifying material parameters described above may be performed. Alternatively, in other embodiments, the computing unit 1701 may be configured by any other suitable means (e.g., by means of firmware) to perform a method for generating a material parameter identification model or a method for identifying material parameters.

[0287] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPG8), application-specific integrated circuits (8SICs), application-specific standard products (8SSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0288] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0289] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0290] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0291] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with embodiments of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0292] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other.

[0293] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0294] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A method for generating a material parameter identification model, comprising: Obtain a target image pair, target text, and a material parameter label for a first object. The target image pair includes a first image and a second image, the second image being an image obtained by projecting the first image onto the surface of the first object. The target text includes environmental parameters. The material parameter label for the first object is used to characterize the parameters of the material to which the surface of the first object belongs. The first object is an object used to present the first image. The target image pair and the target text are encoded separately to obtain feature maps of the target image pair and the target text; the feature maps of the target image pair and the target text are fused to obtain a fused feature map; a material parameter recognition model is obtained by training based on the fused feature map and the material parameter labels of the first object.

2. The method according to claim 1, wherein, The process of fusing the feature maps of the target image pair and the target text to obtain a fused feature map includes: The feature maps of the target image pair and the target text are fused using an attention mechanism to obtain the fused feature map.

3. The method according to any one of claims 1-2, wherein, The target text also includes: the material category of the first object.

4. The method according to claim 3, wherein, The feature map is obtained by fusing the feature maps of the target image pair and the target text using an attention mechanism, including: A first feature map is obtained by fusing the feature maps of the first image and the second image; and a second feature map is obtained by fusing the feature maps of environmental parameters and the feature map of the material category of the first object. A cross-attention mechanism is used to perform feature crossing on the first feature map and the second feature map to obtain a first cross-feature map and a second cross-feature map. The first cross feature map and the second cross feature map are fused to obtain the fused feature map.

5. The method according to claim 4, wherein, The feature map of the first image and the feature map of the second image are fused to obtain the first feature map; A second feature map is obtained by fusing the feature map of environmental parameters and the feature map of the material category of the first object, including: The feature maps of the first image and the second image are fused to obtain a first fused feature map; the first fused feature map is processed using a self-attention mechanism to obtain a first feature map; and the feature maps of environmental parameters and the material category of the first object are fused to obtain a second fused feature map; the second fused feature map is processed using a self-attention mechanism to obtain a second feature map. And / or, The process of fusing the first cross feature map and the second cross feature map to obtain the fused feature map includes: fusing the first cross feature map and the second cross feature map to obtain a third fused feature map; and processing the third fused feature map using a self-attention mechanism to obtain the fused feature map.

6. The method according to claim 3, wherein, The material category of the first object is determined based on the following steps: A preset light is projected onto the surface of the first object using a projection device, and a third image is acquired using a shooting device. The third image is input into a preset material classification model to obtain the material category of the first object.

7. The method according to claim 6, wherein, The step of inputting the third image into a preset material classification model to obtain the material category of the first object includes: According to a preset masking area, a fourth image is obtained from the third image, wherein the fourth image includes an image of the first object; The fourth image is input into a preset material classification model to obtain the material category of the first object.

8. The method according to claim 7, wherein, The masking area is determined based on the following steps: The first image is projected onto the surface of the first object using a projection device, and the second image is obtained using a shooting device. The second image is segmented to obtain a segmented image, wherein the segmented image is an image including the first object; The segmented image is compared with the first image to determine the projection region; A masking area is generated based on the projection area.

9. A method for identifying material parameters, comprising: Obtain a first image pair, the environmental parameters of the second object in the first image pair, and the material type of the second object. The first image pair includes a third image and a fourth image, wherein the fourth image is an image obtained by projecting the third image onto the surface of the second object. The first image pair, the environmental parameters of the second object, and the material category of the second object are input into the material parameter recognition model generated by the method described in any one of claims 1-8 to obtain the material parameters of the second object.

10. The method according to claim 9, further comprising: Based on the material parameters of the second object, determine the material loss of the second object.

11. The method according to claim 9, further comprising: The display parameters are determined based on the material parameters of the second object; Based on the display parameters, the image to be enhanced is optimized for display.

12. The method according to claim 11, wherein, The image to be enhanced includes: an image projected onto the surface of a second object; or, a virtual reality image in virtual reality.

13. An apparatus for generating a material parameter identification model, comprising: The image acquisition module is configured to acquire a target image pair, target text, and a material parameter label of a first object. The target image pair includes a first image and a second image, the second image being an image obtained by projecting the first image onto the surface of the first object. The target text includes environmental parameters. The material parameter label of the first object is used to characterize the parameters of the material to which the surface of the first object belongs. The first object is an object used to present the first image. The model training module includes an encoding unit, a fusion unit, and a training unit. The encoding unit is configured to encode the target image pair and the target text respectively to obtain feature maps of the target image pair and the target text. The fusion unit is configured to fuse the feature maps of the target image pair and the target text to obtain a fused feature map. The training unit is configured to train based on the fused feature map and the material parameter labels of the first object to obtain a material parameter recognition model.

14. A material parameter identification device, comprising: The image acquisition module is configured to acquire a first image pair, environmental parameters of the second object in the first image pair, and the material type of the second object. The first image pair includes a third image and a fourth image, wherein the fourth image is an image obtained by projecting the third image onto the surface of the second object. The parameter recognition module is configured to input the first image pair, the environmental parameters of the second object, and the material category of the second object into the material parameter recognition model generated by the method described in any one of claims 1-8, so as to obtain the material parameters of the second object.

15. An electronic device comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-12.

16. A non-transitory computer-readable storage medium storing computer instructions for causing the computer to perform the method of any one of claims 1-12.

Images