Three-dimensional model matching, and training method and device of multi-modal feature mapping model

Abstract

The embodiment of the specification provides a three-dimensional model matching and a training method and device of a multi-modal feature mapping model, the method comprises the following steps: determining at least one first feature corresponding to a first object according to the first object by using a target multi-modal feature mapping model, wherein the target multi-modal feature mapping model is obtained by training based on a sample graph-text pair, the sample graph-text pair comprises corresponding text and a depth map, and the first object is a first text or a first three-dimensional model; obtaining at least one second feature corresponding to a second three-dimensional model, wherein each second feature is generated by using the target multi-modal feature mapping model according to a depth map of the second three-dimensional model; and determining a matching result of the first object and the second three-dimensional model based on the first feature corresponding to the first object and the plurality of second features corresponding to the second three-dimensional model.

Description

Technical Field

[0001] This specification relates to the field of data processing technology, and in particular to a method and apparatus for training a three-dimensional model matching and multimodal feature mapping model. Background Technology

[0002] In some scenarios, users need to find the 3D models they require. To meet this need, providing a 3D model matching method has become an urgent problem to solve. Summary of the Invention

[0003] This specification provides one or more embodiments of a three-dimensional model matching method and apparatus to achieve the matching of three-dimensional models.

[0004] According to the first aspect, a three-dimensional model matching method is provided, including:

[0005] Based on the first object, a target multimodal feature mapping model is used to determine at least one first feature corresponding to the first object, wherein the target multimodal feature mapping model is obtained by training based on sample image-text pairs, the sample image-text pairs include corresponding text and depth maps, and the first object is a first text or a first 3D model;

[0006] At least one second feature corresponding to the second three-dimensional model is obtained, and each second feature is generated based on the depth map of the second three-dimensional model by using the target multimodal feature mapping model;

[0007] Based on the first feature corresponding to the first object and multiple second features corresponding to the second three-dimensional model, the matching result between the first object and the second three-dimensional model is determined.

[0008] In one alternative implementation, the first object is a first three-dimensional model;

[0009] Determining at least one first feature corresponding to the first object includes:

[0010] Determine the first depth map of the first three-dimensional model under at least one preset first viewpoint;

[0011] Based on each first depth map, the first feature corresponding to each first depth map is determined using the image encoder of the target multimodal feature mapping model.

[0012] In one alternative implementation, the depth map of the second three-dimensional model includes: depth maps of each of the second three-dimensional models from at least one first viewpoint;

[0013] Determining the matching result between the first object and the second 3D model includes:

[0014] Calculate the average value of the first features corresponding to all first depth maps to obtain the average features of the first 3D model;

[0015] For each second three-dimensional model, calculate the average value of the second feature corresponding to that second three-dimensional model to obtain the average feature of each second three-dimensional model;

[0016] Based on the average features of the first 3D model and the average features of each of the second 3D models, the matching results between the first 3D model and each of the second 3D models are determined.

[0017] In one optional implementation, determining the matching results between the first 3D model and each of the second 3D models includes:

[0018] Calculate the average features of the first 3D model and the first similarity value between them and the average features of each of the second 3D models;

[0019] The second 3D model with the highest first similarity value is determined as the 3D model that matches the first 3D model.

[0020] In one alternative implementation, the first object is first text;

[0021] Determining at least one first feature corresponding to the first object includes:

[0022] Based on the first text, the text encoder of the target multimodal feature mapping model is used to determine the first feature corresponding to the first text.

[0023] In one optional implementation, determining the matching result between the first object and the second three-dimensional model includes:

[0024] Calculate the first feature corresponding to the first text and the second similarity value between it and each of the second features;

[0025] Based on each second similarity value, the matching result between the first text and the second three-dimensional model is determined.

[0026] In one alternative implementation, there are multiple second three-dimensional models;

[0027] The step of determining the matching result between the first text and the second 3D model based on each second similarity value includes:

[0028] Based on each second similarity value, N third features with the largest corresponding second similarity values ​​are determined from the plurality of second features, where N is a positive integer;

[0029] Determine the second three-dimensional model corresponding to each third feature;

[0030] The second 3D model with the most corresponding third features is determined as the 3D model that matches the first text.

[0031] According to the second aspect, a training method for a multimodal feature mapping model is provided, including:

[0032] Obtain sample image-text pairs from the training dataset, wherein the sample image-text pairs include sample depth maps and their corresponding sample text;

[0033] Based on the sample depth map and the sample text, the multimodal feature mapping model to be trained is used to determine the graph features corresponding to the sample depth map and the text features corresponding to the sample text.

[0034] The parameters of the multimodal feature mapping model to be trained are adjusted with the goal of maximizing the similarity between the graph features and the text features.

[0035] In one alternative implementation, prior to obtaining the sample image-text pairs from the training dataset, the method further includes:

[0036] Obtain the color image and its corresponding second text;

[0037] Based on the color image, a second depth map corresponding to the color image is obtained using a depth estimation model, wherein the second depth map includes the depth value of each pixel, and the depth estimation model is trained based on the sample color image and its corresponding depth map;

[0038] Based on the second text and the second depth map, a set of sample image-text pairs are formed and included in the training dataset.

[0039] In one alternative implementation, prior to obtaining the sample image-text pairs from the training dataset, the method further includes:

[0040] Obtain the 3D model of the sample;

[0041] The sample 3D model is rendered from at least one second perspective to obtain a 3D model rendering map and a third depth map corresponding to each second perspective, wherein the third depth map includes the depth value of the sample 3D model under the corresponding second perspective.

[0042] Based on the rendering images of each 3D model, an image description generation model is used to determine the third text corresponding to each 3D model rendering image. The image description generation model is used to generate corresponding text descriptions based on the input images.

[0043] Based on the third text corresponding to each 3D model rendering and the third depth map corresponding to each sample 3D model in each second viewpoint, sample image-text pairs corresponding to each second viewpoint are formed and included in the training dataset.

[0044] According to a third aspect, a three-dimensional model matching device is provided, comprising:

[0045] The first determining module is configured to determine at least one first feature corresponding to the first object using a target multimodal feature mapping model, wherein the target multimodal feature mapping model is obtained by training based on sample image-text pairs, the sample image-text pairs include corresponding text and depth maps, and the first object is a first text or a first three-dimensional model;

[0046] The first acquisition module is configured to acquire at least one second feature corresponding to the second three-dimensional model, wherein each second feature is generated based on the depth map of the second three-dimensional model by using the target multimodal feature mapping model;

[0047] The second determining module is configured to determine the matching result between the first object and the second three-dimensional model based on the first feature corresponding to the first object and multiple second features corresponding to the second three-dimensional model.

[0048] According to the fourth aspect, a training device for a multimodal feature mapping model is provided, comprising:

[0049] The second acquisition module is configured to acquire sample image-text pairs in the training dataset, wherein the sample image-text pairs include sample depth maps and their corresponding sample texts.

[0050] The third determining module is configured to determine the graph features corresponding to the sample depth map and the text features corresponding to the sample text using a multimodal feature mapping model to be trained, based on the sample depth map and the sample text.

[0051] The adjustment module is configured to adjust the parameters of the multimodal feature mapping model to be trained with the goal of maximizing the similarity between the graph features and the text features.

[0052] According to a fifth aspect, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed in a computer, causes the computer to perform the method described in the first or second aspect.

[0053] According to a sixth aspect, a computing device is provided, including a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, it implements the method of the first aspect or the second aspect.

[0054] According to the 3D model matching and multimodal feature mapping model training method and apparatus provided in the embodiments of this specification, in the 3D model matching process, based on a first object, at least one first feature corresponding to the first object is determined using a target multimodal feature mapping model, wherein the target multimodal feature mapping model is obtained based on sample image-text pairs, the sample image-text pairs include corresponding text and depth maps, and the first object is a first text or a first 3D model; at least one second feature corresponding to a second 3D model is obtained, each second feature is generated based on the depth map of the second 3D model by using the target multimodal feature mapping model; based on the first feature corresponding to the first object and multiple second features corresponding to the second 3D model, the matching result between the first object and the second 3D model is determined.

[0055] In the above process, by training a target multimodal feature mapping model based on the corresponding text and depth map, at least one first feature corresponding to the first object can be determined. Then, at least one second feature corresponding to the second three-dimensional model is obtained. Each second feature is generated based on the depth map of the second three-dimensional model using the target multimodal feature mapping model. This ensures that at least one first feature of the first object and at least one second feature corresponding to the second three-dimensional model are aligned. Then, based on the aligned first feature of the first object and multiple second features corresponding to the second three-dimensional model, the first object and the second three-dimensional model are matched to determine the matching result. This can achieve matching between text and three-dimensional models, or matching between three-dimensional models. Attached Figure Description

[0056] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are merely some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without any creative effort.

[0057] Figure 1 This is a schematic diagram illustrating the implementation framework of one embodiment disclosed in this specification;

[0058] Figure 2 A flowchart illustrating a training method for a multimodal feature mapping model provided in an embodiment;

[0059] Figure 3 A flowchart illustrating a three-dimensional model matching method provided in this embodiment;

[0060] Figure 4 Another flowchart illustrating the three-dimensional model matching method provided in this embodiment;

[0061] Figure 5 Another flowchart illustrating the 3D model matching method provided in this embodiment:

[0062] Figure 6 A schematic block diagram of a three-dimensional model matching device provided in an embodiment;

[0063] Figure 7 This is a schematic block diagram of a training device for a multimodal feature mapping model provided in an embodiment. Detailed Implementation

[0064] The technical solutions of the embodiments of this specification will now be described in detail with reference to the accompanying drawings.

[0065] As mentioned earlier, in order to meet users' needs for finding the 3D models they require, providing a 3D model matching method has become an urgent problem to be solved.

[0066] Therefore, embodiments of this specification propose a method and apparatus for training a three-dimensional model matching and multimodal feature mapping model to achieve matching of three-dimensional models. Figure 1 This diagram illustrates an implementation scenario according to an embodiment disclosed in this specification. In this implementation scenario, a first object is obtained, which is either first text or a first 3D model; based on the first object, a target multimodal feature mapping model is used to determine at least one first feature corresponding to the first object, wherein the target multimodal feature mapping model is trained based on sample image-text pairs, which include corresponding text and depth maps; then, at least one second feature corresponding to a second 3D model is obtained, wherein each second feature is generated based on the depth map of the second 3D model using the target multimodal feature mapping model; based on the first feature corresponding to the first object and the multiple second features corresponding to the second 3D model, the first object and the second 3D model are matched to determine the matching result between the first object and the second 3D model.

[0067] In the above process, by training a target multimodal feature mapping model based on the corresponding text and depth map, at least one first feature corresponding to the first object can be determined. Then, at least one second feature corresponding to the second three-dimensional model is obtained. Each second feature is generated based on the depth map of the second three-dimensional model using the target multimodal feature mapping model. This ensures that at least one first feature of the first object and at least one second feature corresponding to the second three-dimensional model are aligned. Then, based on the aligned first feature of the first object and multiple second features corresponding to the second three-dimensional model, the first object and the second three-dimensional model are matched to determine the matching result. This can achieve matching between text and three-dimensional models, or matching between three-dimensional models.

[0068] The following detailed description, with reference to specific embodiments, elaborates on the training methods and apparatus for 3D model matching and multimodal feature mapping models provided in this specification.

[0069] In some scenarios, some users want to find the 3D models they need through text, and / or some users want to find other 3D models that match certain known 3D models. To meet the needs of these users, it is crucial to provide a method for matching 3D models. To achieve this, this specification also provides a method for training a multimodal feature mapping model. The trained multimodal feature mapping model can support 3D model matching schemes.

[0070] The training process of the multimodal feature mapping model will be introduced below.

[0071] like Figure 2 This document illustrates a flowchart of a training method for a multimodal feature mapping model according to one embodiment of the present specification. The method is executed by an electronic device, which can be any device, equipment, platform, or cluster of devices with computing and processing capabilities. During the training process of the multimodal feature mapping model, the method includes steps S210-S230:

[0072] In step S210, sample image-text pairs are obtained from the training dataset. These sample image-text pairs include sample depth maps and their corresponding sample texts.

[0073] In one implementation, to train a multimodal feature mapping model, a training dataset for training the model can be obtained first. This training dataset may include several sample image-text pairs. Each sample image-text pair includes a corresponding depth map and text. For clarity, the depth map in the sample image-text pair can be called a sample depth map, and the text in the sample image-text pair can be called sample text. The electronic device can obtain the sample image-text pairs from the training dataset and then execute the subsequent training process.

[0074] In one embodiment of this specification, a process for constructing the training dataset is also provided. The process for constructing the training dataset may include, but is not limited to: using two-dimensional color images and their corresponding text to construct sample image-text pairs containing corresponding depth maps and text; and / or using rendered images and depth maps obtained by rendering a three-dimensional model to construct sample image-text pairs containing corresponding depth maps and text.

[0075] Specifically, in one embodiment, before step S210, the method may further include the following steps 11-13:

[0076] In step 11, the color image and its corresponding second text are obtained. In this step, the electronic device can obtain two-dimensional image-text pairs from any specified public dataset that can provide a large number of two-dimensional image-text pairs. The two-dimensional image-text pairs include two-dimensional color images and their corresponding text. For clarity, the text corresponding to the two-dimensional color image is called the second text. The second text can also be called the image description corresponding to the two-dimensional color image. The second text can be used to generate its corresponding two-dimensional color image.

[0077] Next, in step 12, based on the color image, a second depth map corresponding to the color image is obtained using a depth estimation model. This second depth map includes the depth values ​​of each pixel. The depth estimation model is trained based on the sample color image and its corresponding depth map. The depth map corresponding to the sample color image may include the depth values ​​labeled for each pixel of the sample color image. In this step, the electronic device can obtain the depth estimation model from a preset storage space. This depth estimation model is pre-trained based on the sample color image and its corresponding depth map. For the input color image, the depth estimation model can output a corresponding depth map, which includes the depth values ​​of each pixel. In one case, the depth values ​​can range from [0, 255].

[0078] After an electronic device acquires a color image and its corresponding second text, it can input the color image into a depth estimation model. The depth estimation model processes the color image to obtain a depth map corresponding to the color image, called the second depth map. In one implementation, this depth estimation model can be a monocular depth estimation model, which can be based on the MiDaS model.

[0079] Then, in step 13, a set of sample image-text pairs is formed based on the second text and the second depth map, and these pairs are added to the training dataset. In this step, the second text corresponding to the color image and the second depth map corresponding to the color image can be combined into a set of sample image-text pairs and added to the training data.

[0080] Using the above method, a second depth map corresponding to the color image in multiple two-dimensional image-text pairs can be obtained through a depth estimation model. This allows for the generation of multiple sample image-text pairs using the text in the multiple two-dimensional image-text pairs and the corresponding second depth maps. These sample image-text pairs include the relatively accurate original text (i.e., image description) and the depth map obtained from the two-dimensional color image corresponding to the text through a depth estimation model. In one case, this type of sample image-text pair can be referred to as the first type of sample image-text pair.

[0081] In another embodiment, prior to step S210, the method may further include steps 21-24:

[0082] In step 21, a sample 3D model is acquired. In this step, the electronic device can acquire the sample 3D model from any data source that provides a 3D model. This sample 3D model can be of any type, such as animal models, plant models, human models, architectural models, and various prop models from games. In one implementation, the sample 3D model can exist in the form of an .obj file, which can include the geometric information of the corresponding sample 3D model, such as the spatial position information of each vertex that makes up the sample 3D model, the triangular facet information formed by each vertex, and texture coordinate information, etc.

[0083] Next, in step 22, the sample 3D model is rendered from at least one second perspective to obtain a 3D model rendering image and a third depth map corresponding to each second perspective. The third depth map includes the depth value of the sample 3D model under the corresponding second perspective. In this step, the electronic device can render the sample 3D model from at least one second perspective to obtain a 3D model rendering image and a third depth map corresponding to each second perspective. The at least one second perspective can be randomly determined or preset.

[0084] The spatial position information of each vertex of the sample 3D model can be the spatial position information in a specified 3D spatial coordinate system. In one implementation, the center point of the sample 3D model coincides with the origin O of the specified 3D spatial coordinate system, and the size of the sample 3D model is adjusted to a specified size. For example, the coordinates of each axis in the spatial position information of each vertex of the sample 3D model are normalized to [-1, 1].

[0085] To ensure that the scale of each 3D model rendering obtained from each second perspective is consistent, and that the scale of each third depth map is consistent, the distance between the viewpoint corresponding to each second perspective rendering of the sample 3D model and the center point of the sample 3D model, i.e., the origin of the specified 3D spatial coordinate system, is equal. For example, the sample 3D model can be rendered at at least one position, i.e., a viewpoint, that is, at a specified distance from the center point of the sample 3D model. In the case where there are multiple viewpoints at at least one position, at least one position corresponds to a different second perspective.

[0086] In one scenario, the second perspective can be represented by a set of horizontal and vertical angles. The horizontal angle can be the angle between the line connecting the viewpoint's projection onto the XOZ plane (considered a horizontal plane) of the specified 3D coordinate system and the origin of the specified 3D coordinate system, i.e., the center point of the sample 3D model, and the X-axis. The vertical angle can be the angle between the line connecting the viewpoint to the origin of the specified 3D coordinate system and the line connecting the viewpoint's projection onto the XOZ plane and the origin. The X-axis is the horizontal axis of the specified 3D coordinate system, the Z-axis is the vertical axis of the specified 3D coordinate system, and the Y-axis is the vertical axis of the specified 3D coordinate system.

[0087] In one implementation, if the sample 3D model corresponds to a texture map, then when rendering the sample 3D model from at least one second perspective, the sample 3D model can be rendered based on the correspondence between each vertex in the sample 3D model and each pixel in the texture map, according to the texture in the texture map, to obtain a 3D model rendering map corresponding to at least one second perspective.

[0088] In another implementation, if the sample 3D model does not correspond to a texture map, then when rendering the sample 3D model from at least one second perspective, based on the correspondence between each vertex in the sample 3D model and each pixel in a specified grayscale image, the sample 3D model is rendered according to the specified grayscale image to obtain a 3D model rendering image corresponding to at least one second perspective. In one case, the specified grayscale image can be a solid color image, that is, each pixel in the specified grayscale image corresponds to the same color value.

[0089] At least one third depth map corresponding to a second viewpoint includes the depth values ​​of the sample 3D model under the corresponding second viewpoint. The depth values ​​of the sample 3D model under the corresponding second viewpoint can characterize the relative distances between each vertex of the sample 3D model and the plane containing the viewpoint of the second viewpoint, wherein the plane containing the viewpoint of the second viewpoint is perpendicular to the XOZ plane of the specified 3D spatial coordinate system. The range of depth values ​​in the third depth map is unified to [0, 255].

[0090] In one implementation, the process of determining the depth values ​​in the third depth map may include: determining the distance between each vertex of the sample 3D model and the plane containing the viewpoint corresponding to the second viewpoint j in the specified 3D coordinate system, based on the spatial position information of each vertex of the sample 3D model and the plane containing the viewpoint corresponding to the second viewpoint j; then converting the distance between each vertex of the sample 3D model and the plane containing the viewpoint corresponding to the second viewpoint j to a specified value range, i.e., the aforementioned [0, 255], to obtain the depth values ​​of each vertex of the sample 3D model in the third depth map under the second viewpoint j. Here, j is a positive integer.

[0091] The aforementioned process of converting the distances between each vertex of the sample 3D model and the plane containing the viewpoint corresponding to the second perspective j to a specified range may include: determining the largest distance from the distances between each vertex of the sample 3D model and the plane containing the viewpoint corresponding to the second perspective j; dividing each of the aforementioned distances between each vertex of the sample 3D model and the plane containing the viewpoint corresponding to the second perspective j by the largest distance to obtain the median distances corresponding to each vertex of the sample 3D model; and then multiplying the median distances corresponding to each vertex of the sample 3D model by 255 to convert the distances between each vertex of the sample 3D model and the plane containing the viewpoint corresponding to the second perspective j to a specified range.

[0092] After obtaining the 3D model rendering and third depth map corresponding to each second perspective, in step 23, based on each 3D model rendering, an image description generation model is used to determine the third text corresponding to each 3D model rendering. The image description generation model is used to generate the corresponding text description based on the input image.

[0093] In this step, the electronic device can retrieve an image description generation model from a preset storage space. This model is pre-trained based on two-dimensional image-text pairs and is used to generate corresponding text based on the input two-dimensional image. The electronic device can input each three-dimensional model rendering image into the image description generation model, and the model processes each rendering image to obtain the corresponding third-party text.

[0094] In one implementation, the image description generation model can be implemented using a BLIP (Bootstrapping Language-Image Pre-training) model.

[0095] Next, in step 24, based on the third text corresponding to each 3D model rendering and the third depth map corresponding to each sample 3D model in each second viewpoint, sample image-text pairs corresponding to each second viewpoint are formed and included in the training dataset.

[0096] In this step, the electronic device can combine the third text corresponding to each 3D model rendering with the third depth map corresponding to each 3D model rendering from the second perspective to form sample image-text pairs corresponding to each second perspective, and then incorporate them into the training data.

[0097] Using the above method, at least one set of sample image-text pairs can be obtained for each of multiple sample 3D models. These sample image-text pairs include relatively accurate depth maps and text generated from the 3D model renderings through image description. In one case, these sample image-text pairs can be referred to as the second type of sample image-text pairs.

[0098] The above implementation method, when there are multiple second perspectives, can use a single sample 3D model to obtain multiple sets of sample image-text pairs, which can increase the number of second-type sample image-text pairs to some extent.

[0099] The above method yields a training dataset containing both first-class and second-class sample image-text pairs. Sample image-text pairs can then be extracted from this training dataset; these pairs can belong to either the first-class or second-class sample image-text pairs. In one implementation, to improve the accuracy of the feature mapping results of the multimodal feature mapping model, sample image-text pairs can be extracted from both the first-class and second-class sample image-text pairs on an average basis during the extraction process from the training dataset. This ensures both the accuracy of the multimodal feature mapping model's mapping results and its training efficiency.

[0100] After obtaining the sample image-text pair, in step S220, based on the sample depth map and sample text, the multimodal feature mapping model to be trained is used to determine the image features corresponding to the sample depth map and the text features corresponding to the sample text.

[0101] In one implementation, the multimodal feature mapping model to be trained includes an image encoder and a text encoder. The electronic device can input the sample depth map into the image encoder of the multimodal feature mapping model to be trained, and encode the sample depth map through the image encoder to obtain the graph features corresponding to the sample depth map. The sample text is input into the text encoder of the multimodal feature mapping model to be trained, and the sample text is encoded through the text encoder to obtain the text features corresponding to the sample text, so as to obtain graph features and text features mapped to the same feature space.

[0102] Then, in step S230, the parameters of the multimodal feature mapping model to be trained are adjusted with the goal of maximizing the similarity between graph features and text features.

[0103] In this step, after the electronic device obtains the graph features corresponding to the sample depth map and the text features corresponding to the sample text, it calculates the similarity between the graph features and the text features. For example, it can calculate the cosine similarity value between the graph features and the text features, where a larger cosine similarity value indicates a greater similarity between the two. Alternatively, it can calculate the distance between the graph features and the text features, such as the Euclidean distance, where a smaller distance indicates a greater similarity between the graph features and the text features.

[0104] Next, with the goal of maximizing the similarity between the image features and the text features, the parameters of the multimodal feature mapping model to be trained are adjusted. In one implementation, the electronic device can determine the model loss based on the similarity between the image features and the text features, wherein the model loss is negatively correlated with the similarity between the image features and the text features, and the parameters of the multimodal feature mapping model to be trained are adjusted to minimize the model loss, i.e., with the goal of maximizing the similarity between the image features and the text features.

[0105] Specifically, the target program can determine the gradient of the model parameters of the multimodal feature mapping model to be trained using the backpropagation algorithm based on the model loss. Using the determined gradient and the current model parameters of the multimodal feature mapping model to be trained (i.e., the current values ​​of the model parameters), the updated model parameters of the multimodal feature mapping model to be trained (i.e., the updated values ​​of the model parameters) are determined. Then, based on the updated model parameters, the model parameters of the multimodal feature mapping model to be trained are adjusted. The determination of the gradient of the model parameters of the multimodal feature mapping model to be trained is achieved with the goal of minimizing the model loss. Minimizing the model loss enables supervised training of the mapping results of the multimodal feature mapping model to be trained, maintaining its accuracy and improving the accuracy of the mapping results.

[0106] Steps S210-S230 above constitute one iteration of the model training process for the multimodal feature mapping model to be trained. To train a better multimodal feature mapping model, the above process can be executed iteratively multiple times. That is, after step S230, based on the updated model parameters of the multimodal feature mapping model to be trained, the process returns to step S210.

[0107] The stopping conditions for the above model iterative training process may include: the number of iterations reaches a preset threshold, the iteration training time reaches a preset duration, or the model loss is less than a set loss threshold, etc.

[0108] Through the iterative training process described above, the multimodal feature mapping model is trained repeatedly until the stopping condition is met, resulting in a fully trained multimodal feature mapping model. This model can map depth maps and text to the same feature space, aligning the depth map features with the text features. This trained model can then be used to perform a 3D model matching process.

[0109] then, Figure 3A flowchart illustrating a three-dimensional model matching method in one embodiment of this specification is shown. This method is executed by an electronic device, which can be any device, equipment, platform, or cluster of devices with computing and processing capabilities. The electronic device executing the aforementioned multimodal feature mapping model training method and the electronic device executing the three-dimensional model matching method can be the same physical device or different physical devices. During the three-dimensional model matching process, such as... Figure 3 As shown, the method includes the following steps S310-S330:

[0110] In step S310, based on the first object, at least one first feature corresponding to the first object is determined using a target multimodal feature mapping model, wherein the target multimodal feature mapping model is obtained by training based on sample image-text pairs, the sample image-text pairs include corresponding text and depth maps, and the first object is a first text or a first three-dimensional model.

[0111] In some scenarios, users may need to find the 3D models they require. For example, a user might need to search for a 3D model by text input, meaning they want to be able to input some information to find the 3D model they need. For instance, if a user wants to find a 3D model of a puppy, they input the text "xx puppy," and the electronic device can recommend 3D models of "xx puppy," where "xx" can represent the breed of puppy. Alternatively, a user might need to find other similar 3D models based on a known 3D model. That is, a user wants to input a 3D model, and the electronic device can recommend other 3D models of "xx puppy" similar to the input "xx puppy."

[0112] Accordingly, the electronic device can acquire the user's input content as a first object, which can be text, referred to as the first text, or a three-dimensional model, referred to as the first three-dimensional model. Next, based on the first object, the electronic device uses a target multimodal feature mapping model, that is, inputting the first object into the target multimodal feature mapping model, processing the first object through the target multimodal feature mapping model, and determining at least one first feature corresponding to the first object.

[0113] In one embodiment, the target multimodal feature mapping model can be a multimodal feature mapping model trained using the training method provided in the foregoing embodiments, i.e., the aforementioned trained multimodal feature mapping model. This target multimodal feature mapping model is used to perform feature mapping on text and depth maps respectively, to obtain text features of the text and graph features of the depth map that are in the same feature space.

[0114] In step S320, at least one second feature corresponding to the second three-dimensional model is obtained. Each second feature is generated based on the depth map of the second three-dimensional model by using a target multimodal feature mapping model.

[0115] In one implementation, the electronic device's preset storage space pre-stores a set of three-dimensional models. This set of three-dimensional models may include features of several three-dimensional models and at least one depth map corresponding to each three-dimensional model. The features of the at least one depth map corresponding to each three-dimensional model are generated based on the depth map of the corresponding three-dimensional model using a target multimodal feature mapping model. For example, if a three-dimensional model X corresponds to M depth maps, and the features corresponding to each depth map are Y-dimensional, for example, 512-dimensional, then the features corresponding to the M depth maps corresponding to the three-dimensional model X can be an M*Y(512) matrix.

[0116] Each 3D model in the 3D model collection can be any type of 3D model, such as animal models, plant models, human models, architectural models, and various prop models in games, etc.

[0117] In this step, the electronic device can obtain at least one feature corresponding to the second three-dimensional model from the aforementioned set of three-dimensional models. For clarity, each feature corresponding to the second three-dimensional model is referred to as a second feature. In one case, this set of three-dimensional models can be called a three-dimensional model library. After the electronic device determines at least one first feature corresponding to the first object, it can obtain at least one second feature corresponding to the second three-dimensional model from the aforementioned three-dimensional model library.

[0118] After obtaining at least one first feature corresponding to the first object and at least one second feature corresponding to the second three-dimensional model, in step S330, based on the first feature corresponding to the first object and the multiple second features corresponding to the second three-dimensional model, the matching result between the first object and the second three-dimensional model is determined.

[0119] It is understandable that at least one first feature corresponding to the first object and at least one second feature corresponding to the second three-dimensional model are generated by using a target multimodal feature mapping model. The first feature and the second feature are in the same feature space. In this step, the electronic device can match the first object with the second three-dimensional model based on the first feature corresponding to the first object and the multiple second features corresponding to the second three-dimensional model to determine the matching result between the first object and the second three-dimensional model.

[0120] Understandably, the first object can be first text or a first three-dimensional model; in one embodiment, the aforementioned first object is a first three-dimensional model; such as Figure 4 As shown, the method may include the following steps S410-S460:

[0121] In step S410, a first depth map of the first 3D model under at least one preset first viewpoint is determined. The first 3D model can be any type of 3D model, such as an animal model, plant model, human model, architectural model, or various prop models in a game. In one implementation, the first 3D model can exist in the form of an .obj file, which may include the geometric information of the first 3D model, such as the spatial position information of each vertex constituting the first 3D model, the triangular facet information formed by each vertex, and texture coordinate information, etc.

[0122] In one scenario, after acquiring the first three-dimensional model, the electronic device can, based on the transformation relationship between the coordinate system of the first three-dimensional model and the aforementioned specified three-dimensional spatial coordinate system, transform the spatial position information of each vertex of the first three-dimensional model into the specified three-dimensional spatial coordinate system; then, adjust the size of the first three-dimensional model to the aforementioned specified size, for example, normalize the coordinates of each axis in the spatial position information of each vertex of the first three-dimensional model to [-1, 1], so as to adjust the first three-dimensional model to the specified size; then, adjust the center point of the first three-dimensional model to coincide with the origin O of the specified three-dimensional spatial coordinate system.

[0123] Subsequently, based on the adjusted first 3D model, a first depth map of the first 3D model under at least one preset first viewpoint is determined. In one implementation, the first depth map of the first 3D model under a preset first viewpoint can be determined, wherein the distance between the viewpoint corresponding to the determination of the first depth map of the first 3D model under a preset first viewpoint and the origin of the specified 3D spatial coordinate system can be the aforementioned specified distance or other preset distances.

[0124] In another implementation, a first depth map of the second three-dimensional model can be determined under at least two preset first perspectives, wherein the distance between the viewpoint corresponding to each first perspective and the origin of the specified three-dimensional spatial coordinate system can be the aforementioned specified distance or other preset distances.

[0125] The distance between the viewpoint corresponding to each first perspective and the origin of the specified 3D spatial coordinate system can be determined based on the distance between the viewpoint corresponding to the depth map of the second 3D model and the origin of the specified 3D spatial coordinate system. The center point of the second 3D model coincides with the origin of the specified 3D spatial coordinate system. In one case, to ensure the accuracy of the matching results, the distance between the viewpoint corresponding to each first perspective and the origin of the specified 3D spatial coordinate system is equal to the distance between the viewpoint corresponding to the depth map of the second 3D model and the origin of the specified 3D spatial coordinate system, for example, equal to the aforementioned specified distance.

[0126] In one scenario, each first viewpoint can be represented by a set of horizontal and vertical angles. The horizontal angle can be the angle between the line connecting the viewpoint's projection onto the XOZ plane (considered a horizontal plane) of the specified 3D coordinate system and the origin of the specified 3D coordinate system, i.e., the center point of the first 3D model, and the X-axis. The vertical angle can be the angle between the line connecting the viewpoint to the origin of the specified 3D coordinate system and the line connecting the viewpoint's projection onto the XOZ plane and the origin. The X-axis is the horizontal axis of the specified 3D coordinate system, the Z-axis is the vertical axis of the specified 3D coordinate system, and the Y-axis is the vertical axis of the specified 3D coordinate system.

[0127] In one scenario, the horizontal angle corresponding to at least one preset first-viewpoint may include, but is not limited to, 45 degrees, 135 degrees, 225 degrees, and 315 degrees; the vertical angle corresponding to at least one preset first-viewpoint may include, but is not limited to, 15 degrees and 30 degrees. Horizontal and vertical angles can be arbitrarily combined to obtain at least one first-viewpoint.

[0128] In one implementation, each first depth map includes a depth value of the first 3D model under the corresponding first viewpoint. The depth value of the first 3D model under the corresponding first viewpoint can represent the relative distance between each vertex of the first 3D model and the plane containing the viewpoint of the corresponding first viewpoint. The plane containing the viewpoint of the first viewpoint is perpendicular to the XOZ plane. The depth value of each pixel in each first depth map ranges from [0, 255].

[0129] Taking any one of the at least one first perspectives, i, as an example, the process of determining the first depth map is explained. Specifically, the electronic device can determine the distance between each vertex of the adjusted first 3D model and the plane where the viewpoint corresponding to the first perspective i is located in the specified 3D coordinate system, based on the spatial position information of each vertex of the adjusted first 3D model in the specified 3D coordinate system and the position information of the plane where the viewpoint corresponding to the first perspective i is located in the specified 3D coordinate system. Then, the distance between each vertex of the adjusted first 3D model and the plane where the viewpoint corresponding to the first perspective i is located is converted to a specified value range, namely [0, 255], to obtain the first depth map of the first 3D model under the first perspective i. Here, i is a positive integer.

[0130] The aforementioned process of converting the distances between each vertex of the adjusted first 3D model and the plane containing the viewpoint corresponding to the first viewpoint i to a specified range can be as follows: First, determine the largest distance from the distances between each vertex of the adjusted first 3D model and the plane containing the viewpoint corresponding to the first viewpoint i. Then, divide each distance between each vertex of the adjusted first 3D model and the plane containing the viewpoint corresponding to the first viewpoint i by this largest distance to obtain the intermediate distances corresponding to each vertex of the adjusted first 3D model. Finally, multiply these intermediate distances by 255 to convert the distances between each vertex of the adjusted first 3D model and the plane containing the viewpoint corresponding to the first viewpoint i to a specified range.

[0131] Then, in step S420, based on each first depth map, the image encoder of the target multimodal feature mapping model is used to determine the first feature corresponding to each first depth map. In this step, the electronic device can input each first depth map into the target multimodal feature mapping model, and encode the input first depth map through the image encoder of the target multimodal feature mapping model to obtain the first feature corresponding to each first depth map, that is, to obtain at least one first feature corresponding to the first object.

[0132] Next, in step S430, at least one second feature corresponding to the second three-dimensional model is obtained. Each second feature is generated based on the depth map of the second three-dimensional model using a target multimodal feature mapping model. The depth map of the second three-dimensional model may include: depth maps of each second three-dimensional model under at least one first viewpoint, with one depth map under one first viewpoint corresponding to one second feature. The second three-dimensional model may be a three-dimensional model obtained from the aforementioned three-dimensional model library.

[0133] Then, in step S440, the average value of the first features corresponding to all first depth maps is calculated to obtain the average features of the first 3D model. And in step S450, for each second 3D model, the average value of the second features corresponding to that second 3D model is calculated to obtain the average features of each second 3D model.

[0134] To improve the accuracy of the matching results to a certain extent, the electronic device can calculate the average value of the first features corresponding to all first depth maps to obtain the average features of the first three-dimensional model, and for each second three-dimensional model, calculate the average value of the second features corresponding to that second three-dimensional model to obtain the average features of each second three-dimensional model.

[0135] In one scenario, where there is only one first depth map (i.e., at least one first viewpoint), the first feature corresponding to that first depth map can be used as the average feature of the first 3D model. Correspondingly, each second 3D model has a corresponding second feature, which is the feature corresponding to the depth map of that second 3D model under that first viewpoint. The second feature corresponding to each second 3D model can be determined as the average feature of that second 3D model.

[0136] Next, in step S460, based on the average features of the first three-dimensional model and the average features of each of the second three-dimensional models, the matching result between the first three-dimensional model and each of the second three-dimensional models is determined.

[0137] In one implementation, when there is only one second 3D model, the similarity value between the first 3D model and the second 3D model can be determined based on the average features of the first 3D model and the average features of the second 3D model. If the similarity value exceeds a preset threshold, it can be determined that the first 3D model and the second 3D model match; otherwise, it can be determined that they do not match.

[0138] When there are multiple second three-dimensional models, specifically, in one embodiment, step S460 may include the following steps 31-32:

[0139] In step 31, the average features of the first 3D model are calculated, and the first similarity value between them and the average features of each of the second 3D models is calculated. In this step, the electronic device can calculate the first similarity value between the average features of the first 3D model and the average features of each of the second 3D models based on a preset similarity algorithm, and use this first similarity value as the first similarity value between the first 3D model and each of the second 3D models.

[0140] In one scenario, the preset similarity algorithm is, for example, a cosine similarity algorithm. The first similarity value is positively correlated with the cosine similarity value. For instance, the first similarity value between the first 3D model and the second 3D model is equal to their cosine similarity value. The larger the first similarity value (cosine similarity value) between the first and second 3D models, the more similar they are. In another scenario, the preset similarity algorithm is, for example, an Euclidean distance algorithm. The first similarity value is negatively correlated with the Euclidean distance. The smaller the Euclidean distance between the first and second 3D models, the larger the first similarity value between them, indicating greater similarity between the first and second 3D models.

[0141] Accordingly, in step 32, the second three-dimensional model with the largest first similarity value is determined as the three-dimensional model that matches the first three-dimensional model.

[0142] In another embodiment, to better ensure a better user experience and recommend 3D models that better match the user's search needs, after determining the first similarity value between the first 3D model and each of the second 3D models, the second 3D model with the largest corresponding first similarity value, and whose corresponding first similarity value is greater than a specified similarity threshold, is identified as the 3D model that matches the first 3D model. This specified similarity threshold ensures that second 3D models with high similarity to the first 3D model are recommended to the user.

[0143] In another embodiment, when the aforementioned first object is a first 3D model, and the first 3D model has multiple first depth maps under at least one first viewpoint, and the second 3D model also has multiple depth maps under at least one first viewpoint, the first viewpoints corresponding to the multiple first depth maps can be the same as or different from the first viewpoints corresponding to the multiple depth maps of the second 3D model. For example, the first viewpoints corresponding to the multiple first depth maps may include first viewpoint 1, first viewpoint 2, and first viewpoint 3, and the first viewpoints corresponding to the multiple depth maps of the second 3D model may include first viewpoint 1, first viewpoint 2, and first viewpoint 4. In one case, to ensure higher accuracy of the matching results, the first viewpoints corresponding to the multiple first depth maps may be the same as the first viewpoints corresponding to the multiple depth maps of the second 3D model, i.e., the first viewpoints corresponding to the multiple first depth maps may include first viewpoint 1, first viewpoint 2, and first viewpoint 3, and correspondingly, the first viewpoints corresponding to the multiple depth maps of the second 3D model may include first viewpoint 1, first viewpoint 2, and first viewpoint 3.

[0144] In yet another embodiment, where the aforementioned first object is first text; such as Figure 5 As shown, the method may include the following steps S510-S540:

[0145] In step S510, based on the first text, the text encoder of the target multimodal feature mapping model is used to determine the first feature corresponding to the first text. The first text can be the content input by the user based on the 3D model they need to find. In this step, after the electronic device acquires the first text, it can input the first text into the target multimodal feature mapping model, and the text encoder of the target multimodal feature mapping model encodes the first text to determine the first feature corresponding to the first text, that is, the first feature corresponding to the first object.

[0146] Next, in step S520, at least one second feature corresponding to the second three-dimensional model is obtained. Each second feature is generated based on the depth map of the second three-dimensional model using a target multimodal feature mapping model. The second three-dimensional model can be a three-dimensional model obtained from the aforementioned three-dimensional model library.

[0147] Next, in step S530, the first feature corresponding to the first text is calculated, and the second similarity value between it and each of the second features is calculated. In this step, the electronic device can calculate the similarity value between the first feature corresponding to the first text and each of the second features based on a preset similarity algorithm. This second similarity value is used as the second similarity value between the first feature and each of the second features.

[0148] In one scenario, the preset similarity algorithm is, for example, a cosine similarity algorithm. The second similarity value is positively correlated with the cosine similarity value; the second similarity value between the first feature and the second feature is equal to their cosine similarity value. A larger second similarity value, i.e., the cosine similarity value, indicates a greater similarity between the first and second features. In another scenario, the preset similarity algorithm is, for example, an Euclidean distance algorithm. The second similarity value is negatively correlated with the Euclidean distance; a smaller Euclidean distance between the first and second features indicates a larger second similarity value, correspondingly indicating a greater similarity between the first and second features.

[0149] Then, in step S540, the matching result between the first text and the second three-dimensional model is determined based on each second similarity value.

[0150] In one embodiment, when there is only one second 3D model, each second similarity value includes the similarity value between the first feature of the first text and at least one second feature corresponding to the second 3D model. The magnitude of each second similarity value relative to a specified threshold can be determined. If more than a specified number of second similarity values ​​are greater than the specified threshold, it can be determined that the first text matches the second 3D model; conversely, if less than a specified number of second similarity values ​​are greater than the specified threshold, it can be determined that the first text does not match the second 3D model.

[0151] In yet another embodiment, there are multiple second three-dimensional models; correspondingly, step S540 may include the following steps 41-43:

[0152] In step 41, based on each second similarity value, N third features with the largest corresponding second similarity values ​​are determined from multiple second features, where N is a positive integer. The specific value of N can be set according to requirements. In this step, the electronic device can compare the magnitudes of each second similarity value and, based on the magnitudes of each second similarity value, determine the N features with the largest corresponding second similarity values ​​from multiple second features as the third features.

[0153] Then, in step 42, the second three-dimensional model corresponding to each third feature is determined. Based on the correspondence between each third feature (i.e., the N second features with the largest second similarity values) and the second three-dimensional model, the second three-dimensional model corresponding to each third feature is determined.

[0154] Next, in step 43, the second 3D model with the most corresponding third features is determined as the 3D model matching the first text. For example, N is set to 3, and the third features include third feature 1, third feature 2, and third feature 3. Third feature 1 and third feature 2 correspond to second 3D model A, and third feature 3 corresponds to second 3D model B. In this case, second 3D model A is the second 3D model with the most corresponding third features, and second 3D model A is determined as the 3D model matching the first text. As another example, N is set to 3, and the third features include third feature 1, third feature 2, and third feature 3. Third feature 1 corresponds to second 3D model A, third feature 3 corresponds to second 3D model B, and third feature 2 corresponds to second 3D model C. The second similarity value 1 corresponding to third feature 1 > the second similarity value 2 corresponding to third feature 2 > the second similarity value 2 corresponding to third feature 31. In this case, the second 3D model A with the largest corresponding second similarity value can be determined as the 3D model matching the first text.

[0155] In another embodiment, after determining a second 3D model that matches the first object, such as the first text or the first 3D model, the second 3D model that matches the first object can be displayed to users who need to find 3D models for viewing.

[0156] In this embodiment, by training a target multimodal feature mapping model based on the corresponding text and depth map, at least one first feature corresponding to the first object can be determined. Then, at least one second feature corresponding to the second three-dimensional model is obtained. Each second feature is generated based on the depth map of the second three-dimensional model using the target multimodal feature mapping model. This ensures that at least one first feature of the first object and at least one second feature corresponding to the second three-dimensional model are aligned. Then, based on the aligned first feature of the first object and multiple second features corresponding to the second three-dimensional model, the first object and the second three-dimensional model are matched to determine the matching result between the first object and the second three-dimensional model. This can achieve matching between text and three-dimensional models, or matching between three-dimensional models.

[0157] The foregoing description describes specific embodiments of this specification; other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than those shown in the embodiments, and the desired result may still be achieved. Furthermore, the processes depicted in the drawings do not necessarily need to follow the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0158] Corresponding to the above method embodiments, this specification provides a three-dimensional model matching device 600, the schematic block diagram of which is shown below. Figure 6 As shown, it includes:

[0159] The first determining module 610 is configured to determine at least one first feature corresponding to the first object using a target multimodal feature mapping model, wherein the target multimodal feature mapping model is obtained by training based on sample image-text pairs, the sample image-text pairs include corresponding text and depth maps, and the first object is a first text or a first three-dimensional model.

[0160] The first acquisition module 620 is configured to acquire at least one second feature corresponding to the second three-dimensional model, wherein each second feature is generated based on the depth map of the second three-dimensional model by using the target multimodal feature mapping model;

[0161] The second determining module 630 is configured to determine the matching result between the first object and the second three-dimensional model based on the first feature corresponding to the first object and multiple second features corresponding to the second three-dimensional model.

[0162] In one alternative implementation, the first object is a first three-dimensional model;

[0163] The first determining module 610 includes:

[0164] The first determining unit (not shown in the figure) is configured to determine the first depth map of the first three-dimensional model under at least one preset first viewpoint;

[0165] The second determining unit (not shown in the figure) is configured to determine the first feature corresponding to each first depth map using the image encoder of the target multimodal feature mapping model based on each first depth map.

[0166] In one alternative implementation, the depth map of the second three-dimensional model includes: depth maps of each of the second three-dimensional models from at least one first viewpoint;

[0167] The second determining module 630 includes:

[0168] The first calculation unit (not shown in the figure) is configured to calculate the average value of the first features corresponding to all first depth maps to obtain the average features of the first three-dimensional model.

[0169] The second calculation unit (not shown in the figure) is configured to calculate the average value of the second feature corresponding to each second three-dimensional model, so as to obtain the average feature of each second three-dimensional model.

[0170] The third determining unit (not shown in the figure) is configured to determine the matching result between the first three-dimensional model and each of the second three-dimensional models based on the average features of the first three-dimensional model and the average features of each of the second three-dimensional models.

[0171] In one optional implementation, the third determining unit is specifically configured to calculate a first similarity value between the average features of the first three-dimensional model and the average features of each of the second three-dimensional models;

[0172] The second 3D model with the highest first similarity value is determined as the 3D model that matches the first 3D model.

[0173] In one alternative implementation, the first object is first text;

[0174] The first determining module 610 is specifically configured to determine the first feature corresponding to the first text using the text encoder of the target multimodal feature mapping model based on the first text.

[0175] In one alternative embodiment, the second determining module 630 includes:

[0176] The third calculation unit (not shown in the figure) is configured to calculate the first feature corresponding to the first text and the second similarity value between each of the second features;

[0177] The fourth determining unit (not shown in the figure) is configured to determine the matching result between the first text and the second three-dimensional model based on each second similarity value.

[0178] In one alternative implementation, there are multiple second three-dimensional models;

[0179] The fourth determining unit is specifically configured to determine, based on each second similarity value, the N third features with the largest corresponding second similarity values ​​from the plurality of second features, where N is a positive integer;

[0180] Determine the second three-dimensional model corresponding to each third feature;

[0181] The second 3D model with the most corresponding third features is determined as the 3D model that matches the first text.

[0182] Corresponding to the above method embodiments, this specification provides a training device 700 for a multimodal feature mapping model, the schematic block diagram of which is shown below. Figure 7 As shown, it includes:

[0183] The second acquisition module 710 is configured to acquire sample image-text pairs in the training dataset, wherein the sample image-text pairs include sample depth maps and their corresponding sample texts.

[0184] The third determining module 720 is configured to determine the graph features corresponding to the sample depth map and the text features corresponding to the sample text using a multimodal feature mapping model to be trained, based on the sample depth map and the sample text.

[0185] The adjustment module 730 is configured to adjust the parameters of the multimodal feature mapping model to be trained with the goal of maximizing the similarity between the graph features and the text features.

[0186] In one alternative embodiment, the device further includes:

[0187] The third acquisition module (not shown in the figure) is configured to acquire a color image and its corresponding second text before acquiring the sample image-text pairs in the training dataset;

[0188] The module (not shown in the figure) is configured to obtain a second depth map corresponding to the color image using a depth estimation model, wherein the second depth map includes the depth value of each pixel, and the depth estimation model is trained based on the sample color image and its corresponding depth map.

[0189] The first component module (not shown in the figure) is configured to form a set of sample image-text pairs based on the second text and the second depth map, and to include them in the training dataset.

[0190] In one alternative embodiment, the device further includes:

[0191] The fourth acquisition module (not shown in the figure) is configured to acquire the three-dimensional model of the sample before acquiring the sample image-text pairs in the training dataset;

[0192] The rendering module (not shown in the figure) is configured to render the sample 3D model from at least one second perspective to obtain a 3D model rendering map and a third depth map corresponding to each second perspective, wherein the third depth map includes the depth value of the sample 3D model under the corresponding second perspective.

[0193] The fourth determining module (not shown in the figure) is configured to determine the third text corresponding to each 3D model rendering image by using an image description generation model based on each 3D model rendering image. The image description generation model is used to generate the corresponding text based on the input image.

[0194] The second component module (not shown in the figure) is configured to form sample image-text pairs corresponding to each second viewpoint based on the third text corresponding to each 3D model rendering and the third depth map corresponding to each sample 3D model in each second viewpoint, and to include them in the training dataset.

[0195] The above-described apparatus embodiments correspond to the method embodiments, and detailed descriptions can be found in the description of the method embodiments section, which will not be repeated here. The apparatus embodiments are derived based on the corresponding method embodiments and have the same technical effects as the corresponding method embodiments; detailed descriptions can be found in the corresponding method embodiments.

[0196] This specification also provides a computer-readable storage medium storing a computer program that, when executed in a computer, causes the computer to perform the three-dimensional model matching method or the multimodal feature mapping model training method provided in this specification.

[0197] This specification also provides a computing device, including a memory and a processor. The memory stores executable code, and when the processor executes the executable code, it implements the three-dimensional model matching method or the training method of the multimodal feature mapping model provided in this specification.

[0198] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the embodiments for storage media and computing devices are basically similar to the method embodiments, so they are described more simply; relevant parts can be referred to the descriptions of the method embodiments.

[0199] Those skilled in the art will recognize that the functions described in the embodiments of the present invention in one or more of the above examples can be implemented using hardware, software, firmware, or any combination thereof. When implemented in software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or code on a computer-readable medium.

[0200] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, or improvements made based on the technical solutions of the present invention should be included within the scope of protection of the present invention.

Claims

1. A three-dimensional model matching method, comprising: Based on a first object, a target multimodal feature mapping model is used to determine at least one first feature corresponding to the first object. The target multimodal feature mapping model is trained to maximize the similarity between the text features of the text and the graph features of the depth map in a sample image-text pair. The first object is either a first text or a first 3D model. The target multimodal feature mapping model is used to perform feature mapping on the text and the depth map respectively to obtain the text features of the text and the graph features of the depth map in the same feature space. The sample image-text pair is constructed through at least one of the following methods: based on a two-dimensional color image and its corresponding text, combined with a depth estimation model, a sample image-text pair including a depth map and text with a corresponding relationship is constructed; and / or, based on a rendered image and depth map obtained by rendering a 3D model, combined with an image description generation model, a sample image-text pair including a depth map and text with a corresponding relationship is constructed. At least one second feature corresponding to the second three-dimensional model is obtained, and each second feature is generated based on the depth map of the second three-dimensional model by using the target multimodal feature mapping model; Based on the first feature corresponding to the first object and multiple second features corresponding to the second three-dimensional model, the matching result between the first object and the second three-dimensional model is determined.

2. The method as described in claim 1, wherein the first object is a first three-dimensional model; Determining at least one first feature corresponding to the first object includes: Determine the first depth map of the first three-dimensional model under at least one preset first viewpoint; Based on each first depth map, the first feature corresponding to each first depth map is determined using the image encoder of the target multimodal feature mapping model.

3. The method as described in claim 2, wherein, The depth map of the second three-dimensional model includes: the depth map of each second three-dimensional model under at least one first viewpoint; Determining the matching result between the first object and the second 3D model includes: Calculate the average value of the first features corresponding to all first depth maps to obtain the average features of the first 3D model; For each second three-dimensional model, calculate the average value of the second feature corresponding to that second three-dimensional model to obtain the average feature of each second three-dimensional model; Based on the average features of the first 3D model and the average features of each of the second 3D models, the matching results between the first 3D model and each of the second 3D models are determined.

4. The method of claim 3, wherein, Determining the matching results between the first 3D model and each of the second 3D models includes: Calculate the average features of the first 3D model and the first similarity value between them and the average features of each of the second 3D models; The second 3D model with the highest first similarity value is determined as the 3D model that matches the first 3D model.

5. The method as described in claim 1, wherein the first object is the first text; Determining at least one first feature corresponding to the first object includes: Based on the first text, the text encoder of the target multimodal feature mapping model is used to determine the first feature corresponding to the first text.

6. The method of claim 5, wherein, Determining the matching result between the first object and the second 3D model includes: Calculate the first feature corresponding to the first text and the second similarity value between it and each of the second features; Based on each second similarity value, the matching result between the first text and the second three-dimensional model is determined.

7. The method as described in claim 6, wherein the second three-dimensional model comprises multiple models; The step of determining the matching result between the first text and the second 3D model based on each second similarity value includes: Based on each second similarity value, N third features with the largest corresponding second similarity values ​​are determined from the plurality of second features, where N is a positive integer; Determine the second three-dimensional model corresponding to each third feature; The second 3D model with the most corresponding third features is determined as the 3D model that matches the first text.

8. A training method for a multimodal feature mapping model, comprising: Obtain sample image-text pairs from the training dataset, wherein each sample image-text pair includes a sample depth map and its corresponding sample text; wherein at least some of the sample image-text pairs in the training dataset are constructed through at least one of the following methods: based on a two-dimensional color image and its corresponding text, combined with a depth estimation model, to construct sample image-text pairs including depth maps and text with corresponding relationships; and / or, based on a rendered image and depth map obtained by rendering a three-dimensional model, combined with an image description generation model, to construct sample image-text pairs including depth maps and text with corresponding relationships; Based on the sample depth map and the sample text, the multimodal feature mapping model to be trained is used to determine the graph features corresponding to the sample depth map and the text features corresponding to the sample text. With the goal of maximizing the similarity between the graph features and the text features, the parameters of the multimodal feature mapping model to be trained are adjusted to obtain a target multimodal feature mapping model. The target multimodal feature mapping model is used to perform feature mapping on the text and the depth map respectively to obtain the text features of the text and the graph features of the depth map that are in the same feature space.

9. The method of claim 8, further comprising, before obtaining the sample image-text pairs in the training dataset: Obtain the color image and its corresponding second text; Based on the color image, a second depth map corresponding to the color image is obtained using a depth estimation model, wherein the second depth map includes the depth value of each pixel, and the depth estimation model is trained based on the sample color image and its corresponding depth map; Based on the second text and the second depth map, a set of sample image-text pairs are formed and included in the training dataset.

10. The method of claim 8, further comprising, before obtaining the sample image-text pairs in the training dataset: Obtain the 3D model of the sample; The sample 3D model is rendered from at least one second perspective to obtain a 3D model rendering map and a third depth map corresponding to each second perspective, wherein the third depth map includes the depth value of the sample 3D model under the corresponding second perspective. Based on the rendering images of each 3D model, an image description generation model is used to determine the third text corresponding to each 3D model rendering image. The image description generation model is used to generate the corresponding text based on the input image. Based on the third text corresponding to each 3D model rendering and the third depth map corresponding to each sample 3D model in each second viewpoint, sample image-text pairs corresponding to each second viewpoint are formed and included in the training dataset.

11. A three-dimensional model matching device, comprising: The first determining module is configured to determine at least one first feature corresponding to the first object using a target multimodal feature mapping model, wherein the target multimodal feature mapping model is trained to maximize the similarity between the text features of the text and the graph features of the depth map in the sample image-text pair, the first object is a first text or a first 3D model, the target multimodal feature mapping model is used to perform feature mapping on the text and the depth map respectively to obtain the text features of the text and the graph features of the depth map in the same feature space, and the sample image-text pair is constructed by at least one of the following methods: based on a two-dimensional color image and its corresponding text, combined with a depth estimation model, to construct a sample image-text pair including a depth map and text with a corresponding relationship; and / or, based on a rendered image and a depth map obtained by rendering a 3D model, combined with an image description generation model, to construct a sample image-text pair including a depth map and text with a corresponding relationship; The first acquisition module is configured to acquire at least one second feature corresponding to the second three-dimensional model, wherein each second feature is generated based on the depth map of the second three-dimensional model by using the target multimodal feature mapping model; The second determining module is configured to determine the matching result between the first object and the second three-dimensional model based on the first feature corresponding to the first object and multiple second features corresponding to the second three-dimensional model.

12. A training device for a multimodal feature mapping model, comprising: The second acquisition module is configured to acquire sample image-text pairs in the training dataset, wherein the sample image-text pairs include sample depth maps and their corresponding sample text; wherein at least some of the sample image-text pairs in the training dataset are constructed through at least one of the following methods: based on two-dimensional color images and their corresponding text, combined with a depth estimation model, to construct sample image-text pairs including depth maps and text with corresponding relationships; and / or, based on rendered images and depth maps obtained by rendering a three-dimensional model, combined with an image description generation model, to construct sample image-text pairs including depth maps and text with corresponding relationships; The third determining module is configured to determine the graph features corresponding to the sample depth map and the text features corresponding to the sample text using a multimodal feature mapping model to be trained, based on the sample depth map and the sample text. The adjustment module is configured to adjust the parameters of the multimodal feature mapping model to be trained with the goal of maximizing the similarity between the graph features and the text features, so as to obtain a target multimodal feature mapping model. The target multimodal feature mapping model is used to perform feature mapping on the text and the depth map respectively, so as to obtain the text features of the text and the graph features of the depth map in the same feature space.

13. A computing device comprising a memory and a processor, wherein, The memory stores executable code, and when the processor executes the executable code, it implements the method of any one of claims 1-10.

Images