A method, apparatus, electronic device and storage medium for cross-modal search

Abstract

The application provides a method and device for cross-modal search, electronic equipment and storage medium. The method comprises: performing semantic segmentation on a three-dimensional object based on point cloud data of the three-dimensional object to obtain a plurality of components of the three-dimensional object; obtaining object features of the three-dimensional object according to the plurality of components and the point cloud data of the three-dimensional object, wherein the object features comprise component features of the plurality of components; determining similarity between the object features of the three-dimensional object and semantic features of a plurality of candidate texts, wherein the semantic features comprise word vectors of a plurality of words in the corresponding candidate texts; and determining a candidate text with the greatest similarity between the semantic features and the object features in the plurality of candidate texts as a text most matched with the three-dimensional object searched by the cross-modal search. The method can efficiently and quickly extract the object features of the three-dimensional object, and improves the accuracy of the cross-modal search based on a local alignment search method, so that the cross-modal search is more efficient and accurate.

Description

Technical Field

[0001] This application relates to the field of cross-modal search, and more specifically, to a method, apparatus, electronic device, and storage medium for cross-modal search within the field of cross-modal search. Background Technology

[0002] In the development of internet technology, massive amounts of data have become a product of its evolution. Examples include common data types such as video, audio, images, and text. When people need to search for data, they can use search engines to search for one or more of these data types, such as video, audio, images, and text, to obtain the corresponding search results.

[0003] The process of a user querying or searching using any one of the above data types is called a "monomodular search." Conversely, the process of a user querying or searching using any multiple of the above data types is called a "multimodal search."

[0004] For example, a common search method in multimodal search is cross-modal search, which involves using data of one type or modality to retrieve data of another type or modality. For instance, using image modality data to retrieve text modality data, or using text modality data to retrieve image modality data.

[0005] In one possible implementation, during the cross-modal search of a three-dimensional (3D) model or object and text, the object features and text features or semantic features of the 3D object can be mapped to a global vector, and the similarity score between the 3D object and the text can be calculated on the global vector. The above cross-modal search process is called the "global alignment search" process.

[0006] The global alignment search method described above can only extract a feature vector from 3D objects and text. When encountering 3D objects with complex structures or text with rich semantic information, the extracted information or features are limited, which may lead to low computational accuracy.

[0007] Furthermore, the aforementioned three-dimensional objects are usually represented using voxels or multiple views. Since the voxel representation method has a large overall data volume and a low proportion of effective information, the computational load is relatively large when using deep neural networks for feature extraction. When using multiple views to represent three-dimensional objects, it may be necessary to extract and fuse features from multiple images from different angles, which may lead to errors in feature extraction of three-dimensional objects and a large workload.

[0008] In summary, how to avoid low computational accuracy and inaccurate information extraction during cross-modal search has become an urgent problem to be solved. Summary of the Invention

[0009] This application provides a method, apparatus, electronic device, and storage medium for cross-modal search. The method can efficiently and quickly extract the object features of a 3D object, and the search method based on local alignment improves the accuracy of cross-modal search, making cross-modal search more efficient and accurate.

[0010] Firstly, a cross-modal search method is provided, comprising: semantically segmenting the three-dimensional object based on point cloud data to obtain multiple parts of the three-dimensional object; obtaining object features of the three-dimensional object based on the multiple parts and the point cloud data of the multiple three-dimensional object, the object features including the part features of the multiple parts; determining the similarity between the object features of the three-dimensional object and the semantic features of multiple candidate texts, the semantic features including word vectors of multiple words in the corresponding candidate texts; and determining the candidate text with the highest similarity between the semantic features and the object features among the multiple candidate texts as the text that best matches the three-dimensional object found through cross-modal search.

[0011] In the aforementioned technical solution, this application proposes a cross-modal search method for data processing. Specifically, the process involves first semantically segmenting the 3D object based on its point cloud data to obtain multiple components. Then, using these components and the point cloud data, the object features of the 3D object are derived. By calculating the similarity between these object features and the semantic features of multiple candidate texts in a text library, the text with the highest similarity to the 3D object is obtained. This process, through processing the point cloud data of the 3D object, ultimately yields its object features, making feature extraction more accurate. Furthermore, by using the text with the highest similarity as the best-matching text found in the cross-modal search, the accuracy of the cross-modal search process is ensured.

[0012] In conjunction with the first aspect, in some possible implementations, determining the similarity between the object features of the 3D object and the semantic features of multiple candidate texts includes: inputting the object features of the 3D object and the semantic features of the multiple candidate texts into a cross-modal search model; and using the cross-modal search model to determine the bulldozer distance between the object features of the 3D object and the semantic features of the multiple candidate texts.

[0013] In combination with the first aspect and the above implementation methods, in some possible implementation methods, determining the bulldozer distance between the object features of the three-dimensional object and the semantic features of multiple candidate texts includes: for any one of the multiple candidate texts, determining the bulldozer distance between the component features of each part of the three-dimensional object and the word vector of each word of the candidate text; and performing a fusion process on the bulldozer distance between the component features of each part and the word vector of each word of the candidate text to obtain the bulldozer distance between the object features of the three-dimensional object and the semantic features of the candidate text.

[0014] In the above technical solution, when determining the bulldozer distance between the object features of a 3D object and the semantic features of multiple candidate texts, since the semantic features of the text include multiple word vectors and the object features of the 3D object include multiple component features, a search method of local alignment can be used to calculate the bulldozer distance between each word vector and each component feature pairwise. Then, the final bulldozer distance between the object features of the 3D object and the semantic features of the text is obtained by fusing them. This ensures the accuracy of cross-modal matching of 3D objects and improves the matching precision.

[0015] Combining the first aspect and the above implementation methods, in some possible implementation methods, the point cloud data of the 3D object is used to perform semantic segmentation on the 3D object to obtain multiple components of the 3D object. This includes: inputting the point cloud data of the 3D object into a semantic segmentation model to obtain the point features of each point cloud of the 3D object in different dimensions, wherein the point features of each point cloud in the largest dimension constitute the global features of the 3D object; concatenating the point features of each point cloud in different dimensions and the global features of the 3D object to obtain the target features of the 3D object; and classifying each point cloud corresponding to the target features to obtain the multiple components of the 3D object after semantic segmentation.

[0016] Combining the first aspect and the above implementation methods, in some possible implementation methods, obtaining the object features of the three-dimensional object based on the point cloud data of the multiple components and the three-dimensional object includes: fusing the point features of each point cloud in different dimensions and the color information of each point cloud through the semantic segmentation model to obtain the point embedding features of each point cloud in the multiple point clouds; and fusing the point embedding features of each point cloud with the multiple components to obtain the object features of the three-dimensional object.

[0017] The above technical solution proposes a method for obtaining the object features of a 3D object. First, it fuses the point features of each point cloud of the 3D object across different dimensions with the color information of each point cloud to obtain the point embedding features of each point cloud. Then, it fuses the point embedding features of each point cloud with the obtained multiple components to obtain the final object features. This process considers the matching and correspondence between the point features of each point cloud and the segmented components, ensuring the accuracy and precision of obtaining the object features of the 3D object, thereby guaranteeing the efficiency of cross-modal search.

[0018] In combination with the first aspect and the above implementation, in some possible implementations, the method further includes: inputting each of the multiple candidate texts into a text feature extraction model to obtain multiple word vectors containing contextual information of each candidate text, wherein the semantic features of each candidate text include multiple word vectors corresponding to each candidate text.

[0019] In conjunction with the first aspect and the above implementation methods, in some possible implementation methods, the method further includes: acquiring a first semantic feature of positive sample text corresponding to a sample 3D object, and a second semantic feature of negative sample text; determining the bulldozer distance between the object feature of the sample 3D object and the first semantic feature, and the bulldozer distance between the object feature of the sample 3D object and the second semantic feature; training the cross-modal search model based on the bulldozer distance between the object feature of the 3D object and the first semantic feature, and the bulldozer distance between the object feature of the 3D object and the second semantic feature; or, acquiring a first object feature of a positive sample 3D object corresponding to sample text, and a second object feature of a negative sample 3D object; determining the bulldozer distance between the semantic feature of the sample text and the first object feature, and the bulldozer distance between the semantic feature of the sample text and the second object feature; training the cross-modal search model based on the bulldozer distance between the semantic feature of the sample text and the first object, and the bulldozer distance between the semantic feature of the sample text and the second object.

[0020] In summary, this application proposes a cross-modal search method for data processing. The specific process involves first semantically segmenting the 3D object based on its point cloud data to obtain multiple components. Then, using these components and the point cloud data, the object features of the 3D object are derived. The similarity between these object features and the semantic features of multiple candidate texts in a text library is calculated to obtain the text with the highest similarity to the 3D object. This process, by processing the point cloud data of the 3D object, ultimately yields its object features, making feature extraction more accurate. Furthermore, by using the text with the highest similarity as the best-matching text for cross-modal search, the accuracy of the cross-modal search process is ensured.

[0021] When determining the bulldozer distance between the object features of a 3D object and the semantic features of multiple candidate texts, since the semantic features of the text include multiple word vectors and the object features of the 3D object include multiple component features, a local alignment search method can be used to calculate the bulldozer distance pairwise between each word vector and each component feature. Then, the results are fused to obtain the final bulldozer distance between the object features of the 3D object and the semantic features of the text. This ensures the accuracy of cross-modal matching of 3D objects and improves the matching precision.

[0022] To obtain the object features of a 3D object, a scheme is proposed. First, the point features of each point cloud of the 3D object in different dimensions and the color information of each point cloud are fused to obtain the point embedding features of each point cloud. Then, the point embedding features of each point cloud are fused with the obtained multiple parts to obtain the final object features. This process considers the matching and correspondence between the point features of each point cloud and the multiple segmented parts, ensuring the accuracy and precision of obtaining the object features of the 3D object, thereby guaranteeing the efficiency of cross-modal search.

[0023] Secondly, a cross-modal search apparatus is provided, comprising: a semantic segmentation module for performing semantic segmentation on a 3D object based on point cloud data of the 3D object to obtain multiple parts of the 3D object; a first processing module for obtaining object features of the 3D object based on the multiple parts and the point cloud data of the 3D object, the object features including part features of the multiple parts; and a determination module for determining the similarity between the object features of the 3D object and the semantic features of multiple candidate texts, the semantic features including word vectors of multiple words in the corresponding candidate texts; and determining the candidate text with the highest similarity between the semantic features and the object features among the multiple candidate texts as the text that best matches the 3D object found through cross-modal search.

[0024] In conjunction with the second aspect, in some possible implementations, the determining module is specifically used to: input the object features of the 3D object and the semantic features of the multiple candidate texts into a cross-modal search model; and determine the bulldozer distance between the object features of the 3D object and the semantic features of the multiple candidate texts through the cross-modal search model.

[0025] In conjunction with the second aspect and the above implementation methods, in some possible implementation methods, the determining module is further configured to: for any one of the plurality of candidate texts, determine the bulldozer distance between the component features of each part of the three-dimensional object and the word vector of each word of the candidate text; perform a fusion process on the bulldozer distance between the component features of each part and the word vector of each word of the candidate text to obtain the bulldozer distance between the object features of the three-dimensional object and the candidate text.

[0026] Combining the second aspect and the above implementation methods, in some possible implementation methods, the semantic segmentation module is specifically used to: input the point cloud data of the 3D object into the semantic segmentation model, obtain the point features of each point cloud of the 3D object in different dimensions, wherein the point features of each point cloud in the largest dimension constitute the global features of the 3D object; concatenate the point features of each point cloud in different dimensions and the global features of the 3D object to obtain the target features of the 3D object; classify each point cloud corresponding to the target features to obtain the multiple components of the 3D object after semantic segmentation.

[0027] In combination with the second aspect and the above implementation methods, in some possible implementation methods, the first processing module is specifically used to: fuse the point features of each point cloud in different dimensions and the color information of each point cloud through the semantic segmentation model to obtain the point embedding features of each point cloud in the multiple point clouds; and fuse the point embedding features of each point cloud with the multiple components to obtain the object features of the three-dimensional object.

[0028] In conjunction with the second aspect and the above implementation, in some possible implementations, the device further includes: an acquisition module, used to input each of the plurality of candidate texts into a text feature extraction model to acquire a plurality of word vectors containing contextual information of each candidate text, wherein the semantic features of each candidate text include the plurality of word vectors corresponding to each candidate text.

[0029] In conjunction with the second aspect and the above implementation methods, in some possible implementation methods, the device further includes: a second processing module, configured to acquire a first semantic feature of positive sample text corresponding to a sample 3D object, and a second semantic feature of negative sample text; determine the bulldozer distance between the object feature of the sample 3D object and the first semantic feature, and the bulldozer distance between the object feature of the sample 3D object and the second semantic feature; train the cross-modal search model based on the bulldozer distance between the object feature of the 3D object and the first semantic feature, and the bulldozer distance between the object feature of the 3D object and the second semantic feature; or, acquire a first object feature of a positive sample 3D object corresponding to sample text, and a second object feature of a negative sample 3D object; determine the bulldozer distance between the semantic feature of the sample text and the first object feature, and the bulldozer distance between the semantic feature of the sample text and the second object feature; train the cross-modal search model based on the bulldozer distance between the semantic feature of the sample text and the first object, and the bulldozer distance between the semantic feature of the sample text and the second object.

[0030] Thirdly, an electronic device is provided, including a memory and a processor. The memory is used to store executable program code, and the processor is used to call and run the executable program code from the memory, causing the electronic device to perform the methods of the first aspect or any possible implementation thereof.

[0031] Fourthly, a computer program product is provided, comprising: computer program code, which, when run on a computer, causes the computer to perform the methods described in the first aspect or any possible implementation thereof.

[0032] Fifthly, a computer-readable storage medium is provided that stores computer program code, which, when executed on a computer, causes the computer to perform the methods described in the first aspect or any possible implementation thereof. Attached Figure Description

[0033] Figure 1 This is a schematic flowchart illustrating a cross-modal search method provided in an embodiment of this application;

[0034] Figure 2 This is a schematic diagram of a cross-modal search scenario provided in an embodiment of this application;

[0035] Figure 3 This is a schematic diagram of the structure of a cross-modal search device provided in an embodiment of this application;

[0036] Figure 4This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0037] The technical solutions in this application will be clearly and thoroughly described below with reference to the accompanying drawings. In the description of the embodiments of this application, unless otherwise stated, " / " means "or," for example, A / B can mean A or B. "And / or" in the text is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Furthermore, in the description of the embodiments of this application, "multiple" refers to two or more than two.

[0038] Hereinafter, the terms "first" and "second" are used for descriptive purposes only and should not be construed as implying or suggesting relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature.

[0039] Before introducing a cross-modal search method provided in the embodiments of this application, some terms in the embodiments of this application will be explained first.

[0040] Modality refers to the form in which data exists, such as text, audio, images, video, and other text formats.

[0041] Cross-modal retrieval, also known as cross-modal search, refers to using one type of data as a query to retrieve another related type of data. Examples include retrieving text data or searching for image data, or retrieving image data or searching for video data.

[0042] After introducing the basic concepts, the following is a detailed description of a cross-modal search method provided by the embodiments of this application.

[0043] Figure 1 This is a schematic flowchart of a cross-modal search method provided in an embodiment of this application.

[0044] For example, such as Figure 1 As shown, the method 100 includes:

[0045] 101. Based on the point cloud data of a 3D object, semantic segmentation is performed on the 3D object to obtain multiple parts of the 3D object.

[0046] It should be understood that the three-dimensional objects in this application embodiment are represented using point cloud data. Therefore, each point cloud corresponds to a single point cloud data set. Thus, in this application embodiment, the semantic segmentation process of the three-dimensional object can be achieved based on the point cloud data of each point cloud of the three-dimensional object.

[0047] One possible implementation involves semantic segmentation of a 3D object based on its point cloud data, specifically including:

[0048] The point cloud data of a 3D object is input into a semantic segmentation model to obtain the point features of each point cloud of the 3D object in different dimensions. The point features of each point cloud in the largest dimension constitute the global features of the 3D object.

[0049] The point features of each point cloud in different dimensions and the global features of the 3D object are stitched together to obtain the target features of the 3D object.

[0050] Classify each point cloud corresponding to the target feature to obtain multiple components of the 3D object after semantic segmentation.

[0051] Optionally, the point cloud data of a 3D object includes point cloud coordinates, color information, normal vectors, and other information. In this embodiment, point cloud data refers to point cloud coordinates and color information.

[0052] In this process, the point features of each point cloud of a 3D object in different dimensions can be extracted using the convolutional kernels of multiple convolutional layers from low to high levels in the semantic segmentation model. First, the convolutional kernels of the convolutional layers extract low-level or low-dimensional features (such as vertical features, horizontal edge features, and other local features) of each point cloud of the 3D object. These low-level features may be features that all point clouds of 3D objects possess. As the number of layers increases, the convolutional layers in the middle layers can extract intermediate or intermediate-dimensional features of each point cloud of the 3D object. These intermediate features are, to some extent, a recombination of the low-level features, making the intermediate features closer to the entity of the 3D object. Finally, after passing through the highest-dimensional convolutional layer near the output part, the high-level or high-dimensional features of each point cloud of the 3D object are obtained. The high-level features can also be seen as a combination of intermediate features, so that the final feature is closest to the 3D object.

[0053] Through the above process, the point features of each point cloud of a 3D object in different dimensions can be obtained.

[0054] It should be understood that in the process of semantic segmentation of 3D objects, the input of the semantic segmentation model is point cloud coordinates and other information, such as color information and vector information.

[0055] Semantic segmentation models can be divided into two parts: the first part is used to extract global features of 3D objects, and the second part is used for point cloud classification or point cloud segmentation.

[0056] For the first part, the point cloud data can first be transformed by an affine transformation model (T-Net), and then the shared multilayer perceptron (MLP) extracts features point by point. Then, it is transformed by another T-Net module, and the shared MLP extracts features point by point again. This process is repeated until the last MLP is used, and then max pooling is performed to obtain the global features in the highest dimension.

[0057] For the second part, the global features and the point features in each dimension need to be concatenated. This is equivalent to each point cloud performing a search in the global features (that is, finding out where a single point cloud should be located and which category it should belong to in the global features), and then classifying each point cloud through an MLP.

[0058] In one possible implementation, the semantic segmentation model used in the semantic segmentation of 3D objects described above is the PointNet network.

[0059] PointNet is a deep learning model for processing point cloud data, enabling the classification or segmentation of 3D objects. The PointNet network works by directly inputting point cloud data. For each N*3 point cloud coordinate input, the network first uses a T-Net to spatially align it, then uses an MLP to map it to a 64-dimensional space, performs further alignment, and finally maps it to a 1024-dimensional space. This results in a 1024-dimensional vector representation for each point cloud, and these 1024-dimensional vectors form the global features of the 3D object. For segmentation tasks, since each point needs to be classified, global features alone are insufficient. It's also necessary to extract point features from each point cloud in different dimensions. These point features are then concatenated with the global features (concatenating the 1024-dimensional features with the 64-dimensional point-by-point features of the point cloud, resulting in 1088-dimensional features) to obtain the target features of the 3D object. Finally, an MLP is used to classify each point cloud, resulting in multiple segmented parts.

[0060] The following is a brief introduction to a cross-modal search method provided in this application embodiment, using a circular stool as an example of a three-dimensional object.

[0061] Figure 2 This is a schematic diagram of a cross-modal search scenario provided in an embodiment of this application.

[0062] For example, such as Figure 2 As shown, the 3D object is a circular stool, which can be viewed as a 3D object composed of a large number of point clouds. Each point cloud corresponds to a point cloud coordinate and color information. Based on this, the point cloud coordinates and color information of each point cloud are input into a semantic segmentation network. Through multiple MLP processes, the point features of each point cloud in different dimensions are obtained. Four cuboids of varying thicknesses represent the processing steps for different dimensions. By sequentially concatenating the point features of each point cloud in each dimension, the target features of the circular stool can be obtained. Furthermore, by classifying each point cloud that makes up the circular stool using MLP (e.g., a point cloud belonging to the stool surface, a point cloud belonging to the stool legs), the classification results of all point clouds in the target features can be obtained, i.e., the multiple components of the circular stool. For example... Figure 2 In the image, multiple parts refer to the seat and legs of the round stool, each represented by a different filling shape.

[0063] 102. Based on the point cloud data of multiple components and the 3D object, obtain the object features of the 3D object, which include the component features of multiple components.

[0064] After obtaining multiple components of a 3D object, the object features of the 3D object can be obtained based on the point cloud data of the multiple components and the 3D object. Among them, the object features of the 3D object include the component features of the multiple components that make up the 3D object.

[0065] One possible implementation involves obtaining the object features of a 3D object, specifically including:

[0066] By using a semantic segmentation model, the point features of each point cloud in different dimensions and the color information of each point cloud are fused to obtain the point embedding features of each point cloud in multiple point clouds.

[0067] The point embedding features of each point cloud are fused with multiple components to obtain the object features of the 3D object.

[0068] In this process, the point embedding features of each point cloud are fused with multiple components, which essentially means that each point in the point embedding features corresponds to a different component.

[0069] It should be understood that point features are defined as the most basic features of an image, referring to points whose grayscale signals exhibit significant changes in either two-dimensional or three-dimensional directions. In other words, in this embodiment, the grayscale values ​​of point features differ across dimensions. The point features of each point cloud in different dimensions can be understood as processing only the point cloud coordinates through a semantic segmentation model, without involving the processing of color information. In contrast, the point embedding features of each point cloud consider both the color information and the point features across different dimensions. Therefore, compared to point features, point embedding features not only reflect the different grayscale values ​​of each point cloud but also the possible components that each point cloud might correspond to (different colors represent different components).

[0070] For example, such as Figure 2 As shown, through multiple MLP processes in the semantic segmentation model, the point features of each point cloud in different dimensions can be obtained. By fusing the point features of each point cloud in different dimensions and the color information of each point cloud, the point embedding features of each point cloud can be obtained.

[0071] Furthermore, by fusing the point embedding features and components of each point cloud, the final object features of the circular stool can be obtained.

[0072] The above technical solution proposes a method for obtaining the object features of a 3D object. First, it fuses the point features of each point cloud of the 3D object across different dimensions with the color information of each point cloud to obtain the point embedding features of each point cloud. Then, it fuses the point embedding features of each point cloud with the obtained multiple components to obtain the final object features. This process considers the matching and correspondence between the point features of each point cloud and the segmented components, ensuring the accuracy and precision of obtaining the object features of the 3D object, thereby guaranteeing the efficiency of cross-modal search.

[0073] 103. Determine the similarity between the object features of a 3D object and the semantic features of multiple candidate texts. The semantic features include word vectors of multiple words in the corresponding candidate texts.

[0074] It should be understood that in the process of matching three-dimensional objects, the similarity is calculated one by one between the object features of the three-dimensional object and the semantic features of each candidate text in the text library. The semantic features of each candidate text include word vectors of multiple words.

[0075] It should be understood that before calculating the object features of a 3D object and the semantic features of multiple candidate texts, the text must first be processed through a text extraction model to obtain the semantic features of multiple candidate texts.

[0076] One possible implementation involves obtaining the semantic features of multiple candidate texts, specifically including:

[0077] Each candidate text is input into a text feature extraction model to obtain multiple word vectors containing contextual information of each candidate text. The semantic features of each candidate text include multiple word vectors corresponding to each candidate text.

[0078] After obtaining multiple word vectors, the semantic features corresponding to each candidate text can be obtained by adding each word vector together, which can also be called the text vector of each candidate text.

[0079] In one possible implementation, the text feature extraction model in this application embodiment is a Gate Recurrent Unit (GRU).

[0080] As a type of Recurrent Neural Network (RNN), GRU has the same input-output structure as RNN, including the current input x. t The hidden state passed down from the previous node. t-1 Combination x t and h t-1 This will give you the output y of the current hidden node. t and the hidden state h passed to the next node t .

[0081] Wherein, the output y of the currently hidden node t This can be viewed as a word vector of the text. The semantics of text can be understood as the textual content contained in a piece of text; that is, all the contextual information corresponding to a piece of text constitutes the semantics of the text. In GRU, the relationship between each node can be understood as the relationship between contextual information. Therefore, by processing the text feature extraction model, multiple word vectors of a piece of text can be obtained, and the sum of these multiple word vectors yields the semantic features of the text.

[0082] For example, such as Figure 2 As shown, when the text “A round stool 60cm high” is input into the text feature extraction model, multiple words containing contextual information can be obtained, such as “A round stool”, “60cm” and “high”. Each word corresponds to a word vector, and multiple word vectors together constitute the semantic features of this text.

[0083] Furthermore, after obtaining the semantic features of each candidate text according to the above process, the similarity between the object features of the 3D object and the semantic features of each candidate text can be calculated.

[0084] In one possible implementation, the calculation of the similarity between the object features of the 3D object and the semantic features of each candidate text specifically includes:

[0085] The object features of the 3D object and the semantic features of multiple candidate texts are input into the cross-modal search model;

[0086] By using a cross-modal search model, the bulldozer distance between the object features of a 3D object and the semantic features of multiple candidate texts is determined.

[0087] Specifically, when calculating the bulldozer distance between the object features of a 3D object and the semantic features of multiple candidate texts, the following steps are included:

[0088] For any one of the multiple candidate texts, determine the bulldozer distance between the component features of each part of the 3D object and the word vector of each word in the candidate text.

[0089] The bulldozer distance between the component features of each component and the word vector of each word in the candidate text is fused to obtain the bulldozer distance between the object features of the 3D object and the candidate text.

[0090] For example, such as Figure 2 As shown, by processing the point cloud coordinates and color information of a 3D object—a circular stool—using a semantic segmentation model, the object features of the 3D object are obtained. These object features include features of multiple components, namely the stool surface features and stool leg features. For example... Figure 2 As shown, the features of the stool seat and the features of the stool legs correspond to respectively Figure 2 Two cuboids representing the characteristic features of an object.

[0091] By extracting features from the text, the semantic features corresponding to the text were obtained, such as... Figure 2 As shown, the semantic features consist of three word vectors, each corresponding to... Figure 2 A cuboid representing the semantic features of a character.

[0092] When calculating object features and semantic features, specifically, the bulldozer distance between each component feature in the object features and each word vector in the semantic features is calculated. For example... Figure 2 As shown, the bulldozer distance between each cuboid in the object features and each cuboid in the semantic features is calculated, and then all bulldozer distances are summed to obtain the final bulldozer distance.

[0093] When determining the bulldozer distance between the object features of a 3D object and the semantic features of multiple candidate texts, since the semantic features of the text include multiple word vectors and the object features of the 3D object include multiple component features, a local alignment search method can be used to calculate the bulldozer distance pairwise between each word vector and each component feature. Then, the results are fused to obtain the final bulldozer distance between the object features of the 3D object and the semantic features of the text. This ensures the accuracy of cross-modal matching of 3D objects and improves the matching precision.

[0094] 104. The candidate text with the highest similarity between semantic features and object features among multiple candidate texts is identified as the text that best matches the 3D object in the cross-modal search.

[0095] In step 103, the similarity between the object features of the 3D object and the semantic features of multiple candidate texts can be obtained by calculating the bulldozer distance between the object features of the 3D object and the semantic features of multiple candidate texts. The smaller the bulldozer distance, the greater the similarity between the two modalities or the two feature distributions. The candidate text with the highest similarity or the smallest bulldozer distance is selected from all candidate texts as the text that best matches the 3D object in this cross-modal search process.

[0096] It should be understood that the above process mainly describes the application process of the cross-modal search model. Before application, the accuracy of the cross-modal search model needs to be continuously improved through multiple training sessions.

[0097] In one possible implementation, the training process for the cross-modal search model can be divided into two scenarios: one where the target sample is a 3D object, and the other where the target sample is text. Specifically, these include:

[0098] Obtain the first semantic features of the positive sample text corresponding to the sample 3D object, and the second semantic features of the negative sample text;

[0099] Determine the bulldozer distance between the object features of the sample 3D object and the first semantic feature, and the bulldozer distance between the object features of the sample 3D object and the second semantic feature;

[0100] Train a cross-modal search model based on the bulldozer distance between the object features and the first semantic feature of a 3D object, and the bulldozer distance between the object features and the second semantic feature of a 3D object; or,

[0101] Obtain the first object feature of a positive sample 3D object corresponding to the sample text, and the second object feature of a negative sample 3D object;

[0102] Determine the bulldozer distance between the semantic features of the sample text and the first object feature, and the bulldozer distance between the semantic features of the sample text and the second object feature;

[0103] A cross-modal search model is trained based on the semantic features of the sample text and the bulldozer distance between the first object and the second object.

[0104] In other words, during training, when the sample is a 3D object, a positive sample text and a negative sample text corresponding to the 3D object can be obtained. The desired result during training is that the bulldozer distance between the object features of the 3D object and the semantic features of the positive sample text should be as small as possible, while the bulldozer distance between the object features of the 3D object and the semantic features of the negative sample text should be as large as possible. Similarly, when the sample is text, a positive sample 3D object and a negative sample 3D object corresponding to the text can be obtained. The desired result during training is that the bulldozer distance between the semantic features of the text and the object features of the positive sample 3D object should be as small as possible, while the bulldozer distance between the object features of the text and the semantic features of the negative sample 3D object should be as large as possible.

[0105] In summary, this application proposes a cross-modal search method for data processing. The specific process involves first semantically segmenting the 3D object based on its point cloud data to obtain multiple components. Then, using these components and the point cloud data, the object features of the 3D object are derived. The similarity between these object features and the semantic features of multiple candidate texts in a text library is calculated to obtain the text with the highest similarity to the 3D object. This process, by processing the point cloud data of the 3D object, ultimately yields its object features, making feature extraction more accurate. Furthermore, by using the text with the highest similarity as the best-matching text for the 3D object found through cross-modal search, the accuracy of the cross-modal search process is ensured.

[0106] When determining the bulldozer distance between the object features of a 3D object and the semantic features of multiple candidate texts, since the semantic features of the text include multiple word vectors and the object features of the 3D object include multiple component features, a local alignment search method can be used to calculate the bulldozer distance pairwise between each word vector and each component feature. Then, the results are fused to obtain the final bulldozer distance between the object features of the 3D object and the semantic features of the text. This ensures the accuracy of cross-modal matching of 3D objects and improves the matching precision.

[0107] To obtain the object features of a 3D object, a scheme is proposed. First, the point features of each point cloud of the 3D object in different dimensions and the color information of each point cloud are fused to obtain the point embedding features of each point cloud. Then, the point embedding features of each point cloud are fused with the obtained multiple parts to obtain the final object features. This process considers the matching and correspondence between the point features of each point cloud and the multiple segmented parts, ensuring the accuracy and precision of obtaining the object features of the 3D object, thereby guaranteeing the efficiency of cross-modal search.

[0108] In another scenario, a cross-modal search model can be used to determine the 3D object that matches the text. Similar to the process of matching 3D objects with text, a text feature extraction model can be used to obtain the semantic features of the text to be searched. Then, the similarity between these semantic features and the object features of multiple candidate 3D objects is calculated, and the 3D object with the highest similarity is determined as the 3D object that best matches the text to be searched. The principle is exactly the same as this solution, and will not be elaborated further here.

[0109] Figure 3 This is a schematic diagram of the structure of a cross-modal search device provided in an embodiment of this application.

[0110] For example, such as Figure 3 As shown, the device 300 includes:

[0111] The semantic segmentation module 301 is used to perform semantic segmentation on the point cloud data of the 3D object to obtain multiple parts of the 3D object.

[0112] The first processing module 302 is used to obtain the object features of the three-dimensional object based on the point cloud data of the multiple components and the three-dimensional object, and the object features include the component features of the multiple components.

[0113] The determination module 303 is used to determine the similarity between the object features of the three-dimensional object and the semantic features of multiple candidate texts, wherein the semantic features include word vectors of multiple words in the corresponding candidate texts; and to determine the candidate text with the greatest similarity between the semantic features and the object features among the multiple candidate texts as the text that best matches the three-dimensional object found through cross-modal search.

[0114] In one possible implementation, the determining module 303 is specifically used to: input the object features of the three-dimensional object and the semantic features of the multiple candidate texts into a cross-modal search model; and determine the bulldozer distance between the object features of the three-dimensional object and the semantic features of the multiple candidate texts through the cross-modal search model.

[0115] In one possible implementation, the determining module 303 is further configured to: for any one of the plurality of candidate texts, determine the bulldozer distance between the component features of each part of the three-dimensional object and the word vector of each word of the candidate text; perform a fusion process on the bulldozer distance between the component features of each part and the word vector of each word of the candidate text to obtain the bulldozer distance between the object features of the three-dimensional object and the candidate text.

[0116] In one possible implementation, the semantic segmentation module 301 is specifically used to: input the point cloud data of the 3D object into the semantic segmentation model, obtain the point features of each point cloud of the 3D object in different dimensions, wherein the point features of each point cloud in the largest dimension constitute the global features of the 3D object; concatenate the point features of each point cloud in different dimensions and the global features of the 3D object to obtain the target features of the 3D object; classify each point cloud corresponding to the target features to obtain the multiple components of the 3D object after semantic segmentation.

[0117] In one possible implementation, the first processing module 302 is specifically used to: fuse the point features of each point cloud in different dimensions and the color information of each point cloud through the semantic segmentation model to obtain the point embedding features of each point cloud in the plurality of point clouds; and fuse the point embedding features of each point cloud with the plurality of components to obtain the object features of the three-dimensional object.

[0118] Optionally, the device further includes: an acquisition module, configured to input each of the plurality of candidate texts into a text feature extraction model to acquire a plurality of word vectors containing contextual information of each candidate text, wherein the semantic features of each candidate text include the plurality of word vectors corresponding to each candidate text.

[0119] Optionally, the device further includes: a second processing module, configured to acquire a first semantic feature of positive sample text corresponding to a sample 3D object, and a second semantic feature of negative sample text; determine the bulldozer distance between the object feature of the sample 3D object and the first semantic feature, and the bulldozer distance between the object feature of the sample 3D object and the second semantic feature; train the cross-modal search model based on the bulldozer distance between the object feature of the 3D object and the first semantic feature, and the bulldozer distance between the object feature of the 3D object and the second semantic feature; or, acquire a first object feature of a positive sample 3D object corresponding to sample text, and a second object feature of a negative sample 3D object; determine the bulldozer distance between the semantic feature of the sample text and the first object feature, and the bulldozer distance between the semantic feature of the sample text and the second object feature; train the cross-modal search model based on the bulldozer distance between the semantic feature of the sample text and the first object, and the bulldozer distance between the semantic feature of the sample text and the second object.

[0120] Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application.

[0121] For example, such as Figure 4 As shown, the electronic device 400 includes a memory 401 and a processor 402, wherein the memory 401 stores executable program code 4011, and the processor 402 is used to call and execute the executable program code 4011 to perform a cross-modal search method.

[0122] This embodiment can divide the electronic device into functional modules according to the above method example. For example, each module can correspond to a separate functional module, or two or more functions can be integrated into one processing module. The integrated module can be implemented in hardware. It should be noted that the module division in this embodiment is illustrative and only represents one logical functional division. In actual implementation, there may be other division methods.

[0123] When each functional module is divided according to its corresponding function, the electronic device may include: a semantic segmentation module, a first processing module, a determination module, etc. It should be noted that all relevant content of each step involved in the above method embodiments can be referenced to the functional description of the corresponding functional module, and will not be repeated here.

[0124] The electronic device provided in this embodiment is used to execute the cross-modal search method described above, and therefore can achieve the same effect as the above implementation method.

[0125] When using integrated units, the electronic device may include a processing module and a storage module. The processing module is used to control and manage the operation of the electronic device. The storage module is used to support the execution of program code and data by the electronic device.

[0126] The processing module may be a processor or a controller, which can implement or execute various exemplary logic blocks, modules, and circuits as disclosed in this application. The processor may also be a combination of computing functions, such as a combination of one or more microprocessors, a combination of digital signal processing (DSP) and microprocessors, etc., and the storage module may be a memory.

[0127] This embodiment also provides a computer-readable storage medium storing computer program code. When the computer program code is run on a computer, it causes the computer to execute the above-described related method steps to implement a cross-modal search method in the above embodiment.

[0128] This embodiment also provides a computer program product that, when run on a computer, causes the computer to perform the aforementioned related steps to implement a cross-modal search method as described in the above embodiment.

[0129] In addition, the electronic device provided in the embodiments of this application may specifically be a chip, component or module. The electronic device may include a connected processor and a memory. The memory is used to store instructions. When the electronic device is running, the processor may call and execute the instructions to make the chip perform a cross-modal search method in the above embodiments.

[0130] In this embodiment, the electronic device, computer-readable storage medium, computer program product or chip are all used to execute the corresponding methods provided above. Therefore, the beneficial effects that can be achieved can be referred to the beneficial effects of the corresponding methods provided above, and will not be repeated here.

[0131] Through the above description of the embodiments, those skilled in the art will understand that, for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0132] In the embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0133] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for cross-modal search, characterized in that, The method includes: Based on the point cloud data of a 3D object, semantic segmentation is performed on the 3D object to obtain multiple components of the 3D object; Based on the point cloud data of the multiple components and the three-dimensional object, the object features of the three-dimensional object are obtained, and the object features include the component features of the multiple components; Determine the similarity between the object features of the three-dimensional object and the semantic features of multiple candidate texts, wherein the semantic features include word vectors of multiple words in the corresponding candidate texts; The candidate text with the highest similarity between its semantic features and the object features among the multiple candidate texts is determined as the text that best matches the three-dimensional object through cross-modal search; The point cloud data based on the 3D object is used to perform semantic segmentation on the 3D object to obtain multiple components of the 3D object, including: The point cloud data of the three-dimensional object is input into the semantic segmentation model to obtain the point features of each point cloud of the three-dimensional object in different dimensions. The point features of each point cloud in the largest dimension constitute the global features of the three-dimensional object. The point features of each point cloud in different dimensions and the global features of the three-dimensional object are stitched together to obtain the target features of the three-dimensional object. Classify each point cloud corresponding to the target feature to obtain the multiple components of the three-dimensional object after semantic segmentation; And, determining the similarity between the object features of the three-dimensional object and the semantic features of multiple candidate texts includes: The object features of the 3D object and the semantic features of the multiple candidate texts are input into the cross-modal search model; The bulldozer distance between the object features of the 3D object and the semantic features of the multiple candidate texts is determined using the cross-modal search model.

2. The method according to claim 1, characterized in that, Determining the bulldozer distance between the object features of the 3D object and the semantic features of multiple candidate texts includes: For any one of the multiple candidate texts, determine the bulldozer distance between the component features of each part of the three-dimensional object and the word vector of each word of the candidate text; The bulldozer distance between the component features of each component and the word vector of each word in the candidate text is fused to obtain the bulldozer distance between the object features of the 3D object and the candidate text.

3. The method according to claim 1, characterized in that, The step of obtaining the object features of the three-dimensional object based on the point cloud data of the multiple components and the three-dimensional object includes: The semantic segmentation model is used to fuse the point features of each point cloud in different dimensions and the color information of each point cloud to obtain the point embedding features of each point cloud in the multiple point clouds. The point embedding features of each point cloud are fused with the multiple components to obtain the object features of the three-dimensional object.

4. The method according to claim 1, characterized in that, The method further includes: Each of the multiple candidate texts is input into a text feature extraction model to obtain multiple word vectors containing contextual information of each candidate text, wherein the semantic features of each candidate text include multiple word vectors corresponding to each candidate text.

5. The method according to claim 1, characterized in that, The method further includes: Obtain the first semantic features of the positive sample text corresponding to the sample 3D object, and the second semantic features of the negative sample text; Determine the bulldozer distance between the object features of the sample 3D object and the first semantic feature, and the bulldozer distance between the object features of the sample 3D object and the second semantic feature; The cross-modal search model is trained based on the bulldozer distance between the object features of the 3D object and the first semantic feature, and the bulldozer distance between the object features of the 3D object and the second semantic feature; or, Obtain the first object feature of a positive sample 3D object corresponding to the sample text, and the second object feature of a negative sample 3D object; Determine the bulldozer distance between the semantic features of the sample text and the first object feature, and the bulldozer distance between the semantic features of the sample text and the second object feature; The cross-modal search model is trained based on the semantic features of the sample text and the bulldozer distance between the first object and the sample text and the bulldozer distance between the sample text and the second object.

6. A device for cross-modal search, characterized in that, The device includes: The semantic segmentation module is used to perform semantic segmentation on the three-dimensional object based on the point cloud data of the three-dimensional object, and obtain multiple parts of the three-dimensional object; The first processing module is used to obtain the object features of the three-dimensional object based on the point cloud data of the multiple components and the three-dimensional object, wherein the object features include the component features of the multiple components; The determination module is used to determine the similarity between the object features of the 3D object and the semantic features of multiple candidate texts, wherein the semantic features include word vectors of multiple words in the corresponding candidate texts; and to determine the candidate text with the highest similarity between the semantic features and the object features among the multiple candidate texts as the text that best matches the 3D object through cross-modal search. Specifically, the semantic segmentation module is used for: The point cloud data of the three-dimensional object is input into the semantic segmentation model to obtain the point features of each point cloud of the three-dimensional object in different dimensions. The point features of each point cloud in the largest dimension constitute the global features of the three-dimensional object. The point features of each point cloud in different dimensions and the global features of the three-dimensional object are stitched together to obtain the target features of the three-dimensional object. Classify each point cloud corresponding to the target feature to obtain the multiple components of the three-dimensional object after semantic segmentation; And, the determining module is specifically used for: The object features of the 3D object and the semantic features of the multiple candidate texts are input into the cross-modal search model; The bulldozer distance between the object features of the 3D object and the semantic features of the multiple candidate texts is determined using the cross-modal search model.

7. An electronic device, characterized in that, The electronic device includes: Memory, used to store executable program code; A processor for calling and running the executable program code from the memory, causing the electronic device to perform the method as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed, implements the method as described in any one of claims 1 to 5.

Images