Object retrieval method and system based on point cloud and view fusion

Abstract

The application provides an object retrieval method and system based on point cloud and view fusion, the method comprising: acquiring multiple view data and point cloud data of a stereoscopic vision object, and respectively performing data modal preprocessing and conversion on the point cloud data and the view data; respectively extracting single view features and view modal features of the view data, and point-by-point features and point cloud modal features of the point cloud data through a single modal network model; cross-fusing the single view features and the point cloud modal features, and cross-fusing the view modal features and the point-by-point features to obtain multi-modal fusion features; extracting unified representation features of the stereoscopic vision object from the multi-modal fusion features, and performing classification and retrieval of the stereoscopic vision object according to the unified representation features. The method realizes accurate description of the stereoscopic vision object through joint representation of the point cloud and the view and multi-modal cross-fusion, and further optimizes the performance of classification and retrieval of the stereoscopic vision object.

Description

Technical Field

[0001] The present application relates to the technical field of stereoscopic visual object retrieval, and in particular to an object retrieval method and system based on point cloud and view fusion. Background Art

[0002] Currently, stereoscopic object retrieval has attracted widespread attention from both academia and industry due to its wide range of applications. The development of VR and AR technologies relies heavily on the representation, recognition, and retrieval of stereoscopic objects. Common representations of stereoscopic objects include point clouds, voxels, grids, and views. Stereoscopic object retrieval is typically based on unimodal representations. However, the noise in unimodal data can significantly interfere with its accuracy, making it difficult to implement in real-world scenarios.

[0003] Some multimodal methods in related technologies for stereoscopic visual object representation and retrieval mostly directly fuse or fuse the results of multimodal modalities first or later. Among them, first fusion refers to a simple modal conversion and merging directly on the input data, while later fusion refers to the fusion of probability distribution or feature splicing on the output results. However, neither of these two methods can fully model the relationship between multimodalities, which makes it difficult to have a certain resistance to the noise of stereoscopic visual object data. That is, the stereoscopic visual object retrieval in related technologies has the following problems:

[0004] (1) The limitations of stereoscopic vision object data collection methods: In many scenarios, the data collection method is greatly restricted, and only single-modal point cloud or view data can be obtained.

[0005] (2) The semantic gap between multimodal data: The modeling methods of different modal data of stereoscopic visual objects vary greatly, making it difficult to match and fuse them directly from the original data. At the same time, this also greatly increases the difficulty of cross-modal retrieval tasks. Summary of the Invention

[0006] The present application aims to solve one of the technical problems in the related art at least to a certain extent.

[0007] To this end, the first purpose of this application is to propose an object retrieval method based on the fusion of point cloud and view. This method uses point cloud and view modalities to simultaneously model stereoscopic visual objects, and uses a multimodal cross-fusion algorithm to globally and locally fuse the data of point cloud and view modalities, so that the complementary information between modalities can be used more repeatedly, thereby effectively making up for the differences in modeling capabilities between multiple modalities, improving the expression ability of unified representation features of stereoscopic objects, and enabling data from different modalities to be more deeply integrated, effectively improving the performance and robustness of stereoscopic visual object retrieval.

[0008] The second objective of the present invention is to propose an object retrieval system based on point cloud and view fusion.

[0009] A third object of the present invention is to provide a non-transitory computer-readable storage medium.

[0010] To achieve the above objectives, a first embodiment of the present invention proposes an object retrieval method based on point cloud and view fusion, comprising the following steps:

[0011] S1: photographing a stereoscopic object with multiple cameras set at different angles to obtain multiple view data of the stereoscopic object, sampling the farthest point on the surface of the stereoscopic object to generate point cloud data of the stereoscopic object, and performing data modality preprocessing conversion on the point cloud data and the view data respectively;

[0012] S2: extracting single-view features and view modality features of the view data, and point-by-point features and point cloud modality features of the point cloud data respectively through a single-modality network model;

[0013] S3: cross-fusing the single view feature and the point cloud modal feature, and cross-fusing the view modal feature and the point-by-point feature to obtain a multimodal fusion feature;

[0014] S4: extracting unified representation features of the stereoscopic visual object from the multimodal fusion features, and classifying and retrieving the stereoscopic visual object based on the unified representation features.

[0015] Optionally, in one embodiment of the present application, step S1 specifically includes: obtaining original data and reconstructing the stereoscopic visual object based on the original data; setting the stereoscopic visual object in a three-dimensional space according to a preset direction, and setting multiple cameras at different positions in the horizontal plane of the stereoscopic visual object to shoot the stereoscopic visual object; uniformly collecting a first preset number of points from the surface of the stereoscopic visual object as an initial point cloud set through a farthest point sampling algorithm, and randomly selecting a second preset number of points from the initial point cloud set as initial point cloud modal data; deleting the background of the multiple view data, and randomly rotating and scaling each view data; normalizing the coordinates of the initial point cloud modal data, and randomly jittering each point.

[0016] Optionally, in one embodiment of the present application, step S2 specifically includes: constructing a multi-view full convolutional network, and extracting the underlying features of each view data through the multi-view full convolutional network; constructing a deep convolutional network, and extracting corresponding single-view features from the underlying features of each view data through the deep convolutional network, and performing maximum pooling fusion on the single-view features of each view data to generate view modal features; constructing a point cloud feature extraction network by connecting multiple edge convolutions, pooling layers and activation layers, and extracting point-by-point features of point cloud data through the point cloud feature extraction network; and performing global pooling fusion on each point-by-point feature to generate point cloud modal features.

[0017] Optionally, in one embodiment of the present application, step S3 specifically includes: fusing each of the single view features with the point cloud modal features to obtain the corresponding single view features of the joint point cloud; fusing the single view features of each of the joint point clouds to generate a view-by-view and point cloud cross-fusion feature; fusing each of the point-by-point features with the view modal features to obtain the corresponding point-by-point features of the joint view; fusing the point-by-point features of each joint view to generate a point-by-point and view modal fusion feature.

[0018] Optionally, in one embodiment of the present application, step S4 specifically includes: extracting a unified representation feature of the stereoscopic visual object from the multimodal fusion feature through a deep convolutional network; calculating the distance between the unified representation feature and each object feature in the database, and sorting each distance, and determining the most similar stereoscopic visual object retrieved based on the sorting result.

[0019] To achieve the above objectives, the second embodiment of the present application proposes an object retrieval system based on point cloud and view fusion, including the following modules:

[0020] a data sampling module, configured to capture a stereoscopic object using multiple cameras set at different angles to obtain multiple view data of the stereoscopic object, perform farthest point sampling on the surface of the stereoscopic object to generate point cloud data of the stereoscopic object, and perform data modality preprocessing conversion on the point cloud data and view data;

[0021] a first feature extraction module, configured to extract single-view features and view modality features of the view data, and point-by-point features and point cloud modality features of the point cloud data, respectively, through a single-modality network model;

[0022] a cross fusion module, configured to cross-fuse the single view features and the point cloud modal features, and cross-fuse the view modal features and the point-by-point features to obtain multimodal fusion features;

[0023] The second feature extraction module is used to extract the unified representation features of the stereoscopic visual object from the multimodal fusion features, and classify and retrieve the stereoscopic visual object according to the unified representation features.

[0024] Optionally, in one embodiment of the present application, the data sampling module is specifically used to: obtain raw data and reconstruct the stereoscopic visual object based on the raw data; set the stereoscopic visual object in a three-dimensional space according to a preset direction, and set multiple cameras at different positions in the horizontal plane of the stereoscopic visual object to shoot the stereoscopic visual object; uniformly collect a first preset number of points from the surface of the stereoscopic visual object as an initial point cloud set through a farthest point sampling algorithm, and randomly select a second preset number of points from the initial point cloud set as initial point cloud modal data; delete the background of the multiple view data, and randomly rotate and scale each view data; normalize the coordinates of the initial point cloud modal data, and randomly jitter each point.

[0025] Optionally, in one embodiment of the present application, the first feature extraction module is specifically used to: construct a multi-view full convolutional network, and extract the underlying features of each view data through the multi-view full convolutional network; construct a deep convolutional network, and extract the corresponding single view features from the underlying features of each view data through the deep convolutional network, and perform maximum pooling fusion on the single view features of each view data to generate view modal features; construct a point cloud feature extraction network by connecting multiple edge convolutions, pooling layers and activation layers, and extract point-by-point features of point cloud data through the point cloud feature extraction network; perform global pooling fusion on each point-by-point feature to generate point cloud modal features.

[0026] Optionally, in one embodiment of the present application, the cross-fusion module is specifically used to: fuse each of the single-view features with the point cloud modal features to obtain the corresponding single-view features of the joint point cloud; fuse each of the single-view features of the joint point cloud to generate a view-by-view and point cloud cross-fusion feature; fuse each of the point-by-point features with the view modal features to obtain the corresponding point-by-point features of the joint view; fuse the point-by-point features of each joint view to generate a point-by-point and view modal fusion feature.

[0027] Optionally, in one embodiment of the present application, the second feature extraction module is specifically used to: extract a unified representation feature of the stereoscopic visual object from the multimodal fusion feature through a deep convolutional network; calculate the distance between the unified representation feature and each object feature in the database, and sort each distance, and determine the most similar stereoscopic visual object retrieved based on the sorting result.

[0028] The technical effect of this application: This application uses two modalities, point cloud and view, to simultaneously model stereoscopic visual objects, and uses a multimodal cross-fusion algorithm to globally and locally fuse the data of the two modalities, point cloud and view, respectively, so that the complementary information between the modalities can be used more repeatedly, thereby effectively making up for the differences in modeling capabilities between multiple modalities, improving the ability to express the unified representation features of stereoscopic objects, and enabling data from different modalities to be more deeply integrated, effectively improving the performance and robustness of stereoscopic visual object retrieval.

[0029] To achieve the above-mentioned purpose, the third aspect embodiment of the present application proposes a non-temporary computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, it implements the object retrieval method based on point cloud and view fusion described in the first aspect embodiment of the present application.

[0030] Additional aspects and advantages of the present invention will be set forth in part in the description which follows and, in part, will be obvious from the description which follows, or may be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0032] Figure 1 This is a flowchart of an object retrieval method based on point cloud and view fusion provided by an embodiment of the present application;

[0033] Figure 2 is a schematic diagram of generating point cloud modal data and view modal data provided by an embodiment of the present application;

[0034] Figure 3 This is a schematic diagram of a process for extracting view modality features and point cloud modality features provided by an embodiment of the present application;

[0035] Figure 4 This is a schematic diagram of a process for fusing multimodal features provided by an embodiment of the present application;

[0036] Figure 5 A flowchart of a specific object retrieval method based on point cloud and view fusion provided in one embodiment of the present application;

[0037] Figure 6 This is a structural diagram of an object retrieval system based on point cloud and view fusion provided by an embodiment of the present application. DETAILED DESCRIPTION

[0038] The following describes embodiments of the present invention in detail, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present invention, and are not to be construed as limiting the present invention.

[0039] The object retrieval method and system based on point cloud and view fusion according to an embodiment of the present application will be described below with reference to the accompanying drawings.

[0040] Figure 1 This is a flow chart of an object retrieval method based on point cloud and view fusion provided by an embodiment of the present application. Figure 1 As shown, the method includes the following steps:

[0041] S1: A stereoscopic vision object is photographed by multiple cameras set at different angles to obtain multiple view data of the stereoscopic vision object, and the farthest point sampling is performed on the surface of the stereoscopic vision object to generate point cloud data of the stereoscopic vision object, and the point cloud data and view data are preprocessed and converted into data modalities respectively.

[0042] In one embodiment of the present application, raw data related to the stereoscopic visual object to be retrieved is first obtained, and then the stereoscopic visual object is reconstructed based on the obtained raw data. For example, the stereoscopic visual object is reconstructed based on three-dimensional data in formats such as off in the obtained raw data.

[0043] Then, the reconstructed stereoscopic visual object is set in the three-dimensional space according to the preset direction, and multiple cameras are set at different positions in the horizontal plane of the stereoscopic visual object to shoot the stereoscopic visual object. Among them, the preset direction can be vertically upward, etc., which is not limited here. That is, the stereoscopic visual object is placed vertically upward on a certain plane in the three-dimensional space, and at the same time, a number of cameras are placed around the object in a circle on the horizontal plane where the stereoscopic visual object is located, and it is photographed from different angles, and then the captured images are collected to obtain the data of the stereoscopic visual object view modality. For example, with Figure 2 As shown, three cameras are placed around the stereoscopic object, shooting it from the side front, rear and front to obtain three view data.

[0044] Then, a first preset number of points are uniformly collected from the surface of the stereoscopic visual object by the farthest point sampling algorithm as the initial point cloud set, and a second preset number of points are randomly selected from the initial point cloud set as the initial point cloud modal data. The first preset number and the second preset number can be set according to actual needs such as the accuracy of the retrieval, and are not limited here. For example, continue to refer to Figure 2,Use the farthest point sampling algorithm to uniformly collect 10,000 points from the surface of the stereo vision object as the initial point cloud set, and then randomly select 2048 points from the 10,000 points as the initial point cloud modal data.

[0045] Finally, the background of multiple view data is removed, and each view data is randomly rotated and scaled. The coordinates of the initial point cloud modal data are normalized, and each point is randomly jittered. Specifically, this step preprocesses the point cloud and view data to achieve data enhancement and convert them into a data modality that is easy for deep model learning. For the view data, the background of the view data is removed, and each view data is randomly rotated and scaled for data enhancement. For the point cloud data, its coordinates are normalized to between 0 and 1, and each point is randomly jittered for data enhancement.

[0046] S2: Through the single-modal network model, the single-view features and view modal features of the view data, as well as the point-by-point features and point cloud modal features of the point cloud data are extracted respectively.

[0047] Among them, the single-modal network model can include a view feature extraction network and a point cloud feature extraction network, such as Figure 3 As shown, view modality features can be extracted by a view feature extraction network, and point cloud modality features can be extracted by a point cloud feature extraction network.

[0048] In one embodiment of the present application, a multi-view fully convolutional network is first constructed, and the underlying features of each view data are extracted through the multi-view fully convolutional network. Then, a deep convolutional network is constructed, and the corresponding single-view features are extracted from the underlying features of each view data through the deep convolutional network. The single-view features of each view data are then fused by maximum pooling to generate view modal features. In addition, a point cloud feature extraction network is constructed by connecting multiple edge convolutions, pooling layers, and activation layers. The point cloud feature extraction network extracts point-by-point features of the point cloud data, and then the point-by-point features are globally pooled and fused to generate point cloud modal features.

[0049] For example, refer to Figure 3 As shown in , when extracting view modality features, a multi-view neural network is constructed. Specifically, multiple single-view convolutional neural networks corresponding to the number of view data can be constructed, for example, Figure 3 As shown in the figure, three single-view convolutional neural networks are constructed based on the three view data, and the underlying features of the corresponding single view are extracted through the single-view full convolutional network. A deep convolutional network is further constructed to extract the single-view feature representation, i.e., v o ={v0, v1, ..., v m}, where m represents the number of view data, and then the single view features of all views are fused by maximum pooling to generate the view modality feature v a , which can be achieved through the following formula:

[0050]

[0051] Among them, viewpool() is the view feature pooling layer. As an example, this application can use the maximum pooling function maxpool() as the view feature pooling layer.

[0052] Furthermore, when extracting point cloud features, we first connect multiple edge convolution, pooling, and activation layers to build a point cloud feature extraction network. Then, we extract the point-by-point features of the point cloud through the point cloud feature extraction network. Among them, n means that the point cloud data contains a total of n points. Then the point-by-point features of all points are globally pooled and fused to generate the point cloud modal feature p a , which can be achieved through the following formula:

[0053]

[0054] Among them, pointpool() can be an aggregation function such as maximum pooling or average pooling. As an example, this application uses maximum pooling as an example to generate modal features.

[0055] S3: Cross-fuse single-view features and point cloud modal features, and cross-fuse view modal features and point-by-point features to obtain multimodal fusion features.

[0056] In one embodiment of the present application, each single view feature is first fused with the point cloud modal feature to obtain the corresponding single view feature of the joint point cloud, and then the single view feature of each joint point cloud is fused to generate a view-by-view and point cloud cross-fusion feature, and then each point-by-point feature is fused with the view modal feature to obtain the corresponding point-by-point feature of the joint view, and finally the point-by-point feature of each joint view is fused to generate a point-by-point and view modal fusion feature.

[0057] In specific implementation, as a possible implementation method, Figure 4 The multimodal features are fused in the manner shown in FIG. First, cross-fusion of each view and point cloud is performed, and the generated point cloud modal features p a and single view features Input to the modal pair fusion function h α (), the single view feature VF of the joint point cloud is obtained i Next, all the obtained joint point cloud single view features are fused to generate the cross-fusion feature F of each view and point cloud.vp , the specific formula is as follows:

[0058]

[0059] Among them, Maxpool() is the maximum pooling function.

[0060] Furthermore, the point-by-point cross-fusion with the view is performed to generate the view modal feature v a and point-wise features Input to the modal pair fusion function h β In (·,·), we get the point-by-point feature PF of the joint view i Next, all the point-by-point features of the joint views obtained by fusion are fused to generate point-by-point and view modality fusion features F pv , the specific formula is as follows:

[0061]

[0062] S4: Extract unified representation features of stereoscopic visual objects from multimodal fusion features, and classify and retrieve stereoscopic visual objects based on the unified representation features.

[0063] In one embodiment of the present application, a unified representation feature of a stereoscopic visual object is extracted from a multimodal fusion feature through a deep convolutional network, and then the distance between the unified representation feature and each object feature in the database is calculated, and each distance is sorted. The most similar stereoscopic visual object retrieved is determined based on the sorting result.

[0064] In this embodiment, a unified stereoscopic object representation is first extracted from multimodal features through a deep network for an object retrieval task. For an unknown stereoscopic visual object to be retrieved, after obtaining a unified feature representation of the stereoscopic visual object through the above steps of the present application, the distance between the unified feature representation of the stereoscopic visual object and all object feature representations pre-stored in a database is calculated, and then sorted from small to large according to the distance. Based on the sorting result, the stereoscopic visual object that is most similar to the unknown stereoscopic visual object to be retrieved in the database is retrieved. For example, the object corresponding to the object feature that ranks first after sorting is the most similar stereoscopic visual object to the retrieved unknown stereoscopic visual object.

[0065] To sum up, the object retrieval method based on point cloud and view fusion of the present application uses point cloud and view modalities to model stereoscopic visual objects at the same time, and globally and locally fuses the data of point cloud and view modalities respectively through a multimodal cross-fusion algorithm, so that the complementary information between modalities can be used more repeatedly, thereby effectively making up for the differences in modeling capabilities between multimodalities, improving the expression ability of unified representation features of stereoscopic objects, and enabling data of different modalities to be more deeply integrated, effectively improving the performance and robustness of stereoscopic visual object retrieval.

[0066] Let’s take a specific example to illustrate: Figure 5 As shown, after determining the stereoscopic visual object, the present application collects view modal data through a multi-view camera, and then pre-processes the image. The single view features of a single view are extracted through a view feature extraction network, and then the view modal features are generated through calculations such as maximum pooling fusion. At the same time, point cloud modal data is collected from the surface of the stereoscopic visual object through the farthest point sampling algorithm, and after point cloud normalization processing, the point cloud point-by-point features are extracted through a point cloud feature extraction network, and then the point cloud modal features are generated through calculations such as maximum pooling fusion. Furthermore, multi-modal cross-fusion is performed on single view features, view modal features, point-by-point features, and point cloud modal features, and deep feature extraction is performed on the multi-modal fusion features. A unified representation feature of the stereoscopic visual object is proposed, and the stereoscopic visual object is retrieved through the unified representation feature.

[0067] like Figure 6 As shown, in order to achieve the above-mentioned purpose, the second embodiment of the present application proposes an object retrieval system based on point cloud and view fusion of the present application, including: a data sampling module 100, a first feature extraction module 200, a cross-fusion module 300 and a second feature extraction module 400.

[0068] Among them, the data sampling module 100 is used to shoot a stereoscopic visual object through multiple cameras set at different angles, obtain multiple view data of the stereoscopic visual object, sample the farthest point on the surface of the stereoscopic visual object to generate point cloud data of the stereoscopic visual object, and pre-process and convert the point cloud data and view data into data modes respectively.

[0069] The first feature extraction module 200 is configured to extract single-view features and view modality features of view data, and point-by-point features and point cloud modality features of point cloud data, respectively, through a single-modality network model.

[0070] The cross fusion module 300 is used to cross-fuse single-view features and point cloud modal features, and to cross-fuse view modal features and point-by-point features to obtain multi-modal fusion features.

[0071] The second feature extraction module is used to extract unified representation features of stereoscopic visual objects from multimodal fusion features, and classify and retrieve stereoscopic visual objects based on the unified representation features.

[0072] Optionally, in one embodiment of the present application, the data sampling module is specifically used to: obtain raw data and reconstruct the stereoscopic visual object based on the raw data; set the stereoscopic visual object in a three-dimensional space according to a preset direction, and set multiple cameras at different positions in the horizontal plane of the stereoscopic visual object to shoot the stereoscopic visual object; uniformly collect a first preset number of points from the surface of the stereoscopic visual object as an initial point cloud set through a farthest point sampling algorithm, and randomly select a second preset number of points from the initial point cloud set as initial point cloud modal data; delete the background of multiple view data, and randomly rotate and scale each view data; normalize the coordinates of the initial point cloud modal data, and randomly jitter each point.

[0073] Optionally, in one embodiment of the present application, the first feature extraction module is specifically used to: construct a multi-view full convolutional network, and extract the underlying features of each view data through the multi-view full convolutional network; construct a deep convolutional network, and extract the corresponding single view features from the underlying features of each view data through the deep convolutional network, and perform maximum pooling fusion on the single view features of each view data to generate view modal features; construct a point cloud feature extraction network by connecting multiple edge convolutions, pooling layers and activation layers, and extract point-by-point features of point cloud data through the point cloud feature extraction network; perform global pooling fusion on each point-by-point feature to generate point cloud modal features.

[0074] Optionally, in one embodiment of the present application, the cross-fusion module is specifically used to: fuse each single-view feature with the point cloud modal feature to obtain the corresponding single-view feature of the joint point cloud; fuse the single-view feature of each joint point cloud to generate a view-by-view and point cloud cross-fusion feature; fuse each point-by-point feature with the view modal feature to obtain the corresponding point-by-point feature of the joint view; fuse the point-by-point features of each joint view to generate a point-by-point and view modal fusion feature.

[0075] Optionally, in one embodiment of the present application, the second feature extraction module is specifically used to: extract a unified representation feature of the stereoscopic visual object from the multimodal fusion feature through a deep convolutional network; calculate the distance between the unified representation feature and each object feature in the database, and sort each distance, and determine the most similar stereoscopic visual object retrieved based on the sorting result.

[0076] To sum up, the object retrieval system based on point cloud and view fusion in the embodiment of the present application uses two modalities, point cloud and view, to simultaneously model stereoscopic visual objects, and uses a multimodal cross-fusion algorithm to globally and locally fuse the data of the two modalities, point cloud and view, respectively, so that the complementary information between the modalities can be used more repeatedly, thereby effectively making up for the differences in modeling capabilities between multiple modalities, improving the expression ability of the unified representation features of stereoscopic objects, and enabling data of different modalities to be more deeply integrated, effectively improving the performance and robustness of stereoscopic visual object retrieval.

[0077] In order to implement the above embodiments, the present invention also proposes a non-temporary computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, it implements the object retrieval method based on point cloud and view fusion described in the embodiment of the first aspect of the present application.

[0078] Although the present application is disclosed in detail with reference to the accompanying drawings, it should be understood that these descriptions are merely exemplary and are not intended to limit the application of the present application. The scope of protection of the present application is defined by the appended claims and may include various modifications, alterations and equivalents made to the invention without departing from the scope and spirit of the present application.

[0079] In the description of this specification, the reference terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" mean that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art can combine and combine different embodiments or examples described in this specification and features of different embodiments or examples without contradiction.

[0080] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of the technical features being referred to. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one such feature. In the description of the present invention, "plurality" means at least two, such as two, three, etc., unless otherwise specifically defined.

[0081] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, segment or portion of code comprising one or more executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present invention includes alternative implementations in which functions may be performed out of the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by those skilled in the art to which the embodiments of the present invention pertain.

[0082] The logic and / or steps represented in the flowcharts or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing the logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (e.g., a computer-based system, a system including a processor, or other system that can fetch and execute instructions from an instruction execution system, apparatus, or device). For purposes of this specification, a "computer-readable medium" can be any device that can contain, store, communicate, propagate, or transport a program for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include the following: an electrical connection with one or more wires (electronic devices), a portable computer disk cartridge (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and programmable read-only memory (EPROM or flash memory), fiber optic devices, and a portable compact disc read-only memory (CDROM). Furthermore, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or processing it in another suitable manner if necessary, and then storing it in a computer memory.

[0083] It should be understood that various parts of the present invention can be implemented using hardware, software, firmware, or a combination thereof. In the above-described embodiments, multiple steps or methods can be implemented using software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented using hardware, as in another embodiment, any one of the following technologies known in the art or a combination thereof can be used: a discrete logic circuit having a logic gate circuit for implementing a logic function on a data signal, an application-specific integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0084] Those skilled in the art will understand that all or part of the steps in the method of the above embodiment can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable storage medium. When the program is executed, it includes one or a combination of the steps of the method embodiment.

[0085] In addition, the functional units in the various embodiments of the present invention may be integrated into a single processing module, or each unit may exist physically separately, or two or more units may be integrated into a single module. The aforementioned integrated modules may be implemented in the form of hardware or in the form of software functional modules. If the integrated modules are implemented in the form of software functional modules and sold or used as independent products, they may also be stored in a computer-readable storage medium.

[0086] The storage medium mentioned above may be a read-only memory, a magnetic disk, or an optical disk, etc. Although the embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and are not to be construed as limiting the present invention. Persons skilled in the art may make changes, modifications, substitutions, and variations to the above embodiments within the scope of the present invention.

Claims

1. An object retrieval method based on point cloud and view fusion, characterized in that: The following steps are involved: S1: photographing a stereoscopic object with multiple cameras set at different angles to obtain multiple view data of the stereoscopic object, sampling the farthest point on the surface of the stereoscopic object to generate point cloud data of the stereoscopic object, and performing data modality preprocessing conversion on the point cloud data and the view data respectively; S2: extracting single-view features and view modality features of the view data, and point-by-point features and point cloud modality features of the point cloud data respectively through a single-modality network model; S3: cross-fusing the single view feature and the point cloud modal feature, and cross-fusing the view modal feature and the point-by-point feature to obtain a multimodal fusion feature; Step S3 specifically includes: fusing each of the single view features with the point cloud modal features to obtain the corresponding single view features of the joint point cloud; Fusing the single-view features of each joint point cloud to generate cross-fusion features of each view and point cloud; The cross-fusion features of each view and point cloud are expressed as: Among them, VF i is the single view feature of the joint point cloud, h α (·,·) is the fusion function of single view features and point cloud modal features, p a is the point cloud modal feature, is a single view feature, m represents the number of view data is m, Maxpool() is the maximum pooling function, F vp Cross-fusing features of each view and point cloud; fusing each of the point-by-point features with the view modality features to obtain corresponding point-by-point features of the joint view; Fuse the point-by-point features of each joint view to generate point-by-point and view modality fusion features; The point-by-point and view modality fusion features are expressed as: Among them, PF i is the point-wise feature of the joint view, v a is the view modal feature, is the point-by-point feature of the point cloud, n means that the point cloud data contains n points, h β (·,·) is the fusion function of point-wise features and view modality features, F pv To fuse features point by point with view modality; S4: extracting unified representation features of the stereoscopic visual object from the multimodal fusion features, and classifying and retrieving the stereoscopic visual object based on the unified representation features.

2. The method according to claim 1, characterized in that Step S1 specifically includes: Acquiring original data, and reconstructing the stereoscopic visual object based on the original data; Arranging the stereoscopic visual object in a three-dimensional space according to a preset direction, and arranging a plurality of cameras at different positions in a horizontal plane of the stereoscopic visual object to shoot the stereoscopic visual object; uniformly collecting a first preset number of points from the surface of the stereoscopic visual object as an initial point cloud set using a farthest point sampling algorithm, and randomly selecting a second preset number of points from the initial point cloud set as initial point cloud modal data; Deleting the backgrounds of the plurality of view data, and randomly rotating and scaling each of the view data; The coordinates of the initial point cloud modal data are normalized, and each point is randomly jittered.

3. The method according to claim 1, characterized in that Step S2 specifically includes: Constructing a multi-view fully convolutional network to extract underlying features of each view data; Constructing a deep convolutional network, extracting corresponding single-view features from the underlying features of each view data through the deep convolutional network, and performing maximum pooling on the single-view features of each view data to generate view modality features; Constructing a point cloud feature extraction network by connecting multiple edge convolution layers, pooling layers, and activation layers, and extracting point-by-point features of the point cloud data through the point cloud feature extraction network; The point-by-point features are globally pooled and fused to generate point cloud modal features.

4. The method according to claim 1, wherein Step S4 specifically includes: Extracting unified representation features of the stereoscopic visual object from the multimodal fusion features through a deep convolutional network; The distance between the unified representation feature and each object feature in the database is calculated, and each distance is sorted, and the most similar stereoscopic visual object retrieved is determined according to the sorting result.

5. An object retrieval system based on point cloud and view fusion, characterized in that: include: a data sampling module, configured to capture a stereoscopic object using multiple cameras set at different angles to obtain multiple view data of the stereoscopic object, perform farthest point sampling on the surface of the stereoscopic object to generate point cloud data of the stereoscopic object, and perform data modality preprocessing conversion on the point cloud data and view data; a first feature extraction module, configured to extract single-view features and view modality features of the view data, and point-by-point features and point cloud modality features of the point cloud data, respectively, through a single-modality network model; a cross fusion module, configured to cross-fuse the single view features and the point cloud modal features, and cross-fuse the view modal features and the point-by-point features to obtain multimodal fusion features; The cross-fusion module is specifically used for: fusing each of the single view features with the point cloud modal features to obtain the corresponding single view features of the joint point cloud; Fusing the single-view features of each joint point cloud to generate cross-fusion features of each view and point cloud; The cross-fusion features of each view and point cloud are expressed as: Among them, VF i is the single view feature of the joint point cloud, h α (·,·) is the fusion function of single view features and point cloud modal features, p a is the point cloud modal feature, is a single view feature, m represents the number of view data is m, Maxpool() is the maximum pooling function, F vp Cross-fusing features of each view and point cloud; fusing each of the point-by-point features with the view modality features to obtain corresponding point-by-point features of the joint view; Fuse the point-by-point features of each joint view to generate point-by-point and view modality fusion features; The point-by-point and view modality fusion features are expressed as: Among them, PF i is the point-wise feature of the joint view, v a is the view modal feature, is the point-by-point feature of the point cloud, n means that the point cloud data contains n points, h β (·,·) is the fusion function of point-wise features and view modality features, F pv To fuse features point by point with view modality; The second feature extraction module is used to extract the unified representation features of the stereoscopic visual object from the multimodal fusion features, and classify and retrieve the stereoscopic visual object according to the unified representation features.

6. The system according to claim 5, characterized in that The data sampling module is specifically used for: Acquiring original data, and reconstructing the stereoscopic visual object based on the original data; Arranging the stereoscopic visual object in a three-dimensional space according to a preset direction, and arranging a plurality of cameras at different positions in a horizontal plane of the stereoscopic visual object to shoot the stereoscopic visual object; uniformly collecting a first preset number of points from the surface of the stereoscopic visual object as an initial point cloud set using a farthest point sampling algorithm, and randomly selecting a second preset number of points from the initial point cloud set as initial point cloud modal data; Deleting the backgrounds of the plurality of view data, and randomly rotating and scaling each of the view data; The coordinates of the initial point cloud modal data are normalized, and each point is randomly jittered.

7. The system according to claim 5, characterized in that The first feature extraction module is specifically used to: Constructing a multi-view fully convolutional network to extract underlying features of each view data; Constructing a deep convolutional network, extracting corresponding single-view features from the underlying features of each view data through the deep convolutional network, and performing maximum pooling on the single-view features of each view data to generate view modality features; Constructing a point cloud feature extraction network by connecting multiple edge convolution layers, pooling layers, and activation layers, and extracting point-by-point features of the point cloud data through the point cloud feature extraction network; The point-by-point features are globally pooled and fused to generate point cloud modal features.

8. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the object retrieval method based on point cloud and view fusion as described in any one of claims 1 to 4 is implemented.

Images