Feature fusion method and device of three-dimensional data, electronic equipment and storage medium
By converting three-dimensional data into two-dimensional data for feature extraction and fusion, the problem of large computing resources occupied by the three-dimensional data feature fusion method is solved, and faster processing time is achieved.
Patent Information
- Application Number
- CN202411405791.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-10
- Publication Date
- 2024-11-08
AI Technical Summary
The existing three-dimensional data feature fusion method requires a large amount of computing resources, resulting in large memory usage and excessive calculation time.
The three-dimensional data is processed in two-dimensional processing, divided into at least two-dimensional data, and input the feature extraction model for feature extraction and stacking, and finally input the feature fusion model for fusion to reduce the computing resource requirements.
Through dimensionality reduction processing, the computing resource requirements and video memory usage are reduced, and the computing time is shortened.
Smart Images

Figure CN118918022A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the field of data processing technology, and in particular to a method and device for fusion of three-dimensional data features, an electronic device and a storage medium. Background Art
[0002] During model training, the model needs to extract features from three-dimensional data and fuse the features so that the model can better understand and process three-dimensional data.
[0003] In the related technologies of feature fusion of three-dimensional data, features are usually directly extracted from the three-dimensional data and then fused. Since three-dimensional data is high-dimensional data and high-dimensional data is relatively complex, the feature fusion method of three-dimensional data in the related technologies requires a large amount of computing resources, resulting in large usage of video memory and excessively long computing time. Summary of the invention
[0004] The present invention provides a method and device for fusion of three-dimensional data features, an electronic device and a storage medium. The main purpose is to solve the problem that the method for fusion of three-dimensional data features in the related art requires a large amount of computing resources, resulting in large video memory usage and long computing time.
[0005] According to a first aspect of the present disclosure, a method for feature fusion of three-dimensional data is provided, comprising: Acquire three-dimensional data to be processed, and perform two-dimensional processing on the three-dimensional data to be processed to obtain at least two two-dimensional data of the three-dimensional data to be processed; Inputting the at least two two-dimensional data into a preset feature extraction model respectively, and obtaining two-dimensional features corresponding to the at least two two-dimensional data respectively; Stacking the two-dimensional features to obtain stacked features; The stacked features are input into a preset feature fusion model to obtain fusion data of the stacked features.
[0006] Optionally, the performing two-dimensional processing on the three-dimensional data to be processed to obtain at least two two-dimensional data of the three-dimensional data to be processed includes: Respectively obtain three coordinate axes of the three-dimensional data to be processed; Determine any one of the three coordinate axes as a segmentation direction, and determine the remaining two of the three coordinate axes as segmentation planes; The three-dimensional data to be processed is segmented based on the segmentation direction and the segmentation plane to obtain the at least two two-dimensional data.
[0007] Optionally, stacking the two-dimensional features to obtain stacked features includes: Respectively obtaining a two-dimensional sequence of the at least two two-dimensional data; The two-dimensional features corresponding to the two-dimensional data are stacked according to the two-dimensionalization sequence to obtain the stacked features.
[0008] Optionally, the training method of the preset feature fusion model includes: Acquire at least two training two-dimensional features of the training three-dimensional data, and acquire training stacked features of the training two-dimensional features; Performing average processing on the training two-dimensional features to obtain average two-dimensional features for training; The training stacked feature is divided into at least two training stacked feature blocks with the same shape, and the training average two-dimensional feature is divided into at least two training average two-dimensional feature blocks with the same shape; the training stacked feature block has the same shape as the training average two-dimensional feature block; The initial feature fusion model is trained using the training stacked feature blocks and the corresponding training averaged two-dimensional feature blocks to obtain the preset feature fusion model.
[0009] Optionally, the using the stacked feature blocks for training and the corresponding averaged two-dimensional feature blocks for training to train the initial feature fusion model to obtain the preset feature fusion model includes: The training stacked feature block and the corresponding training average two-dimensional feature block are learned to obtain a loss function; wherein the loss function includes the weight of the initial feature fusion model; The weights are adjusted in a manner that makes the loss function approach a minimum value to obtain the preset feature fusion model.
[0010] Optionally, after inputting the stacked features into a preset feature fusion model to obtain fusion data of the stacked features, the method further includes: Performing confidence evaluation on the fused data to obtain a confidence score of the fused data; If the confidence score is greater than a preset threshold, the fused data is determined as the final fusion result of the stacked features.
[0011] According to a second aspect of the present disclosure, a feature fusion device for three-dimensional data is provided, comprising: A first processing unit is used to obtain three-dimensional data to be processed, and perform two-dimensional processing on the three-dimensional data to be processed to obtain at least two two-dimensional data of the three-dimensional data to be processed; A first input unit, used to input the at least two two-dimensional data into a preset feature extraction model respectively, to obtain two-dimensional features corresponding to the at least two two-dimensional data respectively; A stacking unit, used for stacking the two-dimensional features to obtain stacked features; The second input unit is used to input the stacked features into a preset feature fusion model to obtain fusion data of the stacked features.
[0012] Optionally, the first processing unit includes: A first acquisition module, used to respectively acquire three coordinate axes of the three-dimensional data to be processed; A determination module, used to determine any one of the three coordinate axes as a segmentation direction, and to determine the remaining two of the three coordinate axes as segmentation planes; The segmentation module is used to segment the three-dimensional data to be processed based on the segmentation direction and the segmentation plane to obtain the at least two two-dimensional data.
[0013] Optionally, the stacking unit includes: A second acquisition module is used to respectively acquire the two-dimensionalization sequence of the at least two two-dimensional data; The stacking module is used to stack the two-dimensional features corresponding to the two-dimensional data in the order of the two-dimensionalization to obtain the stacked features.
[0014] Optionally, the device further comprises: an acquisition unit, configured to acquire at least two training two-dimensional features of the training three-dimensional data, and to acquire training stacked features of the training two-dimensional features; A second processing unit is used to perform average processing on the training two-dimensional features to obtain average two-dimensional features for training; A segmentation unit, used to segment the training stacked feature into at least two training stacked feature blocks of the same shape, and to segment the training average two-dimensional feature into at least two training average two-dimensional feature blocks of the same shape; the training stacked feature block and the training average two-dimensional feature block have the same shape; The training unit is used to train the initial feature fusion model using the training stacked feature blocks and the corresponding training averaged two-dimensional feature blocks to obtain the preset feature fusion model.
[0015] Optionally, the training unit includes: A learning module, used for learning the training stacked feature block and the corresponding training average two-dimensional feature block to obtain a loss function; wherein the loss function includes the weight of the initial feature fusion model; The adjustment module is used to adjust the weights in a manner that makes the loss function approach a minimum value to obtain the preset feature fusion model.
[0016] Optionally, the device further comprises: An evaluation unit, configured to, after inputting the stacked features into a preset feature fusion model to obtain fused data of the stacked features, perform confidence evaluation on the fused data to obtain a confidence score for the fused data; A determination unit is used to determine the fused data as a final fusion result of the stacked features when the confidence score is greater than a preset threshold.
[0017] According to a third aspect of the present disclosure, there is provided an electronic device, including: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method described in the first aspect.
[0018] According to a fourth aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute the method described in the first aspect.
[0019] According to a fifth aspect of the present disclosure, a computer program product is provided, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements the method as described in the first aspect above.
[0020] The feature fusion method and device, electronic device and storage medium of three-dimensional data provided by the present disclosure obtain the three-dimensional data to be processed, and perform two-dimensional processing on the three-dimensional data to be processed to obtain at least two two-dimensional data of the three-dimensional data to be processed; respectively input the at least two two-dimensional data into a preset feature extraction model to respectively obtain two-dimensional features corresponding to the at least two two-dimensional data; stack the two-dimensional features to obtain stacked features; input the stacked features into a preset feature fusion model to obtain fused data of the stacked features. Compared with the related art, the embodiment of the present disclosure performs two-dimensional processing on the three-dimensional data to be processed to obtain at least two two-dimensional data of the three-dimensional data to be processed, and performs feature extraction on the at least two two-dimensional data to obtain two-dimensional features corresponding to the at least two two-dimensional data, stacks the two-dimensional features to obtain stacked features, and performs feature fusion on the stacked features to obtain fused data, thereby realizing the conversion of three-dimensional data into two-dimensional data and the feature fusion of two-dimensional data. Since the dimension of two-dimensional data is lower than that of three-dimensional data, the computing resources required for feature fusion of two-dimensional data are smaller, the video memory occupied is also smaller, and the computing time is shorter.
[0021] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present application, nor is it intended to limit the scope of the present application. Other features of the present application will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS
[0022] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure. Figure 1 A schematic diagram of a flow chart of a feature fusion method for three-dimensional data provided in an embodiment of the present disclosure; Figure 2 A schematic diagram of a flow chart of a method for converting three-dimensional data to two-dimensional form provided in an embodiment of the present disclosure; Figure 3 A schematic diagram of a process of processing a training two-dimensional feature and a training stacked feature of a training two-dimensional feature by a preset feature fusion model provided in an embodiment of the present disclosure; Figure 4 A schematic diagram of a processing process of feature fusion of three-dimensional data provided by an embodiment of the present disclosure; Figure 5 A schematic diagram of the structure of a three-dimensional data feature fusion device provided in an embodiment of the present disclosure; Figure 6 A schematic diagram of the structure of another three-dimensional data feature fusion device provided in an embodiment of the present disclosure; Figure 7 A schematic block diagram of an exemplary electronic device provided for an embodiment of the present disclosure. DETAILED DESCRIPTION
[0023] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.
[0024] The following describes a method and device for fusion of three-dimensional data features, an electronic device, and a storage medium according to an embodiment of the present disclosure with reference to the accompanying drawings.
[0025] Figure 1 A flowchart of a feature fusion method for three-dimensional data provided in an embodiment of the present disclosure.
[0026] like Figure 1 As shown, the method is applied in a server, and the method comprises the following steps: Step 101 : acquiring three-dimensional data to be processed, and performing two-dimensional processing on the three-dimensional data to be processed to obtain at least two two-dimensional data of the three-dimensional data to be processed.
[0027] Three-dimensional data is data with a three-dimensional structure (including length, width and height). Three-dimensional data is obtained through three-dimensional scanning, medical imaging and other methods. However, it should be clear that this statement is not intended to limit three-dimensional data to being obtained only through three-dimensional scanning and medical imaging. Three-dimensional data can also be obtained through other methods.
[0028] Two-dimensional data is a two-dimensional projection or slice obtained from three-dimensional data, usually by slicing or projecting the three-dimensional data in a certain dimension, thereby obtaining data containing only two dimensions.
[0029] The method for two-dimensionalizing the three-dimensional data to be processed may be to slice the three-dimensional data to be processed along one dimension direction, thereby obtaining a series of two-dimensional slices, that is, two-dimensional data. However, it should be clear that this statement is not intended to limit the two-dimensional processing method to slicing only, and the three-dimensional data to be processed may also be processed two-dimensionally by other methods.
[0030] Converting three-dimensional data into two-dimensional data can simplify the data structure and improve the efficiency of feature extraction and fusion.
[0031] Step 102: Input the at least two two-dimensional data into a preset feature extraction model respectively to obtain two-dimensional features corresponding to the at least two two-dimensional data respectively.
[0032] The preset feature extraction model is a model used to extract features from two-dimensional data. The preset feature extraction model can be a pre-trained neural network (such as a convolutional neural network) or other machine learning models or algorithms. The preset feature extraction model is specifically used to identify and extract features in two-dimensional images.
[0033] When two-dimensional data is input into a preset feature extraction model, the convolution layer, pooling layer and fully connected layer of the preset feature extraction model can help extract features at different levels in the two-dimensional data. In the convolution layer, local features can be captured through the convolution operation of the filter. The subsequent pooling layer can reduce the dimension and retain the main features. Finally, the fully connected layer can integrate and abstract these features to obtain the final two-dimensional features. However, it should be clear that this statement is not intended to limit the feature extraction process of the preset feature extraction model to extraction through convolution layers, pooling layers and fully connected layers, and it can also be extracted through other methods.
[0034] Three-dimensional data usually contains more information and complexity, and direct processing may be very difficult, especially when the data volume is large or the model complexity is limited. By reducing the three-dimensional data to two dimensions and extracting features from the two-dimensional data, the process of data processing and feature extraction can be simplified and the computational complexity can be reduced.
[0035] Step 103: stack the two-dimensional features to obtain stacked features.
[0036] The stacked features are a collection of all two-dimensional features. Before stacking, you need to determine along which dimension to stack. To stack multiple features along the first dimension of the feature (the first dimension is any of the three dimensions of the feature), you need to consider stacking along the channel dimension, height dimension, or width dimension of the feature, depending on the shape of the feature and the neural network architecture used.
[0037] Stacking features actually increases the dimension and richness of features. By combining multiple two-dimensional features, more information in the three-dimensional data to be processed can be captured, thereby enhancing the expressive power of the features.
[0038] Step 104: input the stacked features into a preset feature fusion model to obtain fusion data of the stacked features.
[0039] The preset feature fusion model is used to fuse the model of stacked features. The preset feature fusion model is a neural network, decision tree or other machine learning model. The preset feature fusion model uses the information in the stacked features to generate a higher-level representation, that is, fused data. The fused features combine the features of all two-dimensional data and have a higher level of abstraction and stronger expression ability. The fused data can be used for subsequent classification, regression or other machine learning tasks.
[0040] Stacked features can be fused by weighted summation. The stacked features are divided into multiple feature blocks of the same shape. Each feature block is assigned a weight, which can be obtained based on experience or through learning. Then, each feature block is multiplied by the corresponding weight and added together to obtain the final fused data.
[0041] The fused features can be adjusted and optimized according to the requirements of different tasks, so as to better adapt to different application scenarios and task objectives.
[0042] The feature fusion method of three-dimensional data provided by the present disclosure obtains the three-dimensional data to be processed, and performs two-dimensional processing on the three-dimensional data to be processed to obtain at least two two-dimensional data of the three-dimensional data to be processed; the at least two two-dimensional data are respectively input into a preset feature extraction model to respectively obtain two-dimensional features corresponding to the at least two two-dimensional data; the two-dimensional features are stacked to obtain stacked features; the stacked features are input into a preset feature fusion model to obtain fusion data of the stacked features. Compared with the related art, the embodiment of the present disclosure performs two-dimensional processing on the three-dimensional data to be processed to obtain at least two two-dimensional data of the three-dimensional data to be processed, and performs feature extraction on the at least two two-dimensional data to obtain two-dimensional features corresponding to the at least two two-dimensional data, stacks the two-dimensional features to obtain stacked features, and performs feature fusion on the stacked features to obtain fusion data, thereby realizing the conversion of three-dimensional data into two-dimensional data and the feature fusion of two-dimensional data. Since the dimension of two-dimensional data is lower than that of three-dimensional data, the computing resources required for feature fusion of two-dimensional data are smaller, the video memory occupied is also smaller, and the computing time is shorter.
[0043] As a refinement of step 101, when performing the two-dimensional processing of the three-dimensional data to be processed to obtain at least two two-dimensional data of the three-dimensional data to be processed, it can be implemented in the following manners but not limited to: Figure 2 As shown, Figure 2 A schematic flow chart of a method for converting three-dimensional data to two-dimensional form provided in an embodiment of the present disclosure includes: Step 201 : respectively obtain three coordinate axes of the three-dimensional data to be processed.
[0044] Each dimension of three-dimensional data corresponds to a coordinate axis. Since three-dimensional data has three dimensions, three-dimensional data corresponds to three coordinate axes, and two of the three coordinate axes are perpendicular to each other.
[0045] For ease of understanding, an example is provided, assuming that the three coordinate axes of the three-dimensional data to be processed are the x-axis, y-axis and z-axis, wherein the x-axis is perpendicular to the y-axis, the x-axis is perpendicular to the z-axis, and the y-axis is perpendicular to the z-axis.
[0046] The three coordinate axes of the three-dimensional data to be processed are obtained to clarify the spatial structure and directionality of the three-dimensional data to be processed, and to provide a basis for the subsequent segmentation of the three-dimensional data to be processed.
[0047] Step 202: determine any one of the three coordinate axes as a segmentation direction, and determine the remaining two of the three coordinate axes as segmentation planes.
[0048] For ease of understanding, an example is provided. Assume that the three coordinate axes are the x-axis, the y-axis and the z-axis. If the x-axis is the segmentation direction, the score composed of the y-axis and the z-axis is determined as the segmentation surface.
[0049] The purpose of determining any one of the three coordinate axes as the segmentation direction and determining the remaining two of the three coordinate axes as the segmentation planes is to achieve flexible data segmentation in data processing and convert complex three-dimensional data into two-dimensional data that is easier to process and analyze, thereby improving the efficiency and accuracy of data processing.
[0050] Step 203 : segment the three-dimensional data to be processed based on the segmentation direction and the segmentation plane to obtain the at least two two-dimensional data.
[0051] The three-dimensional data to be processed is divided along the dividing direction by the dividing plane, and the two-dimensional data of the three-dimensional data to be processed that stays on the dividing plane is intercepted at a preset distance. The preset distance can be any value, which is not limited in the embodiment of the present disclosure.
[0052] Three-dimensional data is usually more complex than two-dimensional data. When analyzing a specific problem, dividing the three-dimensional data into several two-dimensional data can simplify the complexity of the problem and make the analysis more intuitive and simple.
[0053] As a refinement of step 103, when executing the stacking of the two-dimensional features to obtain the stacked features, it can be implemented in the following manner but not limited to: respectively obtain the two-dimensional order of the at least two two-dimensional data; stack the two-dimensional features corresponding to the two-dimensional data according to the two-dimensional order to obtain the stacked features. The two-dimensional order is the order in which each two-dimensional data is generated when the three-dimensional data to be processed is converted into a series of two-dimensional data. For example, if a three-dimensional data to be processed is segmented along the segmentation direction, then the order in which each slice is generated is the two-dimensional order. The purpose of stacking the two-dimensional features corresponding to the two-dimensional data according to the two-dimensional order to obtain the stacked features is to maintain the correlation between the data, facilitate subsequent processing and analysis, enhance the data representation capability, and adapt to different application scenarios.
[0054] To facilitate understanding, an example is provided. Suppose there are three two-dimensional features in the three-dimensional data to be processed, namely a, b, and c. Among them, the two-dimensionalization order of a is 1, the two-dimensionalization order of b is 3, and the two-dimensionalization order of c is 2, then the stacking feature is acb.
[0055] In practical applications, the preset feature fusion model is obtained by training with the training two-dimensional features of the training three-dimensional data and the training stacked features of the training two-dimensional features. When training the preset feature fusion model, it can be implemented in the following ways but not limited to: obtaining at least two training two-dimensional features of the training three-dimensional data, and obtaining the training stacked features of the training two-dimensional features; performing average processing on the training two-dimensional features to obtain the training average two-dimensional features; dividing the training stacked features into at least two training stacked feature blocks of the same shape, and dividing the training average two-dimensional features into at least two training average two-dimensional feature blocks of the same shape; the training stacked feature blocks have the same shape as the training average two-dimensional feature blocks; using the training stacked feature blocks and the corresponding training average two-dimensional feature blocks, the initial feature fusion model is trained to obtain the preset feature fusion model. Converting three-dimensional data into two-dimensional features and stacking and segmenting them can effectively reduce the dimension and compress the data. This helps to reduce computational complexity and improve the efficiency of model training and reasoning.
[0056] In order to better understand the training of the preset feature fusion model using the training two-dimensional features and the training stacked features of the training two-dimensional features, such as Figure 3 As shown, Figure 3 A schematic diagram of the processing process of a preset feature fusion model for training two-dimensional features and training stacked features of training two-dimensional features provided by an embodiment of the present disclosure. The training two-dimensional features and the training stacked features are divided into n*x*y small blocks, where n, x, and y can be any values, for example, n=6, x=y=4, and the small blocks of the training two-dimensional features are averaged and then encoded using linear projection.
[0057] As a refinement of the above embodiment, when executing the method of training the initial feature fusion model using the training stacked feature blocks and the corresponding training average two-dimensional feature blocks to obtain the preset feature fusion model, it can be implemented in but not limited to the following way: learning the training stacked feature blocks and the corresponding training average two-dimensional feature blocks to obtain a loss function; wherein the loss function includes the weights of the initial feature fusion model; and adjusting the weights in a manner that makes the loss function approach a minimum value to obtain the preset feature fusion model.
[0058] For example, the calculation formula of the loss function can be implemented by formula (1): (1) Among them, is the loss function, is the stacked feature block for training, is the average two-dimensional feature block for training, is the weight, and is a constant.
[0059] In practical applications, after the stacked features are input into a preset feature fusion model to obtain the fused data of the stacked features, the fused data output by the preset feature fusion model may be inaccurate, and the fused data needs to be judged for accuracy, which can be achieved by, but not limited to, the following methods: confidence evaluation is performed on the fused data to obtain a confidence score for the fused data; if the confidence score is greater than a preset threshold, the fused data is determined as the final fusion result of the stacked features. Confidence refers to an evaluation of the credibility or accuracy of the fused data. The higher the confidence, the higher the credibility or accuracy of the fused data. The preset threshold can be any value, which is not limited in the embodiments of the present disclosure.
[0060] The confidence score is obtained by inputting the fused data into a preset confidence evaluation model, wherein the preset confidence evaluation is a model obtained by training with the training fused data and the real fused data.
[0061] By evaluating the confidence of the fused data, possible erroneous data can be excluded, thereby improving the accuracy of the fusion results.
[0062] In one possible implementation of the embodiment of the present disclosure, in order to better understand the process of feature fusion of three-dimensional data, as shown in FIG. Figure 4 As shown, Figure 4 A schematic diagram of a feature fusion processing process for three-dimensional data provided in an embodiment of the present disclosure, in which two-dimensional data of the three-dimensional data to be processed is extracted through a preset feature extraction model to obtain two-dimensional features of each two-dimensional data, all two-dimensional features are stacked in order, and feature fusion of the two-dimensional features in the same dimension is performed through a preset feature fusion model to obtain fused data, and confidence calculation is performed on the fused data to detect whether the fused data is accurate.
[0063] In summary, the embodiments of the present disclosure can achieve the following effects: The disclosed embodiment converts three-dimensional data into two-dimensional data and performs feature fusion on the two-dimensional data. Since two-dimensional data has a lower dimension than three-dimensional data, less computing resources are required for feature fusion of the two-dimensional data, less video memory is occupied, and the computing time is shorter.
[0064] Corresponding to the above-mentioned three-dimensional data feature fusion method, the present invention also proposes a three-dimensional data feature fusion device. Since the device embodiment of the present invention corresponds to the above-mentioned method embodiment, the details not disclosed in the device embodiment can be referred to the above-mentioned method embodiment, and will not be repeated in the present invention.
[0065] Figure 5 A structural diagram of a feature fusion device for three-dimensional data provided by an embodiment of the present disclosure, such as Figure 5 As shown, the device is applied to a server and includes: The first processing unit 301 is used to obtain three-dimensional data to be processed, and perform two-dimensional processing on the three-dimensional data to be processed to obtain at least two two-dimensional data of the three-dimensional data to be processed; A first input unit 302, used to input the at least two two-dimensional data into a preset feature extraction model to obtain two-dimensional features corresponding to the at least two two-dimensional data; A stacking unit 303, used for stacking the two-dimensional features to obtain stacked features; The second input unit 304 is used to input the stacked features into a preset feature fusion model to obtain fusion data of the stacked features.
[0066] The feature fusion device of three-dimensional data provided by the present disclosure obtains the three-dimensional data to be processed, and performs two-dimensional processing on the three-dimensional data to be processed to obtain at least two two-dimensional data of the three-dimensional data to be processed; the at least two two-dimensional data are respectively input into a preset feature extraction model to respectively obtain two-dimensional features corresponding to the at least two two-dimensional data; the two-dimensional features are stacked to obtain stacked features; the stacked features are input into a preset feature fusion model to obtain fusion data of the stacked features. Compared with the related art, the embodiment of the present disclosure performs two-dimensional processing on the three-dimensional data to be processed to obtain at least two two-dimensional data of the three-dimensional data to be processed, and performs feature extraction on the at least two two-dimensional data to obtain two-dimensional features corresponding to the at least two two-dimensional data, stacks the two-dimensional features to obtain stacked features, and performs feature fusion on the stacked features to obtain fusion data, thereby realizing the conversion of three-dimensional data into two-dimensional data and the feature fusion of two-dimensional data. Since the dimension of two-dimensional data is lower than that of three-dimensional data, the computing resources required for feature fusion of two-dimensional data are smaller, the video memory occupied is also smaller, and the computing time is shorter.
[0067] Furthermore, in a possible implementation of this embodiment, as Figure 6 As shown, the first processing unit 301 includes: A first acquisition module 3011 is used to respectively acquire three coordinate axes of the three-dimensional data to be processed; A determination module 3012 is used to determine any one of the three coordinate axes as a segmentation direction, and to determine the remaining two of the three coordinate axes as segmentation planes; The segmentation module 3013 is used to segment the three-dimensional data to be processed based on the segmentation direction and the segmentation plane to obtain the at least two two-dimensional data.
[0068] Furthermore, in a possible implementation of this embodiment, as Figure 6 As shown, the stacking unit 303 includes: The second acquisition module 3031 is used to respectively acquire the two-dimensionalization sequence of the at least two two-dimensional data; The stacking module 3032 is used to stack the two-dimensional features corresponding to the two-dimensional data according to the two-dimensionalization sequence to obtain the stacked features.
[0069] Furthermore, in a possible implementation of this embodiment, as Figure 6 As shown, the device also includes: An acquisition unit 305, configured to acquire at least two training two-dimensional features of the training three-dimensional data, and to acquire training stacked features of the training two-dimensional features; The second processing unit 306 is used to perform average processing on the training two-dimensional features to obtain average training two-dimensional features; The segmentation unit 307 is used to segment the training stacked feature into at least two training stacked feature blocks of the same shape, and to segment the training average two-dimensional feature into at least two training average two-dimensional feature blocks of the same shape; the training stacked feature block has the same shape as the training average two-dimensional feature block; The training unit 308 is used to train the initial feature fusion model using the training stacked feature blocks and the corresponding training averaged two-dimensional feature blocks to obtain the preset feature fusion model.
[0070] Furthermore, in a possible implementation of this embodiment, as Figure 6 As shown, the training unit 308 includes: A learning module 3081 is used to learn the training stacked feature block and the corresponding training average two-dimensional feature block to obtain a loss function; wherein the loss function includes the weight of the initial feature fusion model; The adjustment module 3082 is used to adjust the weights in a manner that makes the loss function approach a minimum value, so as to obtain the preset feature fusion model.
[0071] Furthermore, in a possible implementation of this embodiment, as Figure 6 As shown, the device also includes: An evaluation unit 309 is used to, after inputting the stacked features into a preset feature fusion model to obtain fused data of the stacked features, perform confidence evaluation on the fused data to obtain a confidence score of the fused data; The determination unit 310 is configured to determine the fused data as a final fusion result of the stacked features when the confidence score is greater than a preset threshold.
[0072] It should be noted that the above explanation of the method embodiment is also applicable to the device of the embodiment of the present disclosure, and the principle is the same, which is no longer limited in the embodiment of the present disclosure.
[0073] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.
[0074] Figure 7 A schematic block diagram of an example electronic device 400 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.
[0075] like Figure 7 As shown, the device 400 includes a computing unit 401, which can perform various appropriate actions and processes according to a computer program stored in a ROM (Read-Only Memory) 402 or a computer program loaded from a storage unit 408 to a RAM (Random Access Memory) 403. Various programs and data required for the operation of the device 400 can also be stored in the RAM 403. The computing unit 401, the ROM 402, and the RAM 403 are connected to each other via a bus 404. An I / O (Input / Output) interface 405 is also connected to the bus 404.
[0076] A number of components in the device 400 are connected to the I / O interface 405, including: an input unit 406, such as a keyboard, a mouse, etc.; an output unit 407, such as various types of displays, speakers, etc.; a storage unit 408, such as a disk, an optical disk, etc.; and a communication unit 409, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 409 allows the device 400 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.
[0077] The computing unit 401 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 401 include, but are not limited to, CPU (Central Processing Unit), GPU (Graphic Processing Units), various dedicated AI (Artificial Intelligence) computing chips, various computing units running machine learning model algorithms, DSP (Digital Signal Processor), and any appropriate processor, controller, microcontroller, etc. The computing unit 401 performs the various methods and processes described above, such as a feature fusion method for three-dimensional data. For example, in some embodiments, the feature fusion method for three-dimensional data may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 408. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 400 via the ROM 402 and / or the communication unit 409. When the computer program is loaded into the RAM 403 and executed by the computing unit 401, one or more steps of the method described above may be performed. Alternatively, in other embodiments, the computing unit 401 may be configured to execute the aforementioned three-dimensional data feature fusion method in any other appropriate manner (for example, by means of firmware).
[0078] Various embodiments of the systems and techniques described above herein may be implemented in digital electronic circuit systems, integrated circuit systems, FPGAs (Field Programmable Gate Arrays), ASICs (Application-Specific Integrated Circuits), ASSPs (Application Specific Standard Products), SOCs (System On Chips), CPLDs (Complex Programmable Logic Devices), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include: being implemented in one or more computer programs that may be executed and / or interpreted on a programmable system including at least one programmable processor that may be a dedicated or general-purpose programmable processor that may receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.
[0079] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.
[0080] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or apparatus. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any suitable combination of the foregoing. More specific examples of machine-readable storage media may include an electrical connection based on one or more wires, a portable computer disk, a hard disk, RAM, ROM, EPROM (Electrically Programmable Read-Only-Memory) or flash memory, optical fiber, CD-ROM (Compact Disc Read-Only Memory), optical storage device, magnetic storage device, or any suitable combination of the foregoing.
[0081] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (Cathode-Ray Tube) or LCD (Liquid Crystal Display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).
[0082] The systems and techniques described herein may be implemented in a computing system that includes a backend component (e.g., as a data server), or a computing system that includes a middleware component (e.g., an application server), or a computing system that includes a frontend component (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with an implementation of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: LAN (Local Area Network), WAN (Wide Area Network), the Internet, and blockchain networks.
[0083] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship between the client and the server is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services ("Virtual Private Server", or "VPS" for short). The server may also be a server of a distributed system, or a server combined with a blockchain.
[0084] It should be noted that artificial intelligence is a discipline that studies how computers can simulate certain human thought processes and intelligent behaviors (such as learning, reasoning, thinking, planning, etc.), and includes both hardware-level and software-level technologies. Artificial intelligence hardware technologies generally include technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, and big data processing; artificial intelligence software technologies mainly include computer vision technology, speech recognition technology, natural language processing technology, as well as machine learning / deep learning, big data processing technology, knowledge graph technology, and other major directions.
[0085] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.
[0086] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.
Claims
1. A feature fusion method for three-dimensional data, characterized in that: include: Acquire three-dimensional data to be processed, and perform two-dimensional processing on the three-dimensional data to be processed to obtain at least two two-dimensional data of the three-dimensional data to be processed; Inputting the at least two two-dimensional data into a preset feature extraction model respectively, and obtaining two-dimensional features corresponding to the at least two two-dimensional data respectively; Stacking the two-dimensional features to obtain stacked features; The stacked features are input into a preset feature fusion model to obtain fusion data of the stacked features.
2. The method according to claim 1, characterized in that The performing two-dimensional processing on the three-dimensional data to be processed to obtain at least two two-dimensional data of the three-dimensional data to be processed comprises: Respectively obtain three coordinate axes of the three-dimensional data to be processed; Determine any one of the three coordinate axes as a segmentation direction, and determine the remaining two of the three coordinate axes as segmentation planes; The three-dimensional data to be processed is segmented based on the segmentation direction and the segmentation plane to obtain the at least two two-dimensional data.
3. The method according to claim 1, characterized in that The stacking of the two-dimensional features to obtain stacked features comprises: Respectively obtaining a two-dimensional sequence of the at least two two-dimensional data; The two-dimensional features corresponding to the two-dimensional data are stacked according to the two-dimensionalization sequence to obtain the stacked features.
4. The method according to claim 1, characterized in that: The training method of the preset feature fusion model includes: Acquire at least two training two-dimensional features of the training three-dimensional data, and acquire training stacked features of the training two-dimensional features; Performing average processing on the training two-dimensional features to obtain average two-dimensional features for training; The training stacked feature is divided into at least two training stacked feature blocks with the same shape, and the training average two-dimensional feature is divided into at least two training average two-dimensional feature blocks with the same shape; the training stacked feature block has the same shape as the training average two-dimensional feature block; The initial feature fusion model is trained using the training stacked feature blocks and the corresponding training averaged two-dimensional feature blocks to obtain the preset feature fusion model.
5. The method according to claim 4, characterized in that The using the stacked feature blocks for training and the corresponding averaged two-dimensional feature blocks for training to train the initial feature fusion model to obtain the preset feature fusion model includes: The training stacked feature block and the corresponding training average two-dimensional feature block are learned to obtain a loss function; wherein the loss function includes the weight of the initial feature fusion model; The weights are adjusted in a manner that makes the loss function approach a minimum value to obtain the preset feature fusion model.
6. The method according to claim 1, characterized in that After inputting the stacked features into a preset feature fusion model to obtain fusion data of the stacked features, the method further includes: Performing confidence evaluation on the fused data to obtain a confidence score of the fused data; If the confidence score is greater than a preset threshold, the fused data is determined as the final fusion result of the stacked features.
7. A feature fusion device for three-dimensional data, characterized in that: include: A first processing unit is used to obtain three-dimensional data to be processed, and perform two-dimensional processing on the three-dimensional data to be processed to obtain at least two two-dimensional data of the three-dimensional data to be processed; A first input unit, used to input the at least two two-dimensional data into a preset feature extraction model respectively, to obtain two-dimensional features corresponding to the at least two two-dimensional data respectively; A stacking unit, used for stacking the two-dimensional features to obtain stacked features; The second input unit is used to input the stacked features into a preset feature fusion model to obtain fusion data of the stacked features.
8. An electronic device, characterized in that: include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 6.
9. A non-transitory computer-readable storage medium storing computer instructions, characterized in that: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-6.
10. A computer program product, characterized in that The invention comprises a computer program which, when executed by a processor, implements the method according to any one of claims 1 to 6.
