Feature extraction method and device for three-dimensional feature map, storage medium, and electronic device

Abstract

Disclosed are a method and device for extracting features from a three-dimensional feature map, a readable storage medium, and an electronic device, wherein the method comprises: determining a superposition parameter based on depth information of a three-dimensional feature map to be processed; decomposing the three-dimensional feature map into a plurality of target two-dimensional feature maps based on the depth information and the superposition parameter; performing a two-dimensional convolution process on each of the plurality of target two-dimensional feature maps to obtain a plurality of initial feature maps; and determining a target feature map corresponding to the three-dimensional feature map based on the plurality of initial feature maps.

Description

Technical Field

[0001] The present invention relates to a method and device for extracting features of a three-dimensional feature map, a storage medium, and an electronic device. Background Art

[0002] With the increasing demand for the application of 3D models in fields such as autonomous driving, support for three-dimensional convolution processing is imperative. However, since existing hardware components only support two-dimensional convolution processing in most cases, and there is some lag in hardware development relative to the evolution of algorithms, hardware components in the prior art usually cannot support three-dimensional convolution processing. Summary of the invention

[0003] In order to solve the above technical problems, the present disclosure is proposed. The embodiments of the present disclosure provide a method and device for extracting image features of a three-dimensional feature map, a storage medium, and an electronic device.

[0004] According to one aspect of the present disclosure, a method for extracting features from a three-dimensional feature map is provided, comprising:

[0005] Determining a superposition parameter based on depth information of the three-dimensional feature map to be processed;

[0006] Based on the depth information and the superposition parameter, decomposing the three-dimensional feature map into a plurality of target two-dimensional feature maps;

[0007] Performing two-dimensional convolution processing on each of the multiple target two-dimensional feature maps respectively to obtain multiple initial feature maps;

[0008] Based on the multiple initial feature maps, a target feature map corresponding to the three-dimensional feature map is determined.

[0009] According to another aspect of the present disclosure, there is provided a feature extraction device for a three-dimensional feature map, comprising:

[0010] A parameter determination module, used for determining a superposition parameter based on the depth information of the three-dimensional feature map to be processed;

[0011] A feature map decomposition module, configured to decompose the three-dimensional feature map into a plurality of target two-dimensional feature maps based on the depth information and the superposition parameter determined by the parameter determination module;

[0012] A convolution processing module, used for performing two-dimensional convolution processing on each of the multiple target two-dimensional feature maps obtained by the feature map decomposition module to obtain multiple initial feature maps;

[0013] The feature determination module is used to determine the target feature map corresponding to the three-dimensional feature map based on the multiple initial feature maps obtained by the convolution processing module.

[0014] According to another aspect of the present disclosure, a computer-readable storage medium is provided, wherein the storage medium stores a computer program, and the computer program is used to execute the feature extraction method of the three-dimensional feature map described in any of the above embodiments.

[0015] According to another aspect of the present disclosure, an electronic device is provided, the electronic device comprising:

[0016] processor;

[0017] a memory for storing instructions executable by the processor;

[0018] The processor is used to read the executable instructions from the memory and execute the instructions to implement the feature extraction method of the three-dimensional feature map described in any one of the above embodiments.

[0019] Based on the feature extraction method and device of a three-dimensional feature map provided by the above-mentioned embodiments of the present disclosure, a storage medium, and an electronic device are provided. By decomposing the three-dimensional feature map to be processed into multiple target two-dimensional feature maps based on depth information and superposition parameters, a hardware device supporting two-dimensional convolution processing can process the target two-dimensional feature map, and determine the target feature map corresponding to the three-dimensional feature map to be processed based on the initial feature map obtained by processing the target two-dimensional feature map, that is, three-dimensional convolution processing of the three-dimensional feature map is implemented through two-dimensional convolution processing. Since the processing process only depends on the implementation of two-dimensional convolution processing, there is no need to change the hardware structure, and three-dimensional convolution processing of the three-dimensional feature map can be implemented only based on hardware supporting two-dimensional convolution processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The above and other purposes, features and advantages of the present disclosure will become more apparent by describing the embodiments of the present disclosure in more detail in conjunction with the accompanying drawings. The accompanying drawings are used to provide a further understanding of the embodiments of the present disclosure and constitute a part of the specification. Together with the embodiments of the present disclosure, they are used to explain the present disclosure and do not constitute a limitation of the present disclosure. In the drawings, the same reference numerals generally represent the same components or steps.

[0021] Figure 1 It is a schematic diagram of mapping Conv3d to Conv2d provided by an exemplary embodiment of the present disclosure.

[0022] Figure 2 It is a flowchart of a feature extraction method of a three-dimensional feature map provided by an exemplary embodiment of the present disclosure.

[0023] Figure 3 This disclosure Figure 2 A flow chart of step 202 in the embodiment shown is shown.

[0024] Figure 4aThis disclosure Figure 2 A flow chart of step 204 in the embodiment shown is shown.

[0025] Figure 4b This disclosure Figure 4a A flowchart of step 2042 in the embodiment shown is shown.

[0026] Figure 5 This disclosure Figure 2 A flow chart of step 206 in the embodiment shown is shown.

[0027] Figure 6 This disclosure Figure 2 A schematic diagram of a flow chart of step 208 in the embodiment shown.

[0028] Figure 7 It is a structural schematic diagram of a feature extraction device for a three-dimensional feature map provided by an exemplary embodiment of the present disclosure.

[0029] Figure 8 It is a structural schematic diagram of a feature extraction device for a three-dimensional feature map provided by another exemplary embodiment of the present disclosure.

[0030] Fig. 9 is a structural diagram of an electronic device provided by an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0031] Below, the exemplary embodiments according to the present disclosure will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present disclosure, rather than all the embodiments of the present disclosure, and it should be understood that the present disclosure is not limited to the exemplary embodiments described here.

[0032] It should be noted that the relative arrangement of components and steps, the numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present disclosure unless specifically stated otherwise.

[0033] Those skilled in the art can understand that the terms "first" and "second" in the embodiments of the present disclosure are only used to distinguish different steps, devices or modules, etc., and neither represent any specific technical meaning nor indicate the necessary logical order between them.

[0034] It should also be understood that in the embodiments of the present disclosure, “plurality” may refer to two or more than two, and “at least one” may refer to one, two, or more than two.

[0035] It should also be understood that any component, data or structure mentioned in the embodiments of the present disclosure can generally be understood as one or more, unless explicitly limited or otherwise indicated in the context.

[0036] In addition, the term "and / or" in the present disclosure is only a description of the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in the present disclosure generally indicates that the associated objects before and after are in an "or" relationship.

[0037] It should also be understood that the description of the various embodiments in the present disclosure focuses on the differences between the various embodiments, and the same or similar aspects thereof can be referenced to each other, and for the sake of brevity, they will not be described one by one.

[0038] At the same time, it should be understood that for the convenience of description, the sizes of the various parts shown in the drawings are not drawn according to the actual proportional relationship.

[0039] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the present disclosure, its application, or uses.

[0040] Technologies, methods, and equipment known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, the technologies, methods, and equipment should be considered as part of the specification.

[0041] It should be noted that like reference numerals and letters refer to similar items in the following figures, and therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.

[0042] The disclosed embodiments can be applied to electronic devices such as terminal devices, computer systems, servers, etc., which can operate with many other general or special computing system environments or configurations. Examples of well-known terminal devices, computing systems, environments and / or configurations suitable for use with electronic devices such as terminal devices, computer systems, servers, etc. include, but are not limited to: personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, microprocessor-based systems, set-top boxes, programmable consumer electronics, network personal computers, small computer systems, large computer systems, and distributed cloud computing technology environments including any of the above systems, etc.

[0043] Electronic devices such as terminal devices, computer systems, servers, etc. can be described in the general context of computer system executable instructions (such as program modules) executed by computer systems. Generally, program modules can include routines, programs, object programs, components, logic, data structures, etc., which perform specific tasks or implement specific abstract data types. Computer systems / servers can be implemented in a distributed cloud computing environment, where tasks are performed by remote processing devices linked through a communication network. In a distributed cloud computing environment, program modules can be located on local or remote computing system storage media including storage devices.

[0044] Application Overview

[0045] In the process of implementing the present disclosure, the inventors discovered that the hardware devices provided by the prior art can only support two-dimensional convolution processing but cannot support three-dimensional convolution processing.

[0046] The pseudo code for the convolution process of the 3D convolutional (conv3d) network is as follows:

[0047]

[0048] Based on the above pseudo code, we can know that the calculation process of 3D convolution includes 8 layers of loops: the outermost loop is to calculate the output Cout iteration, and the 2nd to 4th layers of loops inward are to perform addition operations in the dimensions of h (height, referring to the height of the feature map), w (width, referring to the width of the feature map), and d (depth, referring to the depth of the feature map, that is, the dimension added by the 3D feature map relative to the 2D feature map); the 5th to 7th layers of loops inward are the sliding of a kernel (the convolution kernel when performing convolution operations, sliding on the feature map for multiplication and addition operations in units of convolution kernels) required by the output point on the input. The kernel sizes in the three directions of h, w, and d need to be accumulated, expressed as kh, kw, and kd; the innermost loop is the accumulation in the input channel (channel, a dimension shared by 2D convolution and 3D convolution, no sliding is done on this dimension, only accumulation is done).

[0049] In general, the outer 4-layer loop is a traversal of the 4-dimensional output point (without considering the batch), and the inner 4-layer loop is a traversal of the multiplication and accumulation data required for an output point.

[0050] The difference between 3D convolution (conv3d) and 2D convolution (conv2d) is that it not only slides in the h and w directions included in conv2d, but also slides in the d direction, and the kernel is four-dimensional and slides in the h, w, and d directions at the same time.

[0051] Figure 1Schematic diagram of mapping Conv3d to Conv2d provided by an exemplary embodiment of the present disclosure. Figure 1 As shown, the disclosed embodiment places the d-dimensional loop in the 3D convolution in the outermost layer, splits the 3D convolution processing of the 3D feature map (Feature) into independent (d / kd) conv2d, and splits the 3D feature map in the D direction, wherein the convolution kernel calculated each time by the 2D convolution is still the size of the 3D convolution kernel (kh*kw*kd), except that the size C of the convolution kernel in the 2D convolution is kd*C (that is, the disclosed embodiment adopts superposition in the C direction to accumulate kd, so as to place the d-dimensional loop in the outermost layer), and the result of each calculation of the 2D convolution is a feature map (H*W) of a point output in the D direction. By superimposing the feature maps output by each 2D convolution in the D direction, the 3D feature map corresponding to the 3D convolution can be obtained.

[0052] The calculation process of splitting conv3d into conv2d can be implemented by the following pseudo code:

[0053]

[0054] Compared with conv2d, conv3d has more d-dimensional data, so converting conv3d to conv2d is to extract the calculation of d-dimensional data, including the accumulation of output point d data and kernel d on input (kd). Here, the traversal of d-dimensional data is placed on the outermost layer of calculation, and the accumulation on kernel d is superimposed on the input channel direction. In this pseudocode, except for the outermost d dimension, the innermost loop is a conv2d, where the conv2d is: input feature shape: h*w*(ci*kd); weight shape is: kh*kw*(ci*kd), and 3d convolution operation is realized by looping conv2d d / kd times.

[0055] Exemplary Methods

[0056] Figure 2 FIG. 1 is a flow chart of a feature extraction method of a three-dimensional feature map provided by an exemplary embodiment of the present disclosure. This embodiment can be applied to electronic devices, such as Figure 2 As shown, the following steps are included:

[0057] Step 202: Determine a superposition parameter based on the depth information of the three-dimensional feature map to be processed.

[0058] Among them, the three-dimensional depth map includes depth information that is not included in the two-dimensional depth map. For example, the depth information of the three-dimensional depth map can be represented by d, and the superposition parameter can be the convolution kernel size of the 3D convolution in the depth direction, for example, the kd provided in the above embodiment.

[0059] Step 204: decompose the three-dimensional feature map into multiple target two-dimensional feature maps based on the depth information and the superposition parameters.

[0060] In this embodiment, the three-dimensional feature map is decomposed in the depth direction to obtain multiple target two-dimensional feature maps. At this time, each target two-dimensional feature map can perform a convolution operation through a 2D convolution.

[0061] Step 206 , performing two-dimensional convolution processing on each of the multiple target two-dimensional feature maps to obtain multiple initial feature maps.

[0062] Among them, each target two-dimensional feature map corresponds to an initial feature map.

[0063] Each of the target two-dimensional feature maps obtained based on the decomposition of the three-dimensional feature map is a two-dimensional feature map. At this time, the two-dimensional feature map can be directly calculated through conv2d to obtain the initial feature map corresponding to each target two-dimensional feature map.

[0064] Step 208: Determine a target feature map corresponding to the three-dimensional feature map based on the multiple initial feature maps.

[0065] Since each of the multiple initial feature maps does not include depth information of the three-dimensional feature map in the depth direction, in order to obtain a target feature map corresponding to the three-dimensional feature map, this embodiment may perform operations such as superimposing the multiple initial feature maps to obtain the target feature map.

[0066] The above-mentioned embodiment of the present disclosure provides a feature extraction method for a three-dimensional feature map, which decomposes the three-dimensional feature map to be processed into multiple target two-dimensional feature maps based on depth information and superposition parameters, so that a hardware device supporting two-dimensional convolution processing can process the target two-dimensional feature map, and determines the target feature map corresponding to the three-dimensional feature map to be processed based on the initial feature map obtained by processing the target two-dimensional feature map, that is, three-dimensional convolution processing of the three-dimensional feature map is implemented through two-dimensional convolution processing. Since the processing process only depends on the implementation of two-dimensional convolution processing, there is no need to change the hardware structure, and three-dimensional convolution processing of the three-dimensional feature map can be implemented only based on hardware supporting two-dimensional convolution processing.

[0067] like Figure 3 As shown in the above Figure 2 Based on the illustrated embodiment, step 202 may include the following steps:

[0068] Step 2021, determining the weight value in the three-dimensional convolution kernel corresponding to the three-dimensional feature map when performing three-dimensional convolution processing.

[0069] When a three-dimensional feature map is processed by an existing three-dimensional convolutional network, the convolution kernel is a three-dimensional convolution sum, wherein the three-dimensional convolution kernel includes information of the depth dimension that is not included in the two-dimensional convolution processing. In this embodiment, in order to decompose the three-dimensional feature map into a two-dimensional feature map that can be processed by a two-dimensional convolutional network, the three-dimensional convolution kernel can be referred to to determine how to decompose the three-dimensional feature map.

[0070] Step 2022: Determine a superposition parameter based on a first weight value corresponding to a depth dimension in the three-dimensional convolution kernel.

[0071] Optionally, the first weight value is used as a superposition parameter.

[0072] In this embodiment, in the three-dimensional convolution kernel, different dimensions correspond to different dimensions in the three-dimensional feature map, that is, the depth dimension in the three-dimensional convolution kernel corresponds to the depth information in the three-dimensional feature map. Therefore, in order to decompose the depth information in the three-dimensional feature map, the first weight value corresponding to the depth dimension in the three-dimensional convolution kernel can be used as a superposition parameter. Based on the superposition parameter and the depth information in the three-dimensional feature map, the three-dimensional feature map can be decomposed from the depth dimension, so that the decomposed feature map can be processed by two-dimensional convolution, thereby improving the decomposition efficiency of the three-dimensional feature map.

[0073] like Figure 4a As shown in the above Figure 2 Based on the illustrated embodiment, step 204 may include the following steps:

[0074] Step 2041, decomposing the three-dimensional feature map in the depth direction, and decomposing the three-dimensional feature map into a first number of two-dimensional feature maps.

[0075] The first number is an integer greater than 1.

[0076] Optionally, the first number may be determined by depth information and a superposition parameter of the three-dimensional feature map, for example, an integer obtained by dividing the depth information by the superposition parameter.

[0077] Step 2042: Expand the channel dimension in each of the first number of two-dimensional feature maps based on the superposition parameter to obtain a first number of target two-dimensional feature maps.

[0078] In this embodiment, since the three-dimensional feature map is split in the depth direction, for example, the three-dimensional feature map is split into independent d / kd two-dimensional feature maps; and the three-dimensional convolution processing performs a convolution operation in the depth direction, it is necessary to compensate for the accumulation of superposition parameters in other parts. In this embodiment, the superposition parameters are accumulated by superimposing them on the channel dimension (for example, the c direction), or the kernel accumulation is performed separately to compensate for the calculation difference caused by splitting the depth information. Each target two-dimensional feature map obtained includes part of the information of the convolution processing of the depth information in the three-dimensional feature map, so as to achieve the purpose of superimposing a first number of target two-dimensional feature maps to obtain the target feature map corresponding to the three-dimensional feature map.

[0079] like Figure 4b As shown in the above Figure 4a Based on the illustrated embodiment, step 2042 may include the following steps:

[0080] Step 401 : Move with a set step length in the depth information direction of the three-dimensional feature map based on the superposition parameter.

[0081] Optionally, the step size is set to the distance that the superposition parameter moves in the direction of the depth information. The value of the step size can be set according to the actual scenario. For example, taking the dimension of the three-dimensional feature map as 1x111x111x111 (the dimension of the depth information) x64 as an example, the dimension of the three-dimensional convolution kernel is 3x3x3, the dimension of the padding is 1x1x1, and the dimension of the stride of the three-dimensional convolution is 1x1x1, which can be split into 111 conv2d. In this example, the step size is set to 1.

[0082] Step 402: Determine a first number of target two-dimensional feature maps based on a first number of groups of two-dimensional image groups obtained by a first number of moves.

[0083] In this embodiment, each two-dimensional image group can be used as a target two-dimensional feature map. By moving in the depth information direction based on a set step size, a two-dimensional image group can be obtained each time the movement is performed. The specific process of obtaining the target two-dimensional feature map may include:

[0084] Based on each movement in the first number of movements, obtaining a set of two-dimensional feature maps including a number of two-dimensional feature maps superimposed with a parameter;

[0085] Each two-dimensional feature map in the two-dimensional feature map group is used as a feature map of a channel in the target two-dimensional feature map, so as to obtain a target two-dimensional feature map in which a first number of channels is obtained and a number of superposition parameters is obtained.

[0086] In this embodiment, a group of two-dimensional feature maps whose depth information is split is obtained through each movement. In order to reduce the amount of information lost in the splitting process of the depth information, this embodiment performs a superposition operation in the channel direction with a superposition parameter. For example, the number of channels of the three-dimensional feature map is 64 (for example, the dimension of the three-dimensional feature map is: 1x111x111x111x64), the superposition parameter is 3, and the number of channels of the target two-dimensional feature map obtained after splitting is 3x64, and the dimension of the two-dimensional feature map required for calculating the first point in the depth dimension is: 1x111x111x(2x64); this embodiment improves the expression of the depth information in the three-dimensional feature map by the target two-dimensional feature map through superposition in the channel dimension, thereby improving the accuracy of the target feature map obtained based on the target two-dimensional feature map.

[0087] like Figure 5 As shown in the above Figure 2 Based on the illustrated embodiment, step 206 may include the following steps:

[0088] Step 2061, based on the three-dimensional convolution kernel corresponding to the three-dimensional convolution processing performed on the three-dimensional feature map, determine the corresponding two-dimensional convolution kernel in the two-dimensional convolution processing.

[0089] Optionally, the two-dimensional convolution kernel obtained based on the three-dimensional convolution kernel can be obtained by directly eliminating the depth dimension, or on the basis of eliminating the depth dimension, supplementing it in other dimensions to obtain the two-dimensional convolution kernel.

[0090] Step 2062, based on the two-dimensional convolution kernel, perform two-dimensional convolution processing on each target two-dimensional feature map in the multiple target two-dimensional feature maps to obtain multiple initial feature maps.

[0091] In this embodiment, a two-dimensional convolution kernel is used to perform a convolution calculation process. Since the object of the convolution calculation (target two-dimensional feature map) and the convolution kernel are both two-dimensional, the calculation process is the same as that of an ordinary two-dimensional convolution calculation. Two-dimensional convolution processing is performed on each target two-dimensional feature map to obtain multiple initial features. The target feature map is determined based on the multiple initial features, thereby realizing three-dimensional convolution processing of the three-dimensional feature map through two-dimensional convolution operation. At this time, there is no need to process the hardware, and only hardware that can support two-dimensional convolution calculation is required to realize feature extraction of the three-dimensional feature map.

[0092] Optionally, step 2061 may include:

[0093] a1, merge the first value of the depth dimension in the three-dimensional convolution kernel with the second value of the channel dimension in the three-dimensional convolution kernel to obtain a third value.

[0094] The third value may be equal to the product of the first value and the second value.

[0095] a2, based on the third value as the corresponding channel dimension value in the two-dimensional convolution kernel, a two-dimensional convolution kernel with compressed dimension number is obtained.

[0096] In this embodiment, since superposition is performed on the channel dimension of the target two-dimensional feature map in the process of decomposing the three-dimensional feature map into multiple target two-dimensional feature maps, the channel dimension is also superimposed in the corresponding convolution kernel, that is, the product of the first value and the second value is used as the channel dimension value in the two-dimensional convolution sum, so that the two-dimensional convolution kernel corresponds to the dimension of the target two-dimensional feature map, which makes it easier to implement the convolution calculation on the target two-dimensional feature map.

[0097] like Figure 6 As shown in the above Figure 2 Based on the illustrated embodiment, step 208 may include the following steps:

[0098] Step 2081, superimpose multiple initial features in the depth direction.

[0099] Step 2082, obtaining a target feature map of the quantity values ​​of the initial features corresponding to the depth information.

[0100] In this embodiment, by superimposing multiple initial features in the depth direction of the feature map, the output target feature map is able to reflect the depth information through the superposition in the depth direction, that is, the result of the three-dimensional feature map after three-dimensional convolution processing is restored, and the technical effect of obtaining the result of three-dimensional convolution processing based on two-dimensional convolution calculation is achieved.

[0101] Any feature extraction method provided in the embodiments of the present disclosure may be executed by any appropriate device with data processing capabilities, including but not limited to: a terminal device and a server, etc. Alternatively, any feature extraction method of a three-dimensional feature map provided in the embodiments of the present disclosure may be executed by a processor, such as the processor executing any feature extraction method of a three-dimensional feature map mentioned in the embodiments of the present disclosure by calling corresponding instructions stored in a memory. This will not be described in detail below.

[0102] Exemplary Devices

[0103] Figure 7 FIG. 1 is a schematic diagram of a structure of a feature extraction device for a three-dimensional feature map provided by an exemplary embodiment of the present disclosure. Figure 7 As shown, the device provided in this embodiment includes:

[0104] The parameter determination module 71 is used to determine the superposition parameters based on the depth information of the three-dimensional feature map to be processed.

[0105] The feature map decomposition module 72 is used to decompose the three-dimensional feature map into multiple target two-dimensional feature maps based on the depth information and the superposition parameters determined by the parameter determination module 71.

[0106] The convolution processing module 73 is used to perform two-dimensional convolution processing on each of the multiple target two-dimensional feature maps obtained by the feature map decomposition module 72 to obtain multiple initial feature maps.

[0107] The feature determination module 74 is used to determine the target feature map corresponding to the three-dimensional feature map based on the multiple initial feature maps obtained by the convolution processing module 73.

[0108] The above-mentioned embodiment of the present disclosure provides a feature extraction device for a three-dimensional feature map, which decomposes the three-dimensional feature map to be processed into multiple target two-dimensional feature maps based on depth information and superposition parameters, so that a hardware device supporting two-dimensional convolution processing can process the target two-dimensional feature map, and determines the target feature map corresponding to the three-dimensional feature map to be processed based on the initial feature map obtained by processing the target two-dimensional feature map, that is, three-dimensional convolution processing of the three-dimensional feature map is implemented through two-dimensional convolution processing. Since the processing process only depends on the implementation of two-dimensional convolution processing, there is no need to change the hardware structure, and three-dimensional convolution processing of the three-dimensional feature map can be implemented only based on hardware supporting two-dimensional convolution processing.

[0109] Figure 8 FIG. 1 is a schematic diagram of a feature extraction device provided by another exemplary embodiment of the present disclosure. Figure 8 As shown, in the device provided by this embodiment, the parameter determination module 71 includes:

[0110] The weight determination unit 711 is used to determine the weight value in the corresponding three-dimensional convolution kernel when performing three-dimensional convolution processing on the three-dimensional feature map.

[0111] The superposition parameter unit 712 is used to determine the superposition parameter based on the first weight value corresponding to the depth dimension in the three-dimensional convolution kernel.

[0112] In some optional embodiments, the feature graph decomposition module 72 includes:

[0113] The depth information decomposition unit 721 is used to decompose the three-dimensional feature map in the depth direction, and decompose the three-dimensional feature map into a first number of two-dimensional feature maps.

[0114] The first number is an integer greater than 2.

[0115] The channel expansion unit 722 is used to expand the channel dimension in each of the first number of two-dimensional feature maps based on the superposition parameter to obtain the first number of target two-dimensional feature maps.

[0116] Optionally, the depth information decomposition unit 721 is specifically used to move the depth information direction of the three-dimensional feature map with a set step size based on the superposition parameter; and determine a first number of target two-dimensional feature maps based on a first number of groups of two-dimensional image groups obtained by a first number of movements.

[0117] Optionally, the depth information decomposition unit 721 is also used to obtain a group of two-dimensional feature maps including a number of superimposed parameters based on each movement in the first number of movements; and use each two-dimensional feature map in the two-dimensional feature map group as a feature map of a channel in the target two-dimensional feature map to obtain a target two-dimensional feature map whose number of channels is the number of superimposed parameters.

[0118] In some optional embodiments, the convolution processing module 73 includes:

[0119] The convolution kernel determination unit 731 is used to determine the corresponding two-dimensional convolution kernel in the two-dimensional convolution processing based on the corresponding three-dimensional convolution kernel when the three-dimensional convolution processing is performed on the three-dimensional feature map.

[0120] The two-dimensional convolution unit 732 is used to perform two-dimensional convolution processing on each target two-dimensional feature map in the multiple target two-dimensional feature maps based on the two-dimensional convolution kernel to obtain multiple initial feature maps.

[0121] Optionally, the convolution kernel determination unit 731 is specifically used to merge a first value corresponding to the depth dimension in the three-dimensional convolution kernel with a second value corresponding to the channel dimension in the three-dimensional convolution kernel to obtain a third value; based on the third value as the corresponding channel dimension value in the two-dimensional convolution kernel, a two-dimensional convolution kernel with compressed dimensionality is obtained.

[0122] In some optional embodiments, the feature determination module 74 includes:

[0123] The superposition unit 741 is used to superimpose multiple initial features in the depth direction.

[0124] The target feature unit 742 is used to obtain a target feature map of the quantity value of the initial feature corresponding to the depth information.

[0125] Exemplary Electronic Devices

[0126] Below, reference Fig. 9 The electronic device according to the embodiment of the present disclosure is described. The electronic device may be any one or both of the first device 100 and the second device 200, or a stand-alone device independent of them, and the stand-alone device may communicate with the first device and the second device to receive the collected input signals from them.

[0127] Fig. 9 A block diagram of an electronic device according to an embodiment of the present disclosure is illustrated.

[0128] like Fig. 9 As shown, the electronic device 90 includes one or more processors 91 and a memory 92 .

[0129] The processor 91 may be a central processing unit (CPU) or other forms of processing units having data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 90 to perform desired functions.

[0130] The memory 92 may include one or more computer program products, and the computer program product may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, a random access memory (RAM) and / or a cache memory (cache), etc. The non-volatile memory may include, for example, a read-only memory (ROM), a hard disk, a flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 91 may run the program instructions to implement the feature extraction method of the three-dimensional feature map of each embodiment of the present disclosure described above and / or other desired functions. Various contents such as input signals, signal components, noise components, etc. may also be stored in the computer-readable storage medium.

[0131] In one example, the electronic device 90 may further include: an input device 93 and an output device 94 , and these components are interconnected via a bus system and / or other forms of connection mechanisms (not shown).

[0132] For example, when the electronic device is the first device 100 or the second device 200, the input device 93 may be the microphone or microphone array described above, for capturing input signals of a sound source. When the electronic device is a stand-alone device, the input device 93 may be a communication network connector, for receiving collected input signals from the first device 100 and the second device 200.

[0133] In addition, the input device 93 may also include, for example, a keyboard, a mouse, and the like.

[0134] The output device 94 can output various information to the outside, including the determined distance information, direction information, etc. The output device 94 can include, for example, a display, a speaker, a printer, a communication network and a remote output device connected thereto, and the like.

[0135] Of course, to simplify, Fig. 9 Only some of the components related to the present disclosure in the electronic device 90 are shown, and components such as a bus, an input / output interface, etc. are omitted. In addition, according to specific application situations, the electronic device 90 may also include any other appropriate components.

[0136] Exemplary computer program products and computer-readable storage media

[0137] In addition to the above-mentioned methods and devices, an embodiment of the present disclosure may also be a computer program product, which includes computer program instructions, which, when executed by a processor, enable the processor to execute the steps of the feature extraction method of the three-dimensional feature map according to various embodiments of the present disclosure described in the above-mentioned "Exemplary Method" section of this specification.

[0138] The computer program product may be written in any combination of one or more programming languages ​​to write program code for performing the operations of the disclosed embodiments, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0139] In addition, an embodiment of the present disclosure may also be a computer-readable storage medium having computer program instructions stored thereon, which, when executed by a processor, causes the processor to execute the steps of the feature extraction method of the three-dimensional feature map according to various embodiments of the present disclosure described in the above “Exemplary Method” section of this specification.

[0140] The computer readable storage medium can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can include, for example, but is not limited to, a system, device or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0141] The basic principles of the present disclosure are described above in conjunction with specific embodiments. However, it should be noted that the advantages, strengths, effects, etc. mentioned in the present disclosure are only examples and not limitations, and it cannot be considered that these advantages, strengths, effects, etc. are required by each embodiment of the present disclosure. In addition, the specific details disclosed above are only for the purpose of illustration and ease of understanding, and are not limitations. The above details do not limit the present disclosure to the necessity of adopting the above specific details to be implemented.

[0142] The block diagrams of the devices, apparatuses, equipment, and systems involved in this disclosure are only illustrative examples and are not intended to require or imply that they must be connected, arranged, and configured in the manner shown in the block diagrams. As will be appreciated by those skilled in the art, these devices, apparatuses, equipment, and systems can be connected, arranged, and configured in any manner. Words such as "including," "comprising," "having," and the like are open words, referring to "including but not limited to," and can be used interchangeably therewith. The words "or" and "and" used herein refer to the words "and / or," and can be used interchangeably therewith, unless the context clearly indicates otherwise. The word "such as" used herein refers to the phrase "such as but not limited to," and can be used interchangeably therewith.

[0143] It should also be noted that in the apparatus, device and method of the present disclosure, each component or each step can be decomposed and / or recombined. Such decomposition and / or recombination should be regarded as equivalent solutions of the present disclosure.

[0144] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use the present disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of the present disclosure. Therefore, the present disclosure is not intended to be limited to the aspects shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

[0145] The above description has been given for the purpose of illustration and description. In addition, this description is not intended to limit the embodiments of the present disclosure to the forms disclosed herein. Although multiple example aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, changes, additions and sub-combinations thereof.

Claims

1. A feature extraction method for a three-dimensional feature map, comprising: Determining a superposition parameter based on depth information of the three-dimensional feature map to be processed; Based on the depth information and the superposition parameter, decomposing the three-dimensional feature map into a plurality of target two-dimensional feature maps; Performing two-dimensional convolution processing on each of the multiple target two-dimensional feature maps respectively to obtain multiple initial feature maps; Superimposing the multiple initial feature maps to determine a target feature map corresponding to the three-dimensional feature map; The step of decomposing the three-dimensional feature map into a plurality of target two-dimensional feature maps based on the depth information and the superposition parameter comprises: Decomposing the three-dimensional feature map in the depth direction to decompose the three-dimensional feature map into a first number of two-dimensional feature maps; wherein the first number n is an integer greater than 2 obtained by dividing the depth information by the superposition parameter; Expanding the channel dimension in each of the two-dimensional feature maps of the first number of two-dimensional feature maps based on the superposition parameter to obtain the first number of target two-dimensional feature maps; The determining of the superposition parameter based on the depth information of the three-dimensional feature map to be processed includes: Determine a weight value in a three-dimensional convolution kernel corresponding to when performing three-dimensional convolution processing on the three-dimensional feature map; The superposition parameter is determined based on a first weight value corresponding to a depth dimension in the three-dimensional convolution kernel.

2. The method according to claim 1, wherein: The expanding the channel dimension in each of the two-dimensional feature maps of the first number of two-dimensional feature maps based on the superposition parameters to obtain the first number of target two-dimensional feature maps includes: Based on the superposition parameter, a movement with a set step length is performed in a depth information direction of the three-dimensional feature map; Based on a first number of groups of two-dimensional image groups obtained by a first number of moves, a first number of target two-dimensional feature maps are determined.

3. The method according to claim 2, wherein: Determining a first number of target two-dimensional feature maps based on a first number of groups of two-dimensional image groups obtained by the first number of moves includes: Based on each movement in the first number of movements, obtaining a set of two-dimensional feature maps including the number of two-dimensional feature maps of the superimposed parameter; Each of the two-dimensional feature maps in the two-dimensional feature map group is used as a feature map of a channel in a target two-dimensional feature map, so as to obtain the target two-dimensional feature map whose number of channels is the number of superposition parameters.

4. The method according to any one of claims 1 to 3, wherein: The performing two-dimensional convolution processing on each target two-dimensional feature map in the multiple target two-dimensional feature maps to obtain multiple initial feature maps includes: Determining a corresponding two-dimensional convolution kernel in a two-dimensional convolution process based on a three-dimensional convolution kernel corresponding to the three-dimensional convolution process performed on the three-dimensional feature map; Based on the two-dimensional convolution kernel, two-dimensional convolution processing is performed on each target two-dimensional feature map in the multiple target two-dimensional feature maps to obtain multiple initial feature maps.

5. The method according to claim 4, wherein: The determining, based on the three-dimensional convolution kernel corresponding to the three-dimensional convolution processing performed on the three-dimensional feature map, the corresponding two-dimensional convolution kernel in the two-dimensional convolution processing includes: Merge a first value corresponding to the depth dimension in the three-dimensional convolution kernel and a second value corresponding to the channel dimension in the three-dimensional convolution kernel to obtain a third value; Based on the third value as the corresponding channel dimension value in the two-dimensional convolution kernel, the two-dimensional convolution kernel with compressed dimension is obtained.

6. The method according to any one of claims 1 to 3, wherein: The step of superimposing the multiple initial features to determine a target feature map corresponding to the three-dimensional feature map includes: Superimposing the multiple initial features in the depth direction; The target feature map of the quantity value of the initial feature corresponding to the depth information is obtained.

7. A feature extraction device for a three-dimensional feature map, comprising: A parameter determination module, used for determining a superposition parameter based on the depth information of the three-dimensional feature map to be processed; A feature map decomposition module, configured to decompose the three-dimensional feature map into a plurality of target two-dimensional feature maps based on the depth information and the superposition parameter determined by the parameter determination module; A convolution processing module, used for performing two-dimensional convolution processing on each of the multiple target two-dimensional feature maps obtained by the feature map decomposition module to obtain multiple initial feature maps; A feature determination module, used to superimpose multiple initial feature maps obtained by the convolution processing module to determine a target feature map corresponding to the three-dimensional feature map; The feature map decomposition module includes: a depth information decomposition unit, used to decompose the three-dimensional feature map in the depth direction, and decompose the three-dimensional feature map into a first number of two-dimensional feature maps; wherein the first number n is an integer greater than 2 obtained by dividing the depth information by the superposition parameter; a channel expansion unit, configured to expand the channel dimension in each of the two-dimensional feature maps of the first number based on the superposition parameter, to obtain the first number of target two-dimensional feature maps; The parameter determination module includes: a weight determination unit, used to determine the weight value in the corresponding three-dimensional convolution kernel when performing three-dimensional convolution processing on the three-dimensional feature map; and a superposition parameter unit, used to determine the superposition parameter based on the first weight value corresponding to the depth dimension in the three-dimensional convolution kernel.

8. A computer-readable storage medium storing a computer program, wherein the computer program is used to execute the feature extraction method of the three-dimensional feature map described in any one of claims 1 to 6.

9. An electronic device, comprising: processor; a memory for storing instructions executable by the processor; The processor is used to read the executable instructions from the memory and execute the instructions to implement the feature extraction method of the three-dimensional feature map described in any one of claims 1-6.

Images