Multimodal feature selection and image data classification method, device and computer equipment

Abstract

The present invention discloses a multimodal feature selection and image data classification method, device and computer equipment. The multimodal feature selection method includes: respectively obtaining data sets of multiple images corresponding to multiple modalities; respectively performing feature extraction on the multiple images, respectively determining multiple first feature sets corresponding to the multiple modalities; and based on a preset multimodal feature selection model, screening the features in the multiple first feature sets to determine a second feature set; wherein the multimodal feature selection model includes: a distance sub-model, which is used to minimize the distance between the mapping points of the image data corresponding to multiple modalities and of the same category in the feature space constructed by the corresponding first feature set and projected onto the feature space constructed by the second feature set. The multimodal feature selection method provided by the present invention can effectively induce the multimodal feature selection model to select features with high discriminability.

Description

Background Art

[0002] Taking the field of medical imaging as an example, multimodal images refer to images obtained from different channels such as MRI (Magnetic Resonance Imaging) and PET (Position Emission Tomography) for the same clinical diagnosis. With the increase of image data, the workload of doctors is increasing due to the manual analysis of multimodal images. Therefore, computer-assisted technology is urgently needed to analyze and process multimodal images.

[0003] In multimodal image analysis, since the features extracted from the original medical images usually have high dimensions, and only a small number of samples in clinical practice contain complete multimodal data, it is very likely to cause dimensionality disaster when applied. Deep learning requires a large number of samples to train the model. If the number of samples is small under high-dimensional features, it will lead to serious overfitting problems. Therefore, how to select and retain some highly discriminative features through feature learning to effectively improve the generalization ability of machine learning algorithms is an important issue currently facing the field of multimodal image analysis.

[0004] It should be noted that the above information disclosed in the background technology section is only used to enhance the understanding of the background of the present invention, and therefore it may include information that does not constitute the prior art known to ordinary technicians in the field. Summary of the invention

[0005] In view of this, the present invention provides a multimodal feature selection and image data classification method, apparatus, computer equipment and computer-readable storage medium.

[0006] Other features and advantages of the present invention will become apparent from the following detailed description, or may be learned in part by practice of the present invention.

[0007] According to one aspect of the present invention, there is provided a multimodal feature selection method, comprising: respectively acquiring data sets of multiple images corresponding to multiple modalities, the data set of each image comprising multiple image data corresponding to multiple position points of the image; respectively performing feature extraction on the multiple images, respectively determining multiple first feature sets corresponding to the multiple modalities; and based on a preset multimodal feature selection model, screening features in the multiple first feature sets to determine a second feature set; wherein the multimodal feature selection model comprises: a distance sub-model, for minimizing the distance between mapping points of image data of the same category corresponding to the multiple modalities in the feature space constructed by the corresponding first feature set after being projected onto the feature space constructed by the second feature set.

[0008] According to an embodiment of the present invention, the distance sub-model includes a label-aligned regularization function.

[0009] According to one embodiment of the present invention, the regularization function of the label alignment is as follows:

[0010]

[0011] Among them, μ is the regularization parameter, p and q represent the modes, and w p 、w q Respectively represent the weight vectors corresponding to the p and q modes, W = [w 1 , w 2 ,…,w M ] is the weight matrix corresponding to M modes, and Respectively represent the mapping points of the image data of the p and q modes in the feature space constructed by the corresponding first feature set, i and j represent the numbers of the N position points of the image data in the corresponding image,

[0012]

[0013] Among them, y i ,y j They are and The category label of the corresponding image data.

[0014] According to one embodiment of the present invention, the objective function of the multimodal feature selection model is solved based on an accelerated approximate gradient algorithm.

[0015] According to an embodiment of the present invention, respectively extracting features from the plurality of images includes: respectively extracting features from images of each modality according to a predefined template.

[0016] According to another aspect of the present invention, there is provided an image data classification method, comprising: acquiring image data to be classified; and classifying the image data to be classified by a pre-trained classifier model in a feature space constructed by a second feature set determined according to any one of the above-mentioned multimodal feature selection methods.

[0017] According to one embodiment of the present invention, the classifier model is a support vector machine model.

[0018] According to one embodiment of the present invention, the support vector machine model is pre-trained based on a linear combination of kernel matrices corresponding to pairwise image data in each modality image, and the pairwise image data have the same position in each modality image.

[0019] According to another aspect of the present invention, there is provided a multimodal feature selection device, comprising: a data acquisition module, for respectively acquiring data sets of multiple images corresponding to multiple modalities, the data set of each image comprising multiple image data corresponding to multiple position points of the image; a feature extraction module, for respectively performing feature extraction on the multiple images, and respectively determining multiple first feature sets corresponding to the multiple modalities; and a feature selection module, for screening features in the multiple first feature sets based on a preset multimodal feature selection model, and determining a second feature set; wherein the multimodal feature selection model comprises: a distance sub-model, for minimizing the distance between mapping points of image data of the same category corresponding to multiple modalities in the feature space constructed by the corresponding first feature set after being projected onto the feature space constructed by the second feature set.

[0020] According to another aspect of the present invention, there is provided an image data classification device, comprising: a data acquisition module for acquiring image data to be classified; and a data classification module for classifying the image data to be classified by a pre-trained classifier model in a feature space constructed according to a second feature set determined by the above-mentioned multimodal feature selection device.

[0021] According to another aspect of the present invention, a computer device is provided, comprising: a memory, a processor, and executable instructions stored in the memory and executable in the processor, wherein the processor implements any of the above methods when executing the executable instructions.

[0022] According to another aspect of the present invention, a computer-readable storage medium is provided, on which computer-executable instructions are stored, and when the executable instructions are executed by a processor, any of the above methods is implemented.

[0023] According to the multimodal feature selection method provided by the present invention, by embedding a distance sub-model into the multimodal feature selection model to fully utilize the complementary information between different modal image data, the model can be effectively induced to select features with high discriminability.

[0024] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The accompanying drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present invention, and together with the specification are used to explain the principles of the present invention. Obviously, the accompanying drawings described below are only some embodiments of the present invention, and for those of ordinary skill in the art, other accompanying drawings can be obtained based on these accompanying drawings without creative work.

[0026] Figure 1is a flowchart of a multimodal feature selection method according to an exemplary embodiment.

[0027] Figure 2 is a flowchart showing a method for classifying image data according to an exemplary embodiment.

[0028] Figure 3 It is a block diagram of a multimodal feature selection device according to an exemplary embodiment.

[0029] Figure 4 is a block diagram of an image data classification device according to an exemplary embodiment.

[0030] Figure 5 It is a schematic diagram showing the structure of a computer device according to an exemplary embodiment. DETAILED DESCRIPTION

[0031] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that the present invention will be more comprehensive and complete and fully convey the concepts of the example embodiments to those skilled in the art. The accompanying drawings are only schematic illustrations of the present invention and are not necessarily drawn to scale. The same reference numerals in the figures represent the same or similar parts, and thus their repeated description will be omitted.

[0032] In addition, the described features, structures or characteristics may be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present invention. However, those skilled in the art will appreciate that the technical solution of the present invention may be practiced while omitting one or more of the specific details, or other methods, devices, steps, etc. may be adopted. In other cases, known structures, methods, devices, implementations or operations are not shown or described in detail to avoid obscuring various aspects of the present invention.

[0033] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the present invention, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0034] In order to solve the overfitting problem in multimodal image analysis tasks, the present invention proposes a feature learning framework capable of performing feature selection on multimodal data, wherein the multimodal feature selection model includes: a first sub-model and a second sub-model.

[0035] The first sub-model contains the empirical risk function, which can be expressed as the minimum The squared loss term in norm form:

[0036]

[0037] in, is the image data matrix of the mth modality among the M modalities, They represent the mapping points of N m-modal image data in the feature space constructed by the m-modal feature set, d m is the feature dimension of the m-modal image data. Represents the weight vector corresponding to the m mode. is the response vector of the N image data corresponding to the analysis task. Taking the image data classification task as an example, y1~y N They are The category label of the corresponding image data.

[0038] It should be noted that, in the embodiment of the present invention, the empirical risk function of the first sub-model is not limited to the form of the above formula (1).

[0039] The second sub-model contains the structural risk function, which can be expressed as the minimum Regularization term in norm form:

[0040]

[0041] According to the definition of vector norm: ‖w m ‖1 is the weight vector w m The sum of the absolute values ​​of each element in is only when the absolute value of each element is the minimum, that is, when w m When as many elements as possible are 0, ‖w m ‖1 to obtain the minimum value. For the target w determined according to the above formula (2) m , where the features corresponding to non-zero elements have high discriminability and will be selected to be retained, while the features corresponding to zero elements will be discarded.

[0042] λ is the first regularization parameter, which is used to balance the relative weights of the first sub-model and the second sub-model in the multimodal feature selection model. When the λ value is set to a larger value, fewer features are selected and retained, and vice versa. For the extreme case of λ = 0, no feature selection is performed.

[0043] It should be noted that, in the embodiment of the present invention, the structural risk function of the second sub-model is not limited to the form of the above formula (2).

[0044] As mentioned above, the feature learning framework proposed in the present invention is capable of performing feature selection on multimodal data. However, although the above-mentioned multimodal feature selection model takes into account the correlation of the same modality data when performing feature selection, it ignores the possible intrinsic connection between different modality data, that is, the complementary information between different modality data is not utilized during feature selection.

[0045] Therefore, the present invention further optimizes the feature learning framework, and proposes a new multimodal feature selection method based on the improved multimodal feature selection model. Figure 1 is a flowchart of a multimodal feature selection method according to an exemplary embodiment.

[0046] refer to Figure 1 , the multimodal feature selection method 10 comprises:

[0047] In step S102, data sets of multiple images corresponding to multiple modalities are obtained respectively. For example, data sets of MRI, PET and other modality images can be obtained respectively.

[0048] The data set of each image includes multiple image data corresponding to multiple position points of the image.

[0049] In step S104, feature extraction is performed on the multiple images respectively, and multiple first feature sets corresponding to the multiple modalities are determined respectively.

[0050] In some embodiments, extracting features from multiple images separately may include: extracting features from images of each modality separately according to a predefined template.

[0051] In step S106, based on a preset multimodal feature selection model, features in the plurality of first feature sets are screened to determine a second feature set.

[0052] Among them, the multimodal feature selection model includes: a distance sub-model, which is used to minimize the distance between the mapping points of image data of the same category corresponding to multiple modalities in the feature space constructed by the corresponding first feature set after being projected onto the feature space constructed by the second feature set.

[0053] The distance sub-model can be embedded in the multimodal feature selection model before the improvement proposed in the present invention, so as to cooperate with the first sub-model and the second sub-model to complete the joint feature selection, which will be explained in detail below. It should be noted that the distance sub-model can also be embedded in any other possible form of multimodal feature selection model, and the present invention does not limit this.

[0054] According to the multimodal feature selection method provided by the embodiment of the present invention, by embedding a distance sub-model into the multimodal feature selection model to fully utilize the complementary information between different modal image data, the model can be effectively induced to select features with high discriminability.

[0055] It should be clearly understood that the present invention describes how to form and use specific examples, but the principles of the present invention are not limited to any details of these examples. On the contrary, based on the teachings of the content disclosed in the present invention, these principles can be applied to many other embodiments.

[0056] In some embodiments, the distance sub-model may include a regularization function for label alignment.

[0057] Furthermore, the regularization function of label alignment can be, for example, as shown in the following formula (3):

[0058]

[0059] Among them, μ is the second regularization parameter, p and q represent the modes, and w p 、w q Respectively represent the weight vectors corresponding to the p and q modes, W = [w 1 , w 2 ,…,w M ] is the weight matrix corresponding to M modes, and Respectively represent the mapping points of the image data of the p and q modes in the feature space constructed by the corresponding first feature set, i and j represent the numbers of the N position points of the image data in the corresponding image,

[0060]

[0061] Among them, y i ,y j They are and The category label of the corresponding image data.

[0062] From a mathematical point of view, if the image data categories corresponding to the two mapping points are the same (for example, both are from brain images of patients with brain disease), they should be close to each other in the feature space after projection, that is, the distance is very small; conversely, if the image data categories corresponding to the two mapping points are different (for example, one is from the brain image of a patient with brain disease and the other is from the brain image of a normal person), they should be far away from each other in the feature space after projection, that is, the distance is large.

[0063] Norm The mapping point and The distance in the projected feature space. When the two modes p and q are the same, the above formula (3) actually maintains the geometric category structure information of the same modality data; when the two modes p and q are different, the above formula (3) can use the complementary information between different modal data to guide the multimodal feature selection model to perform feature selection. Therefore, the above formula (3) not only retains the label correlation within each modality data, but also aligns the data of each modality through the label information, thereby mining the label correlation between different modalities and the same category data.

[0064] Based on the above, the present invention further proposes a new multimodal feature selection model based on label alignment, based on the multimodal feature selection model proposed above and the distance sub-model including the above formula (3), and its objective function is as follows (4):

[0065]

[0066] in, and The empirical risk function in the first sub-model and the structural risk function in the second sub-model are adaptively adjusted from being oriented to multiple modes in order to cooperate with the use of the distance sub-model.

[0067] Specifically, ‖W‖ 2,1 is the weight matrix W norm. According to the definition of matrix norm: ‖W‖ 2,1 For each row of W The sum of the norms is only possible if the When the norm takes the minimum value, that is, when as many elements as possible in each row of W are 0, ‖W‖ 2,1 Only then can the minimum value be obtained, and finally the row sparse effect is produced. Therefore, λ‖W‖ 2,1 It can be called the group sparsity regularization term. During feature selection, the features corresponding to the non-zero rows in W have high discriminability, so they will be selected and retained, while the features corresponding to the zero rows will be discarded. For the extreme case λ=0, then ‖W‖ 2,1 No sparse effect is produced, that is, no feature selection is performed.

[0068] It should be noted that, since the present invention does not limit the form of each function in the first sub-model, the second sub-model and the distance sub-model, the improved multimodal feature selection model and its objective function after the present invention are not limited to those described in the above embodiments.

[0069] In some embodiments, the objective function of the improved multimodal feature selection model can be optimized based on an Accelerated Proximal Gradient (APG) algorithm.

[0070] Continuing with the above formula (4) as an example, the objective function can be divided into a smooth term and a non-smooth term (min omitted), where the smooth term is the following formula (5):

[0071]

[0072] The non-smooth term is as follows (6):

[0073] g(W)=λ‖W‖ 2,1 (6)

[0074] The following formula (7) is constructed to approximate the composite function f(W)+g(W):

[0075]

[0076] in,‖·‖ F represents the Frobenius norm of the matrix, It means that the kth iteration of f(W) is in W k The gradient at l k is the step length. The update step of the APG algorithm is defined as follows:

[0077]

[0078] Among them, l k can be determined by a linear search method, and

[0079] The key to the APG algorithm is how to effectively solve the update problem. Therefore, the above formula (8) can be transformed into d sub-problems as shown in the following formula (9) for solution:

[0080]

[0081] Among them, w i，: and u i，: denote the i-th row of matrices W and U respectively. The d subproblems have the following easy-to-find analytical solutions:

[0082]

[0083] In addition, in some embodiments, the search point Q k Substituting W in the above formula (7) k To speed up the convergence of the APG algorithm. k For example, the following formula (11) can be defined:

[0084] Q k =W k +η k (W k -W k-1 ) (11)

[0085] in, and

[0086] The features selected by the multimodal feature selection method provided in the above embodiments can be applied to various multimodal image data analysis tasks, such as classification, regression, etc. The present invention also does not impose any limitation on the usage scenario of the second feature set.

[0087] Taking the classification task as an example, an application method of the features selected by the multimodal feature selection method provided according to an embodiment of the present invention is specifically described below.

[0088] Figure 2 FIG. 1 is a flowchart of a method for classifying image data according to an exemplary embodiment. Figure 2 The image data classification method shown can be applied, for example, to classify data from brain images of MRI, PET and other modalities, that is, to determine whether the data is from a patient sample or a normal person sample.

[0089] refer to Figure 2 , the image data classification method 20 comprises:

[0090] In step S202, image data to be classified is obtained, such as data from brain images of MRI, PET and other modalities.

[0091] In step S204, in the feature space constructed by the second feature set determined according to the above method 10, the image data to be classified is classified by using a pre-trained classifier model.

[0092] Since the multimodal feature selection method optimized by the embodiment of the present invention can effectively select features with high discriminability, the image data classification method provided by the embodiment of the present invention can significantly improve the accuracy of multimodal image data classification.

[0093] In some embodiments, the classifier model may be a support vector machine model, such as a multi-core support vector machine model established using a traditional SVM (Support Vector Machine) toolbox LIBSVM.

[0094] The support vector machine model can be pre-trained based on the linear combination of kernel matrices corresponding to the image data of each modality, where the positions of the image data of each modality are the same. Take the image data of two image positions n1 and n2 as an example:

[0095] First, calculate the kernel matrix corresponding to the two image data in each modality image Then, the linear combination of the kernel matrices corresponding to each mode is calculated Then, a support vector machine is used to perform supervised training. Finally, the image data of unknown categories can be classified based on the classification decision function of the following formula (12):

[0096]

[0097] Where x is the mapping point of the image data to be classified in the feature space constructed by the second feature set. Combination coefficient β m ≥0, and In some embodiments of the present invention, a grid search strategy can be used to map the points as training data. Perform ten-fold cross validation to determine β m .

[0098] Those skilled in the art will appreciate that all or part of the steps for implementing the above embodiments are implemented as a computer program executed by a CPU. When the computer program is executed by the CPU, the above functions defined by the above method provided by the present invention are performed. The program may be stored in a computer-readable storage medium, which may be a read-only memory, a disk, or an optical disk.

[0099] In addition, it should be noted that the above-mentioned figures are only schematic illustrations of the processes included in the method according to an exemplary embodiment of the present invention, and are not intended to be limiting. It is easy to understand that the processes shown in the above-mentioned figures do not indicate or limit the time sequence of these processes. In addition, it is also easy to understand that these processes can be performed synchronously or asynchronously, for example, in multiple modules.

[0100] The following are embodiments of the device of the present invention, which can be used to implement the method embodiments of the present invention. For details not disclosed in the device embodiments of the present invention, please refer to the method embodiments of the present invention.

[0101] Figure 3 It is a block diagram of a multimodal feature selection device according to an exemplary embodiment.

[0102] refer to Figure 3 The multimodal feature selection device 30 includes: a data acquisition module 302 , a feature extraction module 304 and a feature selection module 306 .

[0103] The data acquisition module 302 is used to respectively acquire data sets of multiple images corresponding to multiple modalities.

[0104] The data set of each image includes multiple image data corresponding to multiple position points of the image.

[0105] The feature extraction module 304 is used to perform feature extraction on the multiple images respectively, and determine multiple first feature sets corresponding to the multiple modalities respectively.

[0106] The feature selection module 306 is used to screen the features in the plurality of first feature sets based on a preset multimodal feature selection model to determine a second feature set.

[0107] Among them, the multimodal feature selection model includes: a distance sub-model, which is used to minimize the distance between the mapping points of image data of the same category corresponding to multiple modalities in the feature space constructed by the corresponding first feature set after being projected onto the feature space constructed by the second feature set.

[0108] According to the multimodal feature selection device provided in the embodiment of the present invention, by embedding a distance sub-model into the multimodal feature selection model, the complementary information between different modal image data is fully utilized, and the model can be effectively induced to select features with high discriminability.

[0109] Figure 4 is a block diagram of an image data classification device according to an exemplary embodiment.

[0110] refer to Figure 4 The image data classification device 40 includes: a data acquisition module 402 and a data classification module 404 .

[0111] The data acquisition module 402 is used to acquire image data to be classified.

[0112] The data classification module 404 is used to classify the image data to be classified by using a pre-trained classifier model in the feature space constructed according to the second feature set determined by the multimodal feature selection device 30 .

[0113] The image data classification device provided according to the embodiment of the present invention can significantly improve the accuracy of multimodal image data classification.

[0114] It should be noted that the block diagrams shown in the above figures are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software form, or in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0115] It will be appreciated by those skilled in the art that various aspects of the present invention may be implemented as a system, method or program product. Therefore, various aspects of the present invention may be specifically implemented in the following forms, namely: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software, which may be collectively referred to herein as a "circuit", "module" or "system".

[0116] Figure 5 is a schematic diagram of a computer device according to an exemplary embodiment. It should be noted that: Figure 5 The computer device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0117] like Figure 5 As shown, the computer device 800 includes a central processing unit (CPU) 801, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 802 or a program loaded from a storage part 808 into a random access memory (RAM) 803. In the RAM 803, various programs and data required for the operation of the device 800 are also stored. The CPU 801, the ROM 802, and the RAM 803 are connected to each other via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0118] The following components are connected to the I / O interface 805: an input section 806 including a keyboard, a mouse, etc.; an output section 807 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 808 including a hard disk, etc.; and a communication section 809 including a network interface card such as a LAN card, a modem, etc. The communication section 809 performs communication processing via a network such as the Internet. A drive 810 is also connected to the I / O interface 805 as needed. A removable medium 811, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 810 as needed, so that a computer program read therefrom is installed into the storage section 808 as needed.

[0119] In particular, according to an embodiment of the present invention, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present invention includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication section 809, and / or installed from a removable medium 811. When the computer program is executed by a central processing unit (CPU) 801, the above-mentioned functions defined in the device of the present invention are executed.

[0120] It should be noted that the computer-readable medium shown in the present invention may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present invention, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0121] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0122] The units involved in the embodiments of the present invention may be implemented in software or hardware. The units described may also be provided in a processor. For example, they may be described as follows: a processor includes a sending unit, an acquiring unit, a determining unit, and a first processing unit. The names of these units do not, in some cases, limit the units themselves. For example, the sending unit may also be described as a "unit that sends a request for obtaining an image to a connected server."

[0123] As another aspect, the present invention further provides a computer-readable medium, which may be included in the device described in the above embodiment; or may exist independently without being assembled into the device. The above computer-readable medium carries one or more programs, and when the above one or more programs are executed by a device, the device includes:

[0124] Respectively obtain data sets of multiple images corresponding to multiple modalities; respectively perform feature extraction on the multiple images, and respectively determine multiple first feature sets corresponding to the multiple modalities; based on a preset multimodal feature selection model, screen the features in the multiple first feature sets to determine the second feature set; wherein the multimodal feature selection model includes: a distance sub-model, which is used to minimize the distance between the mapping points of the image data corresponding to multiple modalities and of the same category in the feature space constructed by the corresponding first feature set and projected onto the feature space constructed by the second feature set. And,

[0125] Obtain image data to be classified; and classify the image data to be classified using a pre-trained classifier model in a feature space constructed using the second feature set determined by the above method.

[0126] The exemplary embodiments of the present invention are specifically shown and described above. It should be understood that the present invention is not limited to the detailed structure, configuration or implementation method described herein; on the contrary, the present invention is intended to cover various modifications and equivalent configurations included in the spirit and scope of the appended claims.

Claims

1. A multimodal feature selection method, characterized in that: include: Respectively acquiring data sets of multiple images corresponding to multiple modalities, wherein the data set of each image includes multiple image data corresponding to multiple position points of the image; Performing feature extraction on the multiple images respectively, and determining multiple first feature sets corresponding to the multiple modalities respectively; as well as Based on a preset multimodal feature selection model, the features in the plurality of first feature sets are screened to determine a second feature set; The multimodal feature selection model includes: a distance sub-model, which is used to minimize the distance between the mapping points of the image data of the same category corresponding to multiple modalities in the feature space constructed by the corresponding first feature set and projected onto the feature space constructed by the second feature set; The distance sub-model includes a regularization function for label alignment; the label alignment means that the regularization function includes a function for aligning category labels; The category labels represent categories of image data corresponding to mapping points of image data of different modalities in the feature space constructed by the corresponding first feature set.

2. The method according to claim 1, characterized in that The regularization function of the label alignment is as follows: ； in, is the regularization parameter, Indicates the mode, Respectively The weight vector corresponding to the mode, for The weight matrix corresponding to each mode is and Respectively The mapping points of the image data of the modality in the feature space constructed by the corresponding first feature set, Indicates the image data in the corresponding image The number of the location point, ； in, They are and The category label of the corresponding image data.

3. The method according to claim 2, characterized in that The objective function of the multimodal feature selection model is solved based on an accelerated approximate gradient algorithm.

4. The method according to claim 1, characterized in that: Extracting features from the multiple images separately includes: extracting features from the images of each modality separately according to a predefined template.

5. A method for classifying image data, characterized in that: include: Obtain image data to be classified; as well as In the feature space constructed by the second feature set determined by the method according to any one of claims 1 to 4, the image data to be classified is classified by a pre-trained classifier model.

6. The method according to claim 5, characterized in that The classifier model is a support vector machine model.

7. The method according to claim 6, characterized in that The support vector machine model is pre-trained based on a linear combination of kernel matrices corresponding to pairwise image data in each modality image, and the pairwise image data are located at the same position in each modality image.

8. A multimodal feature selection device, characterized in that: include: A data acquisition module, used to respectively acquire data sets of multiple images corresponding to multiple modalities, wherein the data set of each image includes multiple image data corresponding to multiple position points of the image; a feature extraction module, configured to perform feature extraction on the plurality of images respectively, and determine a plurality of first feature sets corresponding to the plurality of modalities respectively; and A feature selection module, configured to screen the features in the plurality of first feature sets based on a preset multimodal feature selection model to determine a second feature set; The multimodal feature selection model includes: a distance sub-model, which is used to minimize the distance between the mapping points of the image data of the same category corresponding to multiple modalities in the feature space constructed by the corresponding first feature set and projected onto the feature space constructed by the second feature set; The distance sub-model includes a regularization function for label alignment; the label alignment means that the regularization function includes a function for aligning category labels; The category labels represent categories of image data corresponding to mapping points of image data of different modalities in the feature space constructed by the corresponding first feature set.

9. An image data classification device, characterized in that: include: A data acquisition module, used for acquiring image data to be classified; as well as A data classification module is used to classify the image data to be classified by using a pre-trained classifier model in a feature space constructed according to the second feature set determined by the device according to claim 8.

10. A computer device comprising: A memory, a processor, and executable instructions stored in the memory and executable in the processor, wherein the processor implements the method according to any one of claims 1 to 7 when executing the executable instructions.

11. A computer-readable storage medium having computer-executable instructions stored thereon, characterized in that: When the executable instructions are executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images