Target classification method and device based on multi-modal learning, equipment and medium

Abstract

The present application relates to a target classification method, device, equipment and medium based on multi-modal learning, the method comprising: inputting the to-be-processed data into a pre-trained classification model to obtain a classification result corresponding to the to-be-processed data; the classification model comprises an input layer for receiving the to-be-processed data, a depth separable convolution layer for performing channel-by-channel convolution on each kind of data to obtain a first feature map corresponding to each kind of data, performing point-by-point convolution on the first feature map corresponding to each kind of data to obtain a second feature map corresponding to each kind of data, and obtaining a third feature map corresponding to each kind of data based on the first feature map and the second feature map corresponding to each kind of data; a multi-scale feature fusion layer is used for fusion processing of the third feature map corresponding to all data to obtain a fusion feature map; and an output layer is used for obtaining a classification result according to the fusion feature map. Through the method of the present application, the amount of calculation can be reduced while maintaining the accuracy, so that the classification result obtained based on the classification model is more accurate.

Description

Technical Field

[0001] This invention relates to the field of machine learning technology, and more specifically, to a target classification method, apparatus, device, and medium based on multimodal learning. Background Technology

[0002] In existing technologies, classification models typically employ convolutional layers for feature extraction, followed by classification based on the extracted features. However, since this method uses convolutional kernels within these layers for feature extraction, it involves a large amount of data and is unsuitable for scenarios requiring fast data processing. Summary of the Invention

[0003] The technical problem to be solved by the present invention is to provide a target classification method, apparatus, device and medium based on multimodal learning, which aims to solve at least one of the above-mentioned technical problems.

[0004] In a first aspect, the technical solution of the present invention to solve the above-mentioned technical problems is as follows: a target classification method based on multimodal learning, the method comprising:

[0005] Acquire data to be processed for a target, the data to be processed including images, text and speech to be processed, the target being the type of vehicle;

[0006] The data to be processed is input into a pre-trained classification model to obtain the classification result corresponding to the data to be processed; wherein, the classification model includes an input layer, a depthwise separable convolutional layer, a multi-scale feature fusion layer and an output layer;

[0007] The input layer is used to receive the data to be processed. The depthwise separable convolutional layer is used to perform channel-wise convolution on each type of data in the data to be processed to obtain a first feature map corresponding to each type of data, and to perform point-wise convolution on the first feature map corresponding to each type of data to obtain a second feature map corresponding to each type of data. Based on the first feature map and the second feature map corresponding to each type of data, a third feature map corresponding to each type of data is obtained.

[0008] The multi-scale feature fusion layer is used to fuse the third feature maps corresponding to all types of data in the data to be processed to obtain a fused feature map; the output layer is used to obtain the classification result corresponding to the data to be processed based on the fused feature map.

[0009] The beneficial effects of this invention are: by using the depthwise separable convolutional layers in the pre-trained classification model to extract features from the data to be processed, the amount of computation can be reduced while maintaining accuracy, thereby making the classification results obtained based on the classification model more accurate.

[0010] Based on the above technical solution, the present invention can be further improved as follows.

[0011] Furthermore, the classification model also includes an attention mechanism layer, which is used to enhance the first feature in the fused feature map and suppress the second feature in the fused feature map to obtain a first output feature map. The first feature is a feature related to the target corresponding to the classification result, and the second feature is a feature unrelated to the target corresponding to the classification result. The first output feature map is used as the input of the output layer.

[0012] Furthermore, the attention mechanism layer includes a first average pooling layer, a first max pooling layer, a first activation function layer, and a first fusion layer. The first average pooling layer is used to perform average pooling on the fused feature map to obtain a fourth feature map. The first max pooling layer is used to perform max pooling on the fused feature map to obtain a fifth feature map. The first activation function layer is used to fuse the fourth feature map and the fifth feature map through a multilayer perceptron to obtain a sixth feature map. The first fusion layer is used to fuse the sixth feature map and the fused feature map to obtain a first output feature map.

[0013] Furthermore, the classification model also includes a global attention layer, which is used to extract global information from the first output feature map and obtain a second output feature map based on the global information and the first output feature map. The second output feature map is used as the input of the output layer, and the global information is information representing the global features of the target.

[0014] Furthermore, the global attention layer includes a second average pooling layer, a second max pooling layer, a second activation function layer, and a second fusion layer. The second average pooling layer is used to perform average pooling on the first output feature map to obtain a seventh feature map. The second max pooling layer is used to perform max pooling on the first output feature map to obtain an eighth feature map. The second activation function layer is used to fuse the seventh feature map and the eighth feature map through a multilayer perceptron to obtain a ninth feature map. The second fusion layer is used to fuse the ninth feature map and the first output feature map to obtain a second output feature map.

[0015] Furthermore, the input layer is also used to convert the received data to be processed into target data in a preset format.

[0016] Furthermore, the multi-scale feature fusion layer includes multiple convolutional layers corresponding to convolutional kernels of different sizes, and the multi-scale feature fusion layer is specifically used for:

[0017] For each type of data, features are extracted from the third feature map corresponding to each type of data through convolutional layers with convolutional kernels of various sizes, resulting in a third output feature map.

[0018] The third output feature maps corresponding to all types of data are concatenated to obtain the fused feature map corresponding to the data to be processed.

[0019] Secondly, to solve the above-mentioned technical problems, the present invention also provides a target classification device based on multimodal learning, the device comprising:

[0020] The data acquisition module is used to acquire data to be processed for a target, including images, text, and speech to be processed, where the target is the type of vehicle.

[0021] A classification module is used to input the data to be processed into a pre-trained classification model to obtain the classification result corresponding to the data to be processed; wherein, the classification model includes an input layer, a depthwise separable convolutional layer, a multi-scale feature fusion layer, and an output layer;

[0022] The input layer is used to receive the data to be processed. The depthwise separable convolutional layer is used to perform channel-wise convolution on each type of data in the data to be processed to obtain a first feature map corresponding to each type of data, and to perform point-wise convolution on the first feature map corresponding to each type of data to obtain a second feature map corresponding to each type of data. Based on the first feature map and the second feature map corresponding to each type of data, a third feature map corresponding to each type of data is obtained.

[0023] The multi-scale feature fusion layer is used to fuse the third feature maps corresponding to all types of data in the data to be processed to obtain a fused feature map; the output layer is used to obtain the classification result corresponding to the data to be processed based on the fused feature map.

[0024] Thirdly, in order to solve the above-mentioned technical problems, the present invention also provides an electronic device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the target classification method based on multimodal learning of the present application.

[0025] Fourthly, in order to solve the above-mentioned technical problems, the present invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the target classification method based on multimodal learning of the present application.

[0026] Additional aspects and advantages of this application will be set forth in part in the description which follows, and will become apparent from the description or may be learned by practice of this application. Attached Figure Description

[0027] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments of the present invention will be briefly introduced below.

[0028] Figure 1 A flowchart illustrating a target classification method based on multimodal learning, provided as an embodiment of the present invention;

[0029] Figure 2 This is a schematic diagram of the structure of a target classification device based on multimodal learning, provided in one embodiment of the present invention;

[0030] Figure 3 This is a schematic diagram of the structure of an electronic device provided in one embodiment of the present invention. Detailed Implementation

[0031] The principles and features of the present invention are described below. The examples given are only for explaining the present invention and are not intended to limit the scope of the present invention.

[0032] The technical solution of the present invention and how the technical solution of the present invention solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of the present invention will now be described with reference to the accompanying drawings.

[0033] The solution provided in this invention can be applied to any application scenario that requires classification of vehicle types. The solution provided in this invention can be executed by any electronic device, such as a user's terminal device, including at least one of the following: smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, smart TV, or smart in-vehicle device.

[0034] This invention provides a possible implementation, such as... Figure 1 The diagram shows a flowchart of a target classification method based on multimodal learning. This method can be executed by any electronic device, such as a terminal device, or jointly executed by a terminal device and a server. For ease of description, the method provided in this embodiment will be described below using a terminal device as the execution subject as an example. Figure 1 The flowchart shown indicates that the method may include the following steps:

[0035] S10, acquire the data to be processed for the target, the data to be processed includes the image to be processed, the text to be processed and the voice to be processed, the target is the type of vehicle;

[0036] S20, the data to be processed is input into a pre-trained classification model to obtain the classification result corresponding to the data to be processed; wherein, the classification model includes an input layer, a depthwise separable convolutional layer, a multi-scale feature fusion layer and an output layer;

[0037] The input layer is used to receive the data to be processed. The depthwise separable convolutional layer is used to perform channel-wise convolution on each type of data in the data to be processed to obtain a first feature map corresponding to each type of data, and to perform point-wise convolution on the first feature map corresponding to each type of data to obtain a second feature map corresponding to each type of data. Based on the first feature map and the second feature map corresponding to each type of data, a third feature map corresponding to each type of data is obtained.

[0038] The multi-scale feature fusion layer is used to fuse the third feature maps corresponding to all types of data in the data to be processed to obtain a fused feature map; the output layer is used to obtain the classification result corresponding to the data to be processed based on the fused feature map.

[0039] The method of the present invention extracts features from the data to be processed by using the depthwise separable convolutional layers in the pre-trained classification model, which can reduce the amount of computation and maintain accuracy, thereby making the classification results obtained based on the classification model more accurate.

[0040] The following specific embodiments further illustrate the solution of the present invention. In these embodiments, the target classification method based on multimodal learning may include the following steps:

[0041] S10, acquire the data to be processed for the target, the data to be processed includes the image to be processed, the text to be processed and the voice to be processed, the target is the type of vehicle;

[0042] In different application fields, the meaning of the target varies. For example, in the medical field, the target could be different types of diseases; in the vehicle field, the target could be different types of vehicles; in the agricultural field, the target could be different types of plants; and in the robotics field, the target could be different types of robot actions. This application does not list all the targets. If the target is a type of vehicle, the specific vehicle types could be trucks, off-road vehicles, dump trucks, tractor-trailers, special-purpose vehicles, buses, and cars.

[0043] S20, the data to be processed is input into a pre-trained classification model to obtain the classification result corresponding to the data to be processed; wherein, the classification model includes an input layer, a depthwise separable convolutional layer, a multi-scale feature fusion layer and an output layer;

[0044] The input layer is used to receive the data to be processed. The depthwise separable convolutional layer is used to perform channel-wise convolution on each type of data in the data to be processed to obtain a first feature map corresponding to each type of data, and to perform point-wise convolution on the first feature map corresponding to each type of data to obtain a second feature map corresponding to each type of data. Based on the first feature map and the second feature map corresponding to each type of data, a third feature map corresponding to each type of data is obtained.

[0045] The multi-scale feature fusion layer is used to fuse the third feature maps corresponding to all types of data in the data to be processed to obtain a fused feature map; the output layer is used to obtain the classification result corresponding to the data to be processed based on the fused feature map.

[0046] In this application, the same processing can be performed on different types of data. Before processing, different types of data can be preprocessed. For example, the speech to be processed can be converted into text, and then the text data and image data can be processed separately based on the classification model to obtain the classification results.

[0047] The classification result represents the type of result the target is, and can be represented based on probability values.

[0048] Optionally, before inputting the data to be processed into the classification model, the data can be cleaned, denoised, and formatted to provide a high-quality data foundation for the subsequent classification model.

[0049] Optionally, when the input layer receives the data to be processed, it can usually first convert the received data to be processed into target data in a preset format, wherein the preset format can be a format that the target model can process.

[0050] As an example, the dimensions of the input image (the image to be processed) can be converted to (H, W, C), where H is the height, W is the width, and C is the number of channels.

[0051] Since the processing method is the same for each type of data in this application, the following will take one type of data as an example to describe the subsequent processing method in detail. This type of data can be referred to as the input image.

[0052] Optionally, the specific implementation process of the above depth-separable convolutional layer can be as shown in the first formula:

[0053] Yds=Depthwi seConv(X)+Po i ntwi seConv(Depthwi seConv(X)) (1)

[0054] Where X represents the input feature map corresponding to the input image, DepthwiseConv(X) represents the channel-wise convolution of the input feature map, i.e. the first feature map, PointwiseConv(DepthwiseConv(X)) represents the point-wise convolution of the first feature map (DepthwiseConv(X)), i.e. the second feature map, and Yds represents the third feature map.

[0055] Optionally, the above-mentioned multi-scale feature fusion layer includes multiple convolutional layers corresponding to convolutional kernels of different sizes. One implementation of the multi-scale feature fusion layer is as follows:

[0056] For each type of data, features are extracted from the third feature map corresponding to each type of data through convolutional layers with convolutional kernels of various sizes, resulting in a third output feature map.

[0057] The third output feature maps corresponding to all types of data are concatenated to obtain the fused feature map corresponding to the data to be processed.

[0058] Optionally, one possible way to concatenate the third output feature maps corresponding to all types of data to obtain the fused feature map corresponding to the data to be processed is to perform average pooling on a first number of third output feature maps to obtain a first number of fourth output feature maps, and to concatenate the first number of fourth output feature maps and the second number of third output feature maps to obtain the fused feature map corresponding to the data to be processed, wherein the sum of the first number and the second number is equal to the number of types of data.

[0059] Optionally, the above-mentioned multi-scale feature fusion layer includes multiple convolutional layers corresponding to convolutional kernels of different sizes. The number of convolutional kernels is the same as the number of data types, that is, there are as many convolutional kernels as there are data types. Another way to implement the multi-scale feature fusion layer is as follows:

[0060] For each type of data, the third feature map corresponding to the data is extracted by the convolutional layer corresponding to the convolutional kernel of that type of data, and the third output feature map is obtained.

[0061] All third output feature maps are concatenated to obtain the fused feature map corresponding to the data to be processed.

[0062] As an example, multiple convolutional kernels of different sizes are divided into 3×3 kernels, 5×5 kernels, and 7×7 kernels. If the data to be processed includes the first type of data, the second type of data, and the third type of data, the specific implementation process of the above multi-scale feature fusion layer can be found in the second formula:

[0063] Ymsf=Concat(Conv3×3(Yds), Conv5×5(Yds), AvgPool(Conv7×7(Yds))) (2)

[0064] Where Ymsf represents the fused feature map corresponding to the data to be processed, Convk×k represents the k×k size convolution, AvgPool(Conv7×7(Yds)) represents average pooling of Conv7×7(Yds), Concat represents feature map concatenation, Yds in Conv3×3(Yds) represents the third feature map corresponding to the first type of data, Yds in Conv5×5(Yds) represents the third feature map corresponding to the second type of data, and Yds in Conv7×7(Yds) represents the third feature map corresponding to the third type of data.

[0065] In this application, multi-scale features of each data type in the data to be processed can be obtained by using convolutional kernels of different sizes in the multi-scale feature fusion layer.

[0066] Optionally, the classification model further includes an attention mechanism layer, which is used to enhance a first feature in the fused feature map and suppress a second feature in the fused feature map to obtain a first output feature map. The first feature is a feature related to the target corresponding to the classification result, and the second feature is a feature unrelated to the target corresponding to the classification result. The first output feature map is used as the input of the output layer.

[0067] Suppression can be understood as preventing too many effective features from being lost from the fused feature map. In other words, in addition to the effective first feature, there are some other effective features in the fused feature map. These features can be included in the second feature. The purpose of suppression is to filter out the effective features in the second feature and use them as the first feature.

[0068] Furthermore, the attention mechanism layer includes a first average pooling layer, a first max pooling layer, a first activation function layer, and a first fusion layer. The first average pooling layer is used to perform average pooling on the fused feature map to obtain a fourth feature map. The first max pooling layer is used to perform max pooling on the fused feature map to obtain a fifth feature map. The first activation function layer is used to fuse the fourth feature map and the fifth feature map through a multilayer perceptron to obtain a sixth feature map. The first fusion layer is used to fuse the sixth feature map and the fused feature map to obtain a first output feature map.

[0069] The purpose of average pooling and max pooling is to enhance the first feature in the fused feature map. Max pooling aims to filter out useless information in the fused feature map and extract the first feature, while average pooling aims to fuse the second feature in the feature map, thus preventing the filtering out of too much effective information.

[0070] Specifically, the implementation process of the attention mechanism layer can be found in the third formula:

[0071]

[0072] Where Yatt represents the first output feature map, and Ymsf represents the fused feature map. σ represents element-wise multiplication, σ represents the first activation function layer (which can be a Sigmoid activation function layer), MLP represents a multilayer perceptron, and AvgPool and MaxPool represent average pooling and max pooling, respectively.

[0073] Optionally, the classification model further includes a global attention layer, which is used to extract global information from the first output feature map and obtain a second output feature map based on the global information and the first output feature map, and use the second output feature map as the input of the output layer, wherein the global information is information representing the global features of the target.

[0074] The global features represent the features in the first output feature map that can represent contextual information. They allow for a holistic representation of the target's characteristics based on global information. The second output feature map is a feature map that incorporates global information.

[0075] Further, the global attention layer includes a second average pooling layer, a second max pooling layer, a second activation function layer, and a second fusion layer. The second average pooling layer is used to perform average pooling on the first output feature map to obtain a seventh feature map. The second max pooling layer is used to perform max pooling on the first output feature map to obtain an eighth feature map. The second activation function layer is used to fuse the seventh feature map and the eighth feature map through a multilayer perceptron to obtain a ninth feature map. The second fusion layer is used to fuse the ninth feature map (which contains global features) and the first output feature map to obtain a second output feature map.

[0076] Specifically, the implementation process of the attention mechanism layer can be found in Formula 4 and Formula 5:

[0077] Zglobal=σ(MLP(AvgPool(Yatt))+MLP(MaxPool(Yatt))) (4)

[0078]

[0079] Here, Zglobal represents the ninth feature map, also known as the global attention map; σ represents the second activation function layer, which can be a Sigma-Aldrich activation function layer; MLP represents a multilayer perceptron; AvgPool and MaxPool represent average pooling and max pooling, respectively; Yatt represents the first output feature map corresponding to the input image; and Ygam represents the second output feature map. This indicates element-wise multiplication.

[0080] Optionally, the above output layer is specifically used for:

[0081] Based on the second output feature map, the classification result corresponding to the data to be processed is obtained.

[0082] One possible way to obtain the classification result of the data to be processed based on the second output feature map can be found in Formula 6:

[0083] Y = W·Ygam + b (6)

[0084] Where Y is the classification result corresponding to the data to be processed, W is the weight matrix of the output layer, Ygam represents the second output feature map, and b represents the bias term of the output layer.

[0085] In this application, a user interface can also be provided to display the classification results.

[0086] Furthermore, based on the classification results, feedback information on the classification results can be received through the user interface, so as to optimize the classification model and the user interface based on the feedback information.

[0087] With the continuous advancement of medical technology and the arrival of the big data era, artificial intelligence (AI) technology is gradually demonstrating its enormous potential in the medical field. Traditional medical consultation systems mainly rely on text input, which suffers from incomplete information expression and limited diagnostic accuracy. In recent years, the development of multimodal learning technology has provided new solutions for medical consultation systems. By combining information from multiple sensory modalities such as vision and hearing, it is possible to more comprehensively understand and respond to user needs, improving diagnostic accuracy and user experience. However, most existing multimodal medical systems focus on general medical consultation, and consultation systems targeting specific diseases are still relatively lacking.

[0088] To address the aforementioned issues, the present invention provides a specialized disease consultation method for users based on a multimodal learning-based target classification method. Specifically, if the target in this application is a different type of disease in the medical field, the present invention can identify the symptoms in the data to be processed. Furthermore, it can also provide users with diagnostic results and treatment suggestions through a user interface.

[0089] This method can make full use of multimodal data (such as voice, images, and text) for deep learning, providing accurate and efficient medical consultation services for patients with specialized diseases.

[0090] Based on and Figure 1 Using the same principle as the method shown, this embodiment of the invention also provides a target classification device 20 based on multimodal learning, such as... Figure 2 As shown, the target classification device 20 based on multimodal learning may include a data acquisition module 210 and a classification module 220, wherein:

[0091] The data acquisition module 210 is used to acquire data to be processed for a target, the data to be processed including images, text and voice to be processed, and the target is the type of vehicle;

[0092] The classification module 220 is used to input the data to be processed into a pre-trained classification model to obtain the classification result corresponding to the data to be processed; wherein, the classification model includes an input layer, a convolutional layer, a multi-scale feature fusion layer and an output layer;

[0093] The input layer is used to receive the data to be processed, and the convolutional layer is used to perform channel-wise convolution on each type of data in the data to be processed to obtain a first feature map corresponding to each type of data, and perform point-wise convolution on the first feature map corresponding to each type of data to obtain a second feature map corresponding to each type of data, and obtain a third feature map corresponding to each type of data based on the first feature map and the second feature map corresponding to each type of data.

[0094] The multi-scale feature fusion layer is used to fuse the third feature maps corresponding to all types of data in the data to be processed to obtain a fused feature map; the output layer is used to obtain the classification result corresponding to the data to be processed based on the fused feature map.

[0095] Optionally, the classification model further includes an attention mechanism layer, which is used to enhance a first feature in the fused feature map and suppress a second feature in the fused feature map to obtain a first output feature map. The first feature is a feature related to the target corresponding to the classification result, and the second feature is a feature unrelated to the target corresponding to the classification result. The first output feature map is used as the input of the output layer.

[0096] Optionally, the attention mechanism layer includes a first average pooling layer, a first max pooling layer, a first activation function layer, and a first fusion layer. The first average pooling layer is used to perform average pooling on the fused feature map to obtain a fourth feature map. The first max pooling layer is used to perform max pooling on the fused feature map to obtain a fifth feature map. The first activation function layer is used to fuse the fourth feature map and the fifth feature map through a multilayer perceptron to obtain a sixth feature map. The first fusion layer is used to fuse the sixth feature map and the fused feature map to obtain a first output feature map.

[0097] Optionally, the classification model further includes a global attention layer, which is used to extract global information from the first output feature map and obtain a second output feature map based on the global information and the first output feature map, and use the second output feature map as the input of the output layer, wherein the global information is information representing the global features of the target.

[0098] Optionally, the global attention layer includes a second average pooling layer, a second max pooling layer, a second activation function layer, and a second fusion layer. The second average pooling layer is used to perform average pooling on the first output feature map to obtain a seventh feature map. The second max pooling layer is used to perform max pooling on the first output feature map to obtain an eighth feature map. The second activation function layer is used to fuse the seventh feature map and the eighth feature map through a multilayer perceptron to obtain a ninth feature map. The second fusion layer is used to fuse the ninth feature map and the first output feature map to obtain a second output feature map.

[0099] Optionally, the input layer is further configured to convert the received data to be processed into target data in a preset format.

[0100] Optionally, the multi-scale feature fusion layer includes multiple convolutional layers corresponding to convolutional kernels of different sizes, and the multi-scale feature fusion layer is specifically used for:

[0101] For each type of data, features are extracted from the third feature map corresponding to each type of data through convolutional layers with convolutional kernels of various sizes, resulting in a third output feature map.

[0102] The third output feature maps corresponding to all types of data are concatenated to obtain the fused feature map corresponding to the data to be processed.

[0103] The target classification device based on multimodal learning in this embodiment of the invention can execute the target classification method based on multimodal learning provided in this embodiment of the invention. The implementation principle is similar. The actions performed by each module and unit in the target classification device based on multimodal learning in each embodiment of the invention correspond to the steps in the target classification method based on multimodal learning in each embodiment of the invention. For detailed functional descriptions of each module of the target classification device based on multimodal learning, please refer to the descriptions in the corresponding target classification methods based on multimodal learning shown above, which will not be repeated here.

[0104] The aforementioned target classification device based on multimodal learning can be a computer program (including program code) running on a computer device, for example, the target classification device based on multimodal learning is an application software; the device can be used to execute the corresponding steps in the method provided in the embodiments of the present invention.

[0105] In some embodiments, the target classification device based on multimodal learning provided in this invention can be implemented using a combination of hardware and software. As an example, the target classification device based on multimodal learning provided in this invention can be a processor in the form of a hardware decoding processor, which is programmed to execute the target classification method based on multimodal learning provided in this invention. For example, the processor in the form of a hardware decoding processor can be one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), or other electronic components.

[0106] In other embodiments, the target classification device based on multimodal learning provided in this invention can be implemented in software. Figure 2 A target classification device based on multimodal learning, stored in a memory, is shown. It can be software in the form of programs and plug-ins, and includes a series of modules, including a data acquisition module 210 and a classification module 220, for implementing the target classification method based on multimodal learning provided in the embodiments of the present invention.

[0107] The modules described in the embodiments of the present invention can be implemented in software or hardware. The names of the modules are not, in some cases, limiting the scope of the module itself.

[0108] Based on the same principles as the methods shown in the embodiments of the present invention, the embodiments of the present invention also provide an electronic device, which may include, but is not limited to: a processor and a memory; the memory for storing computer programs; and the processor for executing the methods shown in any embodiment of the present invention by invoking the computer programs.

[0109] In one alternative embodiment, an electronic device is provided, such as Figure 3 As shown, Figure 3 The illustrated electronic device 4000 includes a processor 4001 and a memory 4003. The processor 4001 and the memory 4003 are connected, for example, via a bus 4002. Optionally, the electronic device 4000 may further include a transceiver 4004, which can be used for data interaction between the electronic device and other electronic devices, such as sending and / or receiving data. It should be noted that in practical applications, the transceiver 4004 is not limited to one type, and the structure of the electronic device 4000 does not constitute a limitation on the embodiments of the present invention.

[0110] Processor 4001 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It can implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this invention. Processor 4001 may also be a combination that implements computational functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc.

[0111] Bus 4002 may include a pathway for transmitting information between the aforementioned components. Bus 4002 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. Bus 4002 can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 3 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0112] The memory 4003 may be ROM (Read Only Memory) or other types of static storage devices capable of storing static information and instructions, RAM (Random Access Memory) or other types of dynamic storage devices capable of storing information and instructions, or EEPROM (Electrically Erasable Programmable Read Only Memory), CD-ROM (Compact Disc Read Only Memory) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto.

[0113] The memory 4003 stores the application code (computer program) for executing the present invention, and its execution is controlled by the processor 4001. The processor 4001 executes the application code stored in the memory 4003 to implement the content shown in the foregoing method embodiments.

[0114] Among these, electronic devices can also be terminal devices. Figure 3 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of use of the embodiments of the present invention.

[0115] This invention provides a computer-readable storage medium storing a computer program that, when run on a computer, enables the computer to execute the corresponding content in the aforementioned method embodiments.

[0116] According to another aspect of the present invention, a computer program product or computer program is also provided, comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the methods provided in the various embodiments described above.

[0117] Computer program code for performing the operations of this invention can be written in one or more programming languages ​​or a combination thereof. These programming languages ​​include object-oriented programming languages—such as Java, Smalltalk, and C++—and conventional procedural programming languages—such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0118] It should be understood that the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of methods and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0119] The computer-readable storage medium provided in this invention can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this invention, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0120] The aforementioned computer-readable storage medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to perform the method shown in the above embodiments.

[0121] The above description is merely a preferred embodiment of the present invention and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of disclosure in this invention is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features with similar functions disclosed in this invention.

Claims

1. A target classification method based on multimodal learning, characterized in that, Includes the following steps: Acquire data to be processed for a target, the data to be processed including images, text and speech to be processed, the target being the type of vehicle; The data to be processed is input into a pre-trained classification model to obtain the classification result corresponding to the data to be processed; wherein, the classification model includes an input layer, a depthwise separable convolutional layer, a multi-scale feature fusion layer and an output layer; The input layer is used to receive the data to be processed. The depthwise separable convolutional layer is used to perform channel-wise convolution on each type of data in the data to be processed to obtain a first feature map corresponding to each type of data, and to perform point-wise convolution on the first feature map corresponding to each type of data to obtain a second feature map corresponding to each type of data. Based on the first feature map and the second feature map corresponding to each type of data, a third feature map corresponding to each type of data is obtained. The multi-scale feature fusion layer is used to fuse the third feature maps corresponding to all types of data in the data to be processed to obtain a fused feature map; the output layer is used to obtain the classification result corresponding to the data to be processed based on the fused feature map. The specific implementation process of the depth-separable convolutional layer is shown in the first formula: Yds=DepthwiseConv(X)+PointwiseConv(DepthwiseConv(X)) Where X represents the input feature map corresponding to the input image, DepthwiseConv(X) represents the first feature map obtained by performing channel-wise convolution on the input feature map, PointwiseConv(DepthwiseConv(X)) represents the second feature map obtained by performing point-wise convolution on the first feature map DepthwiseConv(X), and Yds represents the third feature map. The classification model further includes an attention mechanism layer, which is used to enhance a first feature in the fused feature map and suppress a second feature in the fused feature map to obtain a first output feature map. The first feature is a feature related to the target corresponding to the classification result, and the second feature is a feature unrelated to the target corresponding to the classification result. The first output feature map is used as the input of the output layer. The multi-scale feature fusion layer includes multiple convolutional layers corresponding to convolutional kernels of different sizes. Specifically, the multi-scale feature fusion layer is used for: For each type of data, features are extracted from the third feature map corresponding to each type of data through convolutional layers with convolutional kernels of various sizes, resulting in a third output feature map. The third output feature maps corresponding to all types of data are concatenated to obtain the fused feature map corresponding to the data to be processed. The process of concatenating the third output feature maps corresponding to all types of data to obtain the fused feature map corresponding to the data to be processed includes: performing average pooling on a first number of third output feature maps to obtain a first number of fourth output feature maps; and concatenating the first number of fourth output feature maps and the second number of third output feature maps to obtain the fused feature map corresponding to the data to be processed, wherein the sum of the first number and the second number is equal to the number of data types.

2. The method according to claim 1, characterized in that, The attention mechanism layer includes a first average pooling layer, a first max pooling layer, a first activation function layer, and a first fusion layer. The first average pooling layer is used to perform average pooling on the fused feature map to obtain a fourth feature map. The first max pooling layer is used to perform max pooling on the fused feature map to obtain a fifth feature map. The first activation function layer is used to fuse the fourth feature map and the fifth feature map through a multilayer perceptron to obtain a sixth feature map. The first fusion layer is used to fuse the sixth feature map and the fused feature map to obtain a first output feature map.

3. The method according to claim 1, characterized in that, The classification model further includes a global attention layer, which is used to extract global information from the first output feature map and obtain a second output feature map based on the global information and the first output feature map. The second output feature map is used as the input of the output layer. The global information is information that characterizes the global features of the target.

4. The method according to claim 3, characterized in that, The global attention layer includes a second average pooling layer, a second max pooling layer, a second activation function layer, and a second fusion layer. The second average pooling layer is used to perform average pooling on the first output feature map to obtain a seventh feature map. The second max pooling layer is used to perform max pooling on the first output feature map to obtain an eighth feature map. The second activation function layer is used to fuse the seventh feature map and the eighth feature map through a multilayer perceptron to obtain a ninth feature map. The second fusion layer is used to fuse the ninth feature map and the first output feature map to obtain a second output feature map.

5. The method according to any one of claims 1 to 4, characterized in that, The input layer is also used to convert the received data to be processed into target data in a preset format.

6. A target classification device based on multimodal learning, characterized in that, The apparatus employing the multimodal learning-based target classification method as described in claim 1 comprises: The data acquisition module is used to acquire data to be processed for a target, including images, text, and speech to be processed, where the target is the type of vehicle. A classification module is used to input the data to be processed into a pre-trained classification model to obtain the classification result corresponding to the data to be processed; wherein, the classification model includes an input layer, a convolutional layer, a multi-scale feature fusion layer, and an output layer; The input layer is used to receive the data to be processed, and the convolutional layer is used to perform channel-wise convolution on each type of data in the data to be processed to obtain a first feature map corresponding to each type of data, and perform point-wise convolution on the first feature map corresponding to each type of data to obtain a second feature map corresponding to each type of data, and obtain a third feature map corresponding to each type of data based on the first feature map and the second feature map corresponding to each type of data. The multi-scale feature fusion layer is used to fuse the third feature maps corresponding to all types of data in the data to be processed to obtain a fused feature map; the output layer is used to obtain the classification result corresponding to the data to be processed based on the fused feature map.

7. An electronic device, characterized in that, It includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the method of any one of claims 1-5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method of any one of claims 1-5.

Images