Cross-modal task identification method and device, equipment and medium

Abstract

The invention discloses a cross-modal task identification method and device, equipment and a medium. The method comprises the following steps: carrying out feature conversion and splicing on an acquired image and text content through a preset encoder to generate a fusion feature vector of the image and the text; updating the multi-modal clustering center through a preset dynamic updating method according to the fusion feature vector to obtain a new clustering center; performing feature clustering on the fused feature vector and the new clustering center through a preset clustering algorithm to obtain a clustering center vector; and generating a target task identification result from the clustering center vector through a preset neural network sub-model and a preset gating network. By implementing the method provided by the invention, the problem that information of various modes cannot be effectively fused in the prior art can be solved, so that efficient and accurate medical diagnosis is realized.

Description

Technical Field

[0001] The present invention relates to the technical field of task identification and can be applied to the field of medical health, and in particular to a cross-modal task identification method, device, equipment and medium. Background Art

[0002] With technological advancements, large models have gradually become part of our lives, especially in the medical field. Multimodal large models (such as vision-language models) are increasingly being used in key areas such as medical image analysis and diagnostic report generation. However, current medical multimodal data processing often relies on explicit task labels or predefined classifiers, and faces many challenges. For example, due to the complexity of medical data, the interweaving of pathological images and clinical text, the difficulty in obtaining explicit labels, the high cost and error-proneness of expert annotation, and the dynamic evolution of disease spectrums, existing static task partitioning methods are difficult to adapt to changes in data distribution, limiting task recognition accuracy. Medical decision-making requires a deep integration of visual and language information, but existing technologies cannot effectively integrate information from multiple modalities, resulting in poor task recognition results, which in turn affects diagnostic accuracy. Summary of the Invention

[0003] The embodiments of the present invention provide a cross-modal task identification method, apparatus, device, and medium, aiming to solve the problem that the existing technology cannot effectively fuse information of multiple modalities.

[0004] In the first aspect, an embodiment of the present invention provides a cross-modal task recognition method, which is applied to a multimodal large model, including: performing feature conversion and splicing on the acquired collected image and text content through a preset encoder to generate a fusion feature vector of the image and text; updating the multimodal clustering center according to the fusion feature vector through a preset dynamic update method to obtain a new clustering center; performing feature clustering on the fusion feature vector and the new clustering center through a preset clustering algorithm to obtain a cluster center vector; and generating a target task recognition result by passing the cluster center vector through a preset neural network sub-model and a preset gating network.

[0005] In the second aspect, an embodiment of the present invention also provides a cross-modal task identification device, which is applied to a multimodal large model, including: a splicing unit, used to perform feature conversion and splicing on the acquired collected image and text content through a preset encoder to generate a fusion feature vector of the image and text; an updating unit, used to update the multimodal clustering center according to the fusion feature vector through a preset dynamic update method to obtain a new clustering center; a clustering unit, used to perform feature clustering on the fusion feature vector and the new clustering center through a preset clustering algorithm to obtain a cluster center vector; a generation unit, used to generate a target task recognition result by passing the cluster center vector through a preset neural network sub-model and a preset gating network.

[0006] In a third aspect, an embodiment of the present invention further provides a computer device, which includes a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the above method when executing the computer program.

[0007] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium, wherein the storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a processor, the above method can be implemented.

[0008] The embodiment of the present invention provides a cross-modal task recognition method, device, equipment and medium. The method is applied to a large multimodal model, including: performing feature conversion and splicing on the acquired image and text content through a preset encoder to generate a fusion feature vector of the image and text; updating the multimodal cluster center through a preset dynamic update method according to the fusion feature vector to obtain a new cluster center; performing feature clustering on the fusion feature vector and the new cluster center through a preset clustering algorithm to obtain a cluster center vector; and generating a target task recognition result by passing the cluster center vector through a preset neural network sub-model and a preset gating network. The embodiment of the present invention uses a preset encoder to convert and splice the features of the acquired image and text content, so as to fully utilize the information of the multimodal data, provide a rich feature representation for subsequent task recognition, and enable the cluster center to be dynamically updated according to the current fusion feature vector, so as to help the large multimodal model adapt to changes in data distribution in real time and improve the robustness and accuracy of task recognition. Each of the fused feature vectors and the new cluster center is clustered using a preset clustering algorithm to obtain a cluster center vector that accurately reflects the task category of the sample in the multimodal space. This is then passed through a preset neural network sub-model and a preset gating network to generate a target task recognition result, thereby generating an accurate task recognition result. By splicing and clustering multiple types of information to generate the final recognition result, the system fully integrates information from visual and language modalities, dynamically adapts to changes in data distribution, and thus improves the accuracy of task recognition and efficiently integrates information from multiple modalities. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0010] Figure 1 A schematic diagram of a flow chart of a cross-modal task identification method provided by an embodiment of the present invention;

[0011] Figure 2 A schematic diagram of a sub-process of a cross-modal task identification method provided by an embodiment of the present invention;

[0012] Figure 3 A schematic diagram of a sub-process of a cross-modal task identification method provided by an embodiment of the present invention;

[0013] Figure 4 A schematic diagram of a sub-process of a cross-modal task identification method provided by an embodiment of the present invention;

[0014] Figure 5 A schematic diagram of a sub-process of a cross-modal task identification method provided by an embodiment of the present invention;

[0015] Figure 6 A schematic diagram of a sub-process of a cross-modal task identification method provided by an embodiment of the present invention;

[0016] Figure 7 A schematic diagram of a sub-process of a cross-modal task identification method provided by an embodiment of the present invention;

[0017] Figure 8 A schematic block diagram of a cross-modal task identification device provided by an embodiment of the present invention;

[0018] Figure 9 A schematic block diagram of a computer device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0019] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0020] It will be understood that when used in this specification and the appended claims, the terms “comprises” and “comprising” indicate the presence of described features, integers, steps, operations, elements and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof.

[0021] It should also be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the present invention. As used in the specification and appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms unless the context clearly indicates otherwise.

[0022] It should be further understood that the term "and / or" used in the present description and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.

[0023] See also Figure 1 , Figure 1 A flow chart of a cross-modal task recognition method provided in an embodiment of the present invention. The cross-modal task recognition method in this embodiment can be applied to a large multimodal model that performs task recognition based on information from multiple different modalities to generate task recognition results, for example, it can be used in medical image analysis and diagnosis scenarios. Specifically, the cause of the disease can be determined by performing task recognition based on input X-rays and theoretical knowledge of the disease. This method can fully integrate information from visual and language modalities when processing multimodal data, improve the effect of task recognition, and thus efficiently integrate information from multiple modalities.

[0024] Figure 1 FIG. 1 is a flow chart of a cross-modal task identification method provided by an embodiment of the present invention. As shown in the figure, the method includes the following steps S110-S140.

[0025] S110 , performing feature conversion and splicing on the acquired collected image and text content through a preset encoder to generate a fusion feature vector of the image and text.

[0026] In this embodiment, the captured image is an image that needs to be identified for a task, for example, image data of a patient obtained through medical equipment, such as an X-ray, CT scan, or MRI image, and the text content is text information related to the image, such as a patient's medical history, examination report, diagnosis, etc. The preset encoder is an encoder for extracting features from an image or text, such as a visual encoder and a language encoder. The captured image and text content are subjected to feature conversion and splicing through a preset encoder, that is, the captured image and text content are subjected to feature extraction through corresponding encoders to obtain a feature vector with key information, and the image feature vector and the text feature vector are spliced to form a fused feature vector, wherein the fused feature vector contains key information in both the image and the text. By extracting and splicing features based on the captured image and text, more comprehensive data support is provided for subsequent medical decision-making.

[0027] In one embodiment, if Figure 2 As shown, the step S110 also includes steps S111-S113.

[0028] S111, converting the collected image into an image feature vector through a preset visual encoder;

[0029] S112, converting the text content into a text feature vector through a preset language encoder;

[0030] S113: Concatenate the image feature vector and the text feature vector to obtain the fused feature vector.

[0031] In this embodiment, the visual encoder is a deep learning model, such as the CLIP model (Contrastive Language-Image Pre-training, a multimodal pre-trained neural network). It is trained on a large amount of medical imaging data and is capable of learning characteristic patterns in images. In the medical field, these models are optimized based on the characteristics of medical images to extract features relevant to disease diagnosis. The preset language encoder is a pre-trained language model, such as the BERT model (Bidirectional Encoder Representations from Transformers). Both the visual encoder and the language encoder are trained using a dataset from a large multimodal model (such as an image-text dataset), enabling them to learn the correspondence between images and text in a semantic vector space. The captured image is converted into an image feature vector using the preset visual encoder. Specifically, when the captured medical image is input into the preset visual encoder, the encoder performs layer-by-layer feature extraction and abstraction on the image. Ultimately, the image is converted into a fixed-length vector, namely the image feature vector. This vector contains key information in the image, such as the shape, size, and density of the lesion, and can represent the essential characteristics of the image in numerical form, facilitating subsequent calculations and analysis. The text content is converted into a text feature vector through a preset language encoder. Specifically, after the text content is input into the preset language encoder, the encoder will perform operations such as word segmentation and word embedding on the text, and then convert the text into a vector of fixed length, namely the text feature vector. This vector can represent the theme, key information and other features of the text, and convert the text data into a numerical form that can be processed by the computer. The image feature vector and the text feature vector are spliced together to obtain the fused feature vector, wherein the fused feature vector integrates the key features of the medical image and text information. The specific splicing operation is: m =Concat(f visual (I),f text (T)), where x m is the fused feature vector, f visual (I) represents the image feature vector generated by the preset visual encoder to extract features from image I, f text(T) represents the text feature vector generated by the preset text encoder after feature extraction from the text T. Concat(·) represents the concatenation of visual and text features. By fusing information from two different modalities, more comprehensive and accurate patient information can be provided. In medical diagnosis, this fused feature vector helps doctors more accurately diagnose conditions and formulate treatment plans.

[0032] S120 . Update the multimodal cluster center according to the fused feature vector using a preset dynamic update method to obtain a new cluster center.

[0033] In this embodiment, the fused feature vector is a vector obtained by extracting and splicing features from data of different modalities (such as medical images and text information). The preset dynamic update method is an algorithm or strategy for updating cluster centers. It dynamically adjusts the position of the cluster centers according to certain calculation rules based on the current fused feature vector and the existing cluster centers. The multimodal cluster center is the current cluster center of the CTC module (Cross-modal Task Clustering) of the multimodal large model. The multimodal cluster center is updated according to the fused feature vector using the preset dynamic update method to obtain a new cluster center. Specifically, whenever a new batch of fused feature vectors is processed, the cluster center is updated according to the data of the current batch, for example, the cluster center is updated according to parameters such as the preset learning rate. By updating the multimodal cluster center according to the preset dynamic update method to obtain a new cluster center, the dynamic update of the cluster center can more accurately reflect the current task category distribution, thereby improving the accuracy of task recognition.

[0034] In one embodiment, if Figure 3 As shown, the step S120 also includes steps S121-S122.

[0035] S121, performing difference and accumulation calculation on the fused feature vector and the multimodal cluster center to obtain a cumulative difference;

[0036] S122: Perform dynamic update calculation based on the accumulated difference, the preset learning rate, the preset sample set, and the multimodal cluster center to generate the new cluster center.

[0037] In this embodiment, the preset learning rate is a parameter that controls the step size of cluster center updates. A larger learning rate means that the cluster center will adjust faster based on the new data (fused feature vector), while a smaller learning rate means that the adjustment will be slower. The fused feature vector and the multimodal cluster center are subjected to difference and cumulative calculations, and a dynamic update calculation is performed based on the cumulative difference, the preset learning rate, the preset sample set, and the multimodal cluster center, specifically:

[0038]

[0039] in, is the updated new cluster center, is the multimodal cluster center (cluster center before update), α is the learning rate, is the preset sample set, wherein the preset sample set can be determined according to the current fused feature vector, for example, randomly selecting data from the fused feature vector and assigning it to the sample set, which is not limited to this, x m By updating the cluster center, the model can handle new task types that may appear in long-term operation, thereby enhancing the generalization ability of the model and improving the accuracy of task recognition.

[0040] S130 , performing feature clustering on the fused feature vector and the new cluster center using a preset clustering algorithm to obtain a cluster center vector.

[0041] In this embodiment, the new cluster center is a center point representing different categories obtained by dynamically adjusting the distribution of the fused feature vector in the clustering algorithm. The preset clustering algorithm is an algorithm for dividing data points (such as fused feature vectors) into different categories or groups. Common clustering algorithms include K-Means, hierarchical clustering, etc. The clustering algorithm assigns data points to different cluster centers by calculating the similarity or distance between data points. The algorithm iteratively updates the cluster center until the convergence condition is reached (such as the cluster center no longer changes or changes very little). The fused feature vector and the new cluster center are feature clustered by the preset clustering algorithm to obtain a cluster center vector. Specifically, the fused feature vector and the new cluster center are provided as input data to the CTC module, and the preset clustering algorithm calculates the similarity or distance between each fused feature vector and the new cluster center, for example, the Euclidean distance, cosine similarity and other metrics. According to the calculated similarity or distance, each fused feature vector is assigned to the nearest cluster center to obtain a new cluster center vector. These vectors represent the characteristic centers of different categories or groups. The updated cluster centers better reflect the distribution characteristics of the current data points. In the medical field, cluster center vectors can be used for a variety of tasks. For example, they can serve as characteristic representations of disease subtypes for further analysis and modeling; they can also be used to segment patient groups and provide a basis for developing personalized treatment plans. By obtaining cluster center vectors, the model can adapt to changes in data distribution in real time, improving the accuracy of task recognition.

[0042] In one embodiment, if Figure 4 As shown, the step S130 also includes steps S131-S133.

[0043] S131, calculating the Euclidean distance between each fused feature vector and all the new cluster centers using the preset clustering algorithm;

[0044] S132, assigning the fused feature vector to the new cluster center with the smallest Euclidean distance;

[0045] S133: Determine the cluster center vector according to the allocated new cluster center by using the preset clustering algorithm.

[0046] In this embodiment, the Euclidean distance is a commonly used distance measurement method for calculating the straight-line distance between two vectors. In this embodiment, the preset clustering algorithm is the K-Means algorithm (an algorithm for dividing data points into different categories or groups). The K-Means algorithm is used to calculate the Euclidean distance between each fused feature vector and all new cluster centers to measure the similarity or difference between them. According to the calculation result of the Euclidean distance, each fused feature vector is assigned to the new cluster center with the smallest Euclidean distance, that is, each fused feature vector is considered to belong to the same category or group as the new cluster center closest to it. After the fused feature vector is assigned, a new cluster center is generated. The iterative K-Means algorithm updates the position of the cluster center according to the assigned data points, that is, generates the final cluster center vector. These vectors represent the new feature centers of different categories or groups, and the cluster center vector better reflects the distribution characteristics of the current data (i.e., the fused feature vector). It can be expressed as follows:

[0047]

[0048] Where M is the total number of new cluster centers, c l is the lth new cluster center. j is the nearest new cluster center to which the sample is assigned. By determining the cluster center vector according to a preset clustering algorithm, the cluster center vector can accurately reflect the task category of the sample, providing a reliable basis for subsequent task routing, thereby improving the efficiency of multi-tasking processing.

[0049] S140: Generate a target task recognition result by applying the cluster center vector to a preset neural network sub-model and a preset gating network.

[0050] In this embodiment, the preset neural network sub-model is a lightweight neural network module. In this embodiment, the preset neural network sub-model is an Adapter network model, which is used to fine-tune the pre-trained multimodal large model to adapt to specific downstream tasks. The Adapter can adjust the model output by inserting additional layers or modules without changing the main structure of the original model. The preset gated network is a neural network structure that uses a gating mechanism to control the flow of information. The gated network can dynamically adjust the information flow within the network based on the characteristics of the input data, thereby enhancing the model's expressive power and flexibility. The cluster center vector is passed through the preset neural network sub-model and the preset gated network to generate a target task recognition result. Specifically, the cluster center vector is input into the preset neural network sub-model (Adapter). After feature transformation and extraction, a feature representation more suitable for the target task is obtained. These feature representations are then input into the preset gated network, where the gating mechanism dynamically adjusts the information flow to ultimately generate the target task recognition result. By combining the advantages of the cluster center vector, the preset neural network sub-model, and the gated network, more accurate and reliable target task recognition results can be generated, providing strong support for medical decision-making and personalized treatment.

[0051] In one embodiment, if Figure 5 As shown, the step S140 also includes steps S1401-S1402 before the step S140.

[0052] S1401, constructing a preset number of neural network sub-models in the multimodal large model by using a preset ensemble learning method;

[0053] S1402: Construct a gating network in the multimodal large model, and generate a weight coefficient of each of the neural network sub-models through the gating network and the cluster center vector.

[0054] In this embodiment, the multimodal large model is a deep learning model capable of processing data from multiple modalities (such as text, images, audio, etc.). In the medical field, this can involve processing various types of information, such as medical images, electronic medical records, and genetic data. The preset ensemble learning method is a method for constructing neural network sub-models. In this embodiment, the preset ensemble learning method is a MoE (Mixture of Experts) ensemble learning method, which can construct multiple "expert" models (i.e., neural network sub-models), with each expert model responsible for processing a portion of the data or specific features. In the MoE, different expert models can be specialized for training different aspects or modalities of the data, thereby improving the performance and generalization ability of the overall model. In the multimodal large model, a preset number of neural network sub-models are constructed using the MoE method. These sub-models can respectively process different modal data or different data features, such as one sub-model processing medical image features and another sub-model processing text information features. The gating network is a special neural network structure that is used to control the activation level of different expert models (i.e., neural network sub-models). The weight coefficient of each neural network sub-model is generated by the gating network and the cluster center vector. Specifically, the cluster center vector is obtained through the above-mentioned clustering algorithm and other steps, and represents the feature center of different categories or groups. The cluster center vector is input into the gating network, and the gating network generates the weight coefficient of each neural network sub-model according to the characteristics of these vectors. These weight coefficients reflect the importance or contribution of each neural network sub-model under a given cluster center vector (i.e., a specific category or group feature). It should be noted that the cluster center vector is used as a task embedding for subsequent task identification processing, so the cluster center vector can also be called a task embedding. By constructing a neural network sub-model through a preset ensemble learning method (MoE) and combining the gating network to generate weight coefficients, more flexible and efficient feature processing and decision making can be achieved in the multimodal large model, providing strong support for applications in the medical field.

[0055] In one embodiment, if Figure 6 As shown, the step S1402 also includes steps S14021-S14022 before the step S1402.

[0056] S14021. Performing network training on the gated network through preset back-propagation learning, and adjusting network parameters according to the training results;

[0057] S14022. Control the gating network to generate weight coefficients corresponding to the neural network sub-model according to the adjusted network parameters.

[0058] In this embodiment, the preset back propagation learning is an algorithm for training a neural network, which updates the network parameters to minimize the loss function by calculating the gradient of the loss function relative to the network parameters. The gated network is trained through the preset back propagation learning. Specifically, the back propagation learning can randomly initialize the parameters of the gated network (such as weights and biases), forward propagate the cluster center vector through the network, generate the initial weight coefficients of the neural network sub-model, and then calculate the difference between the predicted result and the true label based on these weight coefficients and the preset loss function (such as mean square error, cross entropy, etc.), that is, the loss value. Finally, the gradient of the loss function with respect to the network parameters is calculated through the back propagation algorithm, and the parameters are updated to minimize the loss value. Through back propagation learning, the gated network can learn how to dynamically adjust the weight coefficients of the neural network sub-model according to the patient's characteristics (cluster center vector) to improve the accuracy of tasks such as disease diagnosis and treatment response prediction. After backpropagation learning, the parameters of the gating network are adjusted, allowing the network to better adapt to the input data and generate more accurate weight coefficients. When a new cluster center vector enters the gating network, the network will calculate based on the adjusted parameters and generate the weight coefficients of the corresponding neural network sub-model. The gating network automatically generates the corresponding weight coefficients of the neural network sub-model, allowing the gating network to dynamically control the activation level of each neural network sub-model, thereby selecting the most appropriate sub-model for prediction or decision-making based on the characteristics of the input data.

[0059] In one embodiment, if Figure 7 As shown, the step S140 also includes steps S141-S142.

[0060] S141, passing the cluster center vector through each of the neural network sub-models to generate a corresponding sub-model output;

[0061] S142. Perform weighted summation on the outputs of the multiple sub-models through the weight coefficients of the neural network sub-model to generate the target task recognition result.

[0062] In this embodiment, the sub-model output is the output result generated by each of the neural network sub-models according to the cluster center vector. The cluster center vector is passed through each of the neural network sub-models to generate the corresponding sub-model output. Specifically, when the cluster center vector is input to each neural network sub-model, the sub-model processes the input vector according to its internal structure and parameters and generates corresponding outputs. These outputs represent the sub-model's understanding or prediction of the input vector. The outputs of multiple sub-models are multiplied by their corresponding weight coefficients, and then these products are added to obtain a weighted summation result. This process is actually integrating the information of different sub-models to generate more accurate and comprehensive target task recognition results. For example, there are three sub-models: Adapter 1, for pneumonia diagnosis in X-ray images; Adapter 2: for tumor detection in CT images; and Adapter 3, for brain disease diagnosis in MRI images. The three sub-models generate corresponding sub-model outputs X1, X2, and X3 based on the cluster center vector. The gating network receives the cluster center vector and outputs the weight coefficients of the three sub-models, for example: Adapter 1 (pneumonia diagnosis): 0.8; Adapter 2 (tumor detection): 0.15; Adapter 3 (brain disease diagnosis): 0.05. That is, the target task recognition result S is 0.8X1 + 0.15X2 + 0.05X3. It can be understood that the sum of all weight coefficients is 1. By combining the outputs of multiple neural network sub-models and performing a weighted sum based on their importance, the accuracy of the target task recognition result can be improved, and accurate diagnosis or treatment plans can be generated, promoting the development of medical care.

[0063] Figure 8 is a schematic block diagram of a cross-modal task identification device 200 provided by an embodiment of the present invention. Figure 8 As shown, corresponding to the above cross-modal task identification method, the present invention also provides a cross-modal task identification device. The cross-modal task identification device includes a unit for executing the above cross-modal task identification method, and the device can be configured in a desktop computer, tablet computer, laptop computer, etc. Specifically, please refer to Figure 8 The cross-modal task identification device includes a splicing unit 210 , an updating unit 220 , a clustering unit 230 and a generating unit 240 .

[0064] The splicing unit 210 is used to perform feature conversion and splicing on the acquired image and text content through a preset encoder to generate a fusion feature vector of the image and text.

[0065] In one embodiment, the splicing unit 210 includes an image conversion unit, a text conversion unit, and a fusion unit.

[0066] An image conversion unit, configured to convert the collected image into an image feature vector via a preset visual encoder;

[0067] A text conversion unit, configured to convert the text content into a text feature vector via a preset language encoder;

[0068] A fusion unit is used to concatenate the image feature vector and the text feature vector to obtain the fused feature vector.

[0069] The updating unit 220 is configured to update the multimodal cluster center according to the fused feature vector by a preset dynamic updating method to obtain a new cluster center.

[0070] In one embodiment, the updating unit 220 includes a difference unit and an updating unit.

[0071] A difference unit, configured to perform difference and accumulation calculations on the fused feature vector and the multimodal cluster center to obtain a cumulative difference;

[0072] An updating unit performs dynamic update calculation according to the accumulated difference, a preset learning rate, a preset sample set, and the multimodal cluster center to generate the new cluster center.

[0073] The clustering unit 230 is configured to perform feature clustering on the fused feature vector and the new cluster center using a preset clustering algorithm to obtain a cluster center vector.

[0074] In one embodiment, the clustering unit 230 includes a calculation unit, an allocation unit, and a determination unit.

[0075] a calculation unit, configured to calculate the Euclidean distance between each of the fused feature vectors and all the new cluster centers using the preset clustering algorithm;

[0076] an allocating unit, configured to allocate the fused feature vector to the new cluster center with the smallest Euclidean distance;

[0077] A determining unit is configured to determine the cluster center vector according to the allocated new cluster center by using the preset clustering algorithm.

[0078] The generating unit 240 is configured to generate a target task recognition result by passing the cluster center vector through a preset neural network sub-model and a preset gating network.

[0079] In one embodiment, the generating unit 240 includes a constructing unit and a weight generating unit.

[0080] A construction unit, configured to construct a preset number of neural network sub-models in the multimodal large model by using a preset ensemble learning method;

[0081] A weight generation unit is used to construct a gating network in the multimodal large model, and generate a weight coefficient of each neural network sub-model through the gating network and the cluster center vector.

[0082] In one embodiment, the generating unit 240 includes a training unit and an adjusting unit.

[0083] A training unit, configured to perform network training on the gating network through preset back-propagation learning, and adjust network parameters according to the training results;

[0084] An adjustment unit is used to control the gating network to generate weight coefficients corresponding to the neural network sub-model according to the adjusted network parameters.

[0085] In one embodiment, the generating unit 240 includes an output unit and a weighting unit.

[0086] An output unit, configured to generate a corresponding sub-model output by passing the cluster center vector through each of the neural network sub-models;

[0087] A weighting unit is used to perform weighted summation on the outputs of multiple sub-models through the weight coefficients of the neural network sub-model to generate the target task recognition result.

[0088] It should be noted that those skilled in the art can clearly understand that the specific implementation process of the above-mentioned cross-modal task identification device 200 and each unit can refer to the corresponding description in the aforementioned method embodiment. For the convenience and brevity of description, it will not be repeated here.

[0089] The cross-modal task identification device can be implemented in the form of a computer program. The computer program can be used in Figure 9 Runs on the computer device shown.

[0090] See also Figure 9 , Figure 9 This is a schematic block diagram of a computer device provided in an embodiment of the present application. The computer device 500 can be a terminal or a server. The terminal can be a smart phone, tablet computer, laptop computer, desktop computer, personal digital assistant, wearable device, or other electronic device with communication capabilities. The server can be a standalone server or a server cluster consisting of multiple servers.

[0091] See Figure 9 The computer device 500 includes a processor 502 , a memory, and a network interface 505 connected via a system bus 501 , wherein the memory may include a non-volatile storage medium 503 and an internal memory 504 .

[0092] The non-volatile storage medium 503 may store an operating system 5031 and a computer program 5032. The computer program 5032 includes program instructions, which, when executed, may enable the processor 502 to perform a cross-modal task identification method.

[0093] The processor 502 is used to provide computing and control capabilities to support the operation of the entire computer device 500.

[0094] The internal memory 504 provides an environment for the operation of the computer program 5032 in the non-volatile storage medium 503 . When the computer program 5032 is executed by the processor 502 , the processor 502 can execute a cross-modal task recognition method.

[0095] The network interface 505 is used to communicate with other devices through the network. Figure 9 The structure shown in the figure is merely a block diagram of a portion of the structure related to the solution of the present application, and does not constitute a limitation on the computer device 500 to which the solution of the present application is applied. The specific computer device 500 may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0096] The processor 502 is configured to run a computer program 5032 stored in the memory to implement the steps of the above method.

[0097] It should be understood that in the embodiment of the present application, the processor 502 may be a central processing unit (CPU), and the processor 502 may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0098] Those skilled in the art will appreciate that all or part of the steps in the method of the above-described embodiment can be implemented by instructing the relevant hardware through a computer program. The computer program includes program instructions, which can be stored in a storage medium that is computer-readable. The program instructions are executed by at least one processor in the computer system to implement the steps in the method of the above-described embodiment.

[0099] Therefore, the present invention also provides a storage medium. The storage medium may be a computer-readable storage medium. The storage medium stores a computer program, wherein the computer program includes program instructions. When the program instructions are executed by a processor, the processor performs the steps of the above method.

[0100] The storage medium may be any computer-readable storage medium that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a magnetic disk, or an optical disk.

[0101] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the above description has generally described the composition and steps of each example according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0102] In the several embodiments provided herein, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the various units is merely a logical functional division, and actual implementation may employ other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be omitted or not implemented.

[0103] The steps in the methods of the embodiments of the present invention may be adjusted in order, combined, or deleted as needed. The units in the devices of the embodiments of the present invention may be combined, divided, or deleted as needed. Furthermore, the functional units in the various embodiments of the present invention may be integrated into a single processing unit, each unit may exist physically separately, or two or more units may be integrated into a single unit.

[0104] If this integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the existing technology, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes a number of instructions for causing a computer device (which can be a personal computer, terminal, or network device, etc.) to execute all or part of the steps of the method described in various embodiments of the present invention.

[0105] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and such modifications or substitutions are intended to be within the scope of protection of the present invention. Therefore, the scope of protection of the present invention shall be subject to the scope of protection of the claims.

Claims

1. A cross-modal task identification method, characterized in that: The method is applied to a multimodal large model, and the method comprises: The acquired image and text content are converted and spliced through a preset encoder to generate a fusion feature vector of the image and text; updating the multimodal cluster center according to the fused feature vector using a preset dynamic updating method to obtain a new cluster center; Performing feature clustering on the fused feature vector and the new cluster center using a preset clustering algorithm to obtain a cluster center vector; The cluster center vector is passed through a preset neural network sub-model and a preset gating network to generate a target task recognition result.

2. The method according to claim 1, characterized in that The step of performing feature conversion and splicing on the acquired collected image and text content through a preset encoder to generate a fusion feature vector of the image and text includes: Converting the captured image into an image feature vector through a preset visual encoder; Converting the text content into a text feature vector through a preset language encoder; The image feature vector and the text feature vector are concatenated to obtain the fused feature vector.

3. The method according to claim 1, characterized in that The step of updating the multimodal cluster center by a preset dynamic updating method according to the fused feature vector to obtain a new cluster center includes: Performing difference and accumulation calculation on the fused feature vector and the multimodal cluster center to obtain a cumulative difference; Dynamic update calculation is performed according to the cumulative difference, the preset learning rate, the preset sample set and the multimodal cluster center to generate the new cluster center.

4. The method according to claim 1, wherein The step of performing feature clustering on the fused feature vector and the new cluster center using a preset clustering algorithm to obtain a cluster center vector includes: Calculating the Euclidean distance between each of the fused feature vectors and all the new cluster centers using the preset clustering algorithm; Assigning the fused feature vector to the new cluster center with the smallest Euclidean distance; The cluster center vector is determined according to the allocated new cluster center through the preset clustering algorithm.

5. The method according to claim 1, wherein Before the step of generating a target task recognition result by using the cluster center vector through a preset neural network sub-model and a preset gating network, the method includes: Constructing a preset number of neural network sub-models in the multimodal large model by using a preset ensemble learning method; A gating network is constructed in the multimodal large model, and a weight coefficient of each of the neural network sub-models is generated through the gating network and the cluster center vector.

6. The method according to claim 5, characterized in that Before the step of generating the weight coefficient of each neural network sub-model through the gating network and the cluster center vector, the method includes: The gated network is trained through preset back propagation learning, and the network parameters are adjusted according to the training results; The gated network is controlled according to the adjusted network parameters to generate weight coefficients corresponding to the neural network sub-model.

7. The method according to claim 6, characterized in that The step of generating a target task recognition result by using the cluster center vector through a preset neural network sub-model and a preset gating network includes: Passing the cluster center vector through each of the neural network sub-models to generate a corresponding sub-model output; The outputs of the multiple sub-models are weighted and summed using the weight coefficients of the neural network sub-model to generate the target task recognition result.

8. A cross-modal task recognition device, characterized in that: The device is applied to a multimodal large model, and the device comprises: A splicing unit is used to perform feature conversion and splicing on the acquired image and text content through a preset encoder to generate a fusion feature vector of the image and text; An updating unit, configured to update the multimodal cluster center according to the fused feature vector by a preset dynamic updating method to obtain a new cluster center; A clustering unit, configured to perform feature clustering on the fused feature vector and the new cluster center using a preset clustering algorithm to obtain a cluster center vector; A generating unit is used to generate a target task recognition result by passing the cluster center vector through a preset neural network sub-model and a preset gating network.

9. A computer device, characterized in that: The computer device includes a memory and a processor, the memory stores a computer program, and the processor implements the method according to any one of claims 1 to 7 when executing the computer program.

10. A storage medium, characterized in that: The storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a processor, the method according to any one of claims 1 to 7 can be implemented.

Images