Training method, device and electronic equipment based on cross-modal image recognition model

Abstract

The present disclosure provides a training method, device, and electronic device based on a cross-modal image recognition model, which relates to the field of artificial intelligence technology, especially to the technical fields of computer vision, deep learning, and large models. The specific implementation scheme is as follows: obtaining an image to be trained and determining multimodal data of the image to be trained; the multimodal data includes a text feature vector and a visual feature vector, the text feature vector represents the text data describing the object in the image to be trained, and the visual feature vector represents the object in the image to be trained; the image to be trained is input into an initial model to obtain a feature vector to be trained of the image to be trained; the feature vector to be trained represents the object in the image to be trained; the initial model is trained based on the feature vector to be trained and the multimodal data to obtain an image recognition model; the image recognition model is used to identify objects in the image.

Description

Technical Field

[0001] The present disclosure relates to computer vision, deep learning, large models and other technical fields in the field of artificial intelligence technology, and in particular to a training method, device and electronic device based on a cross-modal image recognition model. Background Art

[0002] Neural network models are used in all aspects of life. They can automatically identify objects in images and determine their categories, bringing convenience to people's lives. For example, neural network models can be used in scenarios such as intelligent traffic detection to identify guardrails and manhole covers on the road.

[0003] Neural network models need to be pre-trained to ensure accurate detection and recognition of objects in images. Therefore, how to train neural network models has become a key research topic. Summary of the Invention

[0004] The present disclosure provides a training method, device, and electronic device for a cross-modal image recognition model.

[0005] According to a first aspect of the present disclosure, a method for training a cross-modal image recognition model is provided, comprising:

[0006] Acquire an image to be trained and determine multimodal data of the image to be trained; wherein the multimodal data includes a text feature vector and a visual feature vector, the text feature vector represents text data describing an object in the image to be trained, and the visual feature vector represents the object in the image to be trained;

[0007] Inputting the image to be trained into the initial model to obtain a feature vector to be trained corresponding to the image to be trained; wherein the feature vector to be trained represents an object in the image to be trained;

[0008] The initial model is trained according to the feature vector to be trained and the multimodal data to obtain a trained image recognition model; wherein the image recognition model is used to recognize objects in an image.

[0009] According to a second aspect of the present disclosure, there is provided a method for image recognition, comprising:

[0010] Obtain the image to be recognized;

[0011] The image to be identified is input into the image recognition model described in the first aspect, and based on the image recognition model, an object in the image to be identified is determined.

[0012] According to a third aspect of the present disclosure, a training device for a cross-modal image recognition model is provided, comprising:

[0013] an acquisition unit, configured to acquire an image to be trained and determine multimodal data of the image to be trained; wherein the multimodal data includes a text feature vector and a visual feature vector, the text feature vector representing text data describing an object in the image to be trained, and the visual feature vector representing an object in the image to be trained;

[0014] An input unit, configured to input the image to be trained into an initial model to obtain a feature vector to be trained corresponding to the image to be trained; wherein the feature vector to be trained represents an object in the image to be trained;

[0015] A training unit is used to train the initial model based on the feature vector to be trained and the multimodal data to obtain a trained image recognition model; wherein the image recognition model is used to recognize objects in an image.

[0016] According to a fourth aspect of the present disclosure, there is provided an apparatus for image recognition, comprising:

[0017] An image acquisition unit, configured to acquire an image to be recognized;

[0018] The image recognition unit is configured to input the image to be recognized into the image recognition model described in the third aspect, and determine the object in the image to be recognized based on the image recognition model.

[0019] According to a fifth aspect of the present disclosure, there is provided an electronic device, including:

[0020] at least one processor; and

[0021] a memory communicatively coupled to the at least one processor;

[0022] The memory stores instructions that can be executed by the at least one processor. The instructions are executed by the at least one processor to enable the at least one processor to perform the method of the first aspect and the second aspect.

[0023] According to a sixth aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause the computer to execute the method according to the first and second aspects.

[0024] According to a seventh aspect of the present disclosure, a computer program product is provided, comprising: a computer program, which implements the methods described in the first and second aspects when executed by a processor.

[0025] According to the technology disclosed in the present invention, the training accuracy of the image recognition model is improved, thereby improving the accuracy of image recognition.

[0026] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] The accompanying drawings are provided to facilitate a better understanding of the present invention and do not constitute a limitation of the present disclosure.

[0028] Figure 1 is a flowchart of a training method for a cross-modal image recognition model provided according to an embodiment of the present disclosure;

[0029] Figure 2 is a flowchart of a training method for a cross-modal image recognition model provided according to an embodiment of the present disclosure;

[0030] Figure 3 is a flowchart of a training method for a cross-modal image recognition model provided according to an embodiment of the present disclosure;

[0031] Figure 4 is a schematic diagram of a multimodal feature extraction process provided according to an embodiment of the present disclosure;

[0032] Figure 5 is a flowchart of an image recognition method provided according to an embodiment of the present disclosure;

[0033] Figure 6 1 is a structural block diagram of a training device based on a cross-modal image recognition model provided according to an embodiment of the present disclosure;

[0034] Figure 7 1 is a structural block diagram of a training device based on a cross-modal image recognition model provided according to an embodiment of the present disclosure;

[0035] Figure 8 is a structural block diagram of an image recognition device provided according to an embodiment of the present disclosure;

[0036] Figure 9 is a block diagram of an electronic device for implementing the cross-modal image recognition model training method and image recognition method according to an embodiment of the present disclosure;

[0037] Figure 10 It is a block diagram of an electronic device used to implement the cross-modal image recognition model training method and image recognition method of the embodiment of the present disclosure. DETAILED DESCRIPTION

[0038] The following description of exemplary embodiments of the present disclosure is made in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding. These details should be considered as merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0039] With the rapid development of artificial intelligence (AI), neural network models are being applied in all aspects of life. Neural network models can detect and recognize images, and can be used in scenarios such as facial recognition and intelligent traffic detection. For example, they can detect damaged guardrails and manhole covers on roads.

[0040] In order for neural network models to meet actual business needs, they must be pre-trained. Model training techniques often utilize single-modal data for training, such as textual or visual data. However, this single-modal training approach can affect the model's generalization capabilities. Single-modal data contains less information than images, resulting in lower accuracy in object recognition, making it difficult to meet actual business needs.

[0041] The present disclosure provides a training method, device, and electronic device based on a cross-modal image recognition model, which are applied to technical fields such as computer vision, deep learning, and large models in the field of artificial intelligence technology to improve the training accuracy of the image recognition model.

[0042] It should be noted that the model in this embodiment is not targeted at a specific user and cannot reflect the personal information of a specific user. It should be noted that the images in this embodiment are from a public dataset.

[0043] In the technical solutions disclosed herein, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0044] In order to enable readers to have a deeper understanding of the implementation principle of this disclosure, the following Figures 1-10 The embodiment is further refined.

[0045] Figure 1 The flowchart of a training method based on a cross-modal image recognition model provided in accordance with an embodiment of the present disclosure is shown in FIG. The method can be executed by a training device based on a cross-modal image recognition model. Figure 1 As shown, the method includes the following steps:

[0046] S101. Acquire an image to be trained and determine multimodal data of the image to be trained; wherein the multimodal data includes a text feature vector and a visual feature vector, the text feature vector represents text data describing an object in the image to be trained, and the visual feature vector represents the object in the image to be trained.

[0047] For example, an image recognition model is pre-built. The image recognition model is a machine learning neural network model that can be used to detect and identify objects in images. For example, it can determine the category of objects in an image. The image recognition model can be applied to scenarios such as face recognition or road detection. For example, it can identify railings, manhole covers, and other objects in road environments. The image recognition model can include network layers such as convolutional layers. Before using the image recognition model, it needs to be trained so that it can accurately identify objects in images.

[0048] The training images are collected in advance. For example, a large number of road environment images can be collected as the training images. The training images can include one or more objects, for example, vehicles, pedestrians, trees, etc.

[0049] The multimodal data of the image to be trained can be determined based on the image to be trained. Multimodal data refers to data of multiple modalities. The multimodality in this embodiment may include textual modality and visual modality, etc. Therefore, the multimodal data may include a feature vector of the textual modality and a feature vector of the visual modality. The feature vector of the textual modality is determined as a textual feature vector, and the feature vector of the visual modality is determined as a visual feature vector. That is to say, after obtaining the image to be trained, the textual feature vector and the visual feature vector of the image to be trained are determined. The textual feature vector can represent the textual data describing the object in the image to be trained, and the visual feature vector can represent the object in the image to be trained.

[0050] The training image can be described in text form to obtain text data describing the objects in the training image. This text data is then converted into a matrix vector form as text feature data. Furthermore, the high-dimensional training image can be converted into a low-dimensional matrix vector form to obtain a visual feature vector.

[0051] S102 , inputting the image to be trained into the initial model to obtain a feature vector to be trained corresponding to the image to be trained; wherein the feature vector to be trained represents an object in the image to be trained.

[0052] For example, a pre-built image recognition model to be trained is determined as an initial model, and the initial model is trained. After the training is completed, a final image recognition model is obtained. The initial model may include network layers such as convolutional layers. In this embodiment, the network structure of the initial model is not specifically limited.

[0053] The training image is input into the initial model. The network layers in the initial model extract features from the training image and output a training feature vector corresponding to the training image. The training feature vector can represent the object in the training image. For example, if there is a black car in the training image, the extracted training feature vector can indicate that the object in the training image is a car and its color is black.

[0054] In this embodiment, the number of training images input to the initial model can be preset. For example, a single training image can be input at a time, or multiple training images can be input in batches. For each training image, a corresponding training feature vector can be obtained. The training feature vector is in the form of a matrix.

[0055] S103. Train the initial model according to the feature vector to be trained and the multimodal data to obtain a trained image recognition model; wherein the image recognition model is used to recognize objects in the image.

[0056] For example, after obtaining the training feature vectors and multimodal data, the initial model can be iteratively trained based on the feature vectors and multimodal data to be trained. If the training is complete, the trained model is determined as the image recognition model; if the training is not complete, the iterative training is continued until the training is complete.

[0057] The feature vector to be trained is obtained from the initial model, and the multimodal data is obtained from models other than the initial model. When training the initial model, the multimodal data can be matched with the feature data to be trained. For example, text feature data can be compared and matched with the training data, and visual feature data can be compared and matched with the training data. This allows the initial model to learn information about objects in the multimodal data, and achieves knowledge transfer between different modalities based on cross-modality. This enhances the initial model's visual representation of objects and significantly improves accuracy.

[0058] When matching multimodal data with feature data to be trained, the degree of consistency between the multimodal data and the feature data to be trained can be determined based on a preset loss function. If the degree of consistency between the multimodal data and the feature data to be trained is high, the model training can be considered complete; if the degree of consistency between the multimodal data and the feature data to be trained is low, training needs to be continued. For example, the degree of consistency between the text feature data and the feature data to be trained can be determined, as can the degree of consistency between the visual feature data and the feature data to be trained. If the degree of consistency between the text feature data and the feature data to be trained is high, and the degree of consistency between the visual feature data and the feature data to be trained is high, the model training is determined to be complete, that is, a trained image recognition model is obtained.

[0059] In the disclosed embodiment, an image to be trained is obtained, and multimodal data such as a text feature vector and a visual feature vector of the image to be trained is determined. The image to be trained is input into the initial model to obtain a feature vector to be trained corresponding to the image to be trained. Through the text feature vector and the visual feature vector, cross-modal knowledge transfer is performed on the feature vector to be trained, that is, through different modalities, the initial model learns more knowledge. The initial model is thereby trained to obtain a trained image recognition model. This effectively solves the problem of low model training accuracy caused by training with single-modal features. Through different modal data, the training accuracy of the model is improved, the visual representation effect of the model on the image is enhanced, and a significant improvement in the recognition accuracy of the image is achieved.

[0060] Figure 2 A flowchart of a method for training a cross-modal image recognition model is provided in an embodiment of the present disclosure.

[0061] In this embodiment, determining the multimodal data of the image to be trained can be further refined as follows: determining the text data describing the image to be trained, and based on a preset text encoder, determining the text feature vector corresponding to the text data; wherein the preset text encoder is used to extract features from the text data; based on a preset visual encoder, determining the visual feature vector of the image to be trained; wherein the visual encoder is used to extract features from the image.

[0062] like Figure 2 As shown, the method includes the following steps:

[0063] S201. Acquire an image to be trained, determine text data describing the image to be trained, and determine a text feature vector corresponding to the text data based on a preset text encoder; wherein the preset text encoder is used to extract features from the text data.

[0064] For example, a pre-collected training image is obtained, and text data describing the training image can be determined based on the object expressed in the training image. For example, if the training image contains a black car, the text data can be "a black car".

[0065] Text data can be determined by pre-annotating the training images. For example, when the training images are collected, the objects they represent are pre-annotated, and the pre-annotated information is used as the text data for the training images. Alternatively, semantic analysis of the content of the training images can be performed based on a pre-set large language model to obtain the text data corresponding to the training images.

[0066] The text data is in the form of textual statements that can be read. A text encoder can be pre-configured to encode the text data. That is, the text data is converted into a matrix format as a text feature vector. The text feature vector can represent the text data. The text encoder can be used to extract features from the text data. The encoding process of the text data is also the feature extraction process. The text encoder can be a predetermined neural network model used for feature extraction from text data. In this embodiment, the network structure of the text encoder is not specifically limited.

[0067] In this embodiment, determining text data describing the image to be trained includes: inputting the image to be trained into a preset large language model, and outputting first text data describing the image to be trained.

[0068] Specifically, a large language model is pre-set. The large language model can be an open source neural network model that can be used to describe the image in the form of text. The image to be trained is input into the large language model. The large language model extracts features of the image to be trained and outputs text data. The output text data can be used to describe the image to be trained. The text data output by the large language model is determined as the first text data. The first text data is the caption (description) information of the image to be trained. The large language model can output caption information, which is a description of the image content. In this embodiment, the model structure of the large language model is not specifically limited.

[0069] The beneficial effect of this setting is that by introducing a large language model to extract caption information, text data describing the image to be trained can be automatically obtained, reducing manual operations and effectively improving the efficiency of model training.

[0070] In this embodiment, based on a preset text encoder, determining a text feature vector corresponding to text data includes: based on the preset text encoder, performing feature extraction on the first text data to obtain a text feature vector corresponding to the first text data as the first text vector.

[0071] Specifically, after obtaining the first text data, the first text data is input into a preset text encoder. The preset text encoder performs feature extraction on the first text data to obtain a feature vector corresponding to the first text data. In other words, the caption information is converted into a feature vector. The feature vector corresponding to the first text data is determined as the first text vector. In other words, the first text data is represented in the form of a matrix vector.

[0072] The beneficial effect of this setting is that the feature vector of the text modality is obtained through the preset text encoder, thereby realizing cross-modal knowledge transfer and improving the accuracy of model training.

[0073] In this embodiment, determining text data describing the image to be trained includes: obtaining image labels pre-labeled on the image to be trained; wherein the image labels represent objects in the image; and obtaining second text data describing the image to be trained based on the image labels and a preset text template.

[0074] Specifically, when acquiring training images, the training images can be pre-labeled, and each training image can have an annotated image label. The image label can represent the object in the image. For example, if there is a car in the image, the image label can be "car." Each object in the training image can be assigned an image label. That is, if there are multiple objects in the training image, there can be multiple image labels. For example, if there is a pedestrian and a car in the training image, the image label "pedestrian" and the image label "car" can be assigned.

[0075] A text template is pre-set, and this text template can be a template for the text data to be generated. In this embodiment, the text template can be a prompt template, that is, the text data to be generated is prompt. Prompt is the prompt text data used during model training and application, which can help the model better understand the input image.

[0076] An image label of the image to be trained is obtained. Based on the image label and a preset text template, text data describing the image to be trained can be obtained, and the text data is determined as the second text data. For example, the second text data containing the image label can be obtained in the form of a preset text template.

[0077] The beneficial effect of this setting is that, based on the preset image labels and the preset text templates, text data can be quickly obtained, thereby improving the efficiency of determining text data and thus improving the training efficiency of the model.

[0078] In this embodiment, obtaining second text data describing the image to be trained based on the image label and a preset text template includes: adding the image label to a preset position in the preset text template to obtain the second text data describing the image to be trained.

[0079] Specifically, the text template may have a preset location for the image tag. After obtaining the image tag, the image tag can be filled in the preset location in the text template to supplement the text template and obtain the complete second text data. For example, if the text template is "an image of XXX", the location of "XXX" is the location of the image tag. Replacing "XXX" with the image tag will obtain the second text data. For example, if the image tag is "car", the second text data obtained will be "an image of a car".

[0080] The beneficial effect of this setting is that, through a simple replacement operation, the second text data can be obtained quickly and accurately, thereby improving the accuracy and efficiency of model training.

[0081] In this embodiment, based on a preset text encoder, determining a text feature vector corresponding to the text data includes: based on the preset text encoder, performing feature extraction on the second text data to obtain a text feature vector corresponding to the second text data as the second text vector.

[0082] Specifically, after obtaining the second text data, the second text data is input into a preset text encoder. The preset text encoder performs feature extraction on the second text data to obtain a feature vector corresponding to the second text data. In other words, prompt is converted into a feature vector. The feature vector corresponding to the second text data is determined as the second text vector. In other words, the second text data is represented in the form of a matrix vector. In this embodiment, the text encoder that outputs the first text vector and the text encoder that outputs the second text vector can be the same text encoder.

[0083] The beneficial effect of this setting is that the feature vector of the text modality is obtained through the preset text encoder, thereby realizing cross-modal knowledge transfer and improving the accuracy of model training.

[0084] S202. Determine a visual feature vector of the image to be trained based on a preset visual encoder; wherein the visual encoder is used to extract features from the image.

[0085] For example, a vision encoder is pre-set. The vision encoder of this embodiment is an open source neural network model that can be used to extract features from images and convert high-dimensional images into low-dimensional matrix vectors. In this embodiment, the network structure of the vision encoder is not specifically limited.

[0086] The training image is fed into a visual encoder, which extracts features from the image and outputs a visual feature vector for the training image. The visual feature vector represents the training image in the form of a matrix vector, for example, it can represent the objects in the training image.

[0087] In this embodiment, the text encoder and visual encoder are used to obtain feature vectors for different modalities, including feature vectors for the caption and prompt, as well as feature vectors for the visual modality. This enables cross-modal knowledge transfer for the model being trained, allowing the model to learn knowledge from different modalities. This enhances the model's visual representation, which helps improve the accuracy of model training and usage.

[0088] S203 , inputting the image to be trained into the initial model to obtain a feature vector to be trained corresponding to the image to be trained; wherein the feature vector to be trained represents an object in the image to be trained.

[0089] For example, this step may refer to the above-mentioned step S102 and will not be described in detail.

[0090] S204. Train the initial model based on the feature vector to be trained and the multimodal data to obtain a trained image recognition model; wherein the image recognition model is used to recognize objects in the image.

[0091] For example, this step may refer to the above-mentioned step S103 and will not be described in detail.

[0092] In the disclosed embodiment, an image to be trained is obtained, and multimodal data such as a text feature vector and a visual feature vector of the image to be trained is determined. The image to be trained is input into the initial model to obtain a feature vector to be trained corresponding to the image to be trained. Through the text feature vector and the visual feature vector, cross-modal knowledge transfer is performed on the feature vector to be trained, that is, through different modalities, the initial model learns more knowledge. The initial model is thereby trained to obtain a trained image recognition model. This effectively solves the problem of low model training accuracy caused by training with single-modal features. Through different modal data, the training accuracy of the model is improved, the visual representation effect of the model on the image is enhanced, and a significant improvement in the recognition accuracy of the image is achieved.

[0093] Figure 3A flowchart of a method for training a cross-modal image recognition model is provided in an embodiment of the present disclosure.

[0094] In this embodiment, the initial model is trained according to the feature vector to be trained and the multimodal data to obtain a trained image recognition model, which can be further refined as follows: according to the feature vector to be trained and the multimodal data, based on a preset loss function, a training loss value is determined; wherein the training loss value is used to represent the training status of the model; according to the training loss value, based on a preset back propagation algorithm, the initial model is trained to obtain a trained image recognition model.

[0095] like Figure 3 As shown, the method includes the following steps:

[0096] S301. Acquire an image to be trained and determine multimodal data of the image to be trained; wherein the multimodal data includes a text feature vector and a visual feature vector, the text feature vector represents text data describing an object in the image to be trained, and the visual feature vector represents the object in the image to be trained.

[0097] For example, this step may refer to the above-mentioned step S101 and will not be described in detail.

[0098] S302 : Input the image to be trained into the initial model to obtain a feature vector to be trained corresponding to the image to be trained; wherein the feature vector to be trained represents an object in the image to be trained.

[0099] For example, this step may refer to the above-mentioned step S102 and will not be described in detail.

[0100] S303. Determine a training loss value based on the feature vector to be trained and the multimodal data and a preset loss function; wherein the training loss value is used to represent the training status of the model.

[0101] For example, a loss function is pre-set. After obtaining the feature vectors and multimodal data to be trained, the feature vectors and multimodal data can be substituted into the pre-set loss function. The loss function is calculated to obtain a calculation result, which is determined as a training loss value. The training loss value can be used to indicate the training status of the model. That is, based on the training loss value, it can be determined whether the image recognition model has been trained.

[0102] In this embodiment, a training loss value is determined based on a preset loss function according to the feature vector to be trained and the multimodal data, including: determining a first loss value based on a preset first loss function according to the feature vector to be trained and the text feature vector; wherein the first loss value represents the degree of consistency between the feature vector to be trained and the text feature vector; determining a second loss value based on a preset second loss function according to the feature vector to be trained and the visual feature vector; wherein the second loss value represents the degree of consistency between the feature vector to be trained and the visual feature vector; determining a third loss value based on a preset third loss function according to the feature vector to be trained and the image label pre-marked on the image to be trained; wherein the third loss value represents the degree of consistency between the feature vector to be trained and the image label; and determining a training loss value based on the first loss value, the second loss value, and the third loss value.

[0103] Specifically, a first loss function is pre-set, and the first loss function can be used to calculate the feature vector to be trained and the text feature vector. The feature vector to be trained and the text feature vector are substituted into the first loss function to obtain the calculation result of the first loss function. The calculation result of the first loss function is determined as the first loss value. The first loss value can characterize the degree of consistency between the feature vector to be trained and the text feature vector, that is, whether the object represented by the feature vector to be trained and the object represented by the text feature vector are close. For example, the first loss function can be a contrastive loss function, and the first loss value can be expressed as s1=contrastive loss(s_f, t_f), where s1 represents the first loss value, s_f represents the feature vector to be trained, and t_f represents the text feature vector.

[0104] A second loss function is pre-set, and the second loss function can be used to calculate the feature vector to be trained and the visual feature vector. The feature vector to be trained and the visual feature vector are substituted into the second loss function to obtain the calculation result of the second loss function. The calculation result of the second loss function is determined as the second loss value. The second loss value can characterize the degree of consistency between the feature vector to be trained and the visual feature vector, that is, whether the object represented by the feature vector to be trained and the object represented by the visual feature vector are close. For example, the second loss function can be mse loss (mean square error loss function), and the second loss value can be expressed as s2=mse loss(s_f, v_f), where s2 represents the second loss value and v_f represents the visual feature vector.

[0105] A third loss function is pre-set, and the third loss function can be used to calculate the feature vector to be trained. The feature vector to be trained is substituted into the third loss function to obtain the calculation result of the third loss function. The calculation result of the third loss function is determined as the third loss value. The third loss value represents the degree of consistency between the feature vector to be trained and the image label. The image to be trained is pre-marked with an image label, and the image label can be used to represent the category of the object in the image. For example, if the image label is "car", it can be determined that the category of the object in the image is car. By determining the degree of consistency between the feature vector to be trained and the image label, it can be determined whether the category of the object represented by the feature vector to be trained is the category represented by the image label. The third loss function can be a classification loss (classification loss function), and the third loss value can be expressed as s3=classification loss(s_f), where s3 represents the third loss value.

[0106] After obtaining the first, second, and third loss values, the first, second, and third loss values ​​can be calculated based on a preset calculation formula, and the calculation result can be determined as the training loss value. For example, the first, second, and third loss values ​​can be added to obtain the training loss value, or the average of the first, second, and third loss values ​​can be calculated to obtain the training loss value. In this embodiment, the calculation formula for calculating the training loss value is not specifically limited.

[0107] The beneficial effect of this setting is that multiple loss values ​​can be obtained based on the data to be trained and the feature vectors of different modalities, which enables cross-modal training of the model through feature vectors of different modalities and improves the training accuracy of the model.

[0108] In this embodiment, a first loss value is determined based on a preset first loss function according to the feature vector to be trained and the text feature vector, including: determining a fourth loss value based on the preset first loss function according to the feature vector to be trained and the first text vector; wherein the fourth loss value represents the degree of consistency between the feature vector to be trained and the first text vector; determining a fifth loss value based on the preset first loss function according to the feature vector to be trained and the second text vector; wherein the fifth loss value represents the degree of consistency between the feature vector to be trained and the second text vector; and determining the first loss value based on the fourth loss value and the fifth loss value.

[0109] Specifically, the text feature vector may include a first text vector and a second text vector. When determining the first loss value, the loss value between the feature vector to be trained and the first text vector can be calculated based on a preset first loss function, as well as the loss value between the feature vector to be trained and the second text vector. The first loss value is determined based on these two loss values.

[0110] The feature vector to be trained and the first text vector can be substituted into a preset first loss function, and the calculated result can be determined as a fourth loss value. The fourth loss value can represent the degree of consistency between the feature vector to be trained and the first text vector, that is, whether the object represented by the feature vector to be trained is close to the object represented by the first text vector. For example, the fourth loss value can be expressed as s4=contrastive loss(s_f, c_f), where s4 represents the fourth loss value and c_f represents the first text vector.

[0111] The feature vector to be trained and the second text vector can be substituted into a preset first loss function, and the calculated result can be determined as a fifth loss value. The fifth loss value can represent the degree of consistency between the feature vector to be trained and the second text vector, that is, whether the object represented by the feature vector to be trained is close to the object represented by the second text vector. For example, the fifth loss value can be expressed as s5=contrastive loss(s_f, p_f), where s5 represents the fifth loss value and p_f represents the second text vector.

[0112] The first loss value is determined based on the fourth loss value and the fifth loss value. For example, the fourth loss value and the fifth loss value can be added to obtain the first loss value. Alternatively, a weight for the fourth loss value and a weight for the fifth loss value can be preset, and the fourth loss value and the fifth loss value can be weighted and summed according to the preset weights to obtain the first loss value. For example, s1 = r1 × s4 + r2 × s5, where r1 is the weight of the fourth loss value and r2 is the weight of the fifth loss value.

[0113] The beneficial effect of this setting is that the feature vectors of the text modality can include two, and different text feature vectors can be calculated separately, so that the model can learn more knowledge of the text modality and effectively improve the accuracy of model training.

[0114] In this embodiment, the training loss value is determined based on the first loss value, the second loss value, and the third loss value, including: according to the preset weight of the first loss value, the weight of the second loss value, and the weight of the third loss value, the first loss value, the second loss value, and the third loss value are weighted and summed to obtain the training loss value.

[0115] Specifically, the training loss value is determined by combining the first loss value, the second loss value, and the third loss value. For example, the first loss value, the second loss value, and the third loss value can be added together. Alternatively, the weights of the first loss value, the second loss value, and the third loss value can be pre-set, and the first loss value, the second loss value, and the third loss value can be weighted and summed according to the preset weights to obtain the training loss value. For example, the training loss value can be expressed as s = r3 × s1 + r4 × s2 + r5 × s3, where s represents the training loss value, r3 represents the weight of the first loss value, r4 represents the weight of the second loss value, and r5 represents the weight of the third loss value.

[0116] The beneficial effect of this setting is that multiple loss values ​​are combined to obtain the final training loss value, so that the model can be trained using data from different modalities, effectively improving the accuracy of model training.

[0117] S304: According to the training loss value, the initial model is trained based on a preset back propagation algorithm to obtain a trained image recognition model.

[0118] For example, after obtaining the training loss value, it can be determined whether the model training is complete based on the training loss value. For example, a training completion condition can be preset, and it can be determined whether the training loss value meets the preset training completion condition. If it does, it is determined that the trained image recognition model is complete; if not, it is necessary to continue training.

[0119] A backpropagation algorithm can be preset to update model parameters, such as the learning rate and convolutional layer weights. Training is then continued based on the updated model until model training is complete. In this embodiment, the preset backpropagation algorithm is not specifically limited.

[0120] In this embodiment, by setting a specific loss function and combining multimodal data to train the model, knowledge transfer between different modalities is achieved, so that the model can learn knowledge from multiple modal data and improve the comprehensiveness of information in the feature vector output by the model, that is, improve the training accuracy of the model and facilitate the subsequent application of the model.

[0121] Figure 4 Schematic diagram of the multimodal feature extraction process. Figure 4The image to be trained is input into a preset large language model, which outputs first text data. Second text data can be obtained based on the image label and text template of the image to be trained. Both the first and second text data can be input into a text encoder, which can output a first text vector and a second text vector. The first text vector corresponds to the first text data, and the second text vector corresponds to the second text data. The image to be trained can also be input into a preset visual encoder, which outputs a visual feature vector. The image to be trained can also be input into an initial model, which outputs a feature vector to be trained. The first text vector, the second text vector, and the feature vector to be trained are all calculated using a first loss function. The first text vector and the feature vector to be trained can be used to generate a fourth loss value, and the second text vector and the feature vector to be trained can be used to generate a fifth loss value. The visual feature vector and the feature vector to be trained can be used to generate a second loss value based on the second loss function. The feature vector to be trained can be used to generate a third loss value based on the third loss function. Based on the fourth loss value, the fifth loss value, the second loss value, and the third loss value, a final loss value to be trained can be obtained, thereby determining whether the model training is complete.

[0122] In the disclosed embodiment, an image to be trained is obtained, and multimodal data such as a text feature vector and a visual feature vector of the image to be trained is determined. The image to be trained is input into the initial model to obtain a feature vector to be trained corresponding to the image to be trained. Through the text feature vector and the visual feature vector, cross-modal knowledge transfer is performed on the feature vector to be trained, that is, through different modalities, the initial model learns more knowledge. The initial model is thereby trained to obtain a trained image recognition model. This effectively solves the problem of low model training accuracy caused by training with single-modal features. Through different modal data, the training accuracy of the model is improved, the visual representation effect of the model on the image is enhanced, and a significant improvement in the recognition accuracy of the image is achieved.

[0123] Figure 5 FIG. 1 is a flow chart of an image recognition method according to an embodiment of the present disclosure, which can be performed by an image recognition device. Figure 5 As shown, the method includes the following steps:

[0124] S501: Obtain an image to be recognized.

[0125] For example, after training the image recognition model, it can be applied. For example, it can be used to identify objects in an image and determine their categories. The image recognition model can also be applied to scenarios such as face recognition or intelligent transportation. For example, it can promptly identify damaged railings on the road.

[0126] The input data for an image recognition model is the image to be recognized. When applying the image recognition model, image acquisition can be performed in real time or periodically to obtain the image to be recognized. For example, a camera mounted on a vehicle can be used to capture images of the road environment, and these captured images can be used as the image to be recognized.

[0127] S502: Input the image to be recognized into the image recognition model, and determine the object in the image to be recognized based on the image recognition model.

[0128] For example, an image to be recognized is input into an image recognition model, which may include multiple network layers such as convolutional layers. The network layers in the image recognition model extract features from the image to be recognized, generating a feature vector for the image to be recognized. Based on the feature vector, the object in the image to be recognized is determined. For example, the object's category, color, size, etc. can be determined.

[0129] The image recognition model is trained using multimodal data. When applying the image recognition model, you can input only the image to be recognized, use the feature vector of the image to be recognized as the feature vector of the visual modality, perform unimodal data processing, and determine the object in the image to be recognized. Alternatively, when inputting the image to be recognized, you can also input the feature vector of the text modality into the image recognition model, allowing the image recognition model to determine the object in the image to be recognized based on both text and visual modalities. In other words, the trained image recognition model has the ability to perform image recognition based on multimodal feature vectors. When actually using the image recognition model, you can use either unimodal or multimodal feature vectors for image recognition, thereby flexibly meeting actual business needs.

[0130] In this embodiment, inputting the image to be recognized into the image recognition model includes: determining the text feature vector of the image to be recognized; wherein the text feature vector of the image to be recognized represents the text data describing the object in the image to be recognized; and inputting the image to be recognized and the text feature vector of the image to be recognized into the image recognition model.

[0131] Specifically, when image recognition is required based on multimodal feature vectors, after acquiring the image to be recognized, a text feature vector for the image to be recognized can be determined. Both the image to be recognized and its text feature vector are input into the image recognition model, allowing the model to extract the feature vector of the visual modality of the image to be recognized. The model then combines the feature vectors of both the visual and text modalities to perform image recognition. The text feature vector of the image to be recognized can represent textual data describing the object in the image to be recognized, i.e., it is a feature vector of the text modality.

[0132] For example, a neural network model for determining text feature vectors can be pre-set. The input of this model is an image, and the output can be a text feature vector corresponding to the image. In other words, the model can determine the text data describing the object in the image and convert the text data into a matrix form. In this embodiment, the model for determining the text feature vector is not specifically limited.

[0133] The beneficial effect of this setting is that the text feature vector of the image to be recognized is determined, so that the image recognition model processes the visual modality features and text feature vectors of the image to be recognized, avoiding the situation where image recognition only uses the feature vector of a single modality to cause erroneous recognition results, and effectively improving the accuracy of image recognition.

[0134] In this embodiment, determining the text feature vector of the image to be identified includes: inputting the image to be identified into a preset large language model, and outputting text data describing the image to be identified; based on a preset text encoder, performing feature extraction on the text data describing the image to be identified, and obtaining the text feature vector of the image to be identified; wherein the preset text encoder is used to perform feature extraction on the text data.

[0135] Specifically, when determining the text feature vector of the image to be recognized, the image to be recognized can be first input into a preset large language model. The large language model can analyze the image to be recognized, determine the color, size and other information of the object expressed in the image to be recognized, and describe the image to be recognized in the form of text, that is, it can output text data describing the image to be recognized.

[0136] A text encoder is pre-configured. This encoder can be the same as the one used during model training and is used to extract features from text data. Text data describing the image to be recognized is input into the encoder, which encodes the text data, i.e., extracts features. The text data is converted into a matrix to obtain a text feature vector for the image to be recognized.

[0137] The beneficial effect of this setting is that the text feature data of the image to be recognized is automatically determined through a large language model and a text encoder, which facilitates image recognition based on feature vectors of both text and visual modalities, thereby improving the efficiency and accuracy of image recognition.

[0138] In the disclosed embodiments, an image to be recognized is obtained and a trained image recognition model is applied to automatically identify objects in the image, thereby improving image recognition efficiency. Furthermore, during the recognition process, both single-modal and multi-modal feature vectors can be processed, meeting actual business needs and improving image recognition accuracy.

[0139] Figure 6This is a structural block diagram of a training device for a cross-modal image recognition model provided by an embodiment of the present disclosure. For ease of explanation, only the parts related to the embodiment of the present disclosure are shown. Figure 6 The training device 600 based on the cross-modal image recognition model includes: an acquisition unit 601, an input unit 602 and a training unit 603.

[0140] An acquisition unit 601 is configured to acquire an image to be trained and determine multimodal data of the image to be trained; wherein the multimodal data includes a text feature vector and a visual feature vector, wherein the text feature vector represents text data describing an object in the image to be trained, and the visual feature vector represents an object in the image to be trained;

[0141] An input unit 602 is configured to input the image to be trained into an initial model to obtain a feature vector to be trained corresponding to the image to be trained; wherein the feature vector to be trained represents an object in the image to be trained;

[0142] The training unit 603 is used to train the initial model according to the feature vector to be trained and the multimodal data to obtain a trained image recognition model; wherein the image recognition model is used to recognize objects in an image.

[0143] Figure 7 A structural block diagram of a training device based on a cross-modal image recognition model provided in an embodiment of the present disclosure, such as Figure 6 As shown, the training device 700 based on the cross-modal image recognition model includes an acquisition unit 701, an input unit 702 and a training unit 703, wherein the acquisition unit 701 includes a first determination module 7011 and a second determination module 7012.

[0144] A first determining module 7011 is configured to determine text data describing the image to be trained, and determine a text feature vector corresponding to the text data based on a preset text encoder, wherein the preset text encoder is configured to perform feature extraction on the text data;

[0145] The second determination module 7012 is used to determine the visual feature vector of the image to be trained based on a preset visual encoder; wherein the visual encoder is used to extract features from the image.

[0146] In one example, the first determining module 7011 includes:

[0147] The first determining submodule is configured to input the image to be trained into a preset large language model and output first text data describing the image to be trained.

[0148] In one example, the first determining module 7011 includes:

[0149] The first extraction submodule is configured to perform feature extraction on the first text data based on a preset text encoder to obtain a text feature vector corresponding to the first text data as a first text vector.

[0150] In one example, the first determining module 7011 includes:

[0151] A label acquisition submodule is used to obtain image labels pre-labeled on the image to be trained; wherein the image labels represent objects in the image;

[0152] The second determining submodule is configured to obtain second text data describing the image to be trained according to the image label and a preset text template.

[0153] In one example, the second determination submodule is specifically configured to:

[0154] The image label is added to a preset position in the preset text template to obtain second text data describing the image to be trained.

[0155] In one example, the first determining module 7011 includes:

[0156] The second extraction submodule is configured to perform feature extraction on the second text data based on a preset text encoder to obtain a text feature vector corresponding to the second text data as a second text vector.

[0157] In one example, the training unit 703 includes:

[0158] A loss value determination module, configured to determine a training loss value based on the feature vector to be trained and the multimodal data and a preset loss function; wherein the training loss value is used to represent the training status of the model;

[0159] The model training module is used to train the initial model according to the training loss value based on a preset back propagation algorithm to obtain a trained image recognition model.

[0160] In one example, the loss value determination module includes:

[0161] A first loss submodule, configured to determine a first loss value based on the feature vector to be trained and the text feature vector and a preset first loss function; wherein the first loss value represents a degree of consistency between the feature vector to be trained and the text feature vector;

[0162] A second loss submodule, configured to determine a second loss value based on the feature vector to be trained and the visual feature vector and a preset second loss function; wherein the second loss value represents a degree of consistency between the feature vector to be trained and the visual feature vector;

[0163] a third loss submodule, configured to determine a third loss value based on the feature vector to be trained and the image label pre-labeled on the image to be trained, based on a preset third loss function; wherein the third loss value represents the degree of consistency between the feature vector to be trained and the image label;

[0164] A loss combination submodule is used to determine the training loss value according to the first loss value, the second loss value, and the third loss value.

[0165] In one example, the first loss submodule is specifically used to:

[0166] Determining a fourth loss value based on the feature vector to be trained and the first text vector and a preset first loss function; wherein the fourth loss value represents a degree of consistency between the feature vector to be trained and the first text vector;

[0167] Determining a fifth loss value based on the feature vector to be trained and the second text vector and a preset first loss function; wherein the fifth loss value represents a degree of consistency between the feature vector to be trained and the second text vector;

[0168] The first loss value is determined according to the fourth loss value and the fifth loss value.

[0169] In one example, the loss combination submodule is used to:

[0170] According to the preset weight of the first loss value, the weight of the second loss value, and the weight of the third loss value, the first loss value, the second loss value, and the third loss value are weightedly summed to obtain the training loss value.

[0171] Figure 8 This is a structural block diagram of an image recognition device provided by an embodiment of the present disclosure. For ease of explanation, only the parts related to the embodiment of the present disclosure are shown. Figure 8 The image recognition device 800 includes: an image acquisition unit 801 and an image recognition unit 802.

[0172] An image acquisition unit 801 is used to acquire an image to be recognized;

[0173] The image recognition unit 802 is configured to input the image to be recognized into the image recognition model according to any one of claims 15 to 25, and determine the object in the image to be recognized based on the image recognition model.

[0174] In one example, the image recognition unit 802 includes:

[0175] A vector determination module, configured to determine a text feature vector of the image to be identified; wherein the text feature vector of the image to be identified represents text data describing an object in the image to be identified;

[0176] An input module is used to input the image to be recognized and the text feature vector of the image to be recognized into the image recognition model.

[0177] In one example, the vector determination module includes:

[0178] A data output submodule, configured to input the image to be recognized into a preset large language model and output text data describing the image to be recognized;

[0179] The vector acquisition submodule is used to perform feature extraction on the text data describing the image to be identified based on a preset text encoder to obtain a text feature vector of the image to be identified; wherein the preset text encoder is used to perform feature extraction on the text data.

[0180] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device.

[0181] Figure 9 A structural block diagram of an electronic device provided in an embodiment of the present disclosure, such as Figure 9 As shown, the electronic device 900 includes: at least one processor 902; and a memory 901 communicatively connected to the at least one processor 902; wherein the memory stores instructions that can be executed by the at least one processor 902, and the instructions are executed by the at least one processor 902 to enable the at least one processor 902 to execute the cross-modal image recognition model-based training method and image recognition method disclosed in the present invention.

[0182] The electronic device 900 further includes a receiver 903 and a transmitter 904. The receiver 903 is used to receive instructions and data sent by other devices, and the transmitter 904 is used to send instructions and data to external devices.

[0183] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0184] According to an embodiment of the present disclosure, the present disclosure also provides a computer program product, which includes: a computer program, the computer program is stored in a readable storage medium, at least one processor of an electronic device can read the computer program from the readable storage medium, and at least one processor executes the computer program so that the electronic device executes the solution provided by any of the above embodiments.

[0185] Figure 10 A schematic block diagram of an example electronic device 1000 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0186] like Figure 10 As shown, the device 1000 includes a computing unit 1001, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1002 or a computer program loaded from a storage unit 1008 into a random access memory (RAM) 1003. Various programs and data required for the operation of the device 1000 can also be stored in the RAM 1003. The computing unit 1001, the ROM 1002, and the RAM 1003 are connected to each other via a bus 1004. An input / output (I / O) interface 1005 is also connected to the bus 1004.

[0187] Various components in device 1000 are connected to I / O interface 1005, including an input unit 1006, such as a keyboard, mouse, etc.; an output unit 1007, such as various types of displays, speakers, etc.; a storage unit 1008, such as a magnetic disk, optical disk, etc.; and a communication unit 1009, such as a network card, modem, wireless communication transceiver, etc. The communication unit 1009 allows device 1000 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0188] The computing unit 1001 can be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 1001 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units that run machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 1001 performs the various methods and processes described above, such as the training method and image recognition method based on the cross-modal image recognition model. For example, in some embodiments, the training method and image recognition method based on the cross-modal image recognition model can be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as the storage unit 808. In some embodiments, part or all of the computer program can be loaded and / or installed on the device 1000 via the ROM 1002 and / or the communication unit 1009. When the computer program is loaded into the RAM 1003 and executed by the computing unit 1001, one or more steps of the training method and image recognition method based on the cross-modal image recognition model described above can be performed. Alternatively, in other embodiments, the computing unit 1001 may be configured in any other appropriate manner (for example, by means of firmware) to execute the cross-modal image recognition model-based training method and the image recognition method.

[0189] Various embodiments of the systems and techniques described above can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-chip systems (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system that includes at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0190] The program code for implementing the method of the present disclosure can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device so that when the program code is executed by the processor or controller, the functions / operations specified in the flow chart and / or block diagram are implemented. The program code can be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0191] In the context of the present disclosure, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in conjunction with an instruction execution system, device or equipment. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium can include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0192] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0193] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0194] A computer system may include a client and a server. The client and server are generally remote from each other and typically interact via a communication network. This client-server relationship is established by computer programs running on the respective computers, establishing a client-server relationship. The server may be a cloud server, also known as a cloud computing server or cloud host, a host product within the cloud computing service ecosystem that addresses the management difficulties and limited scalability of traditional physical hosts and VPS services ("Virtual Private Servers" or simply "VPS"). The server may also be a server in a distributed system or a server integrated with blockchain.

[0195] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved. This is not a limitation herein.

[0196] The above specific embodiments do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure shall be included within the scope of protection of this disclosure.

Claims

1. A training method for a cross-modal image recognition model, comprising: Acquire an image to be trained, and determine multimodal data of the image to be trained using a first model; wherein the multimodal data includes a text feature vector and a visual feature vector, the text feature vector representing text data describing an object in the image to be trained, and the visual feature vector representing the object in the image to be trained; Inputting the image to be trained into an initial model to obtain a feature vector to be trained corresponding to the image to be trained; wherein the feature vector to be trained represents an object in the image to be trained, and the first model is different from the initial model; Matching the feature vector to be trained with the multimodal data, training the initial model, and obtaining a trained image recognition model; wherein the image recognition model is used to recognize objects in an image; Matching the feature vector to be trained with the multimodal data, training the initial model, and obtaining a trained image recognition model, including: Comparing and matching the text feature data with the data to be trained, and comparing and matching the visual feature data with the data to be trained, and determining a training loss value based on a preset loss function; According to the training loss value, based on a preset back propagation algorithm, the initial model is trained to obtain a trained image recognition model.

2. The method according to claim 1, wherein Determining multimodal data of the image to be trained includes: Determining text data describing the image to be trained, and determining a text feature vector corresponding to the text data based on a preset text encoder; wherein the preset text encoder is used to perform feature extraction on the text data; Based on a preset visual encoder, a visual feature vector of the image to be trained is determined; wherein the visual encoder is used to extract features from the image.

3. The method according to claim 2, wherein: Determining text data describing the image to be trained, including: The image to be trained is input into a preset large language model, and first text data describing the image to be trained is output.

4. The method according to claim 3, wherein: Determining a text feature vector corresponding to the text data based on a preset text encoder includes: Based on a preset text encoder, feature extraction is performed on the first text data to obtain a text feature vector corresponding to the first text data, which is a first text vector.

5. The method according to claim 2, wherein: Determining text data describing the image to be trained, including: Obtaining image labels pre-labeled on the image to be trained; wherein the image labels represent objects in the image; According to the image label and a preset text template, second text data describing the image to be trained is obtained.

6. The method according to claim 5, wherein: Obtaining second text data describing the image to be trained based on the image label and a preset text template, including: The image label is added to a preset position in the preset text template to obtain second text data describing the image to be trained.

7. The method according to claim 5 or 6, wherein: Determining a text feature vector corresponding to the text data based on a preset text encoder includes: Based on a preset text encoder, feature extraction is performed on the second text data to obtain a text feature vector corresponding to the second text data, which is a second text vector.

8. The method according to claim 7, wherein: Comparing and matching the text feature data with the data to be trained, and comparing and matching the visual feature data with the data to be trained, and determining a training loss value based on a preset loss function, including: Determining a first loss value based on the feature vector to be trained and the text feature vector and a preset first loss function; Determining a second loss value based on a preset second loss function according to the feature vector to be trained and the visual feature vector; Determining a third loss value based on a preset third loss function according to the feature vector to be trained and the image label pre-marked on the image to be trained; The training loss value is determined according to the first loss value, the second loss value, and the third loss value.

9. The method according to claim 8, wherein Determining a first loss value based on the to-be-trained feature vector and the text feature vector and a preset first loss function includes: Determining a fourth loss value based on the feature vector to be trained and the first text vector and a preset first loss function; Determining a fifth loss value based on the feature vector to be trained and the second text vector and a preset first loss function; The first loss value is determined according to the fourth loss value and the fifth loss value.

10. The method according to claim 8 or 9, wherein: Determining the training loss value according to the first loss value, the second loss value, and the third loss value includes: According to the preset weight of the first loss value, the weight of the second loss value, and the weight of the third loss value, the first loss value, the second loss value, and the third loss value are weightedly summed to obtain the training loss value.

11. A method for image recognition, comprising: Obtain the image to be recognized; The image to be identified is input into the image recognition model according to any one of claims 1 to 10, and the object in the image to be identified is determined based on the image recognition model.

12. The method according to claim 11, wherein Inputting the image to be recognized into the image recognition model includes: Determining a text feature vector of the image to be recognized; wherein the text feature vector of the image to be recognized represents text data describing an object in the image to be recognized; The image to be recognized and the text feature vector of the image to be recognized are input into the image recognition model.

13. The method according to claim 12, wherein: Determining a text feature vector of the image to be recognized includes: Inputting the image to be recognized into a preset large language model, and outputting text data describing the image to be recognized; Based on a preset text encoder, feature extraction is performed on the text data describing the image to be identified to obtain a text feature vector of the image to be identified; wherein the preset text encoder is used to extract features from the text data.

14. A training device for a cross-modal image recognition model, comprising: an acquisition unit, configured to acquire an image to be trained and determine multimodal data of the image to be trained using a first model; wherein the multimodal data includes a text feature vector and a visual feature vector, wherein the text feature vector represents text data describing an object in the image to be trained, and the visual feature vector represents the object in the image to be trained; An input unit, configured to input the image to be trained into an initial model to obtain a feature vector to be trained corresponding to the image to be trained; wherein the feature vector to be trained represents an object in the image to be trained, and the first model and the initial model are different; A training unit, configured to match the feature vector to be trained with the multimodal data, train the initial model, and obtain a trained image recognition model; wherein the image recognition model is used to recognize objects in an image; The training unit comprises: a loss value determination module, configured to compare and match the text feature data with the data to be trained, and to compare and match the visual feature data with the data to be trained, and to determine a training loss value based on a preset loss function; The model training module is used to train the initial model according to the training loss value based on a preset back propagation algorithm to obtain a trained image recognition model.

15. The device according to claim 14, wherein The acquisition unit includes: A first determination module is configured to determine text data describing the image to be trained, and determine a text feature vector corresponding to the text data based on a preset text encoder; wherein the preset text encoder is configured to perform feature extraction on the text data; The second determination module is used to determine the visual feature vector of the image to be trained based on a preset visual encoder; wherein the visual encoder is used to extract features from the image.

16. The device according to claim 15, wherein The first determining module includes: The first determining submodule is configured to input the image to be trained into a preset large language model and output first text data describing the image to be trained.

17. The device according to claim 16, wherein The first determining module includes: The first extraction submodule is configured to perform feature extraction on the first text data based on a preset text encoder to obtain a text feature vector corresponding to the first text data as a first text vector.

18. The device according to claim 15, wherein The first determining module includes: A label acquisition submodule is used to obtain image labels pre-labeled on the image to be trained; wherein the image labels represent objects in the image; The second determining submodule is configured to obtain second text data describing the image to be trained according to the image label and a preset text template.

19. The device according to claim 18, wherein The second determining submodule is specifically configured to: The image label is added to a preset position in the preset text template to obtain second text data describing the image to be trained.

20. The device according to claim 18 or 19, wherein The first determining module includes: The second extraction submodule is configured to perform feature extraction on the second text data based on a preset text encoder to obtain a text feature vector corresponding to the second text data as a second text vector.

21. The device according to claim 20, wherein The loss value determination module includes: A first loss submodule, configured to determine a first loss value based on a preset first loss function according to the feature vector to be trained and the text feature vector; A second loss submodule, configured to determine a second loss value based on a preset second loss function according to the feature vector to be trained and the visual feature vector; A third loss submodule, configured to determine a third loss value based on a preset third loss function according to the feature vector to be trained and the image label pre-marked on the image to be trained; A loss combination submodule is used to determine the training loss value according to the first loss value, the second loss value, and the third loss value.

22. The device according to claim 21, wherein The first loss submodule is specifically configured to: Determining a fourth loss value based on the feature vector to be trained and the first text vector and a preset first loss function; Determining a fifth loss value based on the feature vector to be trained and the second text vector and a preset first loss function; The first loss value is determined according to the fourth loss value and the fifth loss value.

23. The device according to claim 21 or 22, wherein The loss combining submodule is specifically used for: According to the preset weight of the first loss value, the weight of the second loss value, and the weight of the third loss value, the first loss value, the second loss value, and the third loss value are weightedly summed to obtain the training loss value.

24. An image recognition device, comprising: An image acquisition unit, configured to acquire an image to be recognized; An image recognition unit is configured to input the image to be recognized into the image recognition model according to any one of claims 14 to 23, and determine the object in the image to be recognized based on the image recognition model.

25. The apparatus according to claim 24, wherein The image recognition unit includes: A vector determination module, configured to determine a text feature vector of the image to be identified; wherein the text feature vector of the image to be identified represents text data describing an object in the image to be identified; An input module is used to input the image to be recognized and the text feature vector of the image to be recognized into the image recognition model.

26. The device according to claim 25, wherein The vector determination module includes: A data output submodule, configured to input the image to be identified into a preset large language model and output text data describing the image to be identified; The vector acquisition submodule is used to perform feature extraction on the text data describing the image to be identified based on a preset text encoder to obtain a text feature vector of the image to be identified; wherein the preset text encoder is used to perform feature extraction on the text data.

27. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 13.

28. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-13.

29. A computer program product, wherein The invention comprises a computer program, which implements the steps of the method according to any one of claims 1 to 13 when executed by a processor.

Images