CLIP multi-mode fused dish identification method, device and equipment

By combining food image and text information using the CLIP multimodal fusion method, and calculating cosine similarity using the CLIP model, the problem of low accuracy in identifying similar dishes in existing food recognition methods is solved, achieving higher recognition accuracy and model versatility.

CN119007189APending Publication Date: 2024-11-22GUANGZHOU PAIKEPUSHI INFORMATION TECH CO LTD

Patent Information

Application Number
CN202411019703.9
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-26
Publication Date
2024-11-22

AI Technical Summary

Technical Problem

Existing methods for identifying dishes that rely on image features have low accuracy for similar dishes, especially when the ingredients change.

Method used

The CLIP multimodal fusion method is adopted to initially identify the category of dishes through a visual recognition model. Combined with the dish image and text information, the CLIP model is used to extract the feature vectors of the image and text, calculate the cosine similarity matrix, and determine the target dish category.

Benefits of technology

It improves the accuracy of identifying similar dishes and the versatility of the model, reduces false identifications, and enhances the precision and efficiency of dish identification.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119007189A_ABST
    Figure CN119007189A_ABST
Patent Text Reader

Abstract

The invention discloses a CLIP multi-mode fused dish identification method, device and equipment, after a to-be-identified dish image is obtained, the dish image is input into a visual identification model to obtain a dish category set, text information corresponding to each dish category is obtained, and then the dish image and each piece of text information are input into a CLIP model. And the CLIP model extracts an image feature vector of the dish image and a text feature vector of each piece of text information, calculates a cosine similarity matrix of the image feature vector and the text feature vector, and takes the dish category corresponding to the text information with the maximum cosine similarity with the dish image as a target dish category. According to the method, after the visual identification model predicts, the CLIP model performs secondary identification on the basis of the predicted text information of the dish category and the dish image, multi-modal information such as the image and the text is fused, highly similar dishes can be better distinguished, and the universality and accuracy of dish identification are further improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of image data processing, and more specifically, to a dish recognition method, device and equipment integrating CLIP multimodality. Background Art

[0002] With the development of modern technology, people's living standards are gradually improving; in a fast-paced life, dining out is more frequent. With the increase in diners, manual settlement is inefficient and the accuracy cannot be guaranteed. With the development of artificial intelligence, some automatic pricing methods have emerged in recent years.

[0003] The commonly used dish recognition method at present is to use the trained model to identify the color, shape, texture and other characteristic information of the dish in the dish image, and calculate it through the distance formula to obtain the dish category corresponding to the dish image. However, there are many kinds of dishes, the same ingredients are prepared differently in different regions, and some dishes are difficult to distinguish by human eyes. When the dish changes, such as the ingredients of the dish change, at this time, after the pre-trained model recognizes the characteristic information of the dish in the dish image, the accuracy of the dish category determined based on the characteristic information is low. Summary of the invention

[0004] In view of this, the present application provides a dish recognition method, device and equipment integrating CLIP multimodality, which is used to solve the problem that the existing dish recognition method only relies on the collected dish images for recognition and has low recognition accuracy for dishes with high similarity.

[0005] To achieve the above objectives, the proposed solution is as follows:

[0006] A dish recognition method integrating CLIP multimodality, comprising:

[0007] Obtaining a dish image to be identified;

[0008] Inputting the dish image into a pre-trained visual recognition model to obtain a set of dish categories predicted by the visual recognition model, wherein the set of dish categories includes M dish categories predicted by the visual recognition model that are closest to the dish image, where M is a positive integer greater than 0;

[0009] Get the text information corresponding to each dish category in the dish category set;

[0010] Inputting the dish image and each text information into the pre-trained CLIP model, so that the CLIP model determines the target dish category based on the dish image and each text information;

[0011] The process of CLIP model determining the target dish category based on dish images and text information includes:

[0012] Extracting image feature vectors of dish images and text feature vectors of each text information;

[0013] Calculate the cosine similarity matrix between the image feature vector and the text feature vector;

[0014] The dish category corresponding to the text information with the largest cosine similarity with the dish image is taken as the target dish category.

[0015] Preferably, the CLIP model comprises: an image encoder and a text encoder;

[0016] The backbone network of the image encoder is Resnet50;

[0017] The backbone network of the text encoder is RBT3-chinese.

[0018] Preferably, the training process of the CLIP model includes:

[0019] Get training images and training texts;

[0020] Freeze the network layer Resnet50 of the image encoder;

[0021] The training text and training pictures are input into the CLIP model, and the text encoder and image encoder extract the text feature vector and image feature vector respectively;

[0022] Predicting a training similarity matrix between training images and training texts based on the image feature vector and the text feature vector;

[0023] Calculate the loss function of the CLIP model based on the training similarity matrix and the label similarity matrix;

[0024] Optimize the parameters of the CLIP model based on the loss function until the preset training end conditions are met;

[0025] Freeze the network layer RBT3-chinese of the trained text encoder, input the training text and training pictures into the CLIP model, the text encoder and image encoder extract the text feature vector and image feature vector respectively, and return to execute the above training process of inputting the training text and training pictures into the CLIP model, calculating the loss function of the CLIP model according to the text feature vector and image feature vector extracted by the CLIP model, and optimizing the parameters of the CLIP model based on the loss function.

[0026] Preferably, the loss function is:

[0027]

[0028] Among them, N represents the number of training images and training texts input for one training. The training images and training texts correspond one to one. N = number of training images = number of training texts. labels is the label similarity matrix between the training images and the corresponding training texts. i represents the data of the i-th row of the label similarity matrix, p is the training similarity matrix of the predicted training image and training text, p i is the data of the i-th row of the training similarity matrix.

[0029] Preferably, the formula for calculating cosine similarity based on the image feature vector and the text feature vector is:

[0030]

[0031] Among them, F i is the image feature vector, F i =(f i1 ,f i2 ,f i3 ,...f im ), F t is the text feature vector, F t =(f t1 ,f t2 ,f t3 ,...f tm ), m is the dimension of the feature vector, f iz and f tz Indicates F i and F t The zth component of .

[0032] A dish recognition device integrating CLIP multimodality, comprising:

[0033] An image acquisition unit, used to acquire an image of a dish to be identified;

[0034] A category prediction unit, used to input a dish image into a pre-trained visual recognition model to obtain a dish category set predicted by the visual recognition model, wherein the dish category set includes M dish categories predicted by the visual recognition model that are closest to the dish image, where M is a positive integer greater than 0;

[0035] An information acquisition unit, used to acquire text information corresponding to each dish category in the dish category set;

[0036] The category determination unit inputs the dish image and each text information into the pre-trained CLIP model. The CLIP model extracts the image feature vector of the dish image and the text feature vector of each text information, calculates the cosine similarity matrix between the image feature vector and the text feature vector, and takes the dish category corresponding to the text information with the largest cosine similarity with the dish image as the target category.

[0037] A dish recognition device integrating CLIP multimodality, comprising: a memory and a processor;

[0038] The memory is used to store programs;

[0039] The processor is used to execute the program to implement the various steps of the dish recognition method integrating CLIP multimodality as described above.

[0040] It can be seen from the above technical solutions that the dish recognition method integrated with CLIP multimodality provided by the present application, after obtaining the dish image to be identified, inputs the dish image into the pre-trained visual recognition model to obtain a set of dish categories, and obtains the text information corresponding to each dish category, and then inputs the dish image and each text information into the pre-trained CLIP model. The CLIP model extracts the image feature vector of the dish image and the text feature vector of each text information, calculates the cosine similarity matrix of the image feature vector and the text feature vector, and takes the dish category corresponding to the text information with the largest cosine similarity to the dish image as the target dish category. In the present application, after the visual recognition model predicts the dish category of the dish image, the CLIP model will also perform secondary recognition based on the text information of the predicted dish category and the dish image. The CLIP model fuses multimodal information such as images and texts, which can better distinguish highly similar dishes, and further improve the accuracy of dish recognition and the versatility of the model.

[0041] This application provides a dish recognition method integrating CLIP multimodality, which uses a visual model to perform preliminary recognition of the category of dish images and reduce the recognition range. The CLIP model performs secondary recognition of dish categories in a small range to improve the model reasoning rate. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0043] Figure 1 A flow chart of a dish recognition method integrating CLIP multimodality provided in an embodiment of the present application;

[0044] Figure 2 A flow chart of determining a target dish category using a CLIP model provided in an embodiment of the present application;

[0045] Figure 3A CLIP model training flow chart provided in an embodiment of the present application;

[0046] Figure 4 A schematic diagram of the structure of a dish recognition device integrating CLIP multi-modality provided in an embodiment of the present application;

[0047] Figure 5 A hardware structure block diagram of a dish recognition device integrating CLIP multimodality provided in an embodiment of the present application. DETAILED DESCRIPTION

[0048] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0049] First, combine Figure 1 A dish recognition method integrating CLIP multimodality provided in an embodiment of the present application is introduced. Figure 1 As shown, the method includes:

[0050] Step S01, obtaining a dish image to be identified.

[0051] Specifically, the dish image of the dish to be identified can be captured by an image capture device.

[0052] Step S02: input the dish image into a pre-trained visual recognition model.

[0053] Specifically, the dish image is input into the pre-trained visual recognition model, the visual recognition model extracts the characteristic information such as color, shape, texture, etc. in the dish image, calculates the distance between the characteristic information of the dish image and the characteristic information of each dish category, sorts the distance from near to far, selects the first M dish categories with the closest distance, and obtains a dish category set, where M is a positive integer greater than 0;

[0054] Step S03, obtaining text information corresponding to each dish category in the dish category set.

[0055] Specifically, after the visual model recognizes the set of dish categories, it obtains the text information corresponding to each dish category from the pre-stored text information through the mapping relationship between dish categories and text information. The text information includes: Chinese name of the dish, dish ingredient information, dish color (red, yellow, black, white, etc.) and dish status (dry stir-fried, deep-fried, baked, wet with soup), etc.

[0056] In order for the CLIP model to better understand the input text, the vocabulary description can be prompted when entering text information. The prompt operation mainly displays a prompt dialog box. The user can enter text information according to the prompt and combine single ingredients and dish description words into sentences for input, such as: The ingredients of dish XXX (dish name) are: ingredient 1, ingredient 2,..., the color is xxx (color description), and it is a dish of type XXX (status description word).

[0057] Step S04: input the dish image and the text information into a pre-trained CLIP model, so that the CLIP model can determine the target dish category based on the dish image and the text information.

[0058] Specifically, the Chinese CLIP model is used to map the dish image and the text information of the dish to the same semantic space through contrastive learning. Therefore, after obtaining the dish image and each text information, the dish image and each text information are input into the pre-trained CLIP model. The CLIP model calculates the cosine similarity of the dish image and each text information based on the input dish image and each text information, and determines the target dish category from the dish category set according to the cosine similarity. Take M=5 as an example for introduction. For example, the text information corresponding to the top 5 dish categories closest to the visual model recognition prediction result, the dish image image and text information (text1, text2, text3, text4, text5) to be identified, are respectively input into the image encoder CLIP resnet50 and text encoder CLIP rbt3 , respectively obtain the image feature vector F of the dish image i And the text feature vector F of each text information t1 , F t2 , F t3 , F t4 , F t5 , calculate the similarity matrix between the image feature vector and each text feature vector, and arrange the cosine similarity in descending order, and select the dish category corresponding to the text information with the largest cosine similarity as the target dish category.

[0059] Next, the process of determining the target dish category based on the dish image and each text information in step S04 is introduced. Figure 2 As shown, the process includes:

[0060] Step S041, extracting the image feature vector of the dish image and the text feature vector of each text information.

[0061] Specifically, after the dish image and each text information are combined, the feature vectors in the dish image and text information are extracted respectively. The CLIP model includes an image encoder and a text encoder, wherein the backbone network of the image encoder is Resnet50; the backbone network of the text encoder is RBT3-chinese. The CLIP model extracts the image feature vector of the dish image through the trained image encoder, and extracts the text feature vector of each text information through the trained text encoder. When extracting the feature vector, the similarity of the feature vectors of the matching dish image and text information should be as large as possible, and the similarity of the feature vectors of the unmatched dish image and text information should be as small as possible. The feature vectors extracted by the image encoder and the text encoder are shown in formula (1).

[0062]

[0063] Among them, CLIP rbt3 For text encoder, CLIP resnet50 is an image encoder, Image represents the input dish image, text represents the acquired text information, and F i is the image feature vector, F i =(f i1 ,f i2 ,f i3 ,...f im ), F t is the text feature vector, F t =(f t1 ,f t2 ,f t3 ,...f tm ), f i1 、f i2 、f i3 、f im and f t1 ,f t2 ,f t3 , ...f tm Each represents a numerical value, that is, each component of the eigenvector. m represents the dimension of the eigenvector, that is, how many numbers the eigenvector consists of. m can be set, and m is generally a multiple of 8, such as: m = 512.

[0064] Step S042, calculating the cosine similarity matrix between the image feature vector and the text feature vector.

[0065] Specifically, the cosine similarity is calculated by calculating the image feature vector and the text feature vector using the following formula (2).

[0066]

[0067] Among them, f iz and ftz Indicates F i and F t The zth component of .

[0068] According to the cosine similarity between the dish image and each text information, a cosine similarity matrix as shown in formula (3) is generated.

[0069]

[0070] Among them, similarity nn Represents the cosine similarity between the nth dish image and the nth text information.

[0071] Step S043: taking the dish category corresponding to the text information having the largest cosine similarity with the dish image as the target dish category.

[0072] Specifically, the text information corresponds to the dish category one by one, and the dish category with the largest cosine similarity is used as the target dish category corresponding to the dish image.

[0073] The dish recognition method integrating CLIP multimodality provided in the embodiment of the present application, after obtaining the dish image to be recognized, inputs the dish image into the pre-trained visual recognition model to obtain a set of dish categories, obtains the text information corresponding to each dish category, and then inputs the dish image and each text information into the pre-trained CLIP model. The CLIP model calculates the cosine similarity matrix of the image feature vector and the text feature vector based on the image feature vector extracted from the dish image and the text feature vector of each text information, and takes the dish category corresponding to the text information with the largest cosine similarity to the dish image as the target dish category. In the present application, after the visual recognition model predicts the dish category of the dish image, the CLIP model will also perform secondary recognition based on the text information of the predicted dish category and the dish image. The CLIP model fuses multimodal information such as image and text, which can better distinguish highly similar dishes, and further improve the accuracy of dish recognition and the universality of the model.

[0074] The present application embodiment provides a dish recognition method integrating CLIP multimodality, which performs preliminary recognition of the category of dish images through a visual model to reduce the recognition range. The CLIP model performs secondary recognition of dish categories within a small range to improve the model reasoning rate.

[0075] The present application embodiment introduces the training process of the CLIP model. The image encoder and text encoder in the CLIP model in the present application embodiment are trained in two stages. Figure 3 As shown, the training process includes:

[0076] Step S11, obtaining training images and training texts.

[0077] Specifically, training images and training texts are collected from the system, and the collected data is cleaned.

[0078] Step S12, freezing the network layer Resnet50 of the image encoder.

[0079] Specifically, the first stage frozen image encoder CLIP resnet50 The network layer only trains the text encoder CLIP rbt3 .

[0080] Step S13, input the training image and training text into the CLIP model, and the text encoder and image encoder extract the text feature vector and image feature vector respectively.

[0081] Specifically, the training images and training texts are respectively input into the image encoder and text encoder in the CLIP model, and the image feature vectors in each training image and the text feature vectors in each training text are extracted as shown in the aforementioned formula (1).

[0082] Step S14: predicting a training similarity matrix between the training image and the training text according to the image feature vector and the text feature vector.

[0083] Specifically, the cosine similarity between each training image and each training text is calculated based on the above formula (2) according to the image feature vector of each training image and the feature vector in each training text.

[0084] According to the cosine similarity between each training image and each training text, a training similarity matrix as shown in formula (3) is generated.

[0085] Step S15, calculating the loss function of the CLIP model according to the training similarity matrix and the label similarity matrix.

[0086] Specifically, the loss function when optimizing the CLIP model can use the cross entropy loss function, and the loss function is shown in the following formula (4).

[0087]

[0088] Among them, N represents the number of training images and training texts input for one training. The training images and training texts correspond one to one. N = number of training images = number of training texts. labels is the label similarity matrix between the training images and the corresponding training texts. i represents the data of the i-th row of the label similarity matrix, p is the training similarity matrix of the predicted training image and training text, p i is the data of the i-th row of the training similarity matrix.

[0089] The label similarity matrix labels is a cosine similarity matrix representing the dish image and the corresponding text. As shown in formula (5), labels is an N×N diagonal matrix with a diagonal of 1.

[0090]

[0091] Among them, the cosine similarity between the dish image and the matching text information is 1, and the similarity between the dish image and the unmatched text is 0. The larger the value, the more similar it is, and the maximum value is 1.

[0092] Step S16, optimizing the parameters of the CLIP model based on the loss function until the preset training end condition is met.

[0093] Specifically, the parameters of the CLIP model include text encoder parameters and image encoder parameters. In the first stage, the text encoder training process is to optimize the parameters of the text encoder based on the loss function. In the second stage, the image encoder training process is to optimize the parameters of the image encoder based on the loss function.

[0094] Step S17, freezing the network layer RBT3-chinese of the trained text encoder.

[0095] Specifically, the second stage trains the image encoder and uses the trained text encoder CLIP rbt3 The network layer of the image encoder is frozen and the image encoder is trained. The training process of the image encoder is basically the same as the steps in the text encoder training process. The specific process refers to steps S13-S16, except that the second stage only optimizes the parameters of the image encoder in the CLIP model.

[0096] After two stages of training, the trained CLIP model is obtained.

[0097] The embodiment of the present application trains the CLIP model. The trained CLIP model can extract image feature vectors and text feature vectors, fuse multimodal information such as images and texts, perform secondary recognition of dish categories, and achieve more comprehensive and accurate dish recognition.

[0098] The dish recognition device integrating CLIP multimodality provided in an embodiment of the present application is described below. The dish recognition device integrating CLIP multimodality described below and the dish recognition method integrating CLIP multimodality described above can be referenced to each other.

[0099] First combine Figure 4 , the dish recognition device integrating CLIP multimodality is introduced, such as Figure 4 As shown, the dish recognition device integrating CLIP multimodality may include:

[0100] An image acquisition unit 100 is used to acquire an image of a dish to be identified;

[0101] The category prediction unit 200 is used to input the dish image into the pre-trained visual recognition model to obtain a dish category set predicted by the visual recognition model, wherein the dish category set includes M dish categories predicted by the visual recognition model that are closest to the dish image, where M is a positive integer greater than 0;

[0102] The information acquisition unit 300 is used to acquire text information corresponding to each dish category in the dish category set;

[0103] The category determination unit 400 inputs the dish image and each text information into the pre-trained CLIP model. The CLIP model extracts the image feature vector of the dish image and the text feature vector of each text information, calculates the cosine similarity matrix between the image feature vector and the text feature vector, and takes the dish category corresponding to the text information with the largest cosine similarity with the dish image as the target category.

[0104] The dish recognition device integrating CLIP multi-modality provided in the embodiment of the present application can be applied to the dish recognition equipment integrating CLIP multi-modality. Figure 5 The hardware structure diagram of the dish recognition device integrating CLIP multi-modality is shown. Figure 5 , the hardware structure of the device may include: at least one processor 1, at least one communication interface 2, at least one memory 3 and at least one communication bus 4;

[0105] In the embodiment of the present application, the number of the processor 1, the communication interface 2, the memory 3, and the communication bus 4 is at least one, and the processor 1, the communication interface 2, and the memory 3 communicate with each other through the communication bus 4;

[0106] The processor 1 may be a central processing unit CPU, or an application-specific integrated circuit ASIC (Application Specific Integrated Circuit), or one or more integrated circuits configured to implement the embodiments of the present invention, etc.;

[0107] The memory 3 may include a high-speed RAM memory, and may also include a non-volatile memory, such as at least one disk memory;

[0108] Among them, the memory stores a program, and the processor can call the program stored in the memory, and the program is used to implement each processing flow in the aforementioned dish recognition solution that integrates CLIP multimodality.

[0109] An embodiment of the present application also provides a storage medium, which can store a program suitable for execution by a processor, and the program is used to implement each processing flow in the aforementioned dish recognition solution that integrates CLIP multimodality.

[0110] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are merely used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations.

[0111] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A dish recognition method integrating CLIP multimodality, characterized in that: include: Obtaining a dish image to be identified; Inputting the dish image into a pre-trained visual recognition model to obtain a set of dish categories predicted by the visual recognition model, wherein the set of dish categories includes M dish categories predicted by the visual recognition model that are closest to the dish image, where M is a positive integer greater than 0; Get the text information corresponding to each dish category in the dish category set; Inputting the dish image and each text information into the pre-trained CLIP model, so that the CLIP model determines the target dish category based on the dish image and each text information; The process of the CLIP model determining the target dish category based on the dish image and text information includes: Extracting image feature vectors of dish images and text feature vectors of each text information; Calculate the cosine similarity matrix between the image feature vector and the text feature vector; The dish category corresponding to the text information with the largest cosine similarity with the dish image is taken as the target dish category.

2. The dish recognition method integrating CLIP multimodality according to claim 1, characterized in that: CLIP model, including: image encoder and text encoder; The backbone network of the image encoder is Resnet50; The backbone network of the text encoder is RBT3-chinese.

3. The dish recognition method integrating CLIP multimodality according to claim 2 is characterized in that: The training process of the CLIP model includes: Get training images and training texts; Freeze the network layer Resnet50 of the image encoder; The training text and training pictures are input into the CLIP model, and the text encoder and image encoder extract the text feature vector and image feature vector respectively; Predicting a training similarity matrix between training images and training texts based on the image feature vector and the text feature vector; Calculate the loss function of the CLIP model based on the training similarity matrix and the label similarity matrix; Optimize the parameters of the CLIP model based on the loss function until the preset training end conditions are met; Freeze the network layer RBT3-chinese of the trained text encoder, and return to execute the above training process of inputting the training text and training pictures into the CLIP model, calculating the loss function of the CLIP model according to the text feature vector and image feature vector extracted by the CLIP model, and optimizing the parameters of the CLIP model based on the loss function.

4. The dish recognition method integrating CLIP multimodality according to claim 3 is characterized in that: The loss function is: Among them, N represents the number of training images and training texts input for one training. The training images and training texts correspond one to one. N = number of training images = number of training texts. labels is the label similarity matrix between the training images and the corresponding training texts. i represents the data of the i-th row of the label similarity matrix, p is the training similarity matrix of the predicted training image and training text, p i is the data of the i-th row of the training similarity matrix.

5. The dish recognition method integrating CLIP multimodality according to any one of claims 1 to 4, characterized in that: The formula for calculating cosine similarity based on image feature vector and text feature vector is: Among them, F i is the image feature vector, F i =(f i1 ,f i2 ,f i3 ,...f im ), F t is the text feature vector, F t =(f t1 ,f t2 ,f t3 ,...f tm ), m is the dimension of the feature vector, f iz and f tz Indicates F i and F t The zth component of .

6. A dish recognition device integrating CLIP multimodality, characterized in that: include: An image acquisition unit, used to acquire an image of a dish to be identified; A category prediction unit, used to input a dish image into a pre-trained visual recognition model to obtain a dish category set predicted by the visual recognition model, wherein the dish category set includes M dish categories predicted by the visual recognition model that are closest to the dish image, where M is a positive integer greater than 0; An information acquisition unit, used to acquire text information corresponding to each dish category in the dish category set; The category determination unit inputs the dish image and each text information into the pre-trained CLIP model. The CLIP model extracts the image feature vector of the dish image and the text feature vector of each text information, calculates the cosine similarity matrix between the image feature vector and the text feature vector, and takes the dish category corresponding to the text information with the largest cosine similarity with the dish image as the target category.

7. A dish recognition device integrating CLIP multi-modality, characterized in that: include: Memory and processor; The memory is used to store programs; The processor is used to execute the program to implement the various steps of the dish recognition method integrating CLIP multimodality as described in any one of claims 1 to 5.

Citation Information

Patent Citations

  • Method and apparatus for identifying dishes

    CN108256474A

  • Dish identification method and device based on metric learning

    CN111274972A

  • Image classification method and device, computer equipment and medium

    CN114741581A

  • Multi-modal image retrieval method and system based on self-training Chinese CLIP model

    CN117786155A

  • Semantic segmentation model training method and apparatus, and electronic device and storage medium

    WO2024012251A1

Cited By

  • Image depth clustering method and system based on multi-modal large model guidance

    CN119888293A

  • Model training method, electronic equipment and computer readable storage medium

    CN120336859A

  • Construction and application method and system of multi-modal fusion food identification large model

    CN120524371A

  • Asset statistical method and device, medium and equipment

    CN120893424A