Zero-sample food image detection method based on visual semantic bidirectional guidance

Abstract

The invention discloses a zero-sample food image detection method based on visual semantic bidirectional guidance, which belongs to the technical field of image data processing and generation, takes DETR based on Transform as a detection framework, and fuses a food semantic enhancement network and a cross-modal bidirectional guidance module. Firstly, an image is input to an image backbone network branch to extract visual features, a category label is input to a food semantic enhancement network branch to extract semantic features, then the two obtained features are sent to a cross-modal two-way guide fusion module, the module comprises a two-way guide encoder, a two-way guide query vector and a two-way guide decoder, and the cross-modal two-way guide fusion module comprises a two-way guide query vector and a two-way guide decoder. Visual and semantic features are aligned and fused through coding and decoding processes, and finally, regression and classification tasks of zero sample food image detection are completed in a full connection layer. According to the method, detailed food attributes and visual features can be captured in the detection process, effective interaction of visual semantic modals is achieved, and therefore the detection precision of unseen food categories is improved.

Description

Technical Field

[0001] The invention relates to a zero-sample food image detection method based on visual semantic bidirectional guidance, and belongs to the technical field of image data processing and generation. Background Art

[0002] As an emerging multidisciplinary cross-field, food computing has gradually become the focus of researchers due to its broad application prospects. Food image detection, as a core task in food computing, aims to simultaneously locate and identify food objects, and has shown great potential in many practical applications such as food recommendation, dietary management and health testing.

[0003] In real-world scenarios, food is constantly changing in various multimedia data, and new food categories will emerge. Faced with such uncontrollable factors, food detectors that can only identify fixed categories often show poor results when dealing with new categories. In addition, since deep learning methods are data-driven, they rely on a large number of labeled training samples, which also means a lot of manpower and time investment. In this case, it is unrealistic to continuously collect and annotate new food objects. Therefore, food detection in real-world scenarios requires the ability of zero-shot detection to detect new food categories that have no samples during training.

[0004] Zero-shot food image detection aims to simultaneously identify and locate food categories that do not appear in the training data. Existing methods are mainly based on generative methods, that is, on the Faster R-CNN target detection framework, the semantic features of the corresponding category are input into the generative model to synthesize visual features, and then the features of the seen categories and the synthesized features of the unseen categories are input into the target detection framework for supervised learning. These generative methods based on Faster R-CNN have application defects in zero-shot food image detection tasks. On the one hand, the two-stage design of Faster R-CNN limits the deep integration with the semantic module, which is not conducive to multimodal semantic tasks. Specifically, it relies too much on the region proposal network to generate candidate regions, which makes it difficult to capture the detailed features in food images, and is prone to positioning errors when processing scenes with high inter-class similarity in food images. On the other hand, the generative method relies on a multi-step training process and cannot achieve end-to-end detection. The whole process is extremely complex and time-consuming.

[0005] Food is different from other items with rigid shapes and structures. Its visual features are affected by a variety of complex attributes such as cooking methods and types of ingredients. There are similarities between different categories and subtle differences between the same type of food. However, existing methods usually directly input category labels into a pre-trained text encoder and generate semantic features through a single prompt template. This design makes it difficult to distinguish the same type of food that exhibits different visual features due to complex attributes and fine-grained feature differences. In addition, after generating semantic features, existing methods often fail to fully explore and integrate the features of both visual and semantic modalities, limiting the performance of the model in food detection tasks. Summary of the invention

[0006] The purpose of the present invention is to overcome the deficiencies of the above-mentioned prior art and provide a zero-sample food image detection method based on visual semantics bidirectional guidance.

[0007] The technical solution provided by the present invention is as follows: a zero-sample food image detection method based on visual semantic bidirectional guidance, characterized in that it comprises the following steps: Step 1: Divide the data set into training set and test set; Step 2: Input the training set images into the image backbone network branch to extract visual features; Step 3: Input the training set category labels into the food semantic enhancement network branch to extract semantic features; Step 4: The obtained visual features and semantic features are sent to the cross-modal bidirectional guided fusion module for fusing image and text features, and the visual features and semantic features are aligned and fused through the encoding and decoding process; Step 5: Complete the regression and classification tasks of zero-shot food image detection in the fully connected layer; Step 6: Optimize the model by minimizing the loss function; Step 7: Test the performance of narrow zero-shot food image detection and broad zero-shot food image detection.

[0008] Furthermore, in step 1, the UECFOOD-256 and ZSFooD datasets are used, and the datasets consisting of the seen categories are used as training sets. , which includes images of seen categories and its corresponding seen category labels and bounding box coordinates, and use the dataset consisting of seen categories and unseen categories as the test set , which includes images of seen and unseen categories and its corresponding category label and bounding box coordinates.

[0009] Furthermore, in step 2, the training set Seen category images in Input image backbone network branch Extract image features and obtain visual features : .

[0010] Furthermore, in step 3, the image of the category already seen is Corresponding seen category labels Input food semantics enhancement network branch to extract labels Semantic features of : It includes the following steps: Step 31: Label the categories Input into OpenAI's GPT-3 API and give each category label Generate M semantically suggestive descriptions , ; Step 32: Text Encoder Using CLIP , a semantically suggestive description Encode and average these encodings for each seen category label Generate a 512-dimensional semantic feature : .

[0011] Furthermore, the step 4 comprises the following steps: Step 41: Obtain visual features and semantic features Bidirectional guided encoder input into the cross-modal bidirectional guided fusion module; It includes the following steps: Step 411: Use deformable self-attention mechanism to process visual features And get enhanced visual features : ; in, It is a deformable self-attention mechanism; Step 412: Use self-attention mechanism to process semantic features And get enhanced semantic features : ; in, Represents the self-attention mechanism; Step 413: Enhance visual features Enhanced semantic features A fusion of two characteristics; Will enhance visual features As a query vector , enhance semantic features As a key in the attention mechanism and The value vector is input into the cross-attention mechanism to obtain the feature embedding from vision to semantics : ; in, represents the cross attention mechanism; The semantic features will be enhanced As a query vector , enhance visual features As a key in the attention mechanism Sum vector and input it into the cross-attention mechanism to obtain semantic-to-visual feature embedding : ; in, represents the cross attention mechanism; Step 414: Embedding visual features into semantic features and semantic-to-visual feature embedding Perform weighted updates to adjust the influence of visual features and semantic features in the output, and finally obtain integrated visual features and integrate semantic features : ; ; in, , is a trainable weight parameter; Step 42: Object Query Vector and integrating visual features , integrating semantic features Fusion, to filter out the previous object query vector, i.e., a bidirectional guided query vector ; It includes the following steps: Step 421: Integrate visual features and integrate semantic features , the similarity matrix in the feature space is obtained through dot product operation ,in, represents the batch size for training, Represents integrated visual features The number of units, Representation integration semantic features The number of units, Represents its characteristic dimension: ； Indicates Among the samples, The visual feature unit and The similarity value between semantic feature units; Indicates Among the samples, visual feature units in the feature dimension The characteristic value of No. Among the samples, Semantic feature units in feature dimension The characteristic value of Step 422: Take the maximum similarity value between each visual feature unit and all semantic feature units: ; Step 423: For each sample visual feature unit, based on the maximum similarity value Sort in descending order and select the top most relevant semantic feature units. The index of the visual feature unit , the selection process can be expressed as: ; in, Indicates descending sorting; Step 424: Using the obtained Visual feature unit index , query vector from object Filter to get the previous object query vector, i.e., a bidirectional guided query vector ; Step 43: Bidirectionally guide the query vector Input into the bidirectional guided decoder of the cross-modal feature fusion module; It includes the following steps: Step 431: Bidirectionally guide the query vector Feed it into the self-attention mechanism to compute the embedding of the bidirectional guided query vector : ; in, Represents the self-attention mechanism; Step 432: Embed As a query vector , integrating visual features As a key in the attention mechanism Sum vector and feed it into the criss-cross attention mechanism to get the criss-cross embedding : ; in, represents the cross attention mechanism; Step 433: Embed the cross As a query vector , integrating semantic features As a key in the attention mechanism Sum vector, and input it into the cross-attention mechanism to obtain the final feature , to capture category-level contextual information: ; in, represents the cross attention mechanism; Step 434: Final Features Input into the fully connected layer of the bidirectional guided decoder and obtain preliminary results : ; in, represents a fully connected layer.

[0012] Furthermore, the step 5 comprises the following steps: Step 51: Replace the last classification layer in DETR with a fully connected layer to convert the preliminary results Mapping to semantic features The semantic feature space of : ; Step 52: Use cosine similarity to measure embedding features and semantic features The distance between them and apply Function to obtain the predicted classification score : ; Step 53: Use the fully connected layer to get the predicted position coordinates : ; Step 54: Final set of predictions for: .

[0013] Furthermore, in step 6, the Hungarian algorithm is used to find the prediction set and the real set The only match between them, that is, find the match with the minimum matching loss : ; in, Indicates True value, Indicates Prediction values, For all possible matches, It is True value and Predicted values The matching loss between At the same time, multiple loss items are considered, including classification loss , regression loss ; Among them, the classification loss Using focal loss, regression loss By IoU loss ( loss) and generalized IoU loss Composition, the matching loss function is formalized as: + ; in, Represents the classification loss The hyperparameters of It is True value and Predicted values The classification loss between Represents IoU loss The hyperparameters of Indicates True value and Predicted values The IoU loss between Denotes the generalized IoU loss The hyperparameters of Indicates True value and Predicted values The generalized IoU loss between .

[0014] Furthermore, the step 7 comprises the following steps: Step 71: Extract the dataset under ZSFD task settings The category label is The images and corresponding labels are input into the process of steps 2-5 to evaluate the detection performance; Step 72: Under the GZSFD task setting, set the dataset The category label is The images and corresponding labels are input into the process of steps 2-5 to evaluate the detection performance.

[0015] The beneficial effects of the present invention are as follows: the present invention solves the limitations of existing zero-sample food image detection methods. Based on the Transformer-based detection framework, by introducing the food semantic enhancement network branch and the cross-modal bidirectional guided fusion module, the semantic features and visual features of food can be accurately captured during the detection process, and the effective interaction between semantic and visual modalities can be achieved, thereby improving the detection accuracy of unseen food categories and reducing bias, and enhancing the generalization performance of the model, as follows: 1. This invention solves the application defect problem of generative methods based on Faster R-CNN in zero-shot food image detection tasks. A solution based on the DETR framework is proposed. This method avoids excessive reliance on the region proposal network through an end-to-end detection mechanism, thereby effectively reducing the risk of positioning errors. Compared with the traditional multi-step training process, this method simplifies the training process of the model, significantly improves the performance and multimodal adaptability of the model in complex visual scenes, and provides an efficient and accurate solution for zero-shot food image detection.

[0016] 2. In view of the deficiencies in the generation of semantic vectors caused by the complex attributes and fine-grained problems of food, the present invention proposes a food semantic enhancement network branch that makes full use of a large language model to generate high-quality language descriptions of food categories. Compared with traditional designs that rely on a single prompt template, the present invention can generate higher-quality and more distinctive language descriptions of food categories, and comprehensively capture the complex attributes and fine-grained differences of food visual features.

[0017] 3. The cross-modal bidirectional guided fusion module proposed in the present invention includes a bidirectional guided encoder, a bidirectional guided query vector and a bidirectional guided decoder. Through the bidirectional guided fusion of visual semantics, the deep interaction between semantic features and visual features is strengthened. Semantic features provide context clues for visual features, while visual features inject instantiation details into semantic information, thereby achieving efficient fusion of the two modalities, effectively improving the detection robustness and accuracy of unseen categories, and is particularly suitable for fine-grained recognition scenarios of food images. DETAILED DESCRIPTION

[0018] The specific embodiments of the present invention are further described in detail below: A zero-shot food image detection method based on visual semantic bidirectional guidance, which uses Transformer-based DETR as the target detection model framework, and integrates the food semantic enhancement network branch based on the large model and the cross-modal bidirectional guidance fusion module. First, the image is input into the image backbone network branch to extract image features, and the category label is input into the food semantic enhancement network branch to extract semantic features. Then, the obtained visual features and semantic features are sent to the cross-modal bidirectional guidance fusion module for fusing image and text features. The cross-modal bidirectional guidance fusion module includes a bidirectional guidance encoder, a bidirectional guidance query vector and a bidirectional guidance decoder to align and fuse the visual features and semantic features through the encoding and decoding process. Finally, the regression and classification tasks of zero-shot food image detection are completed in the fully connected layer.

[0019] The specific steps include: Step 1: Use the UECFOOD-256 and ZSFooD datasets and the datasets consisting of the seen categories as training sets , which includes images of seen categories and its corresponding seen category labels and bounding box coordinates, and use the dataset consisting of seen categories and unseen categories as the test set , which includes images of seen and unseen categories and its corresponding category label and bounding box coordinates.

[0020] Step 2: The training set Seen category images in Input image backbone network branch Extract image features and obtain visual features : .

[0021] Step 3: Convert the images of the seen categories Corresponding seen category labels Input food semantics enhancement network branch to extract labels Semantic features of : It includes the following steps: Step 31: Label the categories Input into OpenAI's GPT-3 API and give each category label Generate M semantically suggestive descriptions , ; Step 32: Text Encoder Using CLIP , a semantically suggestive description Encode and average these encodings for each seen category label Generate a 512-dimensional semantic feature : .

[0022] Step 4: The obtained visual features and semantic features are sent to the cross-modal bidirectional guided fusion module for fusing image and text features, and the visual features and semantic features are aligned and fused through the encoding and decoding process; It includes the following steps: Step 41: Obtain visual features and semantic features Bidirectional guided encoder input into the cross-modal bidirectional guided fusion module.

[0023] It includes the following steps: Step 411: Use deformable self-attention mechanism to process visual features And get enhanced visual features : ; in, It is a deformable self-attention mechanism.

[0024] Step 412: Use self-attention mechanism to process semantic features And get enhanced semantic features : ; in, Represents the self-attention mechanism.

[0025] Step 413: Enhance visual features Enhanced semantic features A fusion of two characteristics.

[0026] Will enhance visual features As a query vector , enhance semantic features As a key in the attention mechanism and The value vector is input into the cross-attention mechanism to obtain the feature embedding from vision to semantics : ; in, Represents the cross attention mechanism.

[0027] The semantic features will be enhanced As a query vector , enhance visual features As a key in the attention mechanism Sum vector and input it into the cross-attention mechanism to obtain semantic-to-visual feature embedding : ; in, Represents the cross attention mechanism.

[0028] Step 414: Embedding visual features into semantic features and semantic-to-visual feature embedding Perform weighted updates to adjust the influence of visual features and semantic features in the output, and finally obtain integrated visual features and integrate semantic features : ; ; in, , are trainable weight parameters.

[0029] Step 42: The invention of this paper is based on the Transformer DETR as the target detection model framework. The detection process can be regarded as a set prediction problem. According to the learnable object query vector , directly infer a fixed size of The prediction set . The object query vector and integrating visual features , integrating semantic features Fusion, to filter out the previous object query vector, i.e., a bidirectional guided query vector .

[0030] It includes the following steps: Step 421: Integrate visual features and integrate semantic features , the similarity matrix in the feature space is obtained through dot product operation ,in, represents the batch size for training, Represents integrated visual features The number of units, Representation integration semantic features The number of units, Represents its characteristic dimension: ； Indicates Among the samples, The visual feature unit and The similarity value between semantic feature units. Indicates Among the samples, visual feature units in the feature dimension The characteristic value of No. Among the samples, Semantic feature units in feature dimension The characteristic value of .

[0031] Step 422: Take the maximum similarity value between each visual feature unit and all semantic feature units: ; Step 423: For each sample visual feature unit, based on the maximum similarity value Sort in descending order and select the top most relevant semantic feature units. The index of the visual feature unit , the selection process can be expressed as: ; in, Indicates descending sort order.

[0032] Step 424: Using the obtained Visual feature unit index , query vector from object Filter to get the previous object query vector, i.e., a bidirectional guided query vector ; Step 43: Bidirectionally guide the query vector Input into the bidirectional guided decoder of the cross-modal feature fusion module.

[0033] It includes the following steps: Step 431: Bidirectionally guide the query vector Feed it into the self-attention mechanism to compute the embedding of the bidirectional guided query vector : ; in, Represents the self-attention mechanism.

[0034] Step 432: Embed As a query vector , integrating visual features As a key in the attention mechanism Sum vector and feed it into the criss-cross attention mechanism to get the criss-cross embedding : ; in, Represents the cross attention mechanism.

[0035] Step 433: Embed the cross As a query vector , integrating semantic features As a key in the attention mechanism Sum vector, and input it into the cross-attention mechanism to obtain the final feature , to capture category-level contextual information: ; in, Represents the cross attention mechanism.

[0036] Step 434: Final Features Input into the fully connected layer of the bidirectional guided decoder and obtain preliminary results : ; in, represents a fully connected layer.

[0037] Step 5: Complete the regression and classification tasks of zero-shot food image detection in the fully connected layers.

[0038] It includes the following steps:

[0039] Step 51: To apply DETR to zero-shot food image detection, replace the last classification layer in DETR with a fully connected layer to convert the preliminary results Mapping to semantic features The semantic feature space of : ; Step 52: Use cosine similarity to measure embedding features and semantic features The distance between them and apply Function to obtain the predicted classification score : ; Step 53: Use the fully connected layer to get the predicted position coordinates : ; Step 54: Final set of predictions for: .

[0040] Step 6: Optimize the model by minimizing the loss function; Finding a set of predictions using the Hungarian algorithm and the real set The only match between them, that is, find the match with the minimum matching loss : ; in, Indicates True value, Indicates Prediction values, For all possible matches, It is True value and Predicted values The matching loss between At the same time, multiple loss items are considered, including classification loss , regression loss .

[0041] Among them, the classification loss Using focal loss, regression loss By IoU loss ( loss) and generalized IoU loss Composition, the matching loss function is formalized as: + ; in, Represents the classification loss The hyperparameters of It is True value and Predicted values The classification loss between Represents IoU loss The hyperparameters of Indicates True value and Predicted values The IoU loss between Denotes the generalized IoU loss The hyperparameters of Indicates True value and Predicted values The generalized IoU loss between .

[0042] Step 7: The testing phase is divided into two settings, namely narrow zero-shot food image detection (ZSFD) and generalized zero-shot food image detection (GZSFD). In the testing phase, ZSFD only needs to evaluate the unseen categories, while GZSFD needs to evaluate both seen and unseen categories.

[0043] It includes the following steps: Step 71: Extract the dataset under ZSFD task settings The category label is The images and corresponding labels are input into the process of steps 2-5 to evaluate the detection performance.

[0044] Step 72: Under the GZSFD task setting, set the dataset The category label is The images and corresponding labels are input into the process of steps 2-5 to evaluate the detection performance.

[0045] The present invention uses two datasets, UECFOOD-256 and ZSFooD, as the average indicators of model performance. UECFOOD-256 has 205 seen categories and 51 unseen categories, and ZSFooD has 184 seen categories and 44 unseen categories. The specific number of pictures is shown in Table 1.

[0046] Table 1

[0047] The Transformer-based DETR detection framework used in the present invention is specifically its variant model DINO. In addition, the image backbone network branch in step 2 The ResNet-50 network is used, and the specific model of the CLIP text encoder used in the food semantic enhancement network branch in step 32 is Vit-B / 32. , , The values ​​are set to 1.0, 5.0, and 2.0 respectively.

[0048] In the test evaluation of step 7, the average detection accuracy AP at IoU = 0.5 is used as the evaluation indicator in both task settings.

[0049] In order to intuitively understand the impact of the two components proposed in this invention on the detection performance, we conducted experiments under UECFOOD-256 and ZSFooD, testing the average detection accuracy AP of unseen categories under the ZSFD task setting, and the average detection accuracy AP of seen and unseen categories under the GZSFD task setting. In particular, the GZSD setting is more challenging and realistic than the ZSD setting, and the harmonic mean is the key indicator for GZSD performance: ; As shown in Table 2, S represents the seen category, U represents the unseen category, and HM represents the harmonic mean of the seen and unseen categories.

[0050] Table 2

[0051] It should be understood that the parts not elaborated in detail in this specification belong to the prior art. The above embodiments are only descriptions of the preferred implementation methods of the present invention, and are not intended to limit the scope of the present invention. Without departing from the design spirit of the present invention, various modifications and improvements made by ordinary engineers and technicians in this field to the technical solution of the present invention should fall within the protection scope determined by the claims of the present invention.

Claims

1. A zero-shot food image detection method based on visual semantic bidirectional guidance, characterized in that: It includes the following steps: Step 1: Divide the data set into training set and test set; Step 2: Input the training set images into the image backbone network branch to extract visual features; Step 3: Input the training set category labels into the food semantic enhancement network branch to extract semantic features; Step 4: The obtained visual features and semantic features are sent to the cross-modal bidirectional guided fusion module for fusing image and text features, and the visual features and semantic features are aligned and fused through the encoding and decoding process; Step 5: Complete the regression and classification tasks of zero-shot food image detection in the fully connected layer; Step 6: Optimize the model by minimizing the loss function; Step 7: Test the performance of narrow zero-shot food image detection and broad zero-shot food image detection.

2. The zero-shot food image detection method based on visual semantic bidirectional guidance as claimed in claim 1, characterized in that: In step 1, the UECFOOD-256 and ZSFooD datasets are used as training sets. , which includes images of seen categories and its corresponding seen category labels and bounding box coordinates, and use the dataset consisting of seen categories and unseen categories as the test set , which includes images of seen and unseen categories and its corresponding category label and bounding box coordinates.

3. The zero-shot food image detection method based on visual semantic bidirectional guidance as claimed in claim 1, characterized in that: In step 2, the training set Seen category images in Input image backbone network branch Extract image features and obtain visual features : 。 4. The zero-shot food image detection method based on visual semantic bidirectional guidance as claimed in claim 1, characterized in that: In step 3, the image of the category already seen Corresponding seen category labels Input food semantics enhancement network branch to extract labels Semantic features of : It includes the following steps: Step 31: Label the categories Input into OpenAI's GPT-3 API and give each category label Generate M semantically suggestive descriptions , ; Step 32: Text Encoder Using CLIP , a semantically suggestive description Encode and average these encodings for each seen category label Generate a 512-dimensional semantic feature : 。 5. The zero-shot food image detection method based on visual semantic bidirectional guidance as claimed in claim 1, characterized in that: The step 4 comprises the following steps: Step 41: Obtain visual features and semantic features Bidirectional guided encoder input into the cross-modal bidirectional guided fusion module; It includes the following steps: Step 411: Use deformable self-attention mechanism to process visual features And get enhanced visual features : ； in, It is a deformable self-attention mechanism; Step 412: Use self-attention mechanism to process semantic features And get enhanced semantic features : ； in, Represents the self-attention mechanism; Step 413: Enhance visual features Enhanced semantic features A fusion of two characteristics; Will enhance visual features As a query vector , enhance semantic features As a key in the attention mechanism and The value vector is input into the cross-attention mechanism to obtain the feature embedding from vision to semantics : ； in, represents the cross attention mechanism; The semantic features will be enhanced As a query vector , enhance visual features As a key in the attention mechanism Sum vector and input it into the cross-attention mechanism to obtain semantic-to-visual feature embedding : ； in, represents the cross attention mechanism; Step 414: Embedding visual features into semantic features and semantic-to-visual feature embedding Perform weighted updates to adjust the influence of visual features and semantic features in the output, and finally obtain integrated visual features and integrate semantic features : ； ； in, , is a trainable weight parameter; Step 42: Object Query Vector and integrating visual features , integrating semantic features Fusion, to filter out the previous object query vector, i.e., a bidirectional guided query vector ; It includes the following steps: Step 421: Integrate visual features and integrate semantic features , the similarity matrix in the feature space is obtained through dot product operation ,in, represents the batch size for training, Represents integrated visual features The number of units, Representation integration semantic features The number of units, Represents its characteristic dimension: ； Indicates Among the samples, The visual feature unit and The similarity value between semantic feature units; Indicates Among the samples, visual feature units in the feature dimension The characteristic value of No. Among the samples, Semantic feature units in feature dimension The characteristic value of Step 422: Take the maximum similarity value between each visual feature unit and all semantic feature units: ； Step 423: For each sample visual feature unit, based on the maximum similarity value Sort in descending order and select the top most relevant semantic feature units. The index of the visual feature unit , the selection process can be expressed as: ； in, Indicates descending sorting; Step 424: Using the obtained Visual feature unit index , query vector from object Filter to get the previous object query vector, i.e., a bidirectional guided query vector ; Step 43: Bidirectionally guide the query vector Input into the bidirectional guided decoder of the cross-modal feature fusion module; It includes the following steps: Step 431: Bidirectionally guide the query vector Feed it into the self-attention mechanism to compute the embedding of the bidirectional guided query vector : ； in, Represents the self-attention mechanism; Step 432: Embed As a query vector , integrating visual features As a key in the attention mechanism Sum vector and feed it into the criss-cross attention mechanism to get the criss-cross embedding : ； in, represents the cross attention mechanism; Step 433: Embed the cross As a query vector , integrating semantic features As a key in the attention mechanism Sum vector, and input it into the cross-attention mechanism to obtain the final feature , to capture category-level contextual information: ； in, represents the cross attention mechanism; Step 434: Final Features Input into the fully connected layer of the bidirectional guided decoder and obtain preliminary results : ； in, represents a fully connected layer.

6. The zero-shot food image detection method based on visual semantic bidirectional guidance as claimed in claim 1, characterized in that: The step 5 comprises the following steps: Step 51: Replace the last classification layer in DETR with a fully connected layer to convert the preliminary results Mapping to semantic features The semantic feature space of : ； Step 52: Use cosine similarity to measure embedding features and semantic features The distance between them and apply Function to obtain the predicted classification score : ； Step 53: Use the fully connected layer to get the predicted position coordinates : ； Step 54: Final set of predictions for: 。 7. The zero-shot food image detection method based on visual semantic bidirectional guidance as claimed in claim 1, characterized in that: The Hungarian algorithm is used in step 6 to find the prediction set and the real set The only match between them, that is, find the match with the minimum matching loss : ； in, Indicates True value, Indicates Prediction values, For all possible matches, It is True value and Predicted values The matching loss between At the same time, multiple loss items are considered, including classification loss , regression loss ; Among them, the classification loss Using focal loss, regression loss By IoU loss ( loss) and generalized IoU loss Composition, the matching loss function is formalized as: + ； in, Represents the classification loss The hyperparameters of It is True value and Predicted values The classification loss between Represents IoU loss The hyperparameters of Indicates True value and Predicted values The IoU loss between Denotes the generalized IoU loss The hyperparameters of Indicates True value and Predicted values The generalized IoU loss between .

8. The zero-shot food image detection method based on visual semantic bidirectional guidance as claimed in claim 1, characterized in that: The step 7 comprises the following steps: Step 71: Extract the dataset under ZSFD task settings The category label is The images and corresponding labels are input into the process of steps 2-5 to evaluate the detection performance; Step 72: Under the GZSFD task setting, set the dataset The category label is The images and corresponding labels are input into the process of steps 2-5 to evaluate the detection performance.

Images