Multi-modal enhanced food illegal addition monitoring method and system

Abstract

The invention provides a multi-modal enhanced food illegal addition monitoring method and system, and the method comprises the steps: obtaining third-party platform data through a crawler, and converting the third-party platform data into a multi-modal text data set; processing the multi-modal text data set to form a word vector table; fusing the graph attention network into BI-LSTM to construct a text classification model, and iteratively training the text classification model by using a word vector table; and completing illegal food addition monitoring by using the trained text classification model. According to the invention, whether the food is illegally added can be accurately predicted, consumption early warning and risk prompt can be quickly provided for consumers to select and buy efficacy claim common food, and a strategy is provided for supervision departments to targeted attack on illegal food adding behaviors.

Description

Technical Field

[0001] The present invention relates to the field of food safety, and in particular to a multi-modal enhanced food illegal additive monitoring method and system based on BI-LSTM. Background Art

[0002] In contemporary society, food safety has become a global focus. With the rapid development of the food industry and the increasing market competition, some unscrupulous manufacturers, in pursuit of profit maximization, illegally add chemicals to food (hereinafter referred to as illegal additions), which seriously threatens the health and life safety of consumers. The problem of illegal additions not only damages the rights and interests of consumers, but also has an adverse impact on the sustainable development of the food industry.

[0003] With the development of e-commerce economy, the number of cases where ordinary foods on online shopping platforms claim to have effects such as weight loss and aphrodisiac is increasing. Ordinary foods do not have magical effects such as weight loss and aphrodisiac, and Chinese law also stipulates that ordinary foods cannot be claimed for their effects. However, sellers promote the magical effects of products, and the publicity is overwhelming and exaggerated. Consumers who are unaware of the truth are easily trapped in consumption traps and believe the sellers' claims about the effects of the products, buy and eat such foods. Often, sellers will illegally add drug ingredients to such foods to achieve the claimed effects or simply cause some changes in the consumer's body, so that consumers can personally feel the effects of the products and then become dependent on the products. These effects and changes are usually accompanied by damage to physical health. From time to time, the media reports cases of physical discomfort, poisoning, and even death due to the consumption of weight loss coffee / chocolate, aphrodisiac candies, and ancestral pills.

[0004] Traditional methods for detecting illegal additives often rely on laboratory analysis, which has problems such as long detection cycle, high cost, and low efficiency, and it is difficult to meet the market's needs for rapid detection and supervision. Summary of the invention

[0005] In view of the problems existing in the prior art, a multimodal enhanced monitoring method for illegal food additives is provided, which is based on a bidirectional long short-term memory neural network (Bi-LSTM) to infer whether chemical substances are illegally added to food. The model uses artificial intelligence natural language processing technology to analyze the basic information on the labels of ordinary foods with efficacy claims to infer whether chemical substances are illegally added to food.

[0006] The first aspect of the present invention provides a multi-modal enhanced food illegal additive monitoring method, comprising: Obtain data from third-party platforms through crawlers and convert them into multimodal text datasets; Process the multimodal text dataset to form a word vector table; Integrate the graph attention network into BI-LSTM to build a text classification model, and use the word embedding table to iteratively train the text classification model; Use the trained text classification model to complete illegal food additive monitoring.

[0007] As a preferred solution, the third-party platform data is obtained by crawling and converted into a multimodal text dataset, specifically including: Collect video datasets, image datasets, voice datasets, and text datasets related to product ingredient descriptions from third-party platforms through crawler technology; The text information in the video dataset, image dataset, and voice dataset is extracted respectively, and fused with the text dataset to form a multimodal text dataset.

[0008] As a preferred solution, the extracting of text information from the video dataset, the picture dataset, and the voice dataset respectively, and fusing them with the text dataset to form a multimodal text dataset, specifically includes: Perform frame extraction operation on the video data set to obtain a video frame data set, and input the frame data and image data set in the video frame data into the EAST-based text detection model to obtain the text area; The CTC-based OCR text recognition model is used to recognize the text in the text area to obtain the image text dataset; The speech data set is recognized using a CTC-based speech recognition model to obtain a speech text data set; The image text dataset, speech text dataset and text dataset are fused to obtain a multimodal text dataset.

[0009] As a preferred solution, the EAST-based text detection model includes 5 convolutional layers, 3 upsampling layers and 3 feature fusion layers, wherein the first convolutional layer includes a 7x7 convolution and a pooling layer with a step size of 2, and the second, third, fourth and fifth convolutional layers all include 3x3 convolution and a maximum pooling layer with a step size of 2; the upsampling layer includes upsampling with a step size of 2, 1x1 convolution and 3x3 convolution; the feature fusion layer includes a splicing operation; the CTC-based OCR text recognition model includes 6 convolutional layers, 4 pooling layers, and 1 recurrent layer. and 1 loss layer, wherein the convolution layer includes 3x3 convolutions, the pooling layer size is 2x2, the recurrent layer is composed of 256 double-layer BI-LSTMs, and the loss layer is composed of Softmax and a bidirectional alignment algorithm; the CTC-based speech recognition model includes 5 convolution layers, 4 maximum pooling layers, 1 average pooling layer, 1 recurrent layer and 1 connection temporal classification loss layer, wherein the convolution layer includes 3x3 convolutions, the pooling layer size is 2x2, the recurrent layer is composed of 256 double-layer BI-LSTMs, and the loss layer is composed of Softmax and a bidirectional alignment algorithm.

[0010] As a preferred solution, the processing of the multimodal text dataset to form a word vector table specifically includes: Eliminate redundant and irrelevant information in multimodal text datasets; The filtered multimodal text dataset is input into the GrammarGPT model, and the model instructions are fine-tuned and grammatical errors are corrected by adding the dataset related to illegal food additives defined by laws and regulations. Finally, a high-quality text dataset and labels corresponding to each text sequence are obtained. Perform word segmentation on text data to generate structured vocabulary sequences; Map structured vocabulary sequences into a multi-dimensional continuous value word vector table.

[0011] As a preferred solution, the graph attention network is integrated into BI-LSTM to construct a text classification model, and the word vector table is used to iteratively train the text classification model, which specifically includes: Build a text classification model based on graph attention network and BI-LSTM unit; The label embedding vector is used as the node feature to calculate the graph attention; Model the correlation between labels as a weighted graph to calculate the loss; Use the word embedding table to train the model, iteratively save the optimal accuracy to generate a new model, and evaluate the performance of the model on the validation set until the expected accuracy is achieved.

[0012] As a preferred solution, in the text classification model, the LSTM unit is composed of an input gate, a forget gate, a cell state update and an output gate. , the forward hidden state The input of the current time step and the hidden state at the previous time step Calculated, the reverse hidden state The input of the current time step and the hidden state at the next time step The final hidden state at each time step is calculated From the forward hidden state and the reverse hidden state Combining, expressed as:

[0013] in, and Represent the weight matrices for the forward and reverse hidden states, respectively, Represents the bias term.

[0014] As a preferred solution, the specific process of calculating graph attention includes: Define the first i The input and output node feature vector sets of the layer are and in, , N and F are the number of nodes and the number of features respectively; Calculate the The feature vector of the layer is at the node Linear transformation at :

[0015] in, W is the shared weight matrix; Calculate the attention correlation coefficient:

[0016] in, is the activation function, is the weight parameterization of the learnable self-attention parameters, Indicates splicing; For Node All neighboring nodes Normalized attention weights: .

[0018] As a preferred solution, the specific process of calculating the loss includes: Normalize the co-occurrence matrix by the frequency vector:

[0019] in, is the co-occurrence matrix; A frequency vector representing a single label; The first layer output of the graph attention network is expressed as:

[0020] in, is the tanh activation function, K represents the number of heads in the multi-head attention mechanism; No. l The output of the layer is represented as:

[0021] The attention label feature output of the last layer of the graph attention network is defined as:

[0022] in, c Indicates the number of tags, d Represents the size of the label feature, and then performs a dot multiplication operation with the feature vector output by the BI-LSTM network to obtain the final prediction classification result:

[0023] Calculate the cross entropy as the loss function:

[0024] in, represents the true label, .

[0025] The present invention proposes a system including a memory and a processor, wherein the memory stores a computer program that can be loaded by the processor and executed according to the multimodal enhanced food illegal additive monitoring method as described in the first aspect.

[0026] Compared with the prior art, the beneficial effects of adopting the above technical solution are: the present invention can accurately predict whether food is illegally added, can quickly provide consumers with consumption warnings and risk warnings when purchasing ordinary foods with efficacy claims, and provide strategies for regulatory authorities to target and crack down on illegal food additives. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 This is a flow chart of the multi-modal enhanced food illegal additive monitoring method proposed in the present invention.

[0028] Figure 2This is a flow chart of multimodal text data extraction in one embodiment of the present invention.

[0029] Figure 3 Schematic diagram of data structuring processing in one embodiment of the present invention.

[0030] Figure 4 This is a schematic diagram of the structure of a text classification model based on BI-LSTM and graph attention in one embodiment of the present invention. DETAILED DESCRIPTION

[0031] Embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar modules or modules with the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application, and cannot be construed as limitations on the present application. On the contrary, the embodiments of the present application include all changes, modifications and equivalents that fall within the spirit and connotation of the appended claims.

[0032] Traditional methods for detecting illegal additives often rely on laboratory analysis, which has problems such as long detection cycles, high costs, and low efficiency, making it difficult to meet the market's needs for rapid detection and supervision. The application of artificial intelligence deep learning technology in the field of text analysis can provide new ideas and methods for the rapid identification and capture of illegal additives in food. The bidirectional long short-term memory neural network is a special type of recurrent neural network that can process sequence data and maintain long-term memory. Unlike traditional recurrent neural networks, the bidirectional long short-term memory neural network takes into account both past and future information, allowing the model to better capture contextual relationships in sequence data, provide more comprehensive feature representations, and classify text.

[0033] Based on this, the embodiment of the present invention proposes a multi-modal enhanced monitoring method for illegal food additives, which trains the data in multiple times, forms a model and saves it, and then directly detects it through the model. Compared with laboratory testing, using the BI-LSTM model to predict whether food is illegally added can optimize and improve the monitoring efficiency and accuracy of illegal food additives, and can quickly provide consumers with consumption warnings and risk warnings when purchasing ordinary foods with efficacy claims, and provide strategies for regulatory authorities to target and crack down on illegal food additives. Please refer to Figure 1 , the specific plan is as follows: Step 1: Obtain third-party platform data through crawlers and convert it into a multimodal text dataset.

[0034] Please refer to Figure 2 In this embodiment, the data used for model training can be collected from third-party platforms such as Taobao and Tmall through crawler technology. The collected data includes: video data set , Indicates Product description video clips and image datasets , Indicates Product ingredient description images and voice datasets , Indicates A dataset of audio clips and text of product ingredient descriptions , Indicates The text data of product ingredient description.

[0035] After completing the collection of multiple types of data, it is necessary to perform fusion processing to form a multimodal data set. Specifically, for the video data set V Perform frame extraction operation to obtain video frame dataset V’ , and the video frame data V’ Frame data and image datasets in P Input to the EAST-based text detection model to obtain the text area , Indicates k Then, we use the CTC-based OCR text recognition model to identify the text area. Recognize the text in the image to get the image text dataset , Indicates k The recognized text sequence. Use the CTC-based speech recognition model to obtain a speech text dataset , Indicates k Finally, the image text dataset , speech text dataset and text datasets Fusion to obtain a multimodal text dataset , Indicates k A sequence of text.

[0036] In this embodiment, the specific structures of the EAST-based text detection model, the CTC-based OCR text recognition model, and the CTC-based speech recognition model used in the data set fusion process are also given. The EAST-based text detection model consists of 5 convolutional layers, 3 upsampling layers and 3 feature fusion layers. The first convolutional layer contains a 7x7 convolution and a pooling layer with a stride of 2. The second, third, fourth and fifth convolutional layers all contain 3x3 convolutions and maximum pooling layers with a stride of 2; the upsampling layer contains upsampling with a stride of 2, 1x1 convolution and 3x3 convolution; the feature fusion layer contains a splicing operation.

[0037] The CTC-based OCR text recognition model consists of an image feature extraction submodule consisting of 6 convolutional layers, 4 pooling layers, 1 recurrent layer and 1 loss layer. The convolutional layer contains 3x3 convolutions, the pooling layer size is 2x2, the recurrent layer consists of 256 double-layer BI-LSTMs, and the loss layer consists of Softmax and a bidirectional alignment algorithm.

[0038] The CTC-based OCR speech recognition model consists of 5 convolutional layers, 4 maximum pooling layers, 1 average pooling layer, 1 recurrent layer and 1 connection temporal classification loss layer. The convolutional layer contains 3x3 convolutions, the pooling layer size is 2x2, the recurrent layer consists of 256 double-layer BI-LSTMs, and the loss layer consists of Softmax and bidirectional alignment algorithms.

[0039] Step 2: Process the multimodal text dataset to form a word vector table.

[0040] Please refer to Figure 3 For the obtained multimodal text dataset, data cleaning, grammatical error correction and other processing are required. In this embodiment, the multimodal fusion text dataset Use manual review to remove redundant and irrelevant information from text data and obtain a filtered text data set , Indicates k Then, it is fed into the GrammarGPT model and the dataset of illegal food additives defined by laws and regulations is added. , Indicates k Illegally added vocabulary data, completed model instruction fine-tuning and grammatical error correction, and finally obtained a high-quality text dataset , Indicates k Filtered high-quality text sequences and the labels corresponding to each sequence text , Indicates k The text labels corresponding to the filtered high-quality text sequences, where the text label value By using GrammarGPT's error correction algorithm to filter the specified clues, a high-quality dataset can be further obtained.

[0041] For the obtained high-quality text data, in this embodiment, the data is segmented based on the Jieba word segmentation library to generate a structured vocabulary sequence , Indicates k structured vocabulary sequences, where |V| is the vocabulary size. S Mapped to a multi-dimensional continuous value word vector table , Indicates k embedding word vectors, where , W is the word vector matrix, V k is a word embedding vector of size |V|.

[0042] Step 3: Integrate the graph attention network into BI-LSTM to build a text classification model, and use the word vector table to iteratively train the text classification model.

[0043] Please refer to Figure 4 In this embodiment, the graph attention network is integrated into the BI-LSTM to deal with the text classification problem, forming a new text classification model. The text classification method of the bidirectional long short-term memory network and the graph attention network is used to extract the contextual information and global features of the text sequence. The correlation between labels is revealed through the graph attention network, so that the feature vector and label correlation are combined to predict the label text classification task, further improving the accuracy of classification recognition.

[0044] The LSTM unit of the text classification model consists of an input gate, a forget gate, a cell state update, and an output gate. The input gate inputs the time step t The input vector , for each time step , the forward hidden state The input of the current time step and the hidden state at the previous time step Calculated, the reverse hidden state The input of the current time step and the hidden state at the next time step The final hidden state at each time step is calculated From the forward hidden state and the reverse hidden state Combining, expressed as: (1) in, and Represent the weight matrices for the forward and reverse hidden states, respectively, Represents the bias term.

[0045] As for the graph attention calculation of the model, in this embodiment, the embedding vector of the label is used as the node feature to define the first i The input and output node feature vector sets of the layer are and in, , and are the number of nodes and the number of features respectively. Then use formula (2) to calculate the The feature vector of the layer is at the node Linear transformation at .

[0046] (2) in, represents the shared weight matrix, Indicates l The feature vector of the layer is at the node i The linear transformation result at Indicates Layer The feature vector of each node.

[0047] The attention correlation coefficient is calculated using formula (3) and the shared weight matrix to represent the importance between adjacent nodes. (3) in, is the activation function, is the weight parameterization of the learnable self-attention parameters, Indicates splicing; Use formula (4) to calculate the node All neighboring nodes Normalized attention weights: (4) in, Indicates The set of neighbor nodes of a node.

[0048] Finally, this example also gives the loss calculation process of the constructed text classification model. The correlation between labels is modeled as a weighted graph, so that the adjacency matrix and attention weights jointly reflect the relationship between labels to effectively capture the mutual influence between labels. The co-occurrence matrix is ​​normalized by the frequency vector: (5) in, is the normalized co-occurrence matrix, is the co-occurrence matrix; A frequency vector representing a single label.

[0049] The first layer output of the graph attention network can be expressed as: (6) in, is the output feature vector of the i-th node in the first layer, is the attention weight of the kth head between the i-th node and the j-th node in layer 0, is the tanh activation function, and K represents the number of heads in the multi-head attention mechanism.

[0050] No. l+ The output of layer 1 is represented as (7) in, For the The output feature vector of the i-th node in the layer, For the The attention weight of the kth head between the i-th and j-th nodes in the layer.

[0051] The attention label feature output of the last layer of the graph attention network is defined as (8) in, represents the set of real numbers, c Indicates the number of tags, d Represents the size of the label feature, and then performs a dot multiplication operation with the feature vector output by the BI-LSTM network to obtain the final prediction classification result: (9) Calculate cross entropy as loss function (10) in, represents the true label, represents the predicted label, c is the number of labels, Represents the final predicted classification, which refers to the total set, It represents the predicted label result.

[0052] Finally, the obtained word vector table is used to train the model, and the optimal accuracy is saved iteratively to generate a new model; the performance of the model on the validation set is evaluated until the expected accuracy is achieved.

[0053] Step 4: Use the trained text classification model to complete illegal food additive monitoring.

[0054] In actual applications, after collecting data related to food descriptions, it can be directly input into the saved model to determine whether there are illegal additives in the food descriptions.

[0055] The present invention can accurately predict whether food additives are illegally added, and can quickly provide consumers with consumption warnings and risk reminders when purchasing ordinary foods with efficacy claims, and provide strategies for regulatory authorities to target and crack down on illegal food additives.

[0056] An embodiment of the present invention further proposes a system, including a memory and a processor, wherein the memory stores a computer program corresponding to the aforementioned multimodal enhanced food illegal additive monitoring method that can be loaded by the processor and executed.

[0057] In particular, according to an embodiment of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a computer readable medium, and the computer program includes a program code for executing the method shown in the flowchart.

[0058] It should be noted that the computer-readable medium shown in the embodiment of the present application may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, device or device. In the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, which may send, propagate, or transmit programs for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0059] The flowchart and block diagram in the accompanying drawings illustrate the possible architecture, functions and operations of the system, method and computer program product according to various embodiments of the present application. Wherein, each box in the flowchart or block diagram can represent a module, a program segment, or a part of the code, and the above-mentioned module, program segment, or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0060] The units involved in the embodiments described in this application may be implemented by software or hardware, and the units described may also be set in a processor. The names of these units do not, in some cases, constitute limitations on the units themselves.

[0061] As another aspect, the present application also provides a computer program product or a computer program, which includes a computer instruction stored in a computer-readable storage medium. A processor of a computer device reads the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device executes the multi-modal enhanced food illegal additive monitoring method described in the above embodiment.

[0062] As another aspect, the present application also provides a computer-readable medium, which may be included in the electronic device described in the above embodiment; or may exist independently without being assembled into the electronic device. The above computer-readable medium carries one or more programs, and when the above one or more programs are executed by an electronic device, the electronic device implements the multi-modal enhanced food illegal additive monitoring method described in the above embodiment.

[0063] It should be noted that, although several modules or units of the equipment for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present application, the features and functions of two or more modules or units described above can be embodied in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into being embodied by multiple modules or units.

[0064] Through the description of the above implementation methods, it is easy for those skilled in the art to understand that the example implementation methods described here can be implemented by software, or by combining software with necessary hardware. Therefore, the technical solution according to the implementation method of the present application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, a touch terminal, or a network device, etc.) to execute the method according to the implementation method of the present application.

[0065] For those skilled in the art, the specific meanings of the above terms in the present invention can be understood in specific situations; the drawings in the embodiments are used to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings herein can be arranged and designed in various different configurations.

[0066] Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limitations on the present application. Ordinary technicians in this field can change, modify, replace and modify the above embodiments within the scope of the present application.

Claims

1. A multi-modal enhanced food illegal additive monitoring method, characterized in that: include: Obtain data from third-party platforms through crawlers and convert them into multimodal text datasets; Process the multimodal text dataset to form a word vector table; Integrate the graph attention network into BI-LSTM to build a text classification model, and use the word embedding table to iteratively train the text classification model; Use the trained text classification model to complete illegal food additive monitoring.

2. The multi-modal enhanced food illegal additive monitoring method according to claim 1, characterized in that: The method of obtaining third-party platform data through crawlers and converting it into a multimodal text dataset specifically includes: Collect video datasets, image datasets, voice datasets, and text datasets related to product ingredient descriptions from third-party platforms through crawler technology; The text information in the video dataset, image dataset, and voice dataset is extracted respectively, and fused with the text dataset to form a multimodal text dataset.

3. The multi-modal enhanced food illegal additive monitoring method according to claim 2, characterized in that: The extracting of text information from the video dataset, the picture dataset, and the voice dataset respectively, and fusing them with the text dataset to form a multimodal text dataset specifically includes: Perform frame extraction operation on the video data set to obtain a video frame data set, and input the frame data and image data set in the video frame data into the EAST-based text detection model to obtain the text area; The CTC-based OCR text recognition model is used to recognize the text in the text area to obtain the image text dataset; The speech data set is recognized using a CTC-based speech recognition model to obtain a speech text data set; The image text dataset, speech text dataset and text dataset are fused to obtain a multimodal text dataset.

4. The multi-modal enhanced food illegal additive monitoring method according to claim 3, characterized in that: The EAST-based text detection model includes 5 convolutional layers, 3 upsampling layers and 3 feature fusion layers, wherein the first convolutional layer includes a 7x7 convolution and a pooling layer with a step size of 2, and the second, third, fourth and fifth convolutional layers all include 3x3 convolution and a maximum pooling layer with a step size of 2; the upsampling layer includes upsampling with a step size of 2, 1x1 convolution and 3x3 convolution; the feature fusion layer includes a splicing operation; the CTC-based OCR text recognition model includes 6 convolutional layers, 4 pooling layers, 1 recurrent layer and 1 loss Layer, wherein the convolution layer includes 3x3 convolutions, the pooling layer size is 2x2, the recurrent layer is composed of 256 double-layer BI-LSTMs, and the loss layer is composed of Softmax and a bidirectional alignment algorithm; the CTC-based speech recognition model includes 5 convolution layers, 4 maximum pooling layers, 1 average pooling layer, 1 recurrent layer and 1 connection temporal classification loss layer, wherein the convolution layer includes 3x3 convolutions, the pooling layer size is 2x2, the recurrent layer is composed of 256 double-layer BI-LSTMs, and the loss layer is composed of Softmax and a bidirectional alignment algorithm.

5. The multi-modal enhanced food illegal additive monitoring method according to claim 1 or 2, characterized in that: The processing of the multimodal text dataset to form a word vector table specifically includes: Eliminate redundant and irrelevant information in multimodal text datasets; The filtered multimodal text dataset is input into the GrammarGPT model, and the model instructions are fine-tuned and grammatical errors are corrected by adding the dataset related to illegal food additives defined by laws and regulations. Finally, a high-quality text dataset and labels corresponding to each text sequence are obtained. Perform word segmentation on text data to generate structured vocabulary sequences; Map structured vocabulary sequences into a multi-dimensional continuous value word vector table.

6. The multi-modal enhanced food illegal additive monitoring method according to claim 1 or 2, characterized in that: The graph attention network is integrated into BI-LSTM to build a text classification model, and the word vector table is used to iteratively train the text classification model, specifically including: Build a text classification model based on graph attention network and BI-LSTM unit; The label embedding vector is used as the node feature to calculate the graph attention; Model the correlation between labels as a weighted graph to calculate the loss; Use the word embedding table to train the model, iteratively save the optimal accuracy to generate a new model, and evaluate the performance of the model on the validation set until the expected accuracy is achieved.

7. The multi-modal enhanced food illegal additive monitoring method according to claim 6, characterized in that: In the text classification model, the LSTM unit consists of an input gate, a forget gate, a cell state update, and an output gate. , the forward hidden state The input of the current time step and the hidden state at the previous time step Calculated, the reverse hidden state The input of the current time step and the hidden state at the next time step The final hidden state at each time step is calculated From the forward hidden state and the reverse hidden state Combining, expressed as: in, and Represent the weight matrices for the forward and reverse hidden states, respectively, Represents the bias term.

8. The multi-modal enhanced food illegal additive monitoring method according to claim 7, characterized in that: The specific process of computing graph attention includes: Define the first i The input and output node feature vector sets of the layer are and in, , N and F are the number of nodes and the number of features respectively; Calculate the The feature vector of the layer is at the node Linear transformation at : in, W is the shared weight matrix; Calculate the attention correlation coefficient: in, is the activation function, is the weight parameterization of the learnable self-attention parameters, Indicates splicing; For Node All neighboring nodes Normalized attention weights: 。 9. The multi-modal enhanced food illegal additive monitoring method according to claim 8, characterized in that: The specific process of calculating the loss includes: Normalize the co-occurrence matrix by the frequency vector: in, is the co-occurrence matrix; A frequency vector representing a single label; The first layer output of the graph attention network is expressed as: in, is the tanh activation function, K represents the number of heads in the multi-head attention mechanism; No. l The output of the layer is represented as: The attention label feature output of the last layer of the graph attention network is defined as: in, c Indicates the number of tags, d Represents the size of the label feature, and then performs a dot multiplication operation with the feature vector output by the BI-LSTM network to obtain the final prediction classification result: Calculate the cross entropy as the loss function: in, represents the true label, .

10. A system, characterized in that: It comprises a memory and a processor, wherein the memory stores a computer program corresponding to the multimodal enhanced food illegal additive monitoring method as described in any one of claims 1 to 9, which can be loaded by the processor and executed.

Images