A method and system for retaining and re-inspecting food samples based on deep learning

Abstract

The present invention relates to the technical field of dish supervision, and discloses a method and system for dish sampling and sample re-inspection based on deep learning. The method comprises: performing authority verification on the sample retaining personnel, and after the authority verification is passed, determining the header information of the dish sample label based on the current meal section and the basic information of the sample retaining personnel; obtaining a sample image of the dish to be sampled, and detecting the dish target in the sample image based on a target detection algorithm; performing dish recognition on the detected dish target, obtaining dish sample information based on the dish recognition result, and combining and recording it with the sample image to complete the sample retention; selecting a re-inspected dish from the dishes for which the sample retention has been completed, checking the sample, and obtaining the dish sample information and the sample image; checking the other dishes in the sample image, and checking the sample based on the re-inspection score threshold. The present invention intelligently identifies dishes, automatically records the sampled dishes, improves the efficiency and accuracy of sample retention, adds a sample re-inspection function, and improves the safety and credibility of sample retention.

Description

Technical Field

[0001] The present invention relates to the technical field of dish supervision, and in particular to a method and system for retaining and re-inspecting dish samples based on deep learning. Background Art

[0002] In the restaurant industry, retaining food samples is a necessary process mandated by laws and regulations, and a crucial measure for ensuring food safety. By retaining samples from every meal, food safety issues can be quickly verified, providing crucial evidence for tracing the source and analyzing the cause.

[0003] During the food sampling process, greater attention needs to be paid to sample information collection, archiving and retrieval, risk analysis, and traceability monitoring, requiring more scientific and comprehensive management methods. Current food sampling relies primarily on manual operations, including selecting samples, recording relevant information, placing samples in fresh-keeping equipment, and preserving them according to prescribed time and conditions. However, manual sampling is inefficient, especially during peak catering periods, which can affect the normal catering service process. Staff members need to spend time recording samples, writing sample labels, and other operations, increasing their workload. Manual sampling information is prone to errors or omissions, such as incomplete or inaccurate records of key information such as dish name, sampling time, and operator, which makes subsequent tracing and analysis difficult. Manual sampling is prone to modifying sample information, making it less convincing.

[0004] With the advancement of technology, improvements have been made in food sampling. Intelligent systems, software, and equipment have gradually emerged, such as those that use artificial intelligence to automatically identify food samples, automatically weigh and print sample labels, and upload all sample details to backend systems with a single click. However, existing intelligent systems, software, and equipment have relatively limited functionality, often focusing solely on food sampling and lacking safety monitoring and sample review. In the event of a food safety incident, tampering with food samples will hinder tracing and analyzing the cause of the incident, and the security and credibility of sample data are lacking.

[0005] Therefore, how to improve the efficiency and accuracy of food sampling and enhance the security and credibility of sample data has become a technical problem that needs to be urgently solved in this field. Summary of the Invention

[0006] The purpose of the present invention is to provide a method and system for food sampling and sample re-inspection based on deep learning, which can intelligently identify dishes through deep learning, automatically record sample dishes, improve the efficiency and accuracy of food sampling, and add the function of sample re-inspection to improve the security and credibility of sample data.

[0007] To achieve the above object, the present invention provides the following solutions:

[0008] A method for retaining and retesting food samples based on deep learning, the method comprising the following steps:

[0009] S1, perform permission verification on the sample retainer. After the permission verification passes, record the basic information of the sample retainer;

[0010] S2: After the permission check is passed, the current meal segment is determined based on the current time, and the header information of the dish sample label is determined based on the basic information of the sample retainer;

[0011] S3, obtaining a sample image of the dish to be sampled, and automatically detecting whether there is a dish target in the sample image based on a deep learning target detection algorithm. If so, obtaining the location coordinates and target quantity of the dish target;

[0012] S4: For the detected dish target, the dish subgraph is automatically cut according to the positioning coordinates and the feature vector of the dish subgraph is extracted. The feature vector is compared with the dish features in the feature library, and the current dish target is identified. The current dish target is determined to be invalid or pending confirmation. The invalid dish is added as a temporary dish and entered into the feature library. The pending confirmation dish is corrected to obtain the correct category.

[0013] S5, based on the dish recognition result, obtain the dish sample information, combine the header information with the dish sample information to obtain the complete information of the dish sample label, and record it in combination with the sample image to complete the sample. The dish sampled is added to the sample feature library, where the dish sample information includes the sample time, sample category, sample number, sample weight, sample order number, and target sample quantity.

[0014] S6, based on the sample retention time for re-inspection, determine the re-inspection scope of the sample dish feature library, select re-inspected dishes from the dishes for which sampling has been completed, compare the image features of the re-inspected dishes with the features of the dishes within the re-inspection range of the sample dish feature library, obtain the dish sample retention information and sample image corresponding to the re-inspected dish, and complete the sample inspection; inspect other dishes in the sample image, and based on the re-inspection score threshold, determine whether there is a risk of modification for all dishes in the sample image, and complete the sample inspection.

[0015] Furthermore, in S1, the permission of the sample retainer is verified. After the verification is passed, the basic information of the sample retainer is recorded, including:

[0016] Perform facial recognition on the sample retention personnel to determine whether they are authorized personnel. If so, the facial recognition is successful, the authority verification is passed, and the facial image and name of the sample retention personnel are recorded.

[0017] Furthermore, S1 also includes that after the face recognition is successful, the sample retention personnel enters a dedicated authority password. If the password is correct, the authority verification is passed and the face picture and name of the sample retention personnel are recorded.

[0018] Furthermore, in S2, determining the current meal segment according to the current time specifically includes:

[0019] The current time is determined to be the breakfast period, lunch period or dinner period; the header information of the dish sample retention label includes the name of the sample retention person and the sample retention period.

[0020] Furthermore, the S3 further includes:

[0021] If the target detection algorithm misses a detection, manually select the missed target dish with a rectangular frame on the software interface and click OK to add a temporary target.

[0022] Based on the target detection algorithm, the missed target sub-image is cut according to the coordinates of the manually selected rectangular box, the features of the target sub-image are extracted through the depth measurement model, and the features, width and height information of the target sub-image are saved;

[0023] When the target sub-image appears again in the image of the dish to be sampled, the target detection algorithm is used to automatically call the sliding window algorithm and feature comparison algorithm of the limited area based on the features, width and height information to quickly obtain the positioning coordinate information of the target sub-image.

[0024] Furthermore, in S4, the feature vector is compared with the dish features in the feature library, the current dish target is identified, the current dish target is determined to be an invalid dish or a dish to be confirmed, the invalid dish is added as a temporary dish, and entered into the feature library, and the dish to be confirmed is corrected to obtain the correct dish category, specifically including:

[0025] If the feature database is empty, the current dish target is determined to be an invalid dish based on the feature vector, and the current dish target is added as a temporary dish and entered into the feature database;

[0026] If the total feature library is not empty, a dish recognition score is obtained based on the comparison between the feature vector and the dish features in the total feature library. If the dish recognition score is lower than the trustworthy threshold score, the dish target is determined to be a dish to be confirmed; wherein the dish recognition score is the highest value among multiple comparison scores obtained by comparing the feature vector with multiple dish features in the total feature library;

[0027] Correct the dishes to be confirmed, sort the multiple comparison scores from high to low, obtain the top 10 scores, and determine whether there are similar dish categories. If so, correct them to get the correct dish category. If not, determine it as an invalid dish, add it as a temporary dish, and enter it into the feature library.

[0028] Furthermore, in said S5, the complete information of the dish sample label is obtained and recorded in combination with the sample image to complete the sample, and the dish after the sample is completed is added to the sample dish feature library, specifically including:

[0029] Print and paste the sample label on the corresponding sample dish;

[0030] The complete information, positioning coordinates and sample images of each sample dish are automatically saved. At the same time, the completed sample dishes are added to the sample dish feature library for re-inspection.

[0031] Furthermore, in S6, the checking of samples specifically includes:

[0032] Based on the sample retention time for re-inspection, determine the re-inspection scope of the sample dish feature library, and select re-inspection dishes from the dishes for which the samples have been retained;

[0033] The dish to be re-inspected is placed in the sampling area of ​​the image acquisition device to obtain the dish image, and the dish image is automatically positioned, sub-images are cropped, and features are extracted based on the target detection algorithm to obtain the image features of the dish to be re-inspected;

[0034] Compare the image features of the re-inspected dish with the features of the dishes within the re-inspection range in the retained sample dish feature library to obtain the retained sample dish category and retained sample number corresponding to the re-inspected dish;

[0035] According to the category of sample dishes and sample image number, the sample order number and sample image corresponding to the re-inspected dish are retrieved.

[0036] Furthermore, in said S6, said sample inspection specifically includes:

[0037] Enter the sample inspection page, check the sample image obtained in the sample inspection step with other dishes corresponding to the sample image, and combine it with the sample inspection results of the re-inspected sample to obtain the sample information corresponding to all dishes in the sample image;

[0038] Compare the scores of all dishes in the sample image with the dish features within the re-inspection range in the sample dish feature library. If the score is lower than the re-inspection score threshold, it is considered that the current dish has the risk of being modified.

[0039] Compare the actual weight of all dishes in the sample image with the sample weight in the sample information. If the actual weight exceeds or falls below the sample weight, it is considered that the current dish is at risk of being modified.

[0040] When the score is higher than or equal to the re-inspection score threshold and equal to the retained sample weight, the current dish is considered to have passed the re-inspection.

[0041] The present invention also provides a system for retaining and retesting food samples based on deep learning, which is used to execute the above-mentioned method for retaining and retesting food samples based on deep learning, comprising:

[0042] The permission verification module is used to verify the permissions of the sample retention personnel. After the permission verification is passed, the basic information of the sample retention personnel is recorded;

[0043] The label information setting module is used to determine the current meal period based on the current time after the permission verification is passed, and to determine the header information of the dish sample label based on the basic information of the sample retainer;

[0044] The dish positioning module is used to obtain the sample image of the dish to be sampled, and automatically detect whether there is a dish target in the sample image based on the deep learning target detection algorithm. If so, the positioning coordinates and target quantity of the dish target are obtained;

[0045] The dish recognition module is used to automatically cut the detected dish target into a dish sub-graph according to the positioning coordinates and extract the feature vector of the dish sub-graph. The feature vector is compared with the dish features in the feature library, and the current dish target is identified. The current dish target is determined to be invalid or pending confirmation. The invalid dish is added as a temporary dish and entered into the feature library. The pending confirmation dish is corrected to obtain the correct category.

[0046] The dish sampling module is used to obtain dish sample information based on the dish recognition result, combine the header information with the dish sample information to obtain the complete information of the dish sample label, and record it in combination with the sample image to complete the sample. The dish after sampling is added to the sample feature library, wherein the dish sample information includes the sampling time, sample category, sample number, sample weight, sample order number, and sample target quantity;

[0047] The dish re-inspection module is used to determine the re-inspection scope of the sample dish feature library based on the sample retention time for re-inspection, select re-inspected dishes from the dishes for which sampling has been completed, compare the image features of the re-inspected dishes with the features of dishes within the re-inspection range of the sample dish feature library, obtain the dish sample information and sample image corresponding to the re-inspected dish, and complete the sample inspection; inspect other dishes in the sample image, and based on the re-inspection score threshold, determine whether there is a risk of modification for all dishes in the sample image, and complete the sample inspection.

[0048] According to the specific embodiments provided by the present invention, the method and system for retaining and retesting food samples based on deep learning provided by the present invention disclose the following technical effects:

[0049] (1) Verify the permissions of the sample retainers to avoid misoperation by non-sample retainers, obtain the basic information of the sample retainers and add it to the header information of the dish sample label to facilitate subsequent investigation of the sample retainers;

[0050] (2) Automatically locate and identify sample dishes through deep learning algorithms, automatically integrate information such as dish name, dish weight, and sample retention personnel to obtain dish sample labels, greatly reducing the workload of sample retention personnel and significantly improving sample retention efficiency;

[0051] (3) On the basis of completing the sample retention, a sample re-inspection function is added. Through the sample checking link, all dishes of the same sample retention order are efficiently found. After the sample inspection link, the changes of the dishes before and after the sample retention are automatically compared to achieve rapid verification. The sample retention safety mechanism is added to the sample retention system to reduce the risk of tampering with the sample retention, improve the credibility of food problem tracing and cause analysis, make the dish sample retention work more scientific, standardized and reliable, reduce the risk of abnormal manual handling of the sample retention dishes, affect the traceability and cause analysis of food safety accidents, and improve the safety and credibility of the dish sample retention;

[0052] (4) By retaining and retesting food samples, it provides a strong guarantee for food safety and has important application value. It can be applied to dining scenes in canteens of major enterprises, units, hospitals, military units and schools. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0054] Figure 1 is a flow chart of a method for retaining food samples based on deep learning in an embodiment of the present invention;

[0055] Figure 2 This is a flow chart of target detection of dish samples based on deep learning in an embodiment of the present invention;

[0056] Figure 3 Schematic diagram of a sample retention label according to an embodiment of the present invention, wherein (a) is a sample retention label for a single dish in an order, and (b) is a sample retention label for multiple dishes in an order;

[0057] Figure 4 This is a sample checking flow chart for sample retention and re-inspection according to an embodiment of the present invention;

[0058] Figure 5 This is a sample inspection flow chart for sample re-inspection according to an embodiment of the present invention.

[0059] Figure 6 2 is a schematic diagram of sample checking results according to an embodiment of the present invention. DETAILED DESCRIPTION

[0060] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0061] The purpose of the present invention is to provide a method and system for retaining samples and re-inspecting samples of dishes based on deep learning. The method automatically locates and identifies the dishes for retention through a deep learning algorithm, automatically integrates information such as the dish name, dish weight, and the person who retains the sample, and automatically prints the sample label, which greatly reduces the workload of the sample personnel and significantly improves the efficiency of retaining samples. At the same time, a sample re-inspection function is designed to automatically record the sample order main number and dish sub-number for each sample dish. By comparing and verifying the order main number and dish self-number, as well as the dish identification features before and after the sample is retained, the risk of abnormal manual handling of the sample dishes is reduced, which affects the tracing and cause analysis of food safety accidents and improves the safety and credibility of the sample retention.

[0062] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0063] The method for retaining and retesting food samples based on deep learning provided by the present invention includes the following steps:

[0064] S1, perform permission verification on the sample retainer. After the permission verification passes, record the basic information of the sample retainer;

[0065] S2: After the permission check is passed, the current meal segment is determined based on the current time, and the header information of the dish sample label is determined based on the basic information of the sample retainer;

[0066] S3, obtaining a sample image of the dish to be sampled, and automatically detecting whether there is a dish target in the sample image based on a deep learning target detection algorithm, and if so, obtaining the location coordinates and target quantity of the dish target; for example, placing the dish to be sampled in a sampling area of ​​an image acquisition device to obtain a sample image, and the image acquisition device is also equipped with a weighing device, which can weigh the dish to obtain the sample weight. For other configurations, reference can be made to existing dish sampling devices in the art;

[0067] S4: For the detected dish target, the dish subgraph is automatically cut according to the positioning coordinates and the feature vector of the dish subgraph is extracted. The feature vector is compared with the dish features in the feature library, and the current dish target is identified. The current dish target is determined to be invalid or pending confirmation. The invalid dish is added as a temporary dish and entered into the feature library. The pending confirmation dish is corrected to obtain the correct category.

[0068] S5, based on the dish recognition result, obtain the dish sample information, combine the header information with the dish sample information to obtain the complete information of the dish sample label, and record it in combination with the sample image to complete the sample. The dish sampled is added to the sample feature library, where the dish sample information includes the sample time, sample category, sample number, sample weight, sample order number, and target sample quantity.

[0069] S6, based on the sample retention time for re-inspection, determine the re-inspection scope of the sample dish feature library, select re-inspected dishes from the dishes for which sampling has been completed, compare the image features of the re-inspected dishes with the features of the dishes within the re-inspection range of the sample dish feature library, obtain the dish sample retention information and sample image corresponding to the re-inspected dish, and complete the sample inspection; inspect other dishes in the sample image, and based on the re-inspection score threshold, determine whether there is a risk of modification for all dishes in the sample image, and complete the sample inspection.

[0070] Example:

[0071] The method for retaining and retesting food samples based on deep learning provided in an embodiment of the present invention is performed by a system for retaining and retesting food samples based on deep learning. The method specifically includes a method for retaining food samples based on deep learning and a method for retesting food samples based on deep learning, as follows:

[0072] (1) If Figure 1 and Figure 2 As shown in FIG, the dish sampling method based on deep learning includes the following steps:

[0073] S1. Authority verification: The system for retaining and re-inspecting food samples based on deep learning designed by the present invention requires facial recognition verification when logging in. When the sample retaining personnel operates, the face is automatically detected and recognized. Only authorized personnel can successfully recognize the face. After successful recognition, the face picture of the sample retaining personnel is saved. In order to improve the security of authority verification, after successful face recognition, the exclusive authority password of the recognition personnel needs to be entered. Only when the face and password are successfully verified can the sample retaining system be used freely and normally.

[0074] S2. Print label information setting: After the sample retention system passes the authority verification, it automatically sets the current meal period as breakfast, lunch or dinner according to the current time, obtains the person's information according to the identification, and then automatically presets the header information of the print label, such as Figure 3 shown.

[0075] S3. Dish positioning: Enter the sample retention page and start sampling the dish. Place the dish within the shooting range of the camera of the image acquisition device. The target detection algorithm based on deep learning designed by the present invention automatically locates whether there is a dish target in the current image. If it exists, the positioning box and target quantity of the dish target are displayed. If it is found that the target detection algorithm has missed the detection, the temporary target function is added, and the missed target dish is manually selected with a rectangular box on the system interface, and click OK. The target detection algorithm will then cut the missed target sub-graph according to the manually selected rectangular coordinates, extract the features of the target sub-graph through the depth measurement model, and save the features and the width and height information of the target sub-graph to the system. Subsequently, when a new picture of the missed target dish is encountered, the target sub-graph appears again, and the target detection algorithm is used to automatically call the sliding window algorithm and feature comparison algorithm of the limited area based on the features, width and height information to quickly obtain the positioning coordinate information of the target sub-graph.

[0076] S4. Dish Identification: When used for the first time, the algorithm's dish feature database is empty. When retaining a sample, you need to enter the dish to establish the feature database. After detecting the sample target, the algorithm automatically cuts the dish subgraph according to the positioning coordinates and extracts the feature vector of the dish subgraph. Since the current feature database is empty, the current dish will be displayed as "invalid dish". The present invention is designed with the function of adding temporary dishes. By clicking on the picture of the current sample dish displayed on the system, entering the page of adding temporary dishes, and selecting to add the current sample dish as a temporary dish, the dish entry function can be quickly completed. The current dish will be displayed from an invalid dish to the newly entered dish name. When the total feature library is not empty, but the dish recognition score is lower than the algorithm's trust threshold score, the current dish will be displayed as "pending dish"; manually click on the image of the dish on the system to enter the dish correction page, which will display the top 10 results with the highest scores when comparing the current dish with the total feature library, and sort them from high to low by comparison score. The probability of the correct category appearing in these 10 results is 99.9%. Manually select the correct dish from these 10 results to correct the current recognition result. If none of the 10 results given have the correct category, you can enter the Add Temporary Dish page to enter the current dish into the total feature library.

[0077] S5. Samples: After the sample identification result is confirmed to be correct, use the one-key print function to automatically print out the detailed information of the current sample. The sample label information of each dish is as follows: Figure 3If multiple dishes appear in an image, one-click printing will print labels corresponding to the number of dishes. Each label can be affixed to the corresponding sample dish. Once printing is complete, the label information, location coordinates, and main sample image of each sample dish are automatically saved to the system backend. Furthermore, completed samples are automatically added to a dedicated sample dish feature library for subsequent sample re-inspection.

[0078] (2) If Figure 4 and Figure 5 As shown in FIG, the sample retention re-inspection method based on deep learning includes the following steps:

[0079] S6. Sample retention and re-inspection: Sample retention and re-inspection is an important design function of the present invention, which has two steps: sample checking and sample inspection.

[0080] The first step is to check the sample, such as Figure 4 As shown, only a single dish is identified. The system first selects the sampling time for re-inspection and limits the scope of the sample feature library based on the sampling time. Then, a dish that has already been sampled is randomly selected and placed within the camera's field of view. The algorithm automatically locates the dish, crops the pre-sampled features of the dish sub-image, and compares them one by one with the features in the sample feature library to identify the current dish name and sample image number. Based on the dish name and sample image number, the system automatically retrieves the sample order number and its corresponding sample image. The sample image and sample information are then displayed on the system interface, allowing re-inspectors to quickly locate other sample dishes with the same sample order number as the current re-inspection dish.

[0081] The second step is sample inspection, such as Figure 5 As shown, enter the sample inspection page and place all dishes corresponding to the sample images retrieved in the first step within the camera's field of view. Click the one-click re-inspection function to automatically inspect all dishes and compare each dish's features with those on the sample image. This will reveal the dish's type name, sample number, and total weight. If the feature comparison score falls below the re-inspection score threshold (typically set to 0.9), the current sample is considered at risk of being tampered with. The dish will be highlighted in red and displayed as high risk. If the total weight of all dishes being re-inspected exceeds or falls below the re-inspection weight tolerance range, an error message will be displayed indicating that the total re-inspected weight is abnormal, indicating that a dish may have been tampered with, and prompting the re-inspector to conduct further verification. The re-inspection will only be considered passed if the identification score for each dish meets the required criteria and the total weight of the re-inspected dishes falls within the tolerance threshold.

[0082] Specifically, for steps S3 and S4, this embodiment trains a dish target detection model based on the yolov5 target detection framework and a dataset of millions of dishes accumulated by our company in the field of smart group meals. Although the training set already includes dozens of common types of dishes on the market, there is still a high probability of encountering dish types that the model has never been trained on during execution, resulting in a certain risk of missed detection. To solve this problem, the present invention designs a temporary target function in the dish positioning link.

[0083] Manually select the missed target, obtain the roi coordinates, height h and width w information of the target, crop the target sub-image according to the coordinates, and extract the feature vector of the target sub-image through the deep metric learning model Assume that the height of the retained image is H and the width is W. The retained image is divided into equal parts according to the height h and width w of the target sub-image. The cutting rules are as follows:

[0084] Assume that h and w are not equal to 0, and is the quotient of H and h, is the remainder of H and h, similarly is the quotient of W and w, is the remainder of W and w, then the cutting formula of the retained image is:

[0085] ;

[0086] The cutting conditions of each sub-graph of the retained image are:

[0087] ① , cut equally, the roi (x1, y1, x2, y2) coordinates of each sub-graph are as follows:

[0088]

[0089] ② , cut equally, the roi (x1, y1, x2, y2) coordinates of each sub-graph are as follows:

[0090]

[0091] The roi (x1, y1, x2, y2) coordinates of each sub-image in the last group in the x-coordinate direction are as follows:

[0092]

[0093] ③ , cut equally, the roi (x1, y1, x2, y2) coordinates of each sub-graph are as follows:

[0094]

[0095] The roi (x1, y1, x2, y2) coordinates of each sub-image in the last group in the y-coordinate direction are as follows:

[0096]

[0097] ④ , cut equally, the roi (x1, y1, x2, y2) coordinates of each sub-graph are as follows:

[0098]

[0099] The roi (x1, y1, x2, y2) coordinates of each sub-image in the last group in the x-coordinate direction are as follows:

[0100]

[0101] The roi (x1, y1, x2, y2) coordinates of each sub-image in the last group in the y-coordinate direction are as follows:

[0102]

[0103] The roi (x1, y1, x2, y2) coordinates of the last subimage in the x and y coordinate directions are as follows:

[0104]

[0105] Using the sliding window algorithm, the sub-images obtained by cutting are intersected with the detected food areas in the retained sample image one by one, and the sub-image areas with an intersection-and-union ratio greater than the set threshold are discarded. The remaining sub-image areas are limited to not being in the detected food areas, and then the features of each sub-image that meets the conditions are extracted one by one and compared with the features of the temporarily added detection target. Compare the feature vectors and retain the ROI coordinates of the region with the highest score as the ROI coordinates of the temporarily added target. This simple method can make up for the model's lack of detection of certain targets.

[0106] Specifically, the present invention has made special designs for the total feature library and the sample dish feature library that appear in step S4 and step S5. The total feature library contains all the dish features accumulated in the entire cafeteria, which is used for dish identification in the sample retention process, while the sample retention feature library only contains the dishes that have been sampled in the current meal period of the day, which is used for subsequent sample re-inspection and identification. The two feature libraries are not related to each other.

[0107] (1) Feature database

[0108] This database is specifically designed for the dish sample recognition stage. Using the Add Temporary Dish feature, you can enter a dish, save its name and the feature vector of its sub-image, and build a comprehensive library of dish features. Once a sampled dish is located, the sub-image is cropped based on the ROI coordinates. A deep metric learning model is used to extract the feature vectors of each sub-image, which are then compared against the features in the comprehensive library. The dish category with the highest comparison score is the target category.

[0109] (2) Sample dish feature library

[0110] This database is specifically used for sample retention and re-inspection. Once a sample dish is successfully identified, the one-click print function completes the sample retention for the current order. The image of the sample order is automatically saved to the local Order Images directory, using the sample order number as the image name. The dish category, feature vector, sample number, order number, and weight of each sub-image are also entered into the sample feature database.

[0111] Specifically, for the sample re-inspection in step S6, the present invention adds a sample re-inspection function to the sample retention system, enriching the sample retention system's functionality while improving the credibility and security of the sample data. The sample re-inspection consists of two parts: sample checking and sample verification. Sample checking is used to identify a sampled dish and quickly find other dishes with the same sample order number. Sample verification verifies whether there are significant differences between the characteristic information of each sampled dish found through sample checking and that of the dish before the sample was retained.

[0112] ① Check samples

[0113] Place a sampled dish within the camera's shooting range. The target detection algorithm automatically locates the dish, cuts the subgraph according to the positioning coordinates to extract the feature vector, and then compares it one by one with the features in the sampled dish feature library. The one with the highest score is the category of the current dish. After obtaining the dish category, the algorithm automatically analyzes the sampled order number corresponding to the dish category, and indexes the sampled order image in the local orderImages directory according to the sampled order number, and displays it on the system's sample query result page. It also displays the name, sample number, weight, and other information of each dish on the sampled order image, such as Figure 6 shown.

[0114] ② Inspection sample

[0115] The sample checking step quickly identifies the information of each dish in the same order, allowing re-inspection personnel to efficiently find the corresponding sample dishes in the sample cabinet. Each dish and the sample checking dish are placed within the camera's shooting range. The algorithm automatically extracts the feature vectors of each current dish sub-graph and compares them one by one with the sample dish feature library. The recognition score is first determined. If the comparison score is lower than the re-inspection score threshold (generally defaulting to 0.9), the system automatically selects the dish with a red frame and indicates a high risk. The sample sample number of the dish is then compared. If the recognition score meets the requirements, the sample sample number of each dish is further compared with the sub-number of each dish corresponding to the sample order retrieved by the sample checking. If the sample sample number is different, a high risk indicator is also displayed. Finally, a weight check is performed. If the identification score and the sample number meet the requirements, the total weight of each dish is compared with the total weight of the sample order to see if the difference is within the re-inspection weight tolerance threshold. If it exceeds the range, the dishes currently being re-inspected are at high risk, and the re-inspection personnel are reminded to conduct further verification. Finally, the re-inspection results are synchronized to the system backend for archiving.

[0116] Another aspect of the present invention provides a system for retaining and retesting food samples based on deep learning, and designs corresponding software to execute the above-mentioned method for retaining and retesting food samples based on deep learning through program code, including:

[0117] The permission verification module is used to verify the permissions of the sample retention personnel. After the permission verification is passed, the basic information of the sample retention personnel is recorded;

[0118] The label information setting module is used to determine the current meal period based on the current time after the permission verification is passed, and to determine the header information of the dish sample label based on the basic information of the sample retainer;

[0119] The dish positioning module is used to obtain the sample image of the dish to be sampled, and automatically detect whether there is a dish target in the sample image based on the deep learning target detection algorithm. If so, the positioning coordinates and target quantity of the dish target are obtained;

[0120] The dish recognition module is used to automatically cut the detected dish target into a dish sub-graph according to the positioning coordinates and extract the feature vector of the dish sub-graph. The feature vector is compared with the dish features in the feature library, and the current dish target is identified. The current dish target is determined to be invalid or pending confirmation. The invalid dish is added as a temporary dish and entered into the feature library. The pending confirmation dish is corrected to obtain the correct category.

[0121] The dish sampling module is used to obtain dish sample information based on the dish recognition result, combine the header information with the dish sample information to obtain the complete information of the dish sample label, and record it in combination with the sample image to complete the sample. The dish after sampling is added to the sample feature library, wherein the dish sample information includes the sampling time, sample category, sample number, sample weight, sample order number, and sample target quantity;

[0122] The dish re-inspection module is used to determine the re-inspection scope of the sample dish feature library based on the sample retention time for re-inspection, select re-inspected dishes from the dishes for which sampling has been completed, compare the image features of the re-inspected dishes with the features of dishes within the re-inspection range of the sample dish feature library, obtain the dish sample information and sample image corresponding to the re-inspected dish, and complete the sample inspection; inspect other dishes in the sample image, and based on the re-inspection score threshold, determine whether there is a risk of modification for all dishes in the sample image, and complete the sample inspection.

[0123] In summary, the method and system for retaining and rechecking dishes based on deep learning provided by the present invention, based on the deep learning algorithm for retaining dishes, automatically identifies dishes, automatically weighs and prints with one click, reduces the manual recording of sample information by sample personnel, and the writing of sample labels, greatly improves work efficiency and reduces workload. The sample retention process automatically records detailed information of the sampled dishes, such as dish name, sample number, sample order number, sample personnel, etc., and automatically saves them locally on the device and pushes them to the system background. Local sample data facilitates subsequent sample rechecks, problem tracing and cause analysis, and system background data facilitates systematic data management. Based on the sample retention system, the present invention adds a sample recheck function, efficiently searches for all dishes of the same sample order through the sample checking link, and automatically compares the changes in the dishes before and after the sample retention through the sample inspection link, realizing rapid verification, and adding a sample security mechanism to the sample retention system to reduce the risk of tampering with the sample, improve the credibility of food problem tracing and cause analysis, and make the sample retention work more scientific, standardized and reliable, providing a strong guarantee for food safety. The present invention adds a temporary target detection function in the dish target detection link. By manually selecting the target, the target sub-image is cropped according to the manually selected coordinate information to extract the feature vector, and the sliding window algorithm and feature comparison algorithm of the limited area are used to identify the coordinate information of the target in the subsequent sample image, thereby making up for the shortcomings of the detection model.

[0124] The present invention also provides an electronic device comprising one or more processors; a memory; and one or more applications, wherein the one or more applications are stored in the memory and configured to be executed by the one or more processors, and the one or more programs are configured to execute the method for retaining and retesting food samples based on deep learning as described above.

[0125] The above description of the disclosed embodiments is intended to enable one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be readily apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention is not limited to the embodiments shown herein but is intended to conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for retaining and retesting food samples based on deep learning, characterized in that: The following steps are involved: S1, perform permission verification on the sample retainer. After the permission verification passes, record the basic information of the sample retainer; S2: After the permission check is passed, the current meal segment is determined based on the current time, and the header information of the dish sample label is determined based on the basic information of the sample retainer; S3, obtaining a sample image of the dish to be sampled, and automatically detecting whether there is a dish target in the sample image based on a deep learning target detection algorithm. If so, obtaining the location coordinates and target quantity of the dish target; S4: For the detected dish target, the dish subgraph is automatically cut according to the positioning coordinates and the feature vector of the dish subgraph is extracted. The feature vector is compared with the dish features in the feature library, and the current dish target is identified. The current dish target is determined to be invalid or pending confirmation. The invalid dish is added as a temporary dish and entered into the feature library. The pending confirmation dish is corrected to obtain the correct category. S5, based on the dish recognition result, obtain the dish sample information, combine the header information with the dish sample information to obtain the complete information of the dish sample label, and record it in combination with the sample image to complete the sample. The dish sampled is added to the sample feature library, where the dish sample information includes the sample time, sample category, sample number, sample weight, sample order number, and target sample quantity. S6, based on the sample retention time for re-inspection, determine the re-inspection scope of the sample dish feature library, select re-inspected dishes from the dishes for which sampling has been completed, compare the image features of the re-inspected dishes with the features of the dishes within the re-inspection range of the sample dish feature library, obtain the dish sample retention information and sample image corresponding to the re-inspected dish, and complete the sample inspection; inspect other dishes in the sample image, and based on the re-inspection score threshold, determine whether there is a risk of modification for all dishes in the sample image, and complete the sample inspection.

2. The method for retaining and retesting food samples based on deep learning according to claim 1, characterized in that: In S1, the permission of the sample retainer is verified. After the verification is passed, the basic information of the sample retainer is recorded, including: Perform facial recognition on the sample retention personnel to determine whether they are authorized personnel. If so, the facial recognition is successful, the authority verification is passed, and the facial image and name of the sample retention personnel are recorded.

3. The method for retaining and retesting food samples based on deep learning according to claim 2, characterized in that: Said S1 also includes that after the face recognition is successful, the sample retainer enters the exclusive authority password. If the password is correct, the authority verification is passed and the face picture and name of the sample retainer are recorded.

4. The method for retaining and retesting food samples based on deep learning according to claim 1, characterized in that: In S2, determining the current meal segment according to the current time specifically includes: The current time is determined to be the breakfast period, lunch period or dinner period; the header information of the dish sample retention label includes the name of the sample retention person and the sample retention period.

5. The method for retaining and retesting food samples based on deep learning according to claim 1, characterized in that: Said S3 further includes: If the target detection algorithm misses a detection, manually select the missed target dish with a rectangular frame on the software interface and click OK to add a temporary target. Based on the target detection algorithm, the missed target sub-image is cut according to the coordinates of the manually selected rectangular box, the features of the target sub-image are extracted through the depth measurement model, and the features, width and height information of the target sub-image are saved; When the target sub-image appears again in the image of the dish to be sampled, the target detection algorithm is used to automatically call the sliding window algorithm and feature comparison algorithm of the limited area based on the features, width and height information to quickly obtain the positioning coordinate information of the target sub-image.

6. The method for retaining and retesting food samples based on deep learning according to claim 1, characterized in that: In S4, the feature vector is compared with the dish features in the feature library, the current dish target is identified, and the current dish target is determined to be an invalid dish or a dish to be confirmed. The invalid dish is added as a temporary dish and entered into the feature library. The dish to be confirmed is corrected to obtain the correct dish category, which specifically includes: If the feature database is empty, the current dish target is determined to be an invalid dish based on the feature vector, and the current dish target is added as a temporary dish and entered into the feature database; If the total feature library is not empty, a dish recognition score is obtained based on the comparison between the feature vector and the dish features in the total feature library. If the dish recognition score is lower than the trustworthy threshold score, the dish target is determined to be a dish to be confirmed; wherein the dish recognition score is the highest value among multiple comparison scores obtained by comparing the feature vector with multiple dish features in the total feature library; Correct the dishes to be confirmed, sort the multiple comparison scores from high to low, obtain the top 10 scores, and determine whether there are similar dish categories. If so, correct them to get the correct dish category. If not, determine it as an invalid dish, add it as a temporary dish, and enter it into the feature library.

7. The method for retaining and retesting food samples based on deep learning according to claim 1, characterized in that: In S5, the complete information of the dish sample label is obtained and recorded in combination with the sample image to complete the sample. The dish after the sample is completed is added to the sample dish feature library, which specifically includes: Print and paste the sample label on the corresponding sample dish; The complete information, positioning coordinates and sample images of each sample dish are automatically saved. At the same time, the completed sample dishes are added to the sample dish feature library for re-inspection.

8. The method for retaining and retesting food samples based on deep learning according to claim 1, characterized in that: In S6, the checking of samples specifically includes: Based on the sample retention time for re-inspection, determine the re-inspection scope of the sample dish feature library, and select re-inspection dishes from the dishes for which the samples have been retained; The dish to be re-inspected is placed in the sampling area of ​​the image acquisition device to obtain the dish image, and the dish image is automatically positioned, sub-images are cropped, and features are extracted based on the target detection algorithm to obtain the image features of the dish to be re-inspected; Compare the image features of the re-inspected dish with the features of the dishes within the re-inspection range in the retained sample dish feature library to obtain the retained sample dish category and retained sample number corresponding to the re-inspected dish; According to the category of sample dishes and sample image number, the sample order number and sample image corresponding to the re-inspected dish are retrieved.

9. The method for retaining and retesting food samples based on deep learning according to claim 1, characterized in that: In said S6, said sample inspection specifically includes: Enter the sample inspection page, check the sample image obtained in the sample inspection step with other dishes corresponding to the sample image, and combine it with the sample inspection results of the re-inspected sample to obtain the sample information corresponding to all dishes in the sample image; Compare the scores of all dishes in the sample image with the dish features within the re-inspection range in the sample dish feature library. If the score is lower than the re-inspection score threshold, it is considered that the current dish has the risk of being modified. Compare the actual weight of all dishes in the sample image with the sample weight in the sample information. If the actual weight exceeds or falls below the sample weight, it is considered that the current dish is at risk of being modified. When the score is higher than or equal to the re-inspection score threshold and equal to the retained sample weight, the current dish is considered to have passed the re-inspection.

10. A system for retaining and retesting food samples based on deep learning, used to execute the method for retaining and retesting food samples based on deep learning according to any one of claims 1 to 9, characterized in that: include: The permission verification module is used to verify the permissions of the sample retention personnel. After the permission verification is passed, the basic information of the sample retention personnel is recorded; The label information setting module is used to determine the current meal period based on the current time after the permission verification is passed, and to determine the header information of the dish sample label based on the basic information of the sample retainer; The dish positioning module is used to obtain the sample image of the dish to be sampled, and automatically detect whether there is a dish target in the sample image based on the deep learning target detection algorithm. If so, the positioning coordinates and target quantity of the dish target are obtained; The dish recognition module is used to automatically cut the detected dish target into a dish sub-graph according to the positioning coordinates and extract the feature vector of the dish sub-graph. The feature vector is compared with the dish features in the feature library, and the current dish target is identified. The current dish target is determined to be invalid or pending confirmation. The invalid dish is added as a temporary dish and entered into the feature library. The pending confirmation dish is corrected to obtain the correct category. The dish sampling module is used to obtain dish sample information based on the dish recognition result, combine the header information with the dish sample information to obtain the complete information of the dish sample label, and record it in combination with the sample image to complete the sample. The dish after sampling is added to the sample feature library, wherein the dish sample information includes the sampling time, sample category, sample number, sample weight, sample order number, and sample target quantity; The dish re-inspection module is used to determine the re-inspection scope of the sample dish feature library based on the sample retention time for re-inspection, select re-inspected dishes from the dishes for which sampling has been completed, compare the image features of the re-inspected dishes with the features of dishes within the re-inspection range of the sample dish feature library, obtain the dish sample information and sample image corresponding to the re-inspected dish, and complete the sample inspection; inspect other dishes in the sample image, and based on the re-inspection score threshold, determine whether there is a risk of modification for all dishes in the sample image, and complete the sample inspection.

Images