Deep learning-based dish sample reserving method and system and reserved sample reinspection method and system

Abstract

The invention relates to the technical field of dish supervision, and discloses a dish sample reserving and reserved sample reinspection method and system based on deep learning, and the method comprises the steps: carrying out the authority verification of a sample reserving person, and determining the head information of a dish sample reserving label based on the current meal section and the basic information of the sample reserving person after the authority verification is passed; obtaining a reserved sample image of a dish to be subjected to sample reservation, and detecting a dish target in the reserved sample image based on a target detection algorithm; performing dish identification on the detected dish target, obtaining dish sample information based on a dish identification result, and performing combined recording on the dish sample information and a sample image to complete sample reservation; selecting reinspection dishes from the dishes after sample reservation, and checking the samples to obtain dish sample reservation information and sample reservation images; other dishes in the reserved sample image are subjected to sample checking, and sample checking is carried out based on a rechecking score threshold value. According to the invention, dishes are intelligently identified, sample reserving dishes are automatically recorded, the sample reserving efficiency and accuracy are improved, a reserved sample reinspection function is added, and the sample reserving safety and credibility are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of dish supervision, and in particular to a method and system for retaining and re-inspecting dish samples based on deep learning. Background Art

[0002] In the catering industry, keeping samples of dishes is a necessary process required by explicit laws and regulations, and is also one of the important measures to ensure food safety. By keeping samples of dishes for each meal, when food safety issues occur, they can be quickly verified, providing key evidence for tracing the source and analyzing the cause.

[0003] In the process of retaining samples of dishes, information collection, archiving and retrieval, risk analysis, and traceability monitoring of samples need to be given greater attention, and require more scientific and comprehensive management methods. The current retention of dishes mainly relies on manual operations, including selecting samples, recording relevant information, placing samples in fresh-keeping equipment, and preserving them according to the prescribed time and conditions, but manual sampling is inefficient, especially during peak catering periods, which will affect the normal catering service process. Staff members need to spend time recording samples, writing sample labels, and other operations, which increases their workload; manual sampling information is prone to errors or omissions, such as incomplete or inaccurate records of key information such as dish name, sampling time, and operator, which brings difficulties to subsequent tracing and analysis; manual sampling is prone to modifying sample information and lacks credibility.

[0004] With the development of science and technology, some improvements have been made in the sampling of dishes. Some intelligent systems, software and equipment have gradually emerged, such as automatic identification of sample dishes based on artificial intelligence, automatic weighing and printing of sample labels, and one-click uploading of all sample details to the background system. However, the functions of existing intelligent systems, software and equipment are relatively single, and they often only focus on the sampling of dishes, lacking safety monitoring and sample review of sample retention. When a food safety accident occurs, if the sample dishes are tampered with by humans, it will affect the tracing and analysis of the cause of the accident, and the safety and credibility of the sample retention data lack persuasiveness.

[0005] Therefore, how to improve the efficiency and accuracy of food sampling and enhance the security and credibility of sample data has become a technical problem that needs to be urgently solved in this field. Summary of the invention

[0006] The purpose of the present invention is to provide a method and system for retaining samples of dishes and re-inspecting samples based on deep learning, which can intelligently identify dishes through deep learning, automatically record retained samples, improve the efficiency and accuracy of dish retention, and add the function of re-inspecting samples, thereby improving the security and credibility of retained sample data.

[0007] To achieve the above object, the present invention provides the following solutions: A method for retaining and retesting food samples based on deep learning, the method comprising the following steps: S1, perform permission verification on the sample retainer. After the permission verification passes, record the basic information of the sample retainer; S2: After the permission check is passed, the current meal period is determined according to the current time, and the header information of the dish sample label is determined in combination with the basic information of the sample retainer; S3, obtaining a sample image of the dish to be sampled, and automatically detecting whether there is a dish target in the sample image based on a deep learning target detection algorithm, and if so, obtaining the location coordinates and target quantity of the dish target; S4, for the detected dish target, automatically cut the dish sub-graph according to the positioning coordinates and extract the feature vector of the dish sub-graph, compare the feature vector with the dish feature in the feature library, perform dish recognition on the current dish target, determine that the current dish target is an invalid dish or a dish to be confirmed, add the invalid dish as a temporary dish, enter it into the feature library, and correct the dish to be confirmed to obtain the correct category; S5, based on the dish recognition result, obtain the dish sample information, combine the header information with the dish sample information, obtain the complete information of the dish sample label, and record it in combination with the sample image to complete the sample, and add the dish after the sample is completed to the sample dish feature library, wherein the dish sample information includes the sample time, sample dish category, sample sample number, sample weight, sample order number, and sample target quantity; S6, based on the sampling time for re-inspection, determine the re-inspection scope of the sample dish feature library, select re-inspected dishes from the dishes for which sampling has been completed, compare the image features of the re-inspected dishes with the features of dishes within the re-inspection range of the sample dish feature library, obtain the dish sampling information and sample image corresponding to the re-inspected dish, and complete the sample inspection; inspect other dishes in the sample image, and based on the re-inspection score threshold, determine whether there is a risk of modification for all dishes in the sample image, and complete the sample inspection.

[0008] Furthermore, in S1, the permission of the sample retainer is verified. After the verification is passed, the basic information of the sample retainer is recorded, including: Perform facial recognition on the sample retention personnel to determine whether they are authorized personnel. If so, the facial recognition is successful, the authority check is passed, and the facial image and name of the sample retention personnel are recorded.

[0009] Furthermore, S1 also includes that after the face recognition is successful, the sample retention personnel enters a dedicated authority password. If the password is verified to be correct, the authority verification is passed and the face picture and name of the sample retention personnel are recorded.

[0010] Further, in S2, determining the current meal segment according to the current time specifically includes: The current time is used to determine whether it is the breakfast period, the lunch period or the dinner period; the header information of the dish sample retention label includes the name of the sample retention person and the sample retention period.

[0011] Furthermore, the S3 further includes: If the target detection algorithm misses a detection, manually select the missed target dish with a rectangular frame on the software interface and click OK to add a temporary target. Based on the target detection algorithm, the missed target sub-image is cut according to the coordinates of the manually selected rectangular frame, the features of the target sub-image are extracted through the deep measurement model, and the features, width and height information of the target sub-image are saved; When the target sub-image appears again in the image of the dish to be sampled, the target detection algorithm is used to automatically call the sliding window algorithm and feature comparison algorithm of the limited area based on the feature, width and height information to quickly obtain the positioning coordinate information of the target sub-image.

[0012] Furthermore, in S4, the feature vector is compared with the dish features in the feature library, the current dish target is identified, the current dish target is determined to be an invalid dish or a dish to be confirmed, the invalid dish is added as a temporary dish, entered into the feature library, and the dish to be confirmed is corrected to obtain the correct dish category, specifically including: If the feature database is empty, the current dish target is determined to be an invalid dish based on the feature vector, and the current dish target is added as a temporary dish and entered into the feature database; If the total feature library is not empty, then based on the feature vector and the dish features in the total feature library, a dish recognition score is obtained. If the dish recognition score is lower than the trustworthy threshold score, the dish target is determined to be a dish to be confirmed; wherein the dish recognition score is the highest value among multiple comparison scores obtained by comparing the feature vector with multiple dish features in the total feature library; Correct the dishes to be confirmed, sort the multiple comparison scores from high to low, get the top 10 scores, and determine whether there are similar dish categories. If so, correct them to get the correct dish category. If not, determine them as invalid dishes, add them as temporary dishes, and enter them into the feature library.

[0013] Furthermore, in S5, the complete information of the dish sample label is obtained and recorded in combination with the sample image to complete the sample retention. The dish after the sample retention is added to the sample dish feature library, which specifically includes: Print and paste the sample labels on the corresponding sample dishes; The complete information, location coordinates and sample images of each sampled dish are automatically saved. At the same time, the completed sampled dishes are added to the sampled dish feature library for re-inspection.

[0014] Further, in S6, the checking of samples specifically includes: Based on the sample retention time for re-inspection, determine the re-inspection scope of the sample dish feature library, and select the re-inspection dishes from the dishes for which the samples have been retained; Placing the re-inspected dish in the sampling area of ​​the image acquisition device to obtain the dish image, automatically positioning the dish image, cropping the sub-image and extracting the features based on the target detection algorithm, and obtaining the image features of the re-inspected dish; Compare the image features of the re-inspected dish with the features of the dishes within the re-inspection range in the retained sample dish feature library to obtain the retained sample dish category and retained sample number corresponding to the re-inspected dish; According to the sample dish category and sample image number, retrieve the sample order number and sample image corresponding to the re-inspected dish.

[0015] Further, in said S6, said sample inspection specifically includes: Enter the sample inspection page, check the sample image obtained in the sample inspection step, check other dishes corresponding to the sample image, and combine it with the sample inspection result of the re-inspected sample to obtain the sample information corresponding to all dishes in the sample image; The scores obtained by comparing the features of all dishes in the sample image with the features of dishes within the re-inspection range of the sample dish feature library are compared with the re-inspection score threshold. If the score is lower than the re-inspection score threshold, it is considered that the current dish is at risk of being modified. The actual weight of all dishes in the sample image is compared with the sample weight in the sample information. If it exceeds or falls below the sample weight, it is considered that the current dish is at risk of being modified. When it is higher than or equal to the re-inspection score threshold and equal to the retained sample weight, the current dish is considered to have passed the re-inspection.

[0016] The present invention also provides a system for retaining samples and re-inspecting samples of dishes based on deep learning, which is used to execute the method for retaining samples and re-inspecting samples of dishes based on deep learning, comprising: The permission verification module is used to verify the permissions of the sample retention personnel. After the permission verification passes, the basic information of the sample retention personnel is recorded; The label information setting module is used to determine the current meal period according to the current time after the permission verification is passed, and to determine the header information of the dish sample label in combination with the basic information of the sample retainer; The dish positioning module is used to obtain the sample image of the dish to be sampled, and automatically detect whether there is a dish target in the sample image based on the deep learning target detection algorithm. If so, the positioning coordinates and target quantity of the dish target are obtained; The dish recognition module is used to automatically cut the detected dish target into a dish sub-graph according to the positioning coordinates and extract the feature vector of the dish sub-graph, compare the feature vector with the dish features in the feature library, perform dish recognition on the current dish target, determine whether the current dish target is an invalid dish or a dish to be confirmed, add the invalid dish as a temporary dish, enter it into the feature library, and correct the dish to be confirmed to obtain the correct category; The dish sample retention module is used to obtain the dish sample retention information based on the dish recognition result, combine the header information with the dish sample retention information, obtain the complete information of the dish sample retention label, and record it in combination with the sample retention image to complete the sample retention, and add the dish after the sample retention to the sample retention dish feature library, wherein the dish sample retention information includes the sample retention time, sample retention dish category, sample retention sample number, sample retention weight, sample retention order number, and sample retention target quantity; The dish re-inspection module is used to determine the re-inspection scope of the sample dish feature library based on the sample retention time for re-inspection, select re-inspected dishes from the dishes that have completed sampling, compare the image features of the re-inspected dishes with the features of dishes within the re-inspection range of the sample dish feature library, obtain the dish sample information and sample image corresponding to the re-inspected dish, and complete the sample inspection; check the other dishes in the sample image, and based on the re-inspection score threshold, determine whether there is a risk of modification for all the dishes in the sample image, and complete the sample inspection.

[0017] According to the specific embodiments provided by the present invention, the method and system for retaining samples and re-inspecting samples based on deep learning provided by the present invention disclose the following technical effects: (1) Perform permission verification on sample retention personnel to avoid misoperation by non-sample retention personnel, obtain basic information of sample retention personnel and add it to the header information of the dish sample retention label to facilitate subsequent investigation of the sample retention personnel; (2) Through deep learning algorithms, the system automatically locates and identifies sample dishes, automatically integrates information such as dish name, dish weight, and sample retention personnel, and obtains dish sample labels, which greatly reduces the workload of sample retention personnel and significantly improves sample retention efficiency; (3) On the basis of completing the sample retention, add the sample re-inspection function, efficiently find all dishes of the same sample retention order through the sample checking link, automatically compare the changes of dishes before and after the sample retention through the sample inspection link, and realize rapid verification. Add a sample retention safety mechanism to the sample retention system to reduce the risk of tampering with the sample retention, improve the credibility of food problem tracing and cause analysis, make the sample retention work more scientific, standardized and reliable, reduce the risk of abnormal manual handling of sample retention dishes, affect the traceability and cause analysis of food safety accidents, and improve the safety and credibility of sample retention; (4) By retaining samples of dishes and conducting re-inspection, it provides a strong guarantee for food safety and has important application value. It can be applied to dining scenes in canteens of major enterprises, units, hospitals, troops and schools. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0019] Figure 1 is a flow chart of a method for retaining food samples based on deep learning in an embodiment of the present invention; Figure 2 is a flow chart of target detection of dish samples based on deep learning in an embodiment of the present invention; Figure 3 Schematic diagram of a sample retention label according to an embodiment of the present invention, wherein (a) is a sample retention label for a single sampled dish in an order, and (b) is a sample retention label for multiple sampled dishes in an order; Figure 4 This is a sample checking flow chart for sample retention and re-inspection according to an embodiment of the present invention; Figure 5 This is a sample inspection flow chart for sample retention and re-inspection according to an embodiment of the present invention. Figure 6 Schematic diagram of sample checking results according to an embodiment of the present invention. DETAILED DESCRIPTION

[0020] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0021] The purpose of the present invention is to provide a method and system for retaining samples and re-inspecting samples of dishes based on deep learning, which automatically locates and identifies the dishes for retaining samples through deep learning algorithms, automatically integrates information such as dish names, dish weights, and sample retainers, and automatically prints sample labels, which greatly reduces the workload of sample retainers and significantly improves sample retention efficiency. At the same time, a sample retention and re-inspection function is designed to automatically record the sample retention order main number and dish sub-number for each sampled dish, and compare and verify the order main number and dish self-number, as well as the dish identification features before and after the dish is sampled, to reduce the risk of abnormal manual processing of sampled dishes, affect the traceability and cause analysis of food safety accidents, and improve the safety and credibility of dish retention.

[0022] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0023] The method for retaining and retesting food samples based on deep learning provided by the present invention comprises the following steps: S1, perform permission verification on the sample retainer. After the permission verification passes, record the basic information of the sample retainer; S2: After the permission check is passed, the current meal period is determined according to the current time, and the header information of the dish sample label is determined in combination with the basic information of the sample retainer; S3, obtaining a sample image of the dish to be retained, and automatically detecting whether there is a dish target in the sample image based on a deep learning target detection algorithm, and if so, obtaining the positioning coordinates and target quantity of the dish target; for example, placing the dish to be retained in a sampling area of ​​an image acquisition device to obtain a sample image, and the image acquisition device is also equipped with a weighing device, which can weigh the dish to obtain the sample weight. For other configurations, reference can be made to existing dish sample retention devices in the art; S4, for the detected dish target, automatically cut the dish sub-graph according to the positioning coordinates and extract the feature vector of the dish sub-graph, compare the feature vector with the dish feature in the feature library, perform dish recognition on the current dish target, determine that the current dish target is an invalid dish or a dish to be confirmed, add the invalid dish as a temporary dish, enter it into the feature library, and correct the dish to be confirmed to obtain the correct category; S5, based on the dish recognition result, obtain the dish sample information, combine the header information with the dish sample information, obtain the complete information of the dish sample label, and record it in combination with the sample image to complete the sample, and add the dish after the sample is completed to the sample dish feature library, wherein the dish sample information includes the sample time, sample dish category, sample sample number, sample weight, sample order number, and sample target quantity; S6, based on the sampling time for re-inspection, determine the re-inspection scope of the sample dish feature library, select re-inspected dishes from the dishes for which sampling has been completed, compare the image features of the re-inspected dishes with the features of dishes within the re-inspection range of the sample dish feature library, obtain the dish sampling information and sample image corresponding to the re-inspected dish, and complete the sample inspection; inspect other dishes in the sample image, and based on the re-inspection score threshold, determine whether there is a risk of modification for all dishes in the sample image, and complete the sample inspection.

[0024] Example: The method for retaining samples and re-examining samples based on deep learning provided in an embodiment of the present invention is performed by a system for retaining samples and re-examining samples based on deep learning. The method specifically includes a method for retaining samples based on deep learning and a method for retaining samples and re-examining samples based on deep learning, which are as follows: (1) If Figure 1 and Figure 2 As shown, the dish sampling method based on deep learning includes the following steps: S1. Authority verification: The system for retaining and re-inspecting dishes samples based on deep learning designed by the present invention requires facial recognition verification when logging in. When the sample retaining personnel operates, the face is automatically detected and recognized. Only authorized personnel can successfully recognize the face. After successful recognition, the face picture of the sample retaining personnel is saved. In order to improve the security of authority verification, after successful face recognition, the exclusive authority password of the recognition personnel needs to be entered. Only when the face and password are successfully verified can the sample retaining system be used freely and normally.

[0025] S2. Print label information setting: After the sample retention system passes the permission verification, the current meal period is automatically set to breakfast, lunch or dinner according to the current time, and the information of the person is obtained according to the identification person, and then the header information of the print label is automatically preset, such as Figure 3 shown.

[0026] S3. Dish positioning: Enter the sample retention page and start sampling dishes. Place the dish within the shooting range of the camera of the image acquisition device. The target detection algorithm based on deep learning designed by the present invention automatically locates whether there is a dish target in the current image. If it exists, the positioning box and target quantity of the dish target are displayed. If it is found that the target detection algorithm misses the detection, the temporary target function is added to manually select the missed target dish with a rectangular box on the system interface, and click OK. The target detection algorithm will cut the missed target sub-graph according to the manually selected rectangular coordinates, extract the features of the target sub-graph through the deep measurement model, and save the features and the width and height information of the target sub-graph to the system. Subsequently, when a new picture of the missed target dish is encountered, the target sub-graph appears again, and the target detection algorithm is used to automatically call the sliding window algorithm and feature comparison algorithm of the limited area based on the features, width and height information to quickly obtain the positioning coordinate information of the target sub-graph.

[0027] S4, dish identification: When used for the first time, the algorithm's dish feature library is empty. When retaining a dish sample, it is necessary to enter the dish to establish a feature library. After detecting the sample dish target, the algorithm automatically cuts the dish sub-graph according to the positioning coordinates and extracts the feature vector of the dish sub-graph. Since the current feature library is empty, the current dish will be displayed as an "invalid dish". The present invention is designed with a function of adding temporary dishes. By clicking on the picture of the current sample dish displayed on the system, entering the page of adding temporary dishes, and choosing to add the current sample dish as a temporary dish, the dish entry function can be quickly completed, and the current dish will be displayed from an invalid dish to the newly entered dish name. When the total feature library is not empty, but the dish recognition score is lower than the algorithm's trust threshold score, the current dish will be displayed as "pending dish"; manually click on the picture of the dish on the system to enter the dish correction page, which will display the top 10 results with the highest scores when comparing the current dish with the total feature library, and sort them from high to low by comparison score. The probability of the correct category appearing in these 10 results is 99.9%. Manually select the corresponding correct dish from these 10 results to correct the current recognition result. If none of the 10 results given have the correct category, you can enter the Add Temporary Dish page to enter the current dish into the total feature library.

[0028] S5. Samples: After the sample identification result is confirmed to be correct, use the one-key print function to automatically print out the detailed information of the current sample. Figure 3 As shown. If there are multiple dishes in a picture, one-click printing will print labels corresponding to the number of dishes, and each label paper can be pasted on the corresponding sample dishes. After printing, the label information, positioning coordinates and sample main image of each sample dish will be automatically saved to the system background. At the same time, the dishes that have been sampled will be automatically added to the special sample dish feature library for subsequent sample re-inspection.

[0029] (2) If Figure 4 and Figure 5 As shown, the sample retention re-inspection method based on deep learning includes the following steps: S6. Sample retention and re-inspection: Sample retention and re-inspection is an important design function of the present invention, and the design has two links: sample checking and sample testing.

[0030] The first step is to check the sample, such as Figure 4As shown, only a single dish is identified. First, the sampling time for re-inspection is selected, and the range of the sample dish feature library is limited by the sampling time. Then, the dishes that have been sampled are randomly selected and placed within the shooting field of view of the device camera. The algorithm automatically locates the dish, cuts the features of the dish sub-image in advance, and compares them one by one with the features in the sample dish feature library to identify the current dish name and dish sample number. Then, according to the dish name and dish sample number, the sample order number and the corresponding sample image of the sample at that time are automatically retrieved, and the sample image and sample information are displayed on the system interface, which is convenient for the re-inspection personnel to quickly find and locate other sample dishes with the same sample order number as the current re-inspection dish.

[0031] The second step is to check the sample, such as Figure 5 As shown, enter the sample inspection page, place all the dishes corresponding to the sample images retrieved in the first step under the camera shooting range, click the one-click re-inspection function, automatically check all dishes, and compare the features of each dish on the sample image one by one to obtain the type name of each dish, the sample number of the sample and the total weight of all dishes. If the score of the feature comparison is lower than the re-inspection score threshold (generally set to 0.9), it is considered that the current sample dish is at risk of being modified, and the current dish will be selected with a red frame and prompted to be high risk. If the total weight of all dishes currently re-inspected exceeds or is lower than the tolerance threshold range of the re-inspection weight, it will be prompted that the current re-inspection total weight is abnormal, and there is a risk that the dishes have been modified, reminding the re-inspector to further check. Only when the recognition score of each dish meets the standard and the total weight of the re-inspected dishes is also within the tolerance threshold range will the re-inspection be displayed as qualified.

[0032] Specifically, for step S3 and step S4, this embodiment trains a dish target detection model based on the yolov5 target detection framework and a dataset of millions of dishes accumulated by our company in the field of smart group meals. Although the training set already contains dozens of common types of dishes on the market, there is still a high possibility of encountering types of dishes that the model has never been trained on during the execution process, resulting in a certain risk of missed detection. To solve this problem, the present invention is designed to add a temporary target function in the dish positioning link.

[0033] Manually select the missed target, obtain the roi coordinates, height h and width w information of the target, crop the target sub-image according to the coordinates, and extract the feature vector of the target sub-image through the deep metric learning model Assume that the height of the retained image is H and the width is W. The retained image is cut into equal parts according to the height h and width w of the target sub-image. The cutting rules are as follows: Assume that h and w are not equal to 0, and is the quotient of H and h, is the remainder of H and h. Similarly is the quotient of W and w, is the remainder of W and w, then the cutting formula of the retained image is: ; The cutting conditions of each sub-image of the retained image are: ① , cut equally, the roi (x1, y1, x2, y2) coordinates of each sub-image are as follows: ② , cut equally, the roi (x1, y1, x2, y2) coordinates of each sub-image are as follows: The roi (x1, y1, x2, y2) coordinates of each sub-image in the last group in the x-coordinate direction are as follows: ③ , cut equally, the roi (x1, y1, x2, y2) coordinates of each sub-image are as follows: The roi (x1, y1, x2, y2) coordinates of each sub-image in the last group in the y-coordinate direction are as follows: ④ , cut equally, the roi (x1, y1, x2, y2) coordinates of each sub-image are as follows: The roi (x1, y1, x2, y2) coordinates of each sub-image in the last group in the x-coordinate direction are as follows: The roi (x1, y1, x2, y2) coordinates of each sub-image in the last group in the y-coordinate direction are as follows: The roi (x1, y1, x2, y2) coordinates of the last subimage in the x and y coordinate directions are as follows: Using the sliding window algorithm, the sub-images obtained by cutting are compared with the detected food areas in the retained sample image one by one. The sub-image areas with an intersection and union ratio greater than the set threshold are discarded, and the remaining sub-image areas are limited to not being in the detected food areas. Then, the features of each sub-image that meets the conditions are extracted one by one and compared with the features of the temporarily added detection target. Compare the feature vectors and keep the ROI coordinates of the area with the highest score as the ROI coordinates of the temporarily added target. This simple method can make up for the lack of the model's missed detection of some targets.

[0034] Specifically, the present invention has made a special design for the total feature library and the sample dish feature library that appear in step S4 and step S5, wherein the total feature library contains all dish features accumulated in the entire cafeteria, which is used for dish identification in the sample retention process, while the sample retention feature library only contains dishes that have been sampled in the current meal period of the day, which is used for subsequent sample re-inspection and identification, and the two feature libraries are unrelated to each other.

[0035] (1) Feature Library This database is specifically used for the dish sample recognition stage. Use the Add Temporary Dish function to enter dishes, save the dish name and the feature vector of the dish sub-graph, and establish a total feature library of dishes. When the sample dish target is located, the dish sub-graph will be cropped according to the roi coordinates, and the feature vectors of each sub-graph will be extracted through the deep metric learning model. The features are compared one by one with the features in the total feature library of dishes. The dish category with the highest comparison score is the category of the recognition target.

[0036] (2) Sample dish feature library This database is specifically used for the sample re-inspection stage. After the sample dishes are successfully identified, the one-click printing function is used to complete the sample of the dishes for the current order, and the sample order pictures are automatically saved to the local order Images directory, with the sample order number as the name of the picture. At the same time, the dish category, feature vector, sample number, sample order number and sample weight of each sub-image are entered into the sample dish feature library.

[0037] Specifically, for the sample re-inspection in step S6, the present invention adds a sample re-inspection function on the basis of the sample retention of dishes, enriching the sample retention system function while improving the credibility and security of the sample retention data. The sample re-inspection consists of two parts: sample checking and sample inspection. Sample checking is used to give a sampled dish and quickly find other dishes with the same sample retention order number through identification. Sample inspection is used to verify whether there is a large difference between the characteristic information of each sampled dish found by sample checking and before the sample retention.

[0038] ① Check the sample Place a sampled dish within the camera's shooting range. The target detection algorithm automatically locates the dish, cuts the subgraph according to the location coordinates to extract the feature vector, and then compares it with the features in the sampled dish feature library one by one. The one with the highest score is the category of the current dish. After obtaining the dish category, the algorithm automatically analyzes the sampled order number corresponding to the dish category, and indexes the sampled order image in the local orderImages directory according to the sampled order number, displays it on the system's sample query result page, and displays the name, sampled image number, weight, and other information of each dish on the sampled order image. Figure 6 shown.

[0039] ② Inspection sample Through the sample checking step, the information of each dish of the same order can be quickly identified, so that the re-inspection personnel can efficiently find the corresponding sample dishes from the sample cabinet. Place each dish and the sample checking dish under the camera shooting range, and the algorithm automatically extracts the feature vectors of each current dish sub-graph and compares them one by one with the sample dish feature library. First, make an identification score judgment. If there is a comparison score lower than the re-inspection score threshold (generally the default is 0.9), the system automatically selects the dish with a red frame and prompts a high risk. Then compare the sample number of the dish. If the identification score meets the conditions, further compare the sample number of each dish with the sub-number of each dish corresponding to the sample order retrieved by the sample checking. If there is a difference in the sample number, it also prompts a high risk. Finally, a weight check is performed. If the identification score and the sample number meet the requirements, the total weight of each dish is compared with the total weight of the sample order to see if the difference is within the re-inspection weight tolerance threshold. If it exceeds the range, the dishes currently being re-inspected are at high risk, and the re-inspection personnel are reminded to conduct further verification. Finally, the re-inspection results are synchronized to the system backend for archiving.

[0040] Another aspect of the present invention provides a system for retaining and rechecking dishes based on deep learning, and designs corresponding software to execute the method for retaining and rechecking dishes based on deep learning through program code, including: The permission verification module is used to verify the permissions of the sample retention personnel. After the permission verification passes, the basic information of the sample retention personnel is recorded; The label information setting module is used to determine the current meal period according to the current time after the permission verification is passed, and to determine the header information of the dish sample label in combination with the basic information of the sample retainer; The dish positioning module is used to obtain the sample image of the dish to be sampled, and automatically detect whether there is a dish target in the sample image based on the deep learning target detection algorithm. If so, the positioning coordinates and target quantity of the dish target are obtained; The dish recognition module is used to automatically cut the detected dish target into a dish sub-graph according to the positioning coordinates and extract the feature vector of the dish sub-graph, compare the feature vector with the dish features in the feature library, perform dish recognition on the current dish target, determine whether the current dish target is an invalid dish or a dish to be confirmed, add the invalid dish as a temporary dish, enter it into the feature library, and correct the dish to be confirmed to obtain the correct category; The dish sample retention module is used to obtain the dish sample retention information based on the dish recognition result, combine the header information with the dish sample retention information, obtain the complete information of the dish sample retention label, and record it in combination with the sample retention image to complete the sample retention, and add the dish after the sample retention to the sample retention dish feature library, wherein the dish sample retention information includes the sample retention time, sample retention dish category, sample retention sample number, sample retention weight, sample retention order number, and sample retention target quantity; The dish re-inspection module is used to determine the re-inspection scope of the sample dish feature library based on the sample retention time for re-inspection, select re-inspected dishes from the dishes that have completed sampling, compare the image features of the re-inspected dishes with the features of dishes within the re-inspection range of the sample dish feature library, obtain the dish sample information and sample image corresponding to the re-inspected dish, and complete the sample inspection; check the other dishes in the sample image, and based on the re-inspection score threshold, determine whether there is a risk of modification for all the dishes in the sample image, and complete the sample inspection.

[0041] In summary, the method and system for retaining samples and re-examination of dishes based on deep learning provided by the present invention, the retaining samples of dishes based on deep learning algorithms, automatic identification of dishes, automatic weighing and one-click printing, reducing the manual recording of sample information by the retaining personnel, writing sample labels and other work, greatly improving work efficiency and reducing work burden. The sample retention completes the automatic recording of detailed information of the sampled dishes, such as the name of the dish, the sample number of the retaining sample, the sample order number, the sample retaining personnel, etc., and automatically saves them to the local device and pushes them to the system background. The local sample data facilitates subsequent sample re-examination, problem tracing and cause analysis, and the system background data facilitates systematic data management. On the basis of the sample retention system, the present invention adds a sample re-examination function, efficiently searches for all dishes of the same sample order through the sample checking link, automatically compares the changes of dishes before and after the sample retention through the sample inspection link, realizes rapid verification, and adds a sample retention safety mechanism to the sample retention system to reduce the risk of tampering with the sample retention sample, improve the credibility of food problem tracing and cause analysis, and make the sample retention work of dishes more scientific, standardized and reliable, providing a strong guarantee for food safety. The present invention adds a temporary target detection function in the dish target detection link. By manually selecting the target, the target sub-image is cropped according to the manually selected coordinate information to extract the feature vector, and the sliding window algorithm and feature comparison algorithm of the limited area are used to identify the coordinate information of the target in the subsequent retained sample image, thereby making up for the shortcomings of the detection model.

[0042] The present invention also provides an electronic device, comprising one or more processors; a memory; and one or more applications, wherein the one or more applications are stored in the memory and configured to be executed by the one or more processors, and the one or more programs are configured to execute the method of retaining and re-inspecting dishes based on deep learning as described above.

[0043] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for retaining and retesting food samples based on deep learning, characterized in that: The following steps are involved: S1, perform permission verification on the sample retainer. After the permission verification passes, record the basic information of the sample retainer; S2: After the permission check is passed, the current meal period is determined according to the current time, and the header information of the dish sample label is determined in combination with the basic information of the sample retainer; S3, obtaining a sample image of the dish to be sampled, and automatically detecting whether there is a dish target in the sample image based on a deep learning target detection algorithm, and if so, obtaining the location coordinates and target quantity of the dish target; S4, for the detected dish target, automatically cut the dish sub-graph according to the positioning coordinates and extract the feature vector of the dish sub-graph, compare the feature vector with the dish feature in the feature library, perform dish recognition on the current dish target, determine that the current dish target is an invalid dish or a dish to be confirmed, add the invalid dish as a temporary dish, enter it into the feature library, and correct the dish to be confirmed to obtain the correct category; S5, based on the dish recognition result, obtain the dish sample information, combine the header information with the dish sample information, obtain the complete information of the dish sample label, and record it in combination with the sample image to complete the sample, and add the dish after the sample is completed to the sample dish feature library, wherein the dish sample information includes the sample time, sample dish category, sample sample number, sample weight, sample order number, and sample target quantity; S6, based on the sampling time for re-inspection, determine the re-inspection scope of the sample dish feature library, select re-inspected dishes from the dishes for which sampling has been completed, compare the image features of the re-inspected dishes with the features of dishes within the re-inspection range of the sample dish feature library, obtain the dish sampling information and sample image corresponding to the re-inspected dish, and complete the sample inspection; inspect other dishes in the sample image, and based on the re-inspection score threshold, determine whether there is a risk of modification for all dishes in the sample image, and complete the sample inspection.

2. The method for retaining and retesting food samples based on deep learning according to claim 1, characterized in that: In S1, the permission check is performed on the sample retainer. After the check is passed, the basic information of the sample retainer is recorded, including: Perform facial recognition on the sample retention personnel to determine whether they are authorized personnel. If so, the facial recognition is successful, the authority check is passed, and the facial image and name of the sample retention personnel are recorded.

3. The method for retaining and retesting food samples based on deep learning according to claim 2, characterized in that: The S1 also includes that after the face recognition is successful, the sample retention personnel enters the exclusive authority password. If the password is verified to be correct, the authority verification is passed and the face picture and name of the sample retention personnel are recorded.

4. The method for retaining and retesting food samples based on deep learning according to claim 1, characterized in that: In S2, determining the current meal segment according to the current time specifically includes: The current time is used to determine whether it is the breakfast period, the lunch period or the dinner period; the header information of the dish sample retention label includes the name of the sample retention person and the sample retention period.

5. The method for retaining and retesting food samples based on deep learning according to claim 1, characterized in that: The S3 further includes: If the target detection algorithm misses a detection, manually select the missed target dish with a rectangular frame on the software interface and click OK to add a temporary target. Based on the target detection algorithm, the missed target sub-image is cut according to the coordinates of the manually selected rectangular frame, the features of the target sub-image are extracted through the deep measurement model, and the features, width and height information of the target sub-image are saved; When the target sub-image appears again in the image of the dish to be sampled, the target detection algorithm is used to automatically call the sliding window algorithm and feature comparison algorithm of the limited area based on the feature, width and height information to quickly obtain the positioning coordinate information of the target sub-image.

6. The method for retaining and retesting food samples based on deep learning according to claim 1, characterized in that: In S4, the feature vector is compared with the dish features in the feature library, the current dish target is identified, the current dish target is determined to be an invalid dish or a dish to be confirmed, the invalid dish is added as a temporary dish, entered into the feature library, and the dish to be confirmed is corrected to obtain the correct dish category, specifically including: If the feature database is empty, the current dish target is determined to be an invalid dish based on the feature vector, and the current dish target is added as a temporary dish and entered into the feature database; If the total feature library is not empty, then based on the feature vector and the dish features in the total feature library, a dish recognition score is obtained. If the dish recognition score is lower than the trustworthy threshold score, the dish target is determined to be a dish to be confirmed; wherein the dish recognition score is the highest value among multiple comparison scores obtained by comparing the feature vector with multiple dish features in the total feature library; Correct the dishes to be confirmed, sort the multiple comparison scores from high to low, get the top 10 scores, and determine whether there are similar dish categories. If so, correct them to get the correct dish category. If not, determine them as invalid dishes, add them as temporary dishes, and enter them into the feature library.

7. The method for retaining and retesting food samples based on deep learning according to claim 1, characterized in that: In S5, the complete information of the dish sample label is obtained and recorded in combination with the sample image to complete the sample retention. The dish after the sample retention is added to the sample dish feature library, which specifically includes: Print and paste the sample labels on the corresponding sample dishes; The complete information, location coordinates and sample images of each sampled dish are automatically saved. At the same time, the completed sampled dishes are added to the sampled dish feature library for re-inspection.

8. The method for retaining and retesting food samples based on deep learning according to claim 1, characterized in that: In S6, the checking of samples specifically includes: Based on the sample retention time for re-inspection, determine the re-inspection scope of the sample dish feature library, and select the re-inspection dishes from the dishes for which the samples have been retained; Placing the re-inspected dish in the sampling area of ​​the image acquisition device to obtain the dish image, automatically positioning the dish image, cropping the sub-image and extracting the features based on the target detection algorithm, and obtaining the image features of the re-inspected dish; Compare the image features of the re-inspected dish with the features of the dishes within the re-inspection range in the retained sample dish feature library to obtain the retained sample dish category and retained sample number corresponding to the re-inspected dish; According to the sample dish category and sample image number, retrieve the sample order number and sample image corresponding to the re-inspected dish.

9. The method for retaining and retesting food samples based on deep learning according to claim 1, characterized in that: In S6, the sample inspection specifically includes: Enter the sample inspection page, check the sample image obtained in the sample inspection step, check other dishes corresponding to the sample image, and combine it with the sample inspection result of the re-inspected sample to obtain the sample information corresponding to all dishes in the sample image; The scores obtained by comparing the features of all dishes in the sample image with the features of dishes within the re-inspection range of the sample dish feature library are compared with the re-inspection score threshold. If the score is lower than the re-inspection score threshold, it is considered that the current dish is at risk of being modified. The actual weight of all dishes in the sample image is compared with the sample weight in the sample information. If it exceeds or falls below the sample weight, it is considered that the current dish is at risk of being modified. When it is higher than or equal to the re-inspection score threshold and equal to the retained sample weight, the current dish is considered to have passed the re-inspection.

10. A system for retaining and retesting food samples based on deep learning, used to execute the method for retaining and retesting food samples based on deep learning according to any one of claims 1 to 9, characterized in that: include: The permission verification module is used to verify the permissions of the sample retention personnel. After the permission verification passes, the basic information of the sample retention personnel is recorded; The label information setting module is used to determine the current meal period according to the current time after the permission verification is passed, and to determine the header information of the dish sample label in combination with the basic information of the sample retainer; The dish positioning module is used to obtain the sample image of the dish to be sampled, and automatically detect whether there is a dish target in the sample image based on the deep learning target detection algorithm. If so, the positioning coordinates and target quantity of the dish target are obtained; The dish recognition module is used to automatically cut the detected dish target into a dish sub-graph according to the positioning coordinates and extract the feature vector of the dish sub-graph, compare the feature vector with the dish features in the feature library, perform dish recognition on the current dish target, determine whether the current dish target is an invalid dish or a dish to be confirmed, add the invalid dish as a temporary dish, enter it into the feature library, and correct the dish to be confirmed to obtain the correct category; The dish sample retention module is used to obtain the dish sample retention information based on the dish recognition result, combine the header information with the dish sample retention information, obtain the complete information of the dish sample retention label, and record it in combination with the sample retention image to complete the sample retention, and add the dish after the sample retention to the sample retention dish feature library, wherein the dish sample retention information includes the sample retention time, sample retention dish category, sample retention sample number, sample retention weight, sample retention order number, and sample retention target quantity; The dish re-inspection module is used to determine the re-inspection scope of the sample dish feature library based on the sample retention time for re-inspection, select re-inspected dishes from the dishes that have completed sampling, compare the image features of the re-inspected dishes with the features of dishes within the re-inspection range of the sample dish feature library, obtain the dish sample information and sample image corresponding to the re-inspected dish, and complete the sample inspection; check the other dishes in the sample image, and based on the re-inspection score threshold, determine whether there is a risk of modification for all the dishes in the sample image, and complete the sample inspection.

Images