Meal delivery detection method and device based on computer vision and storage medium

Abstract

The invention discloses a meal delivery detection method and device based on computer vision and a storage medium, and relates to the technical field of computers, and the method comprises the steps: carrying out the first segmentation operation of a target meal delivery reference image through a preset instance segmentation model, carrying out the frame extraction of a meal delivery video stream, and carrying out the second segmentation operation of an obtained real-time image; based on the obtained second segmentation result, judging whether a meal is in a meal delivery state, and if the meal is in the meal delivery state, determining a first dinner plate corner point and a second dinner plate corner point based on a first mask in the first segmentation result and a second mask in the second segmentation result and by utilizing a preset minimum bounding rectangle function; and respectively correcting the target meal delivery reference image and the real-time image by using the homography matrix, and carrying out comparison judgment on the quantity of dishes and comparison judgment on the positions of the dishes by using the corrected reference image and the corrected real-time image so as to obtain a comprehensive detection result of whether the meal delivery of the real-time image meets a preset meal delivery condition or not. The accuracy of meal delivery detection is improved.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a meal delivery detection method, device, and storage medium based on computer vision. Background Art

[0002] The demand for food delivery standard testing is closely tied to the development of the retail and catering industries. With the increasing chainization of these sectors, store management has gradually shifted from offline supervision to online inspections. The standardization and uniformity of dishes are crucial to the chain's reputation and reputation, and are a key concern for operators. Inconsistency in food delivery standards across chain stores can lead to a decrease in consumer trust and desire to consume at all stores due to irregularities in individual stores, significantly impacting the chain's brand. Therefore, food delivery testing technology is needed to address these issues. However, due to the complexities of food delivery testing scenarios, where the quantity and order of ingredients vary from dish to dish, the current common practice is for chefs to rely on their experience while serving the food. However, this method is labor-intensive and lacks standardization, making effectiveness difficult to guarantee.

[0003] As can be seen from the above, how to improve the accuracy of meal delivery detection is an urgent problem to be solved. Summary of the Invention

[0004] In view of this, the purpose of the present invention is to provide a computer vision-based meal delivery detection method, device, equipment, and storage medium that can improve the accuracy of meal delivery detection. The specific solution is as follows:

[0005] In a first aspect, the present application provides a method for detecting food delivery based on computer vision, comprising:

[0006] Using a preset instance segmentation model, a first segmentation operation is performed on the target food delivery reference image to obtain a first segmentation result, a frame is extracted from the food delivery video stream, and a second segmentation operation is performed on the obtained real-time image to obtain a second segmentation result;

[0007] Determining whether the food is in the serving state based on the second segmentation result; if the food is in the serving state, determining a first mask in the first segmentation result and a second mask in the second segmentation result; and determining corresponding first and second plate corner points based on the first and second masks and using a preset minimum enclosing rectangle function;

[0008] Based on the first plate corner point and the second plate corner point and using the homography matrix, the target food presentation reference image and the real-time image are respectively corrected to obtain a corrected reference image and a corrected real-time image. The corrected reference image and the corrected real-time image are used to compare and judge the quantity of dishes and the position of dishes to obtain a comprehensive detection result of whether the food presentation in the real-time image meets the preset food presentation conditions.

[0009] Optionally, before performing a first segmentation operation on the target meal delivery reference image using a preset instance segmentation model to obtain a first segmentation result, the method further includes:

[0010] Obtaining a dish image that appears for the first time after the food serving device is started, determining the dish image as an initial food serving reference image, and determining whether the initial food serving reference image meets a preset food serving condition;

[0011] If the initial meal serving reference diagram meets the preset meal serving condition, the initial meal serving reference diagram is determined as the target meal serving reference diagram;

[0012] If the initial meal serving reference diagram does not meet the preset meal serving conditions, a prompt is created for placing the dishes in the initial meal serving reference diagram, and a new dish diagram is obtained after a preset time period, and the process jumps to the step of determining the dish diagram as the initial meal serving reference diagram until the initial meal serving reference diagram meets the preset meal serving conditions to obtain the target meal serving reference diagram.

[0013] Optionally, performing a first segmentation operation on the target meal delivery reference image using a preset instance segmentation model to obtain a first segmentation result includes:

[0014] Determine the features of each dish in the target meal reference image using the backbone network in the preset instance segmentation model, and fuse the features of each dish based on the feature fusion layer in the preset instance segmentation model to obtain fused features;

[0015] Based on the fused features and using the detection head in the preset instance segmentation model, the first dish position, the first dish quantity, the first dish category and the first mask corresponding to the target meal serving reference diagram are determined.

[0016] Optionally, extracting frames from the food delivery video stream and performing a second segmentation operation on the obtained real-time image to obtain a second segmentation result includes:

[0017] Using a preset camera to collect food delivery equipment in real time to obtain a food delivery video stream, and extracting frames from the food delivery video stream based on a preset frame extraction frequency to obtain a real-time image;

[0018] A second segmentation operation is performed on the real-time image using the preset instance segmentation model to obtain a second dish position, a second dish quantity, a second dish category, and a second mask corresponding to the real-time image.

[0019] Optionally, the determining whether the food is in a serving state based on the second segmentation result, and if the food is in a serving state, determining a first mask in the first segmentation result and a second mask in the second segmentation result, includes:

[0020] Determining whether the second mask in the second segmentation result is empty;

[0021] If the second mask in the second segmentation result is empty, it indicates that the food is not in the serving state;

[0022] If the second mask in the second segmentation result is not empty, determining the coordinates of the center point of the plate based on the second mask, and determining whether the coordinates of the center point of the plate meet a preset continuity condition;

[0023] If the preset continuous condition is met, it indicates that the food is in the serving state;

[0024] The preset continuity condition is that the coordinates of the center point of the dinner plate of the second mask in the second segmentation results corresponding to a continuous target number of real-time images are consistent.

[0025] Optionally, the determining the corresponding first and second plate corner points based on the first mask and the second mask and using a preset minimum bounding rectangle function includes:

[0026] Based on the first mask and the second mask, a minimum rotated rectangle is fitted using a preset minimum bounding rectangle function in a computer vision library to obtain a corresponding first dinner plate rectangle and a second dinner plate rectangle;

[0027] The first dinner plate rectangle and the second dinner plate rectangle are extracted respectively to obtain corresponding first dinner plate corner points and second dinner plate corner points.

[0028] Optionally, the use of the corrected reference image and the corrected real-time image to compare and determine the number of dishes and the location of dishes to obtain a comprehensive detection result of whether the food delivery in the real-time image meets the preset food delivery conditions includes:

[0029] Determining a target dish quantity and a target dish category corresponding to the corrected reference reference based on the first dish category and the first dish quantity in the first segmentation result;

[0030] Determining the actual number of dishes and the actual category of dishes in the corrected real-time image using the second dish quantity and the second dish quantity in the second segmentation result;

[0031] Determining whether the actual number of dishes is consistent with the target number of dishes;

[0032] If the actual number of dishes is consistent with the target number of dishes, then determining whether the actual dish category is consistent with the target dish category;

[0033] If the actual dish category is consistent with the target dish category, determining a first detection frame for all dishes in the corrected reference image, and determining a second detection frame for all dishes in the corrected real-time image, determining an intersection-over-union ratio (IoU) corresponding to the first detection frame and the second detection frame, and determining whether the IoU ratio is not less than a target overlap threshold;

[0034] If the intersection-over-union ratio is not less than the target overlap threshold, determining that the comprehensive detection result is that the food delivery of the real-time image meets the preset food delivery conditions;

[0035] If the intersection-over-union ratio is less than the target overlap threshold, it is determined that the comprehensive detection result is that the food presentation of the real-time image does not meet the preset food presentation conditions, and a prompt that the food presentation is not aesthetically pleasing is created and output.

[0036] Optionally, the correcting the target meal presentation reference image and the real-time image based on the first plate corner point and the second plate corner point using a homography matrix to obtain a corrected reference image and a corrected real-time image includes:

[0037] Determining a first homography matrix from the target food presentation reference to the preset graphic template based on the first food plate corner point and the preset graphic template;

[0038] Determining a second homography matrix from the real-time image to the preset graphic template using the second dinner plate corner points and the preset graphic template;

[0039] Mapping the target meal presentation reference image onto the preset graphic template based on the first homography matrix to obtain a corrected reference image;

[0040] The real-time image is mapped onto the preset graphic template using the second homography matrix to obtain a corrected real-time image.

[0041] In a second aspect, the present application provides a meal delivery detection device based on computer vision, comprising:

[0042] An image segmentation module is configured to perform a first segmentation operation on the target food delivery reference using a preset instance segmentation model to obtain a first segmentation result, extract frames from the food delivery video stream, and perform a second segmentation operation on the obtained real-time image to obtain a second segmentation result;

[0043] a corner point determination module, configured to determine whether the food tray is in a serving state based on the second segmentation result; if the food tray is in a serving state, determine a first mask in the first segmentation result and a second mask in the second segmentation result; and determine corresponding first and second food tray corner points based on the first and second masks and using a preset minimum enclosing rectangle function;

[0044] The food delivery detection module is used to correct the target food delivery reference image and the real-time image based on the first and second food plate corner points and using the homography matrix to obtain a corrected reference image and a corrected real-time image, and use the corrected reference image and the corrected real-time image to compare and judge the number of dishes and the position of dishes to obtain a comprehensive detection result of whether the food delivery in the real-time image meets the preset food delivery conditions.

[0045] In a third aspect, the present application provides an electronic device, comprising:

[0046] Memory, used to store computer programs;

[0047] A processor is used to execute the computer program to implement the aforementioned computer vision-based meal delivery detection method.

[0048] In a fourth aspect, the present application provides a computer-readable storage medium for storing a computer program, wherein the computer program, when executed by a processor, implements the aforementioned computer vision-based meal delivery detection method.

[0049] This application uses a preset instance segmentation model to perform a first segmentation operation on the target food delivery reference image to obtain a first segmentation result, extracts frames from the food delivery video stream, and performs a second segmentation operation on the obtained real-time image to obtain a second segmentation result; based on the second segmentation result, it is determined whether the food delivery state is in place. If the food delivery state is in place, the first mask in the first segmentation result and the second mask in the second segmentation result are determined, and the corresponding first and second plate corner points are determined based on the first and second masks and using a preset minimum circumscribed rectangle function; based on the first and second plate corner points and using a homography matrix, the target food delivery reference image and the real-time image are respectively corrected to obtain a corrected reference image and a corrected real-time image, and the corrected reference image and the corrected real-time image are used to compare and judge the number of dishes and the position of dishes to obtain a comprehensive detection result of whether the food delivery of the real-time image meets the preset food delivery conditions.

[0050] As can be seen from the above, the present application segments the reference image and the real-time image separately through a preset instance segmentation model, and can accurately extract masks of target areas such as plates and dishes. Based on the second segmentation result, it is determined whether the food is in the serving state, and non-serving scenes can be filtered out, and subsequent comparisons are triggered only when the food is served. The homography matrix is ​​calculated using the corner points of the plate, and the reference image and the real-time image can be corrected to the same perspective, so that the two are consistent in spatial scale, solving the spatial misalignment problem caused by different shooting angles and improving the accuracy of the comparison. In this way, by comparing the corrected reference image and the real-time image, the number of dishes and the location information of the dishes can be used to comprehensively determine whether the food in the real-time image meets the preset serving conditions. Abnormal dishes can be discovered in a timely manner and prompts can be given, saving human management costs, which is particularly helpful in the retail and catering industries. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are merely embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0052] Figure 1 This is a flow chart of a meal delivery detection method based on computer vision disclosed in this application;

[0053] Figure 2 A schematic diagram of the segmentation results of an example segmentation model disclosed in this application;

[0054] Figure 3 This is a schematic diagram of the structure of a meal delivery detection device based on computer vision disclosed in this application;

[0055] Figure 4 This is a structural diagram of an electronic device disclosed in this application. DETAILED DESCRIPTION

[0056] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0057] Currently, due to the complexity of meal delivery detection scenarios, the number and order of ingredients placed in different dishes are different. The more common practice is for chefs to judge based on experience while serving the food. However, this method requires a lot of manpower costs and lacks standardization to ensure the effect. To this end, this application provides a meal delivery detection method based on computer vision. By comparing the corrected reference map and the real-time map, it can comprehensively judge whether the meal delivery in the real-time map meets the preset meal delivery conditions based on the number of dishes and the location of the dishes. It can promptly detect abnormal dishes and provide prompts, saving manpower management costs, which is particularly helpful in the retail and catering industries.

[0058] See also Figure 1 As shown, an embodiment of the present invention discloses a meal delivery detection method based on computer vision, comprising:

[0059] Step S11: Use a preset instance segmentation model to perform a first segmentation operation on the target food delivery reference image to obtain a first segmentation result, extract frames from the food delivery video stream, and perform a second segmentation operation on the obtained real-time image to obtain a second segmentation result.

[0060] In this embodiment, before performing the first splitting operation, the target meal serving reference map needs to be determined. Specifically, the first dish image when the meal serving device is started is determined as the initial meal serving reference map, and then it is determined whether the initial meal serving reference map meets the preset meal serving conditions; if the preset meal serving conditions are met, the initial meal serving reference map is determined as the target meal serving reference map; if the preset meal serving conditions are not met, a prompt is created for placing the dishes in the initial meal serving reference map, and a new dish image is obtained after a preset time period, and the process jumps to the step of determining the dish image as the initial meal serving reference map until the initial meal serving reference map meets the preset meal serving conditions to obtain the target meal serving reference map. It is also possible to directly create a standard dish image that meets the preset meal serving conditions and import the standard dish image into the meal serving device.

[0061] Specifically, the method of performing a first segmentation operation on the target meal serving reference map using a preset instance segmentation model to obtain a first segmentation result also includes: obtaining a dish map that appears for the first time after the meal serving equipment is started, determining the dish map as the initial meal serving reference map, and judging whether the initial meal serving reference map meets the preset meal serving conditions; if the initial meal serving reference map meets the preset meal serving conditions, determining the initial meal serving reference map as the target meal serving reference map; if the initial meal serving reference map does not meet the preset meal serving conditions, creating a prompt for placing the dishes in the initial meal serving reference map, obtaining a new dish map after a preset time period, and jumping to the step of determining the dish map as the initial meal serving reference map until the initial meal serving reference map meets the preset meal serving conditions to obtain the target meal serving reference map.

[0062] It can be understood that the CSPDarknet53 (Cross-Stage Partial Darknet53, i.e., cross-batch processing) in the YOLOv8-Seg model (i.e., instance segmentation model) is used to determine the dish features of the target meal serving reference, such as the shape and color of the dish, and the PANet (Path Aggregation Network, i.e., path aggregation network) in the preset instance segmentation model is used to fuse the dish features to obtain fused features; based on the fused features and the detection head in the preset instance segmentation model, the dish position, dish quantity, dish category, and the first mask corresponding to the target meal serving reference are determined; the first mask includes a mask of the plate and masks corresponding to each dish on the plate. In a specific embodiment, Figure 2 This is a schematic diagram of the segmentation results of an example segmentation model provided in this embodiment. The YOLOv8-Seg model can accurately detect the presence of five sushi on a Chinese plate and generate a mask corresponding to each sushi.

[0063] Specifically, the preset instance segmentation model is used to perform a first segmentation operation on the target meal delivery reference map to obtain a first segmentation result, including: using the backbone network in the preset instance segmentation model to determine the features of each dish in the target meal delivery reference map, and fusing the features of each dish based on the feature fusion layer in the preset instance segmentation model to obtain a fused feature; based on the fused feature and using the detection head in the preset instance segmentation model, the first dish position, the first dish quantity, the first dish category and the first mask corresponding to the target meal delivery reference map are determined.

[0064] In this embodiment, a preset camera is used to capture a video of the food delivery outlet to obtain a food delivery video stream, and the food delivery video stream is frame-drawn at a preset frame rate to obtain a real-time image. It should be noted that the image resolution corresponding to the real-time image can be between 720p and 1080p, and the preset frame rate can be determined based on actual conditions. A second segmentation operation is performed on the real-time image using the preset instance segmentation model to obtain a second dish position, a second dish quantity, a second dish category, and a second mask corresponding to the real-time image. Specifically, the frame-drawing of the food delivery video stream and the second segmentation operation on the obtained real-time image to obtain a second segmentation result include: using a preset camera to capture real-time data from the food delivery device to obtain a food delivery video stream, frame-drawing the food delivery video stream at a preset frame rate to obtain a real-time image; and performing a second segmentation operation on the real-time image using the preset instance segmentation model to obtain a second dish position, a second dish quantity, a second dish category, and a second mask corresponding to the real-time image.

[0065] Step S12: Determine whether the food is being served based on the second segmentation result. If the food is being served, determine the first mask in the first segmentation result and the second mask in the second segmentation result. Based on the first mask and the second mask, determine the corresponding first and second plate corner points using a preset minimum circumscribed rectangle function.

[0066] In this embodiment, determining whether the second mask in the second segmentation result is empty, that is, determining whether a plate exists in the real-time image; if not, indicating that the food is not in the serving state; if present, determining the coordinates of the center point of the plate in the real-time image, and determining whether the coordinates of the center point of the plate are consistent in a target number of consecutive real-time images; if they are consistent, satisfying the preset continuity condition, that is, the food is in the serving state. Specifically, determining whether the food is in the serving state based on the second segmentation result, and if it is in the serving state, determining the first mask in the first segmentation result and the second mask in the second segmentation result, includes: determining whether the second mask in the second segmentation result is empty; if the second mask in the second segmentation result is empty, indicating that the food is not in the serving state; if the second mask in the second segmentation result is not empty, determining the coordinates of the center point of the plate based on the second mask, and determining whether the coordinates of the center point of the plate meet a preset continuity condition; if the preset continuity condition is met, indicating that the food is in the serving state; wherein the preset continuity condition is that the coordinates of the center point of the plate in the second mask in the second segmentation results corresponding to the target number of consecutive real-time images are consistent.

[0067] It is understood that OpenCV's minAreaRect function (i.e., a preset bounding rectangle function) is used to detect the minimum rotated rectangle of the plate in the target serving reference image and the real-time image, respectively, to obtain a first plate rectangle and a second plate rectangle. The four corner points of the first plate rectangle are then determined to obtain first plate corner points, and the four corner points of the second plate rectangle are then determined to obtain second plate corner points. Specifically, determining the corresponding first and second plate corner points based on the first and second masks and using the preset minimum bounding rectangle function includes: performing minimum rotated rectangle fitting based on the first and second masks and using the preset minimum bounding rectangle function in a computer vision library to obtain the corresponding first and second plate rectangles; and extracting the first and second plate rectangles to obtain the corresponding first and second plate corner points.

[0068] Step S13: Based on the first plate corner point and the second plate corner point and using the homography matrix, the target food presentation reference image and the real-time image are respectively corrected to obtain a corrected reference image and a corrected real-time image. The corrected reference image and the corrected real-time image are used to compare and judge the number of dishes and the positions of dishes to obtain a comprehensive detection result of whether the food presentation in the real-time image meets the preset food presentation conditions.

[0069] In this embodiment, after obtaining the first and second plate corner points, the first and second plate corner points are respectively mapped to a preset rectangular template of a preset size using a homography matrix to eliminate interference such as plate rotation and camera offset, thereby obtaining a corrected reference image and a corrected real-time image. The corresponding formulas are as follows:

[0070] ;

[0071] in, is the corrected coordinate, that is, the pixel coordinate mapped to the preset rectangular template, To correct the coordinates before, the s vector includes the submatrix Therefore, the s vector is defined as a linear transformation matrix, which controls the scaling and rotation of the image. For example, if the plate is tilted by 30°, s will adjust the rotation angle to make the corrected image straight. The a1 and a2 vectors are translation components, which control the position of the image translation. For example, if the plate is tilted to the left in the original image, a will move it to the center of the template. The b1 and b2 vectors are perspective adjustment components, which assist in adjusting the projection relationship. The preset rectangular template is a rectangular template obtained by scaling down the plate. It should be noted that the homography matrix H is a 3×3 reversible matrix that describes the projection relationship of the same plane in three-dimensional space from two different perspectives. That is, the target serving reference image and the real-time image may appear to have different dish sizes due to different placement angles and camera positions. The homography matrix can unify the projections from different perspectives to the preset template perspective. Projecting the target serving reference image and the real-time image onto a plane of the same scale avoids standard matching misdetection caused by changes in the serving position of the plate, plate rotation, and camera offset.

[0072] Specifically, the target food presentation reference reference and the real-time image are corrected based on the first plate corner points and the second plate corner points and using the homography matrix to obtain a corrected reference reference and a corrected real-time image, including: determining a first homography matrix from the target food presentation reference reference to the preset graphic template based on the first plate corner points and the preset graphic template; determining a second homography matrix from the real-time image to the preset graphic template using the second plate corner points and the preset graphic template; mapping the target food presentation reference reference to the preset graphic template based on the first homography matrix to obtain a corrected reference reference; and mapping the real-time image to the preset graphic template using the second homography matrix to obtain a corrected real-time image.

[0073] It can be understood that the target dish quantity and target dish category corresponding to the corrected reference diagram are determined based on the first dish category and the first dish quantity in the first segmentation result; the actual dish quantity and actual dish category in the corrected real-time diagram are determined using the second dish quantity and the second dish quantity in the second segmentation result; the dish category and quantity of the corrected real-time diagram are first judged, that is, whether the actual dish quantity is consistent with the target dish quantity; if they are consistent, whether the actual dish category is consistent with the target dish category is judged; if they are consistent, the dish arrangement is judged on the corrected real-time diagram, first determining the first detection frame of all dishes in the corrected reference diagram, and determining the second detection frame of all dishes in the corrected real-time diagram, and determining the intersection over union (IoU) corresponding to the first detection frame and the second detection frame. Union), and determine whether the intersection-union ratio is not less than the target overlap threshold; if the intersection-union ratio is not less than the target overlap threshold, it indicates that the food presentation of the real-time image meets the preset food presentation conditions; if the intersection-union ratio is less than the target overlap threshold, it is determined that the comprehensive detection result is that the food presentation of the real-time image does not meet the preset food presentation conditions, and a prompt is created and output that the food presentation is not aesthetically pleasing. It is worth noting that when judging the type of dishes, the number of dishes, and the presentation of dishes in the corrected real-time image, if there is any inconsistency, the corresponding situation is immediately reported for placement adjustment, and subsequent operations are stopped. In addition, the preset overlap threshold can be adjusted according to actual conditions, and is not specifically limited here.

[0074] Specifically, the use of the corrected reference image and the corrected real-time image to compare and judge the number of dishes and the position of dishes to obtain a comprehensive detection result of whether the food delivery of the real-time image meets the preset food delivery conditions, including: determining the target food quantity and target food category corresponding to the corrected reference image based on the first food category and the first food quantity in the first segmentation result; determining the actual food quantity and actual food category in the corrected real-time image using the second food quantity and the second food quantity in the second segmentation result; judging whether the actual food quantity is consistent with the target food quantity; if the actual food quantity is consistent with the target food quantity, judging the actual food quantity. whether the actual dish category is consistent with the target dish category; if the actual dish category is consistent with the target dish category, then determine the first detection frame of all dishes in the corrected reference image, and determine the second detection frame of all dishes in the corrected real-time image, determine the intersection-and-union ratio corresponding to the first detection frame and the second detection frame, and judge whether the intersection-and-union ratio is not less than the target overlap threshold; if the intersection-and-union ratio is not less than the target overlap threshold, then determine that the comprehensive detection result is that the food presentation of the real-time image meets the preset food presentation conditions; if the intersection-and-union ratio is less than the target overlap threshold, then determine that the comprehensive detection result is that the food presentation of the real-time image does not meet the preset food presentation conditions, and create and output a prompt that the food presentation is not aesthetically pleasing.

[0075] As can be seen from the above, the present application segments the reference image and the real-time image separately through a preset instance segmentation model, and can accurately extract masks of target areas such as plates and dishes. Based on the second segmentation result, it is determined whether the food is in the serving state, and non-serving scenes can be filtered out, and subsequent comparisons are triggered only when the food is served. The homography matrix is ​​calculated using the corner points of the plate, and the reference image and the real-time image can be corrected to the same perspective, so that the two are consistent in spatial scale, solving the spatial misalignment problem caused by different shooting angles and improving the accuracy of the comparison. In this way, by comparing the corrected reference image and the real-time image, the number of dishes and the location information of the dishes can be used to comprehensively determine whether the food in the real-time image meets the preset serving conditions. Abnormal dishes can be discovered in a timely manner and prompts can be given, saving human management costs, which is particularly helpful in the retail and catering industries.

[0076] Accordingly, see Figure 3 As shown, the present application also provides a meal delivery detection device based on computer vision, comprising:

[0077] An image segmentation module 11 is configured to perform a first segmentation operation on the target food delivery reference using a preset instance segmentation model to obtain a first segmentation result, extract frames from the food delivery video stream, and perform a second segmentation operation on the obtained real-time image to obtain a second segmentation result;

[0078] a corner point determination module 12 for determining whether the food tray is in the serving state based on the second segmentation result; if the food tray is in the serving state, determining a first mask in the first segmentation result and a second mask in the second segmentation result; and determining corresponding first and second food tray corner points based on the first and second masks and using a preset minimum enclosing rectangle function;

[0079] The food delivery detection module 13 is used to correct the target food delivery reference image and the real-time image based on the first plate corner point and the second plate corner point and using the homography matrix to obtain a corrected reference image and a corrected real-time image, and use the corrected reference image and the corrected real-time image to compare and judge the number of dishes and the position of dishes to obtain a comprehensive detection result of whether the food delivery of the real-time image meets the preset food delivery conditions.

[0080] In some specific embodiments, the computer vision-based meal delivery detection device may further include:

[0081] An initial reference image judging unit, configured to obtain a dish image that appears for the first time after the food serving device is started, determine the dish image as an initial food serving reference image, and judge whether the initial food serving reference image meets a preset food serving condition;

[0082] a target reference map determining unit, configured to determine the initial meal serving reference map as a target meal serving reference map if the initial meal serving reference map satisfies the preset meal serving condition;

[0083] A prompt creation unit is used to create a prompt for placing the dishes in the initial meal serving reference diagram if the initial meal serving reference diagram does not meet the preset meal serving conditions, obtain a new dish diagram after a preset time period, and jump to the step of determining the dish diagram as the initial meal serving reference diagram until the initial meal serving reference diagram meets the preset meal serving conditions to obtain a target meal serving reference diagram.

[0084] In some specific implementations, the image segmentation module 11 may specifically include:

[0085] a feature fusion unit, configured to determine the features of each dish in the target meal reference image using a backbone network in a preset instance segmentation model, and fuse the features of each dish based on a feature fusion layer in the preset instance segmentation model to obtain a fused feature;

[0086] A first mask determination unit is used to determine the first dish position, first dish quantity, first dish category and first mask corresponding to the target meal serving reference diagram based on the fused features and using the detection head in the preset instance segmentation model.

[0087] In some specific implementations, the image segmentation module 11 may specifically include:

[0088] A video stream frame extraction unit is used to use a preset camera to collect data from the food delivery device in real time to obtain a food delivery video stream, and extract frames from the food delivery video stream based on a preset frame extraction frequency to obtain a real-time image;

[0089] The second mask determination unit is used to perform a second segmentation operation on the real-time image using the preset instance segmentation model to obtain a second dish position, a second dish quantity, a second dish category and a second mask corresponding to the real-time image.

[0090] In some specific implementations, the corner point determination module 12 may specifically include:

[0091] a second mask judging unit, configured to judge whether the second mask in the second segmentation result is empty;

[0092] a first state determining unit, configured to indicate that the food is not in a serving state if the second mask in the second segmentation result is empty;

[0093] a center point coordinate determining unit, configured to determine the coordinates of the center point of the plate based on the second mask if the second mask in the second segmentation result is not empty, and to determine whether the coordinates of the center point of the plate meet a preset continuity condition;

[0094] The second state determining unit is used to indicate that the food is in the food serving state if the preset continuous condition is met.

[0095] In some specific implementations, the corner point determination module 12 may specifically include:

[0096] a rectangle fitting unit, configured to perform minimum rotation rectangle fitting based on the first mask and the second mask and using a preset minimum bounding rectangle function in a computer vision library to obtain a corresponding first dinner plate rectangle and a corresponding second dinner plate rectangle;

[0097] The rectangle extraction unit is used to extract the first dinner plate rectangle and the second dinner plate rectangle respectively to obtain corresponding first dinner plate corner points and second dinner plate corner points.

[0098] In some specific implementations, the meal delivery detection module 13 may specifically include:

[0099] a first matrix determining unit, configured to determine a first homography matrix from the target meal presentation reference to the preset graphic template based on the first plate corner points and the preset graphic template;

[0100] a second matrix determining unit, configured to determine a second homography matrix from the real-time image to the preset graphic template using the second dinner plate corner points and the preset graphic template;

[0101] a reference mapping unit, configured to map the target meal presentation reference image onto the preset graphic template based on the first homography matrix to obtain a corrected reference image;

[0102] The real-time image mapping unit is configured to map the real-time image onto the preset graphic template using the second homography matrix to obtain a corrected real-time image.

[0103] In some specific implementations, the meal delivery detection module 13 may specifically include:

[0104] a target category determining unit, configured to determine a target dish quantity and a target dish category corresponding to the corrected reference based on the first dish category and the first dish quantity in the first segmentation result;

[0105] an actual category determining unit, configured to determine an actual number of dishes and an actual category of dishes in the corrected real-time image using the second dish quantity and the second dish quantity in the second segmentation result;

[0106] A quantity determination unit, configured to determine whether the actual quantity of dishes is consistent with the target quantity of dishes;

[0107] a category determination unit, configured to determine whether the actual dish category is consistent with the target dish category if the actual dish quantity is consistent with the target dish quantity;

[0108] an IoU determining unit, configured to, if the actual dish category is consistent with the target dish category, determine a first detection frame for all dishes in the corrected reference image, determine a second detection frame for all dishes in the corrected real-time image, determine an IoU corresponding to the first detection frame and the second detection frame, and determine whether the IoU ratio is not less than a target overlap threshold;

[0109] a first result determination unit, configured to determine, if the intersection-over-union ratio is not less than a target overlap threshold, that the comprehensive detection result is that the food delivery of the real-time image meets a preset food delivery condition;

[0110] The second result determination unit is used to determine that the comprehensive detection result is that the food presentation of the real-time image does not meet the preset food presentation conditions if the intersection-over-union ratio is less than the target overlap threshold, and to create and output a prompt that the food presentation is not aesthetically pleasing.

[0111] Furthermore, the embodiment of the present application also discloses an electronic device, Figure 4This is a structural diagram of an electronic device 20 according to an exemplary embodiment. The content in the diagram cannot be considered as any limitation on the scope of use of this application. The electronic device 20 may specifically include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 is used to store a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the computer vision-based meal detection method disclosed in any of the aforementioned embodiments. In addition, the electronic device 20 in this embodiment may specifically be an electronic computer.

[0112] In this embodiment, the power supply 23 is used to provide operating voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and the external device. The communication protocol it follows is any communication protocol that can be applied to the technical solution of this application and is not specifically limited here; the input and output interface 25 is used to obtain external input data or output data to the outside world. Its specific interface type can be selected according to specific application needs and is not specifically limited here.

[0113] In addition, the memory 22, as a carrier for resource storage, can be a read-only memory, random access memory, disk or CD, etc. The resources stored thereon can include an operating system 221, a computer program 222, etc., and the storage method can be temporary storage or permanent storage.

[0114] The operating system 221 is used to manage and control the hardware devices on the electronic device 20 and the computer program 222. The operating system 221 can be Windows Server, NetWare, Unix, Linux, etc. In addition to including a computer program capable of implementing the computer vision-based meal delivery detection method disclosed in any of the aforementioned embodiments and executed by the electronic device 20, the computer program 222 can further include computer programs capable of performing other specific tasks.

[0115] Furthermore, this application also discloses a computer-readable storage medium for storing a computer program; wherein, when executed by a processor, the computer program implements the aforementioned computer vision-based meal delivery detection method. The specific steps of this method can be referred to the corresponding contents disclosed in the aforementioned embodiments and will not be repeated here.

[0116] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from the other embodiments. Reference can be made to the descriptions of the identical or similar parts between the various embodiments. For the devices disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple, and the relevant parts can be referred to the descriptions of the methods.

[0117] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the above description has generally described the components and steps of each example according to their functions. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0118] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in random access memory (RAM), internal memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0119] Finally, it should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or device comprising the element.

[0120] The above is a detailed introduction to the technical solution provided by the present application. Specific examples are used herein to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea. At the same time, for those skilled in the art, according to the ideas of the present application, there may be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A meal delivery detection method based on computer vision, characterized in that: include: Using a preset instance segmentation model, a first segmentation operation is performed on the target food delivery reference image to obtain a first segmentation result, a frame is extracted from the food delivery video stream, and a second segmentation operation is performed on the obtained real-time image to obtain a second segmentation result; Determining whether the food is in the serving state based on the second segmentation result; if the food is in the serving state, determining a first mask in the first segmentation result and a second mask in the second segmentation result; and determining corresponding first and second plate corner points based on the first and second masks and using a preset minimum enclosing rectangle function; Based on the first plate corner point and the second plate corner point and using the homography matrix, the target food presentation reference image and the real-time image are respectively corrected to obtain a corrected reference image and a corrected real-time image. The corrected reference image and the corrected real-time image are used to compare and judge the quantity of dishes and the position of dishes to obtain a comprehensive detection result of whether the food presentation in the real-time image meets the preset food presentation conditions.

2. The method for detecting meal delivery based on computer vision according to claim 1, characterized in that: Before performing a first segmentation operation on the target meal delivery reference image using the preset instance segmentation model to obtain a first segmentation result, the method further includes: Obtaining a dish image that appears for the first time after the food serving device is started, determining the dish image as an initial food serving reference image, and determining whether the initial food serving reference image meets a preset food serving condition; If the initial meal serving reference diagram meets the preset meal serving condition, the initial meal serving reference diagram is determined as the target meal serving reference diagram; If the initial meal serving reference diagram does not meet the preset meal serving conditions, a prompt is created for placing the dishes in the initial meal serving reference diagram, and a new dish diagram is obtained after a preset time period, and the process jumps to the step of determining the dish diagram as the initial meal serving reference diagram until the initial meal serving reference diagram meets the preset meal serving conditions to obtain the target meal serving reference diagram.

3. The method for detecting meal delivery based on computer vision according to claim 1, wherein: The step of performing a first segmentation operation on the target meal delivery reference image using a preset instance segmentation model to obtain a first segmentation result includes: Determine the features of each dish in the target meal reference image using the backbone network in the preset instance segmentation model, and fuse the features of each dish based on the feature fusion layer in the preset instance segmentation model to obtain fused features; Based on the fused features and using the detection head in the preset instance segmentation model, the first dish position, the first dish quantity, the first dish category and the first mask corresponding to the target meal serving reference diagram are determined.

4. The method for detecting meal delivery based on computer vision according to claim 3, wherein: The extracting frames from the food delivery video stream and performing a second segmentation operation on the obtained real-time image to obtain a second segmentation result includes: Using a preset camera to collect food delivery equipment in real time to obtain a food delivery video stream, and extracting frames from the food delivery video stream based on a preset frame extraction frequency to obtain a real-time image; A second segmentation operation is performed on the real-time image using the preset instance segmentation model to obtain a second dish position, a second dish quantity, a second dish category, and a second mask corresponding to the real-time image.

5. The meal delivery detection method based on computer vision according to claim 4, characterized in that: The determining whether the food is in the serving state based on the second segmentation result, and if the food is in the serving state, determining a first mask in the first segmentation result and a second mask in the second segmentation result, includes: Determining whether the second mask in the second segmentation result is empty; If the second mask in the second segmentation result is empty, it indicates that the food is not in the serving state; If the second mask in the second segmentation result is not empty, determining the coordinates of the center point of the plate based on the second mask, and determining whether the coordinates of the center point of the plate meet a preset continuity condition; If the preset continuous condition is met, it indicates that the food is in the serving state; The preset continuity condition is that the coordinates of the center point of the dinner plate of the second mask in the second segmentation results corresponding to a continuous target number of real-time images are consistent.

6. The meal delivery detection method based on computer vision according to claim 5, characterized in that: The determining the corresponding first and second corner points of the dinner plate based on the first mask and the second mask and using a preset minimum bounding rectangle function includes: Based on the first mask and the second mask, a minimum rotated rectangle is fitted using a preset minimum bounding rectangle function in a computer vision library to obtain a corresponding first dinner plate rectangle and a second dinner plate rectangle; The first dinner plate rectangle and the second dinner plate rectangle are extracted respectively to obtain corresponding first dinner plate corner points and second dinner plate corner points.

7. The method for detecting meal delivery based on computer vision according to claim 4, characterized in that: The comparison and judgment of the number of dishes and the position of dishes by using the corrected reference image and the corrected real-time image to obtain a comprehensive detection result of whether the food delivery in the real-time image meets the preset food delivery conditions includes: Determining a target dish quantity and a target dish category corresponding to the corrected reference reference based on the first dish category and the first dish quantity in the first segmentation result; Determining the actual number of dishes and the actual category of dishes in the corrected real-time image using the second dish quantity and the second dish quantity in the second segmentation result; Determining whether the actual number of dishes is consistent with the target number of dishes; If the actual number of dishes is consistent with the target number of dishes, then determining whether the actual dish category is consistent with the target dish category; If the actual dish category is consistent with the target dish category, determining a first detection frame for all dishes in the corrected reference image, and determining a second detection frame for all dishes in the corrected real-time image, determining an intersection-over-union ratio (IoU) corresponding to the first detection frame and the second detection frame, and determining whether the IoU ratio is not less than a target overlap threshold; If the intersection-over-union ratio is not less than the target overlap threshold, determining that the comprehensive detection result is that the food delivery of the real-time image meets the preset food delivery conditions; If the intersection-over-union ratio is less than the target overlap threshold, it is determined that the comprehensive detection result is that the food presentation of the real-time image does not meet the preset food presentation conditions, and a prompt that the food presentation is not aesthetically pleasing is created and output.

8. The meal delivery detection method based on computer vision according to any one of claims 1 to 7, characterized in that: The correcting the target meal presentation reference image and the real-time image based on the first plate corner point and the second plate corner point using a homography matrix to obtain a corrected reference image and a corrected real-time image, including: Determining a first homography matrix from the target food presentation reference to the preset graphic template based on the first food plate corner point and the preset graphic template; Determining a second homography matrix from the real-time image to the preset graphic template using the second dinner plate corner points and the preset graphic template; Mapping the target meal presentation reference image onto the preset graphic template based on the first homography matrix to obtain a corrected reference image; The real-time image is mapped onto the preset graphic template using the second homography matrix to obtain a corrected real-time image.

9. A meal delivery detection device based on computer vision, characterized in that: include: An image segmentation module is configured to perform a first segmentation operation on the target food delivery reference using a preset instance segmentation model to obtain a first segmentation result, extract frames from the food delivery video stream, and perform a second segmentation operation on the obtained real-time image to obtain a second segmentation result; a corner point determination module, configured to determine whether the food tray is in a serving state based on the second segmentation result; if the food tray is in a serving state, determine a first mask in the first segmentation result and a second mask in the second segmentation result; and determine corresponding first and second food tray corner points based on the first and second masks and using a preset minimum enclosing rectangle function; The food delivery detection module is used to correct the target food delivery reference image and the real-time image based on the first and second food plate corner points and using the homography matrix to obtain a corrected reference image and a corrected real-time image, and use the corrected reference image and the corrected real-time image to compare and judge the number of dishes and the position of dishes to obtain a comprehensive detection result of whether the food delivery in the real-time image meets the preset food delivery conditions.

10. A computer-readable storage medium, characterized in that Used to store a computer program, wherein when the computer program is executed by a processor, the computer vision-based meal delivery detection method according to any one of claims 1 to 8 is implemented.

Images