A computer vision-based dish delivery detection method and device, and a storage medium

Abstract

The application discloses a computer vision-based dish delivery detection method and device and a storage medium, and relates to the technical field of computers, which comprises the following steps: performing a first segmentation operation on a target dish delivery reference image by using a preset instance segmentation model, performing frame extraction on a dish delivery video stream, and performing a second segmentation operation on the obtained real-time image; judging whether a dish delivery state is present based on the obtained second segmentation result; if the dish delivery state is present, determining first and second dish plate corner points based on a first mask in the first segmentation result and a second mask in the second segmentation result and by using a preset minimum circumscribed rectangle function; correcting the target dish delivery reference image and the real-time image by using a homography matrix, and performing comparison judgment on the number of dishes and the position of dishes by using the obtained corrected reference image and corrected real-time image, so as to obtain a comprehensive detection result of whether the dish delivery of the real-time image meets preset dish delivery conditions. The accuracy of dish delivery detection is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer, in particular to a meal delivery detection method and device based on computer vision and a storage medium. BACKGROUND

[0002] The generation of meal delivery standard detection demand is closely related to the development of retail and catering industry. With the increase of the chain rate of retail and catering industry, the management of stores is gradually transferred from on-site supervision to online supervision. Among them, the standardization and unification of dishes are related to the reputation and reputation of chain stores, and are the focus of operators. If the meal delivery standards of different stores of chain stores are not unified, it will cause consumers to reduce the trust and consumption desire of all stores due to the abnormal meal delivery of individual stores, which has a huge impact on chain brands. Therefore, meal delivery detection technology is needed to correct. However, due to the complexity of meal delivery detection scene, the number and order of food materials placed for different dishes are different, and the current common method is for the chef to judge according to experience while delivering meals, but this method needs to consume a lot of labor cost, and it is difficult to guarantee the standard effect.

[0003] From the above, how to improve the accuracy of meal delivery detection is a problem to be solved at present. SUMMARY

[0004] Therefore, the purpose of the present application is to provide a meal delivery detection method and device based on computer vision, which can improve the accuracy of meal delivery detection. The specific scheme is as follows:

[0005] In the first aspect, the present application provides a meal delivery detection method based on computer vision, comprising:

[0006] performing a first segmentation operation on the target meal delivery reference image by using a preset instance segmentation model to obtain a first segmentation result, performing frame extraction on the meal delivery video stream, and performing a second segmentation operation on the obtained real-time image to obtain a second segmentation result;

[0007] determining whether it is in a meal delivery state based on the second segmentation result, if it is in a meal delivery state, determining a first mask in the first segmentation result and a second mask in the second segmentation result, and determining corresponding first and second dish plate corner points based on the first and second masks and by using a preset minimum bounding rectangle function;

[0008] correcting the target meal delivery reference image and the real-time image based on the first and second dish plate corner points and by using a homography matrix to obtain a corrected reference image and a corrected real-time image, and performing comparison judgment of dish quantity and comparison judgment of dish position by using the corrected reference image and the corrected real-time image to obtain a comprehensive detection result of whether the real-time image meets the preset meal delivery condition.

[0009] Optionally, before the first segmentation operation is performed on the target delivery reference image by using the preset instance segmentation model to obtain a first segmentation result, the method further includes:

[0010] acquiring a dish image first appearing after the delivery device is started, determining the dish image as an initial delivery reference image, and judging whether the initial delivery reference image meets a preset delivery condition;

[0011] If the initial delivery reference image meets the preset delivery condition, the initial delivery reference image is determined as the target delivery reference image.

[0012] If the initial delivery reference image does not meet the preset delivery condition, a prompt for placing dishes in the initial delivery reference image is created, a new dish image is acquired after a preset time period, and the step of determining the dish image as the initial delivery reference image is jumped to until the initial delivery reference image meets the preset delivery condition to obtain the target delivery reference image.

[0013] Optionally, the first segmentation operation is performed on the target delivery reference image by using the preset instance segmentation model to obtain a first segmentation result, including:

[0014] determining features of each dish in the target delivery reference image by using a backbone network in the preset instance segmentation model, and fusing the features of each dish based on a feature fusion layer in the preset instance segmentation model to obtain fused features;

[0015] determining a first dish position, a first dish quantity, a first dish category and a first mask corresponding to the target delivery reference image based on the fused features and by using a detection head in the preset instance segmentation model.

[0016] Optionally, the out-delivery video stream is frame-extracted, and a real-time image obtained is subjected to a second segmentation operation to obtain a second segmentation result, including:

[0017] real-time collection of the delivery device by using a preset camera to obtain an out-delivery video stream, and frame extraction of the out-delivery video stream based on a preset frame extraction frequency to obtain a real-time image;

[0018] the real-time image is subjected to a second segmentation operation by using the preset instance segmentation model to obtain a second dish position, a second dish quantity, a second dish category and a second mask corresponding to the real-time image.

[0019] Optionally, based on the second segmentation result, it is judged whether it is in a delivery state, and if it is in a delivery state, the first mask in the first segmentation result and the second mask in the second segmentation result are determined, including:

[0020] determining whether a second mask in the second segmentation result is empty;

[0021] if the second mask in the second segmentation result is empty, representing not in a serving state;

[0022] if the second mask in the second segmentation result is not empty, determining a dish plate center point coordinate based on the second mask, and determining whether the dish plate center point coordinate satisfies a preset continuity condition;

[0023] if the preset continuity condition is satisfied, representing in a serving state;

[0024] wherein the preset continuity condition is that the dish plate center point coordinates of the second masks in the second segmentation results corresponding to a continuous target number of real-time images are consistent.

[0025] Optionally, the determination of the corresponding first dish plate corner point and second dish plate corner point based on the first mask and the second mask and using a preset minimum circumscribed rectangle function comprises:

[0026] performing minimum rotating rectangle fitting based on the first mask and the second mask and using a preset minimum circumscribed rectangle function in a computer vision library to obtain a corresponding first dish plate rectangle and a second dish plate rectangle;

[0027] extracting the first dish plate rectangle and the second dish plate rectangle respectively to obtain a corresponding first dish plate corner point and a second dish plate corner point.

[0028] Optionally, the comparison judgment of the number of dishes and the comparison judgment of the position of dishes using the corrected reference image and the corrected real-time image to obtain a comprehensive detection result of whether the serving of the real-time image meets a preset serving condition comprises:

[0029] determining a target dish number and a target dish category corresponding to the corrected reference image based on a first dish category and a first dish number in the first segmentation result;

[0030] determining an actual dish number and an actual dish category in the corrected real-time image using a second dish number and a second dish number in the second segmentation result;

[0031] determining whether the actual dish number and the target dish number are consistent;

[0032] if the actual dish number and the target dish number are consistent, determining whether the actual dish category and the target dish category are consistent;

[0033] If the actual dish category is consistent with the target dish category, the first detection box of all dishes in the corrected reference image is determined, the second detection box of all dishes in the corrected real-time image is determined, the intersection-over-union corresponding to the first detection box and the second detection box is determined, and it is judged whether the intersection-over-union is not less than a target overlap threshold value;

[0034] If the intersection-over-union is not less than the target overlap threshold value, it is determined that the comprehensive detection result is that the dish in the real-time image meets the preset dish serving condition;

[0035] If the intersection-over-union is less than the target overlap threshold value, it is determined that the comprehensive detection result is that the dish in the real-time image does not meet the preset dish serving condition, and a prompt that the dish arrangement is not beautiful is created and output.

[0036] Optionally, the target dish serving reference image and the real-time image are corrected based on the first dish tray corner point and the second dish tray corner point and by using a homography matrix, to obtain a corrected reference image and a corrected real-time image, which comprises:

[0037] A first homography matrix from the target dish serving reference image to a preset graphic template is determined based on the first dish tray corner point and the preset graphic template;

[0038] A second homography matrix from the real-time image to the preset graphic template is determined by using the second dish tray corner point and the preset graphic template;

[0039] The target dish serving reference image is mapped onto the preset graphic template based on the first homography matrix, to obtain a corrected reference image;

[0040] The real-time image is mapped onto the preset graphic template by using the second homography matrix, to obtain a corrected real-time image.

[0041] In a second aspect, the present application provides a dish serving detection device based on computer vision, which comprises:

[0042] An image segmentation module is configured to perform a first segmentation operation on a target dish serving reference image by using a preset instance segmentation model, to obtain a first segmentation result, perform frame extraction on a dish serving video stream, and perform a second segmentation operation on a real-time image obtained, to obtain a second segmentation result;

[0043] A corner point determination module is configured to determine whether a dish serving state is present based on the second segmentation result, determine a first mask in the first segmentation result and a second mask in the second segmentation result if the dish serving state is present, and determine corresponding first and second dish tray corner points based on the first and second masks and by using a preset minimum circumscribed rectangle function;

[0044] The dish delivery detection module is configured to correct the target dish delivery reference image and the real-time image respectively based on the first dish tray corner point and the second dish tray corner point and by using a homography matrix to obtain a corrected reference image and a corrected real-time image, and to perform comparison judgment of dish quantity and comparison judgment of dish position by using the corrected reference image and the corrected real-time image, so as to obtain a comprehensive detection result of whether the dish delivery of the real-time image meets the preset dish delivery condition.

[0045] In a third aspect, the present application provides an electronic device, comprising:

[0046] a memory configured to store a computer program;

[0047] a processor configured to execute the computer program to implement the computer vision-based dish delivery detection method.

[0048] In a fourth aspect, the present application provides a computer readable storage medium configured to store a computer program, wherein the computer program is executed by a processor to implement the computer vision-based dish delivery detection method.

[0049] The preset instance segmentation model is used to perform a first segmentation operation on the target dish delivery reference image to obtain a first segmentation result, the dish delivery video stream is frame-extracted, and a second segmentation operation is performed on the obtained real-time image to obtain a second segmentation result; whether the dish delivery state is in a dish delivery state is determined based on the second segmentation result, if the dish delivery state is in a dish delivery state, a first mask in the first segmentation result and a second mask in the second segmentation result are determined, the first mask and the second mask are used to determine corresponding first dish tray corner points and second dish tray corner points by using a preset minimum enclosing rectangle function; the target dish delivery reference image and the real-time image are corrected respectively based on the first dish tray corner point and the second dish tray corner point and by using a homography matrix to obtain a corrected reference image and a corrected real-time image, comparison judgment of dish quantity and comparison judgment of dish position are performed by using the corrected reference image and the corrected real-time image, so as to obtain a comprehensive detection result of whether the dish delivery of the real-time image meets the preset dish delivery condition.

[0050] From the above, the preset instance segmentation model is used for segmenting the reference image and the real-time image respectively, the mask of the target region such as the dish tray and the dish can be accurately extracted, whether it is in the dish-out state is judged based on the second segmentation result, the non-dish-out scene can be filtered, and the subsequent comparison is triggered only when the dish is out; the dish corner point is used to calculate the homography matrix, the reference image and the real-time image are corrected to the same perspective, the spatial dislocation problem caused by different shooting angles is solved, and the accuracy of the comparison is improved. In this way, by comparing the corrected reference image and the real-time image, whether the dish-out of the real-time image meets the preset dish-out condition can be comprehensively judged through the dish quantity and the dish position information, the abnormal dish can be found in time and prompted, the labor management cost is saved, and obvious help is provided in the retail and catering industries. BRIEF DESCRIPTION OF DRAWINGS

[0051] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or the prior art description. Obviously, the drawings in the following description are only a part of the embodiments of the present application, and for those skilled in the art, other drawings can be obtained without creative labor on the basis of the provided drawings.

[0052] Figure 1 A flow chart of a dish-out detection method based on computer vision disclosed by the present application;

[0053] Figure 2 A segmentation result schematic diagram of an instance segmentation model disclosed by the present application;

[0054] Figure 3 A structure schematic diagram of a dish-out detection device based on computer vision disclosed by the present application;

[0055] Figure 4 A structure diagram of an electronic device disclosed by the present application. DETAILED DESCRIPTION

[0056] The technical solutions in the embodiments of the present application will be described clearly and completely in combination with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0057] Currently, due to the complexity of the dish delivery detection scene, the number and order of food materials placed for different dishes are different, and the common method is that the chef judges according to experience at the same time of dish delivery, but this method needs to consume a large amount of labor cost, and it is difficult to guarantee the standard effect. Therefore, the present application provides a dish delivery detection method based on computer vision. By comparing the corrected reference map and the real-time map, the number of dishes and the position information of the dishes can be used to comprehensively judge whether the real-time map meets the preset dish delivery condition, abnormal dishes can be found in time and prompted, and the labor management cost is saved, which is obviously helpful in retail and catering industries.

[0058] Referring to Figure 1 The embodiment of the present application discloses a dish delivery detection method based on computer vision, which comprises:

[0059] Step S11, a first segmentation operation is performed on the target dish delivery reference map by using a preset instance segmentation model to obtain a first segmentation result, the dish delivery video stream is frame extracted, and a second segmentation operation is performed on the obtained real-time image to obtain a second segmentation result.

[0060] In this embodiment, before the first segmentation operation, the target dish delivery reference map needs to be determined. Specifically, the first dish map when the dish delivery device starts is determined as an initial dish delivery reference map, and then it is judged whether the initial dish delivery reference map meets the preset dish delivery condition; if the initial dish delivery reference map meets the preset dish delivery condition, the initial dish delivery reference map is determined as the target dish delivery reference map; if the initial dish delivery reference map does not meet the preset dish delivery condition, a prompt for placing dishes in the initial dish delivery reference map is created, and a new dish map is obtained after a preset time period, and the step of determining the dish map as the initial dish delivery reference map is jumped to until the initial dish delivery reference map meets the preset dish delivery condition to obtain the target dish delivery reference map. A standard dish map meeting the preset dish delivery condition can also be directly created and imported into the dish delivery device.

[0061] Specifically, before the first segmentation operation is performed on the target dish delivery reference map by using a preset instance segmentation model to obtain a first segmentation result, it further comprises: obtaining a dish map first appearing after the dish delivery device starts, determining the dish map as an initial dish delivery reference map, judging whether the initial dish delivery reference map meets the preset dish delivery condition; if the initial dish delivery reference map meets the preset dish delivery condition, the initial dish delivery reference map is determined as the target dish delivery reference map; if the initial dish delivery reference map does not meet the preset dish delivery condition, a prompt for placing dishes in the initial dish delivery reference map is created, and a new dish map is obtained after a preset time period, and the step of determining the dish map as the initial dish delivery reference map is jumped to until the initial dish delivery reference map meets the preset dish delivery condition to obtain the target dish delivery reference map.

[0062] It can be understood that the CSPDarknet53 (Cross-Stage Partial Darknet53) in the YOLOv8-Seg model (i.e., an instance segmentation model) is used to determine the dish features of the target dish reference image, such as the shape and color of the dish, the PANet (Path Aggregation Network) in the preset instance segmentation model is used to fuse each dish feature to obtain a fused feature; based on the fused feature and the detection head in the preset instance segmentation model, the dish position, dish quantity, dish category and first mask corresponding to the target dish reference image are determined; the first mask includes a mask of a dish plate and a mask corresponding to each dish in the dish plate. In a specific embodiment, Figure 2 An instance segmentation model provided in this embodiment is a segmentation result schematic diagram, and the YOLOv8-Seg model can accurately detect five sushi in a Chinese dish plate and generate a mask corresponding to each sushi.

[0063] Specifically, the first segmentation operation of the target dish reference image by the preset instance segmentation model to obtain a first segmentation result includes: determining each dish feature of the target dish reference image by the backbone network in the preset instance segmentation model, and fusing each dish feature based on the feature fusion layer in the preset instance segmentation model to obtain a fused feature; determining the first dish position, first dish quantity, first dish category and first mask corresponding to the target dish reference image based on the fused feature and using the detection head in the preset instance segmentation model.

[0064] In this embodiment, a preset camera is used to shoot a video of a dish outlet to obtain a dish video stream, and the dish video stream is frame-extracted based on a preset frame extraction frequency to obtain a real-time image; it should be noted that the picture definition corresponding to the real-time image can be 720p to 1080p, and the preset frame extraction frequency can be determined according to actual conditions. The real-time image is subjected to a second segmentation operation by the preset instance segmentation model to obtain a second dish position, a second dish quantity, a second dish category and a second mask corresponding to the real-time image. Specifically, the frame extraction of the dish video stream and the second segmentation operation of the obtained real-time image to obtain a second segmentation result include: real-time collection of a dish device by a preset camera to obtain a dish video stream, frame extraction of the dish video stream based on a preset frame extraction frequency to obtain a real-time image; the real-time image is subjected to a second segmentation operation by the preset instance segmentation model to obtain a second dish position, a second dish quantity, a second dish category and a second mask corresponding to the real-time image.

[0065] In step S12, it is determined whether the second segmentation result is in a serving state. If the second segmentation result is in the serving state, the first mask in the first segmentation result and the second mask in the second segmentation result are determined, and the first dish tray corner point and the second dish tray corner point are determined based on the first mask and the second mask and by using a preset minimum circumscribed rectangle function.

[0066] In this embodiment, it is determined whether the second mask in the second segmentation result is empty, i.e., whether there is a dish tray in the real-time image. If there is no dish tray, it is indicated that the serving state is not in the serving state. If there is a dish tray, the dish tray center point coordinates of the real-time image are determined, and it is determined whether the dish tray center point coordinates of the continuous target number of real-time images are consistent. If they are consistent, the preset continuous condition is met, i.e., the serving state is in the serving state. Specifically, the determination of whether the second segmentation result is in the serving state, if the second segmentation result is in the serving state, the first mask in the first segmentation result and the second mask in the second segmentation result are determined, includes: determining whether the second mask in the second segmentation result is empty; if the second mask in the second segmentation result is empty, it is indicated that the serving state is not in the serving state; if the second mask in the second segmentation result is not empty, the dish tray center point coordinates are determined based on the second mask, and it is determined whether the dish tray center point coordinates meet the preset continuous condition; if the preset continuous condition is met, it is indicated that the serving state is in the serving state; wherein the preset continuous condition is that the dish tray center point coordinates of the second mask in the second segmentation result of the continuous target number of real-time images are consistent.

[0067] It can be understood that the minAreaRect function (i.e., the preset circumscribed rectangle function) of OpenCV is used to detect the minimum rotating rectangle of the dish tray in the target serving reference image and the real-time image, respectively, to obtain the first dish tray rectangle and the second dish tray rectangle, and to determine the four corner points of the first dish tray rectangle to obtain the first dish tray corner point, and to determine the four corner points of the second dish tray rectangle to obtain the second dish tray corner point. Specifically, the determination of the first dish tray corner point and the second dish tray corner point based on the first mask and the second mask and by using the preset minimum circumscribed rectangle function includes: based on the first mask and the second mask and by using the preset minimum circumscribed rectangle function in the computer vision library, the minimum rotating rectangle fitting is performed to obtain the first dish tray rectangle and the second dish tray rectangle; the first dish tray rectangle and the second dish tray rectangle are extracted respectively to obtain the first dish tray corner point and the second dish tray corner point.

[0068] In step S13, the target dish-out reference image and the real-time image are respectively corrected based on the first dish plate corner point and the second dish plate corner point and by using a homography matrix to obtain a corrected reference image and a corrected real-time image. The corrected reference image and the corrected real-time image are used to perform comparison and judgment on the number of dishes and the position of dishes to obtain a comprehensive detection result of whether the dish-out of the real-time image meets the preset dish-out condition.

[0069] In the embodiment, after the first dish plate corner point and the second dish plate corner point are obtained, the first dish plate corner point and the second dish plate corner point are respectively mapped to a preset rectangular template of a preset size by using a homography matrix to eliminate interference such as dish plate rotation, camera offset, and the like, so as to obtain a corrected reference image and a corrected real-time image. The corresponding formula is as follows:

[0070] ;

[0071] wherein, is a corrected coordinate, i.e., a pixel coordinate mapped to the preset rectangular template, is a coordinate before correction, and the s vector includes a sub-matrix Therefore, the s vector is defined as a linear transformation matrix, which is used to control the scaling and rotation of the image. For example, if the dish plate is tilted by 30°, the s vector adjusts the rotation angle so that the corrected image is straightened. The a1 and a2 vectors are translation components, which are used to control the position of the image translation. For example, if the dish plate is offset to the left in the original image, the a vector moves it to the middle position of the template. The b1 and b2 vectors are perspective adjustment components, which are used to assist in adjusting the projection relationship. The preset rectangular template is a rectangular template obtained based on the scaling of the dish plate. It should be noted that the homography matrix H is a 3x3 invertible matrix, which is used to describe the projection relationship of the same plane in a three-dimensional space under two different visual angles, i.e., the target dish-out reference image and the real-time image may have different sizes of dishes due to different placement angles and camera positions. The homography matrix can unify the projection of different visual angles to a preset template visual angle. The target dish-out reference image and the real-time image are projected to the same scale plane, which avoids the standard matching false detection problem caused by changes in the dish-out position of the dish plate, rotation of the dish plate, and offset of the camera, and the like.

[0072] Specifically, the correcting the target dish-out reference image and the real-time image based on the first dish plate corner point and the second dish plate corner point and using a homography matrix respectively to obtain a corrected reference image and a corrected real-time image comprises: determining a first homography matrix of the target dish-out reference image to a preset graphic template based on the first dish plate corner point and the preset graphic template; determining a second homography matrix of the real-time image to the preset graphic template using the second dish plate corner point and the preset graphic template; mapping the target dish-out reference image onto the preset graphic template based on the first homography matrix to obtain a corrected reference image; and mapping the real-time image onto the preset graphic template using the second homography matrix to obtain a corrected real-time image.

[0073] It can be understood that the target dish quantity and the target dish category corresponding to the corrected reference image are determined based on the first dish category and the first dish quantity in the first segmentation result; the actual dish quantity and the actual dish category in the corrected real-time image are determined using the second dish quantity and the second dish quantity in the second segmentation result; the corrected real-time image is first judged for dish category and quantity, i.e., whether the actual dish quantity is consistent with the target dish quantity; if consistent, whether the actual dish category is consistent with the target dish category is judged; if consistent, the corrected real-time image is judged for dish arrangement, the first detection frame of all dishes in the corrected reference image is determined, the second detection frame of all dishes in the corrected real-time image is determined, the intersection over union (IoU) corresponding to the first detection frame and the second detection frame is determined, and whether the IoU is not less than a target overlap threshold is judged; if the IoU is not less than the target overlap threshold, it is represented that the dish-out of the real-time image meets the preset dish-out condition; if the IoU is less than the target overlap threshold, it is determined that the comprehensive detection result is that the dish-out of the real-time image does not meet the preset dish-out condition, and a prompt that the dish arrangement is not beautiful is created and output. It is worth noting that when the corrected real-time image is judged for dish category, dish quantity and dish arrangement, if inconsistent, the corresponding situation is immediately reported for arrangement adjustment, and subsequent operation is stopped. In addition, the preset overlap threshold can be adjusted according to actual conditions, which is not specifically limited here.

[0074] Specifically, the comparison judgment of the dish quantity and the comparison judgment of the dish position are performed by using the corrected reference image and the corrected real-time image to obtain a comprehensive detection result of whether the dish delivery of the real-time image meets the preset dish delivery condition, which includes: determining a target dish quantity and a target dish category corresponding to the corrected reference image based on the first dish category and the first dish quantity in the first segmentation result; determining an actual dish quantity and an actual dish category in the corrected real-time image by using the second dish quantity and the second dish quantity in the second segmentation result; judging whether the actual dish quantity is consistent with the target dish quantity; if the actual dish quantity is consistent with the target dish quantity, judging whether the actual dish category is consistent with the target dish category; if the actual dish category is consistent with the target dish category, determining first detection boxes of all dishes in the corrected reference image and second detection boxes of all dishes in the corrected real-time image, determining an intersection over union corresponding to the first detection boxes and the second detection boxes, and judging whether the intersection over union is not less than a target overlap threshold; if the intersection over union is not less than the target overlap threshold, determining that the comprehensive detection result is that the dish delivery of the real-time image meets the preset dish delivery condition; if the intersection over union is less than the target overlap threshold, determining that the comprehensive detection result is that the dish delivery of the real-time image does not meet the preset dish delivery condition, and creating and outputting a prompt that the dish arrangement is not beautiful.

[0075] As can be seen from the above, the reference image and the real-time image are respectively segmented by the preset instance segmentation model, the mask of the target region such as the dish tray and the dish can be accurately extracted, whether it is in the dish delivery state is judged based on the second segmentation result, the non-dish delivery scene can be filtered, and the subsequent comparison is triggered only when the dish is delivered; the homography matrix is calculated by the dish corner point, the reference image and the real-time image can be corrected to the same perspective, the space dislocation problem caused by different shooting angles is solved, and the accuracy of the comparison is improved. In this way, by comparing the corrected reference image and the real-time image, whether the dish delivery of the real-time image meets the preset dish delivery condition can be comprehensively judged through the dish quantity and the dish position information, the abnormal dish can be found in time and prompted, the labor management cost is saved, and obvious help is provided in the retail and catering industries.

[0076] Correspondingly, referring to Figure 3 The application further provides a dish delivery detection device based on computer vision, which comprises:

[0077] An image segmentation module 11 is configured to perform a first segmentation operation on a target dish delivery reference image by using a preset instance segmentation model to obtain a first segmentation result, perform frame extraction on a dish delivery video stream, and perform a second segmentation operation on the obtained real-time image to obtain a second segmentation result.

[0078] The corner point determination module 12 is configured to determine whether the serving state is in a serving state based on the second segmentation result, and if the serving state is in the serving state, determine a first mask in the first segmentation result and a second mask in the second segmentation result, and determine corresponding first and second dish tray corner points based on the first and second masks and by using a preset minimum circumscribed rectangle function.

[0079] The serving detection module 13 is configured to correct the target serving reference image and the real-time image respectively based on the first and second dish tray corner points and by using a homography matrix to obtain a corrected reference image and a corrected real-time image, and perform comparison judgment of the number of dishes and comparison judgment of the position of dishes based on the corrected reference image and the corrected real-time image to obtain a comprehensive detection result of whether the serving of the real-time image meets the preset serving condition.

[0080] In some embodiments, the computer vision-based serving detection device can further include:

[0081] The initial reference image judgment unit is configured to obtain a dish image first appearing after the serving device is started, determine the dish image as an initial serving reference image, and judge whether the initial serving reference image meets a preset serving condition.

[0082] The target reference image determination unit is configured to determine the initial serving reference image as a target serving reference image if the initial serving reference image meets the preset serving condition.

[0083] The prompt creation unit is configured to create a prompt for placing dishes in the initial serving reference image if the initial serving reference image does not meet the preset serving condition, obtain a new dish image after a preset time period, and jump to the step of determining the dish image as an initial serving reference image until the initial serving reference image meets the preset serving condition to obtain a target serving reference image.

[0084] In some embodiments, the image segmentation module 11 can include:

[0085] The feature fusion unit is configured to determine features of each dish in the target serving reference image by using a backbone network in a preset instance segmentation model, and fuse each feature of the dish based on a feature fusion layer in the preset instance segmentation model to obtain fused features.

[0086] The first mask determination unit is configured to determine a first dish position, a first dish quantity, a first dish category, and a first mask corresponding to the target serving reference image based on the fused features and by using a detection head in the preset instance segmentation model.

[0087] In some embodiments, the image segmentation module 11 can specifically include:

[0088] a video stream frame extraction unit configured to capture a serving device in real time by using a preset camera to obtain a serving video stream, and extract frames from the serving video stream based on a preset frame extraction frequency to obtain real-time images;

[0089] a second mask determination unit configured to perform a second segmentation operation on the real-time images by using the preset instance segmentation model to obtain a second dish location, a second dish quantity, a second dish category, and a second mask corresponding to the real-time images.

[0090] In some embodiments, the corner point determination module 12 can specifically include:

[0091] a second mask judgment unit configured to judge whether the second mask in the second segmentation result is empty;

[0092] a first state determination unit configured to represent that the serving state is not in a serving state if the second mask in the second segmentation result is empty;

[0093] a center point coordinate determination unit configured to determine a dish center point coordinate based on the second mask in the second segmentation result if the second mask in the second segmentation result is not empty, and judge whether the dish center point coordinate satisfies a preset continuous condition;

[0094] a second state determination unit configured to represent that the serving state is in a serving state if the preset continuous condition is satisfied.

[0095] In some embodiments, the corner point determination module 12 can specifically include:

[0096] a rectangle fitting unit configured to perform minimum rotating rectangle fitting based on the first mask and the second mask and by using a preset minimum enclosing rectangle function in a computer vision library to obtain corresponding first and second dish plate rectangles;

[0097] a rectangle extraction unit configured to extract the first and second dish plate rectangles respectively to obtain corresponding first and second dish plate corner points.

[0098] In some embodiments, the serving detection module 13 can specifically include:

[0099] a first matrix determination unit configured to determine a first homography matrix of the target serving reference image to a preset graphic template based on the first dish plate corner point and the preset graphic template;

[0100] a second matrix determination unit configured to determine a second homography matrix of the real-time image to the preset graphic template by using the second dish tray corner point and the preset graphic template;

[0101] a reference map mapping unit configured to map the target dish-out reference map to the preset graphic template based on the first homography matrix to obtain a corrected reference map;

[0102] a real-time image mapping unit configured to map the real-time image to the preset graphic template by using the second homography matrix to obtain a corrected real-time map.

[0103] In some embodiments, the dish-out detection module 13 can specifically include:

[0104] a target category determination unit configured to determine a target dish quantity and a target dish category corresponding to the corrected reference map based on a first dish category and a first dish quantity in the first segmentation result;

[0105] an actual category determination unit configured to determine an actual dish quantity and an actual dish category in the corrected real-time map by using a second dish quantity and a second dish category in the second segmentation result;

[0106] a quantity judgment unit configured to judge whether the actual dish quantity is consistent with the target dish quantity;

[0107] a category judgment unit configured to judge whether the actual dish category is consistent with the target dish category if the actual dish quantity is consistent with the target dish quantity;

[0108] an intersection-over-union determination unit configured to determine first detection boxes of all dishes in the corrected reference map and second detection boxes of all dishes in the corrected real-time map if the actual dish category is consistent with the target dish category, determine an intersection-over-union corresponding to the first detection boxes and the second detection boxes, and judge whether the intersection-over-union is not less than a target overlap threshold value;

[0109] a first result determination unit configured to determine that a comprehensive detection result is that the dish-out of the real-time image meets a preset dish-out condition if the intersection-over-union is not less than the target overlap threshold value;

[0110] a second result determination unit configured to determine that the comprehensive detection result is that the dish-out of the real-time image does not meet the preset dish-out condition if the intersection-over-union is less than the target overlap threshold value, and create and output a prompt that the dish arrangement is not beautiful.

[0111] Further, the embodiment of the present application further discloses an electronic device, Figure 4is a structural diagram of the electronic device 20 according to an exemplary embodiment, and the content in the diagram should not be considered as any limitation on the use range of the present application. The electronic device 20 can specifically include at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 is configured to store a computer program, and the computer program is loaded and executed by the processor 21 to implement the related steps in the computer vision-based dish delivery detection method disclosed in any of the preceding embodiments. In addition, the electronic device 20 in the present embodiment can be an electronic computer.

[0112] In the present embodiment, the power supply 23 is configured to provide operating voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and external devices, and the communication protocol followed by the communication interface 24 can be any communication protocol applicable to the technical solution of the present application, which is not limited here; the input / output interface 25 is configured to obtain external input data or output data to the outside, and the specific interface type can be selected according to the specific application needs, which is not limited here.

[0113] In addition, the memory 22 as a carrier for resource storage can be a read-only memory, a random access memory, a magnetic disk, or an optical disk, etc., and the resources stored thereon can include an operating system 221, a computer program 222, etc., and the storage mode can be temporary storage or permanent storage.

[0114] The operating system 221 is configured to manage and control each hardware device on the electronic device 20 and the computer program 222, and can be Windows Server, Netware, Unix, Linux, etc. In addition to the computer program capable of completing the computer vision-based dish delivery detection method executed by the electronic device 20 disclosed in any of the preceding embodiments, the computer program 222 can further include a computer program capable of completing other specific work.

[0115] Further, the present application further discloses a computer readable storage medium for storing a computer program; wherein the computer program is executed by a processor to implement the computer vision-based dish delivery detection method disclosed above. For the specific steps of the method, refer to the corresponding content disclosed in the preceding embodiments, which will not be repeated here.

[0116] The embodiments in the specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. For the same or similar parts between the embodiments, refer to each other. For the device disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple, and the relevant parts refer to the method part.

[0117] Those skilled in the art will further appreciate that the various illustrative logical blocks, modules, circuits, and algorithm steps described in connection with the embodiments disclosed herein can be implemented as electronic hardware, computer software, or combinations of both. To clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps have been described above generally in terms of their functionality, without reference to a specific sequence of operations for implementing the functions, and / or without reference to a specific hierarchy of circuit elements. The sequence of operations described in connection with the illustrative embodiments can be embodied directly in hardware, in a software module executed by an electronic processor, or in a combination of the two. A software module can reside in RAM (random access memory), flash memory, ROM (read-only memory), EPROM (erasable programmable ROM), EEPROM (electrically erasable programmable ROM), registers, hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor such that the processor can read information from, and write information to, the storage medium. In the alternative, hardwired circuitry can be used in place of or in combination with software modules. The foregoing disclosure has set forth various embodiments of the present application at a particular level of particularity. This disclosure is not to be taken in a limiting sense, but is made merely to provide those skilled in the art with a complete descriptive record of the present application. Numerous alternative embodiments will be readily apparent to one of ordinary skill in the art having the benefit of this disclosure. Thus, the scope of the present application should be determined by the appended claims and equivalents thereof.

[0118] The steps of a method or algorithm described in connection with the embodiments disclosed herein can be embodied directly in hardware, in a software module executed by an electronic processor, or in a combination of the two. A software module can reside in RAM (random access memory), flash memory, ROM (read-only memory), EPROM (erasable programmable ROM), EEPROM (electrically erasable programmable ROM), registers, hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor such that the processor can read information from, and write information to, the storage medium. In the alternative, hardwired circuitry can be used in place of or in combination with software modules. The foregoing disclosure has set forth various embodiments of the present application at a particular level of particularity. This disclosure is not to be taken in a limiting sense, but is made merely to provide those skilled in the art with a complete descriptive record of the present application. Numerous alternative embodiments will be readily apparent to one of ordinary skill in the art having the benefit of this disclosure. Thus, the scope of the present application should be determined by the appended claims and equivalents thereof.

[0119] Finally, it should be noted that the terms "comprises", "comprising", or other variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can also include other elements not expressly listed or inherent to such process, method, article, or apparatus. The terms "comprises", "comprising", or other variations thereof, do not exclude the presence of elements other than those listed in a process, method, article, or apparatus that comprises the elements unless specifically stated otherwise. The words "including" and "comprising" and variations thereof, mean that the processes, methods, articles, or apparatuses contain the specified elements, but not excluding others.

[0120] The above detailed description has set forth various embodiments of the present application at a particular level of particularity. This description is not to be interpreted in a limiting sense, but merely to provide those skilled in the art with a complete descriptive record of the present application. Numerous alternative embodiments will be readily apparent to one of ordinary skill in the art having the benefit of this disclosure. Thus, the scope of the present application should be determined by the appended claims and equivalents thereof.

Claims

1. A computer vision-based order-out detection method, characterized in that, The method comprises the following steps: performing a first segmentation operation on a target delivery reference image by using a preset instance segmentation model to obtain a first segmentation result, performing frame extraction on a delivery video stream, and performing a second segmentation operation on the obtained real-time image to obtain a second segmentation result; determining whether a delivery state exists based on the second segmentation result, and if the delivery state exists, determining a first mask in the first segmentation result and a second mask in the second segmentation result, and determining corresponding first and second dish plate corner points based on the first and second masks and by using a preset minimum circumscribed rectangle function; the process of determining whether the delivery state exists based on the second segmentation result comprises: determining whether the second mask in the second segmentation result is empty, and if the second mask is not empty, determining a dish plate center point coordinate based on the second mask, and determining whether the dish plate center point coordinate satisfies a preset continuous condition, and if the preset continuous condition is satisfied, it is indicated that the delivery state exists, and corresponding first and second dish plate corner points are determined based on the first and second masks and by using a preset minimum circumscribed rectangle function; the preset continuous condition is that the dish plate center point coordinates of the second masks in the second segmentation results of a plurality of continuous target images are consistent; based on the first and second dish plate corner points and by using a homography matrix, the target delivery reference image and the real-time image are respectively corrected to obtain a corrected reference image and a corrected real-time image, and the corrected reference image and the corrected real-time image are used to perform comparison judgment on the number of dishes and comparison judgment on the position of dishes to obtain a comprehensive detection result of whether the delivery of the real-time image meets a preset delivery condition; the homography matrix is an invertible matrix.

2. The computer vision-based order pickup detection method of claim 1, wherein, Before the first segmentation operation on the target delivery reference image by using the preset instance segmentation model to obtain the first segmentation result, the following steps are further included: acquiring a dish image appearing for the first time after a delivery device is started, determining the dish image as an initial delivery reference image, and determining whether the initial delivery reference image meets a preset delivery condition; if the initial delivery reference image meets the preset delivery condition, the initial delivery reference image is determined as a target delivery reference image; if the initial delivery reference image does not meet the preset delivery condition, a prompt for placing dishes in the initial delivery reference image is created, a new dish image is acquired after a preset time period, and the step of determining the dish image as an initial delivery reference image is jumped to until the initial delivery reference image meets the preset delivery condition to obtain a target delivery reference image.

3. The computer vision-based order pickup detection method of claim 1, wherein, the first segmentation operation on the target delivery reference image by using the preset instance segmentation model to obtain the first segmentation result comprises: determining features of each dish in the target delivery reference image by using a backbone network in the preset instance segmentation model, and fusing the features of each dish based on a feature fusion layer in the preset instance segmentation model to obtain fused features; Determine the first dish position, the first dish quantity, the first dish category and the first mask corresponding to the target delivery reference image based on the fused features and using a detection head in the preset instance segmentation model.

4. The computer vision-based order pickup detection method of claim 3, wherein, The method comprises the following steps: Real-time collection of the delivery equipment by using a preset camera to obtain a delivery video stream, and frame extraction of the delivery video stream based on a preset frame extraction frequency to obtain real-time images; Second segmentation operation of the real-time images by using the preset instance segmentation model to obtain the second dish position, the second dish quantity, the second dish category and the second mask corresponding to the real-time images.

5. The computer vision-based order pickup detection method of claim 4, wherein, The method comprises the following steps: Determine the first mask in the first segmentation result and the second mask in the second segmentation result if the delivery state is determined based on the second segmentation result, and the method comprises the following steps: Determine whether the second mask in the second segmentation result is empty; If the second mask in the second segmentation result is empty, it indicates that the delivery state is not met; If the second mask in the second segmentation result is not empty, determine the center point coordinates of the dish plate based on the second mask, and determine whether the center point coordinates of the dish plate meet a preset continuous condition; If the preset continuous condition is met, it indicates that the delivery state is met.

6. The computer vision-based order pickup detection method of claim 5, wherein, The preset continuous condition is that the center point coordinates of the second mask in the second segmentation result of a continuous target quantity of real-time images are consistent. The method comprises the following steps: Determine the first dish plate corner point and the second dish plate corner point based on the first mask and the second mask and using a preset minimum enclosing rectangle function in a computer vision library, and the method comprises the following steps:

7. The computer vision-based order pickup detection method of claim 4, wherein, Perform minimum rotating rectangle fitting based on the first mask and the second mask and using the preset minimum enclosing rectangle function in the computer vision library to obtain the first dish plate rectangle and the second dish plate rectangle; Extract the first dish plate corner point and the second dish plate corner point from the first dish plate rectangle and the second dish plate rectangle, respectively. The method comprises the following steps: Determine the target dish quantity and the target dish category corresponding to the corrected reference image based on the first dish category and the first dish quantity in the first segmentation result; Determine the actual dish quantity and the actual dish category in the corrected real-time image based on the second dish quantity and the second dish quantity in the second segmentation result; Determine whether the actual dish quantity is consistent with the target dish quantity; If the actual dish quantity is consistent with the target dish quantity, determine whether the actual dish category is consistent with the target dish category; If the actual dish category is consistent with the target dish category, first detection boxes of all dishes in the corrected reference image are determined, second detection boxes of all dishes in the real-time image are determined, an intersection-over-union corresponding to the first detection boxes and the second detection boxes is determined, and whether the intersection-over-union is not less than a target overlap threshold is judged. If the intersection-over-union is not less than the target overlap threshold, it is determined that the comprehensive detection result is that the dish in the real-time image meets the preset dish-out condition. If the intersection-over-union is less than the target overlap threshold, it is determined that the comprehensive detection result is that the dish in the real-time image does not meet the preset dish-out condition, and a prompt that the dish arrangement is not beautiful is created and output.

8. The computer vision-based order detection method of any one of claims 1-7, wherein, The first dish plate corner point and the second dish plate corner point are used to correct the target dish-out reference image and the real-time image based on the first dish plate corner point and the second dish plate corner point and by using a homography matrix, to obtain a corrected reference image and a corrected real-time image, including: A first homography matrix from the target dish-out reference image to a preset graphic template is determined based on the first dish plate corner point and the preset graphic template. A second homography matrix from the real-time image to the preset graphic template is determined by using the second dish plate corner point and the preset graphic template. The target dish-out reference image is mapped onto the preset graphic template based on the first homography matrix, to obtain a corrected reference image. The real-time image is mapped onto the preset graphic template by using the second homography matrix, to obtain a corrected real-time image.

9. A computer vision based order detection device, comprising: It includes: An image segmentation module is configured to perform a first segmentation operation on a target dish-out reference image by using a preset instance segmentation model to obtain a first segmentation result, perform frame extraction on a dish-out video stream, and perform a second segmentation operation on a real-time image obtained to obtain a second segmentation result. A corner point determination module is configured to determine a first mask in the first segmentation result and a second mask in the second segmentation result if the second segmentation result indicates that a dish-out state is present, and determine corresponding first and second dish plate corner points based on the first and second masks and by using a preset minimum enclosing rectangle function. The process of determining whether the second segmentation result indicates that a dish-out state is present includes: determining whether the second mask in the second segmentation result is empty, determining a dish plate center point coordinate based on the second mask if the second mask is not empty, and determining whether the dish plate center point coordinate meets a preset continuous condition if the dish plate center point coordinate meets the preset continuous condition, which indicates that a dish-out state is present, determining corresponding first and second dish plate corner points based on the first and second masks and by using a preset minimum enclosing rectangle function. The preset continuous condition is that the dish plate center point coordinates of the second masks in the second segmentation results of a plurality of target number of real-time images are consistent. The dish delivery detection module is configured to rectify the target dish delivery reference image and the real-time image respectively based on the first dish tray corner point and the second dish tray corner point and by using a homography matrix, to obtain a rectified reference image and a rectified real-time image, and to perform comparison and judgment on the number of dishes and the position of dishes by using the rectified reference image and the rectified real-time image, so as to obtain a comprehensive detection result of whether the real-time image meets the preset dish delivery condition. The homography matrix is a reversible matrix.

10. A computer-readable storage medium, characterized in that, A computer program product is provided for saving a computer program, wherein the computer program is executed by a processor to implement the computer vision-based dish delivery detection method according to any one of claims 1 to 8.

Images