Intelligent settlement machine, intelligent settlement machine dinner plate identification method and system, and medium

Abstract

The invention relates to an intelligent settlement machine, an intelligent settlement machine dinner plate recognition method and system and a medium, and the method comprises the steps: detecting a dinner plate contour and a dish contour in a dinner plate image through a convolutional neural network model, and generating a mask image; and identifying color information of the mask image, and filling the color features into the contour of the dish to generate an original dinner plate image without the dish. And performing feature extraction on the original dinner plate image through the convolutional neural network model to obtain a plurality of feature vectors of the target dinner plate. And comparing the plurality of feature vectors of the target dinner plate with dinner plate feature vectors in a database, and determining that the category of the target dinner plate is a corresponding dinner plate category in the database based on a comparison result. Accurate confirmation of the types of the dinner plates is achieved, and then the names, prices and total prices of the dishes are determined. The problem that in the prior art, intelligent settlement machine dinner plate recognition is low in recognition accuracy is solved.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and in particular to an intelligent settlement machine and an intelligent settlement machine meal plate recognition method, system and medium. Background Art

[0002] In the modern catering industry, as consumers' requirements for service speed and accuracy continue to increase, the manual billing method of traditional restaurants has gradually revealed many disadvantages. Manual billing mainly relies on the experience and manual operation of the cashier, which not only increases labor costs, but also easily leads to billing errors due to human factors. Especially during peak hours, cashiers may make miscalculations or omissions due to heavy workload, fatigue or distraction, which may directly affect the restaurant's revenue and the customer's dining experience. In addition, the manual billing method is inefficient and easily forms queues at checkout, prolonging the customer's waiting time, thereby affecting the restaurant's service quality and operational efficiency.

[0003] In addition to the inherent defects of manual billing, some existing automated billing systems also have technical limitations in practical applications. Some restaurant automated billing systems currently on the market usually rely on auxiliary identifiers such as RFID tags or QR codes for identification and billing. Although such systems have improved settlement efficiency to a certain extent, they require pre-attachment of labels or QR codes to plates or dishes, which not only increases the management cost of tableware, but may also lead to identification failures due to loss, damage or contamination of labels. In addition, this tag-based identification method is powerless when dealing with items without labels or identification, and cannot meet the diverse operational needs of restaurants.

[0004] At present, with the rapid development of computer vision and deep learning technology, automatic settlement systems based on image recognition have gradually attracted attention. However, traditional target recognition algorithms often have difficulty in ensuring sufficient accuracy when dealing with complex plate scenes. There are many kinds of dishes on the plate, and the colors, shapes, and placement methods vary greatly. Especially when multiple dishes are stacked on the plate or close to each other, the existing target recognition algorithms are prone to missed detection, false detection, or recognition errors. Therefore, the plate recognition of intelligent settlement machines in the existing related technologies has the problem of low recognition accuracy. Summary of the invention

[0005] The embodiments of the present application provide a smart settlement machine and a method, system and medium for identifying meal trays of smart settlement machines, so as to at least solve the problem of low recognition accuracy of meal trays of smart settlement machines in the existing related technologies.

[0006] A camera is used to photograph the tray recognition area and obtain a panoramic image of the tray placement area; A user display screen is used to display the price, name and total price of the dish selected by the user; Operation display screen, used by operators to modify prices of dishes that have been misidentified and to set up the smart checkout machine; A plate identification area, used for placing a tray, wherein one tray is provided with a plurality of positions for placing plates, and each plate is provided with a dish; The tray identification area is used to place trays to be settled; The computing unit runs various functions of the intelligent settlement machine, including settlement, setting and output display functions.

[0007] In one embodiment, the dinner plates include multiple types, wherein the same dish is placed on the same type of dinner plate, and different dishes are distinguished by different types of the dinner plates.

[0008] In a second aspect, an embodiment of the present application provides a method for identifying a meal tray of an intelligent checkout machine, the method comprising: According to the panoramic image of the tray placement area obtained in advance, the panoramic image is analyzed by a pre-trained convolutional neural network model to obtain the position coordinates of each of the dinner plates, and based on the position coordinates, an image of the dinner plate is obtained; Detecting the outline of the dinner plate and the outline of the dish in the dinner plate image by a convolutional neural network model to generate a mask image, wherein the mask image includes color information of a visible area of ​​the dinner plate, and the visible area is an area of ​​the dinner plate that is not covered by the dish; Identify the color information of the mask image, fill the color features into the inside of the dish outline, and generate the original plate image without the dish; Using the convolutional neural network model, extracting features from the original plate image to obtain multiple feature vectors of the target plate; The plurality of feature vectors of the target plate are compared with the plate feature vectors in the database, and based on the comparison result, the category of the target plate is determined to be the corresponding plate category in the database.

[0009] In one embodiment, before obtaining the panoramic image of the tray placement area in advance and analyzing the panoramic image by a pre-trained convolutional neural network model to obtain the position coordinates of each of the dinner plates, the method further includes: The camera parameters and angles are adjusted to capture images including the tray and all the plates, thereby obtaining a panoramic image of the tray placement area.

[0010] In one embodiment, analyzing the panoramic image by a pre-trained convolutional neural network model to obtain the position coordinates of each of the dinner plates, and obtaining the dinner plate image based on the position coordinates, comprises: Performing an enhancement operation on the panoramic image by an image preprocessing algorithm, wherein the enhancement operation includes denoising, contrast adjustment and edge sharpening operations; Call a pre-trained convolutional neural network model to analyze the panoramic image after the enhancement operation. The convolutional neural network model is trained with dining scene data to identify the edge contour of each plate and output the position coordinates of each plate P1=(x11,y11,x12,y12), P2=(x21,y21,x22,y22), … Pi=(xi1,yi1,xi2,yi2); Based on the position coordinates of the dinner plate, a corresponding dinner plate image is extracted from the panoramic image after the enhancement operation.

[0011] In one embodiment, detecting the dinner plate contour and the dish contour in the dinner plate image by a convolutional neural network model to generate a mask image includes: Calling a pre-trained convolutional neural network model to detect the dinner plate image, the convolutional neural network model inputs the dinner plate image and outputs a dinner plate outline and a dish outline; Based on the dinner plate outline and the dish outline, a mask image is generated, in which the area inside the dinner plate outline and not covered by the dish is retained as visible color information, while all other areas are set to black or other preset background colors, wherein all other areas include the area outside the dinner plate and the area covered by the dish.

[0012] In one embodiment, the identifying color information of the mask image, filling the color features into the inside of the dish outline, and generating the original plate image without the dish, includes: Extracting color information of a visible area from a mask image, and identifying the color information of the mask image through a convolutional neural network model, wherein the color information includes hue, saturation, and brightness; Filling the color information into the outline of the dish by texture synthesis, color transfer or neural network learning; The boundary between the filled area and the surrounding original plate color is smoothed to generate the original plate image without the dishes.

[0013] In one embodiment, comparing the multiple feature vectors of the target plate with the plate feature vectors in the database, and determining the category of the target plate as the corresponding plate category in the database based on the comparison result, includes: Constructing a database, wherein the constructed database includes a feature vector of each known category of dinner plates; Calculate the cosine similarity between the feature vector of the target plate and the feature vectors of plates of known categories in the database; When the cosine similarity is less than or equal to a preset threshold, it is determined that the category of the target plate is the corresponding plate category in the database.

[0014] In a third aspect, an embodiment of the present application provides a system for intelligent checkout machine plate recognition, the system comprising a plate image acquisition module, a mask image generation module, an original plate image module, a target plate feature vector acquisition module, and a plate category determination module, wherein: A module for acquiring a dinner plate image is used to acquire a panoramic image of the tray placement area in advance, analyze the panoramic image through a pre-trained convolutional neural network model, acquire the position coordinates of each dinner plate, and acquire the dinner plate image based on the position coordinates; The mask image generation module is used to detect the outline of the dinner plate and the outline of the dish in the dinner plate image through a convolutional neural network model to generate a mask image, wherein the mask image includes color information of a visible area of ​​the dinner plate, and the visible area is an area of ​​the dinner plate that is not covered by the dish; The original plate image module is used to identify the color information of the mask image, fill the color features into the inside of the dish outline, and generate the original plate image without the dish; The module for obtaining the target plate feature vector is used to extract features of the original plate image through the convolutional neural network model to obtain multiple feature vectors of the target plate; The module for determining the category of the plate is used to compare the multiple feature vectors of the target plate with the feature vectors of the plates in the database, and based on the comparison result, determine the category of the target plate as the corresponding category of the plate in the database.

[0015] In a fourth aspect, an embodiment of the present application provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, an intelligent checkout machine plate recognition method as described in the second aspect above is implemented.

[0016] In a fifth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a method for identifying a meal plate of an intelligent checkout machine as described in the second aspect above.

[0017] The embodiments of the present application provide an intelligent settlement machine and an intelligent settlement machine plate recognition method, system and medium which have at least the following technical effects.

[0018] The panoramic image of the tray placement area is obtained in advance, and the panoramic image is analyzed by a pre-trained convolutional neural network model to obtain the position coordinates of each plate, and the plate image is obtained based on the position coordinates. The plate outline and the dish outline in the plate image are detected by the convolutional neural network model to generate a mask image, wherein the mask image includes the color information of the visible area of ​​the plate, and the visible area is the area not covered by the dish in the plate. The color information of the mask image is identified, and the color feature is filled into the inside of the dish outline to generate the original plate image without the dish. The original plate image is feature extracted by the convolutional neural network model to obtain multiple feature vectors of the target plate. The multiple feature vectors of the target plate are compared with the plate feature vectors in the database, and based on the comparison results, the category of the target plate is determined to be the corresponding plate category in the database. The accurate confirmation of the plate category is achieved, and then the name, price and total price of the dish are determined. The problem of low recognition accuracy of intelligent checkout machine plate recognition in the existing related technology is solved.

[0019] Details of one or more embodiments of the present application are set forth in the following drawings and description to make other features, objects, and advantages of the present application more readily apparent. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings: Figure 1 It is a structural diagram of an intelligent settlement machine; Figure 2 It is a flow chart of a method for identifying a meal plate of an intelligent checkout machine; Figure 3 is a schematic structural diagram of a tray, a dining plate and dishes according to an exemplary embodiment; Figure 4 is a schematic diagram of a mask image according to an exemplary embodiment; Figure 5 is a block diagram of a system for identifying meal trays in an intelligent checkout machine according to an exemplary embodiment; Figure 6 It is a structural block diagram of an electronic device according to an exemplary embodiment. DETAILED DESCRIPTION

[0021] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is described and illustrated below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. Based on the embodiments provided in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application.

[0022] Obviously, the drawings described below are only some examples or embodiments of the present application. For ordinary technicians in this field, the present application can also be applied to other similar scenarios based on these drawings without creative work. In addition, it can also be understood that although the efforts made in this development process may be complicated and lengthy, for ordinary technicians in this field related to the content disclosed in this application, some changes in design, manufacturing or production based on the technical content disclosed in this application are just conventional technical means, and should not be understood as insufficient content disclosed in this application.

[0023] Reference to "embodiments" in this application means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those of ordinary skill in the art that the embodiments described in this application may be combined with other embodiments without conflict.

[0024] Unless otherwise defined, the technical terms or scientific terms involved in this application should be understood by people with ordinary skills in the technical field to which this application belongs. The words "one", "a", "a", "the" and the like involved in this application do not indicate a quantity limitation, and may indicate the singular or plural. The terms "include", "comprise", "have" and any of their variations involved in this application are intended to cover non-exclusive inclusions; for example, a process, method, system, product or device that includes a series of steps or modules (units) is not limited to the listed steps or units, but may also include steps or units that are not listed, or may also include other steps or units inherent to these processes, methods, products or devices. The words "connect", "connected", "coupled" and the like involved in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. The "multiple" involved in this application refers to two or more. "And / or" describes the association relationship of associated objects, indicating that there may be three relationships, for example, "A and / or B" can mean: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the objects before and after are in an "or" relationship. The terms "first", "second", "third", etc. involved in this application are only used to distinguish similar objects and do not represent a specific ordering of the objects.

[0025] The following are the definitions of terms used in this application: A dinner plate refers to a container used to hold specific dishes, which is usually in various shapes, such as round, rectangular, etc. Each dinner plate only holds one specific dish, so that the system can accurately identify the type of the dinner plate and thus determine the type of the dish. In the present invention, the shape, color, and edge contour of the dinner plate are key identification features. The system determines the type of the dinner plate by analyzing these features, thereby inferring the dish held on the dinner plate.

[0026] Dishes refer to: Dishes are foods served on a plate, which may include various types of dishes, such as main dishes, side dishes, soups, etc. The characteristics of dishes, such as color, shape, ingredients, etc., may vary, which brings challenges to recognition. In order to improve the accuracy of recognition, the present invention adopts special processing steps, such as color filling and contour recognition, to reduce the interference of dish characteristics on plate recognition.

[0027] Convolutional neural network refers to: a deep learning model that is widely used in image recognition and processing tasks. CNN extracts various levels of features from the input image, such as edges, corners, textures, etc. through multi-layer convolution operations and pooling operations. In this application, the CNN model is used in multiple key steps, including plate position detection, image segmentation, color recognition, and feature vector extraction. Through training on a large number of plate and dish images, CNN can effectively identify important features in the image and has a high recognition accuracy.

[0028] Contours are the boundary lines of objects in an image. In the field of image processing and computer vision, contours are often used to define the shape of an object. In this application, there are two main contours that need to be identified: Plate edge contour (C1): It is the boundary line outside the plate, defining the shape and size of the plate. By identifying this contour, the system can segment the plate area from the image.

[0029] Dish outline (C2): It is the boundary line of the dish on the plate, defining the area occupied by the dish. By identifying this outline, the system can further segment the dish area and perform operations such as color filling on the area in subsequent steps to prevent the dish features from interfering with plate recognition.

[0030] In a first aspect, an embodiment of the present application provides an intelligent settlement machine. Figure 1 It is a schematic diagram of an intelligent settlement machine, such as Figure 1 As shown, an intelligent settlement machine includes: A camera is used to photograph the tray recognition area and obtain a panoramic image of the tray placement area; A user display screen is used to display the price, name and total price of the dish selected by the user; Operation display screen, used by operators to modify prices of dishes that have been misidentified and to set up the smart checkout machine; A tray identification area is used to place trays, wherein a tray is provided with multiple positions for placing trays, and each tray is placed with one dish; The tray identification area is used to place trays to be settled; The calculation unit runs various functions of the intelligent settlement machine, including settlement, setting and output display functions.

[0031] Optionally, a camera is installed above the plate recognition area to capture a panoramic image of the tray placement area. The camera is optimized to ensure that the captured image is clear and includes all plates, providing high-resolution image data and providing basic data for subsequent image processing and dish recognition.

[0032] User display screen: located in front of the checkout machine, it displays the name, price and total price of the dishes in the user's current tray, and the user can intuitively view the identification and billing results.

[0033] Operation display screen: Provided for operators to use, allowing them to correct incorrectly identified dish information and make various settings for the smart checkout machine, such as adjusting system parameters or updating the database.

[0034] Plate Identification Area: Specially designed to hold trays with multiple dishes, each tray has a fixed number of positions for different types of plates, ensuring that only one dish is placed on each plate.

[0035] Plate identification area: The area where users place trays to be settled. After the system identifies the current tray, it automatically moves the tray to be identified into the identification area to ensure the continuity and efficiency of the identification process.

[0036] Arithmetic unit: A high-performance computing unit responsible for running all functions of the smart checkout machine, including but not limited to checkout, system settings, and output display functions. The arithmetic unit is connected to the camera and other hardware to ensure fast and stable data transmission. It runs all algorithms for image processing, dish recognition, and price calculation. The high-performance arithmetic unit has strong computing power to support complex neural network models and large-scale data processing.

[0037] In one embodiment, the dinner plates include multiple types, wherein the same dish is placed on the same type of dinner plate, and different dishes are distinguished by different types of dinner plates.

[0038] Optionally, each type of plate corresponds to a specific dish. For example, a blue plate represents a vegetable salad, a red plate represents a main course, a green plate represents a dessert, and so on. In this way, each plate type becomes a unique identifier for identifying the type of dish on the plate. All plates of the same type are placed with the same type of dish, ensuring the accuracy and consistency of recognition. This practice not only simplifies the recognition process, but also reduces errors caused by misplacement or confusion. In addition to color, different shapes, patterns or textures can be used to further distinguish between plate types. For example, round plates are used for cold dishes, square plates are used for hot dishes, and so on. This provides the system with more feature information and increases the reliability of recognition. To enhance the recognition accuracy, small labels or logos can be added to the plates. These logos can be barcodes, QR codes or simple icons, which can serve as auxiliary recognition features.

[0039] Through color coding, shape and other visual features, the smart checkout machine can efficiently and accurately identify different types of plates and the dishes on them, reducing the need for manual intervention. Customers can quickly understand menu options through intuitive colors and shapes, while the automated checkout process also improves service speed and accuracy, improving the overall dining experience.

[0040] In summary, the smart checkout machine can quickly identify the information of plates and dishes, reducing the waiting time for customers. Through automated processing, the process from image capture to result display can be completed in seconds, greatly improving the settlement speed. Using advanced convolutional neural networks (CNN) and other machine learning algorithms, the smart checkout machine can accurately detect the position of the plate, identify the type of plate and dishes, and remove interference factors through methods such as color feature filling to ensure high recognition accuracy. The smart checkout machine is equipped with a clear and intuitive user display that displays the price, name and total price of the selected dish, allowing customers to easily confirm order information, enhancing transparency and trust.

[0041] In a second aspect, the present application provides an intelligent settlement machine and an intelligent settlement machine plate recognition method. Figure 2 It is a flow chart of a method for recognizing a plate in an intelligent checkout machine. Figure 2 As shown, an intelligent settlement machine and a method for identifying a meal plate of an intelligent settlement machine include: Step S101: Based on the panoramic image of the tray placement area obtained in advance, the panoramic image is analyzed by a pre-trained convolutional neural network model to obtain the position coordinates of each dinner plate, and based on the position coordinates, the dinner plate image is obtained.

[0042] Step S102: Detect the outline of the plate and the outline of the dish in the plate image through a convolutional neural network model to generate a mask image, wherein the mask image includes color information of a visible area of ​​the plate, and the visible area is an area of ​​the plate not covered by the dish.

[0043] Step S103: Identify the color information of the mask image, fill the color features into the outline of the dish, and generate the original plate image without the dish.

[0044] Step S104: extract features of the original plate image through a convolutional neural network model to obtain multiple feature vectors of the target plate.

[0045] Step S105: compare the multiple feature vectors of the target plate with the plate feature vectors in the database, and determine the category of the target plate as the corresponding plate category in the database based on the comparison result.

[0046] In summary, the embodiments of the present application provide an intelligent checkout machine and an intelligent checkout machine plate recognition method. The panoramic image of the tray placement area is obtained in advance, and the panoramic image is analyzed by a pre-trained convolutional neural network model to obtain the position coordinates of each plate, and the plate image is obtained based on the position coordinates. The plate outline and the dish outline in the plate image are detected by the convolutional neural network model to generate a mask image, wherein the mask image includes the color information of the visible area of ​​the plate, and the visible area is the area not covered by the dish in the plate. The color information of the mask image is identified, and the color feature is filled into the inside of the dish outline to generate the original plate image without the dish. The original plate image is feature extracted by the convolutional neural network model to obtain multiple feature vectors of the target plate. The multiple feature vectors of the target plate are compared with the plate feature vectors in the database, and based on the comparison results, the category of the target plate is determined to be the corresponding plate category in the database. Accurate confirmation of the plate category is achieved, and then the name, price and total price of the dish are determined. The problem of low recognition accuracy in the intelligent checkout machine plate recognition in the existing related technology is solved.

[0047] In one embodiment, step S101, based on the panoramic image of the tray placement area acquired in advance, the panoramic image is analyzed by a pre-trained convolutional neural network model to acquire the position coordinates of each plate, and based on the position coordinates, the plate image is acquired. Specifically, it includes: Performing enhancement operations on the panoramic image through an image preprocessing algorithm, the enhancement operations include denoising, contrast adjustment and edge sharpening operations; Call the pre-trained convolutional neural network model to analyze the panoramic image after the enhancement operation. The convolutional neural network model is trained with the data of the dining scene to identify the edge contour of each plate and output the position coordinates of each plate P1=(x11,y11,x12,y12), P2=(x21,y21,x22,y22), … Pi=(xi1,yi1,xi2,yi2), where Pi=(xi1,yi1,xi2,yi2) is the coordinate of each plate in the image, xi1,yi1 is the coordinate of the upper left corner of the plate, and xi2,yi2 is the coordinate of the lower right corner of the plate; Based on the position coordinates of the dinner plate, the corresponding dinner plate image is extracted from the panoramic image after the enhancement operation.

[0048] Optionally, Figure 3 FIG. 1 is a schematic diagram of a tray, a dining plate and dishes according to an exemplary embodiment. Figure 3As shown, first, the camera pre-sets parameters (such as resolution, exposure time, white balance, etc.) to ensure that high-quality panoramic images of the tray placement area are captured. These parameters are optimized to improve image clarity and detail performance. A series of enhancement operations are performed on the captured panoramic image, including but not limited to, denoising: applying filtering algorithms (such as Gaussian filtering or bilateral filtering) to remove random noise in the image and maintain the smoothness of the image. Contrast adjustment: Improving image contrast through methods such as histogram equalization or adaptive histogram equalization to make different colors and textures more obvious. Edge sharpening: Using edge enhancement operators (such as Laplacian operators or Sobel operators) to enhance the boundary information in the image is helpful for subsequent contour detection. Then, a convolutional neural network (CNN) model specially trained for catering scenes is called. The model has been trained with a large amount of catering scene data and can accurately identify the edge contours of the plates. The CNN model is used to analyze the preprocessed panoramic image, identify the edge contours of each plate, and output the position coordinates of each plate. The position coordinates of each plate are expressed as a rectangular box, for example, P1=(x11,y11,x12,y12), where ( x 11, y 11) is the coordinate of the upper left corner of the plate, and (x12,y12) is the coordinate of the lower right corner. For multiple plates, the corresponding coordinate sets P2=(x21,y21,x22,y22),…,Pi=(xi1,yi1,xi2,yi2) are given respectively. Finally, according to the obtained plate position coordinates, the corresponding small plate image is accurately cropped from the enhanced panoramic image. This step ensures that each plate image contains only the information of a single plate, reduces interference factors, and facilitates subsequent feature analysis and recognition.

[0049] Step S101 enhances the quality of the original image through image preprocessing, so that the CNN model can detect the edge contour of the plate more accurately and improve the accuracy of the position coordinates. The accurate position coordinates make it possible to quickly extract the small plate image from the panoramic image, reduce unnecessary calculations, and speed up the entire processing flow. The preprocessing operation improves the robustness of the system under different lighting conditions and background complexity, ensuring stable operation even in changing environments.

[0050] In one embodiment, step S102, a convolutional neural network model is used to detect the outline of the plate and the outline of the dish in the plate image, and a mask image is generated, wherein the mask image includes color information of the visible area of ​​the plate, and the visible area is the area of ​​the plate not covered by the dish. Specifically, it includes: Call the pre-trained convolutional neural network model to detect the plate image. The convolutional neural network model inputs the plate image and outputs the plate outline and dish outline. Based on the outline of the dinner plate and the outline of the dish, a mask image is generated. In the mask image, the area inside the outline of the dinner plate and not covered by the dish is retained as visible color information, while all other areas are set to black or other preset background colors, where all other areas include the area outside the dinner plate and the area covered by the dish.

[0051] Optionally, Figure 4 FIG. 1 is a schematic diagram of a mask image according to an exemplary embodiment. Figure 4 As shown, first, a convolutional neural network (CNN) model specially trained for dining scenes is called, and the model has been trained with a large number of image data sets containing plates and different dishes. Ensure that the model can accurately identify the contours of plates and dishes. Subsequently, the plate image extracted from step S101 is provided as input to the CNN model. After forward propagation, the model outputs the contour information of each plate and the contour information of each dish. These contour information are usually given in the form of a bounding box or a segmentation mask, depending on the model architecture. Next, a binary mask image is created on the original plate image based on the plate contour and dish contour information output by the CNN model. In this mask image, the areas inside the plate contour and not covered by the dishes are retained as visible color information, that is, these areas retain the color characteristics of the original plate. All other areas, including the area outside the plate and the area covered by the dishes, are set to black or other preset background colors (such as white). This step is implemented in the following way. Then, a full black (or full white) mask image of the same size as the original plate image is created. For each plate outline that is not covered by the dish, the corresponding pixel value is copied from the original plate image to the mask image, maintaining its original color. For the area within the plate outline but covered by the dish, its corresponding position in the mask image is set to black (or a preset background color) to remove the influence of the dish. Finally, edge smoothing algorithms are applied to optimize the quality of the mask image to ensure that the color information of the visible area of ​​the plate is clearer while eliminating possible noise and artifacts.

[0052] Step S102 uses deep learning technology, especially the CNN model trained for dining scenes, to detect the outlines of plates and dishes very accurately, even in complex backgrounds. By generating a mask image, only the color information of the visible area of ​​the plate that is not covered by the dishes is retained, effectively removing the interference of the dishes on subsequent color recognition and feature extraction. The generated mask image provides a clean and clear basis for subsequent color information extraction, feature analysis, etc., reducing unnecessary computational burden and improving the efficiency of the overall system.

[0053] In one embodiment, step S103, identifying the color information of the mask image, filling the color features into the inside of the dish outline, and generating the original plate image without the dish. Specifically includes: Extracting the color information of the visible area from the mask image, and identifying the color information of the mask image through a convolutional neural network model, wherein the color information includes hue, saturation and brightness; Fill the color information into the outline of the dish through texture synthesis, color transfer or neural network learning; The boundary between the filled area and the surrounding original plate color is smoothed to generate the original plate image without the dishes.

[0054] Optionally, a mask image is first used in which the visible areas of the plate (the parts not covered by the dishes) are marked. For these visible areas, color information is extracted. The RGB image is converted to the HSV color space to better handle hue, saturation, and value, which are important features of color. Subsequently, the extracted color information is analyzed using a pre-trained convolutional neural network (CNN) model. The model can understand and classify different color attributes and predict the color characteristics that the inside of the dish outline should have. This step ensures that colors can be accurately identified and matched even under complex or changing lighting conditions. Depending on specific needs and technical implementation choices, one of the following three methods can be used to fill the inside of the dish outline: Texture synthesis: Mimic the texture pattern of the visible area to create a similar texture effect in the dish area. Color transfer: Directly copy the color characteristics of the visible area (such as HSV values) to the dish area to ensure color consistency between the two. Neural network learning: Use deep learning models (such as generative adversarial networks GANs) to learn how to naturally map the color characteristics of the visible area to the dish area, producing a more realistic visual effect. Finally, in order to make the filled image look more natural, the boundary between the newly filled color and the surrounding original plate color needs to be smoothed. Poisson editing, gradient domain fusion and other algorithms can be used to ensure a natural color transition while keeping the edges clear. Finally, an image of the original plate without the influence of the dish is output. This image retains the color and texture characteristics of the original plate, but without the presence of the dish, and can be used for subsequent applications, such as tableware status recording, automatic inventory, etc.

[0055] Step S103 achieves high-precision capture and reproduction of color information through HSV color space conversion and CNN model, ensuring the authenticity and consistency of the image. Combining a variety of advanced image processing technologies, it ensures that the generated image not only has good color matching, but also has a natural and harmonious overall visual effect without obvious splicing marks.

[0056] In one embodiment, step S104 , features of the original plate image are extracted using a convolutional neural network model to obtain multiple feature vectors of the target plate.

[0057] Optionally, before this step, the original plate image after removing the dishes has been generated through step S103. Ensure that the image input to the convolutional neural network (CNN) model is preprocessed, including but not limited to resizing, normalization and other operations to meet the model input requirements. According to specific application requirements and performance considerations, finally, the convolutional neural network extracts a 1024-dimensional feature vector of the image. This feature vector is a highly abstract representation of the plate image, which can effectively capture the shape, color, texture and other features of the plate.

[0058] In step S104, a pre-trained convolutional neural network model is used to perform deep feature extraction on the original plate image after the dishes are removed, thereby obtaining multiple feature vectors that can characterize the characteristics of the target plate.

[0059] In one embodiment, step S105 compares multiple feature vectors of the target plate with the plate feature vectors in the database, and based on the comparison result, determines that the category of the target plate is the corresponding plate category in the database. Specifically, it includes: Construct a database, including feature vectors of each known category of plates in the database; Calculate the cosine similarity between the feature vector of the target plate and the feature vectors of plates of known categories in the database; When the cosine similarity is less than or equal to a preset threshold, it is determined that the category of the target plate is the corresponding plate category in the database.

[0060] Optionally, first, a large number of known types of plate sample images are collected, and it is ensured that these samples cover all possible plate types and variants. There should be a sufficient number of samples for each plate type to ensure the effectiveness and generalization ability of model training. Subsequently, for each plate sample, a pre-trained convolutional neural network (CNN) or other feature extraction method is used to generate its feature vector. These feature vectors should be able to fully characterize the key characteristics of the plate, such as color, texture, shape, etc. Next, the extracted feature vectors are stored in an efficient data structure, such as an index database or a vector search engine (such as Faiss, Annoy). Each feature vector is associated with a unique plate category label. This step provides a basis for subsequent rapid retrieval and similarity calculation. Subsequently, multiple feature vectors of the target plate obtained in step S104 are used as input. For each target plate feature vector, the cosine similarity between it and all known category plate feature vectors in the database is calculated. Cosine similarity is a method of measuring the cosine value of the angle between two non-zero vectors, ranging from -1 to 1, and the closer the value is to 1, the more similar the two vectors are. According to the actual application requirements and experimental results, a reasonable cosine similarity threshold (for example, 0.8) is set. This threshold is used to determine whether two feature vectors are similar enough to be confirmed as the same category. Next, if the cosine similarity between the feature vector of the target plate and the feature vector of a known category plate is greater than or equal to the preset threshold, the target plate is determined to belong to the known category. Otherwise, if all similarities are below the threshold, further processing may be required, such as marking it as an unknown category or prompting the operator for manual confirmation. Finally, once the plate category is determined, the system displays the recognition results (including plate category, dish name, price and total price) on the user display for the customer to confirm and prepare to pay. If there is a recognition error, the operator can correct it through the operating display.

[0061] Step S105 can accurately measure the similarity between feature vectors by calculating cosine similarity, thereby achieving high-precision classification of plate categories. This method not only considers the direction of feature vectors, but also their length information, making classification more reliable. By using efficient database structure and indexing technology, the most similar candidates can be quickly found in a large-scale feature vector set, greatly improving the response speed and efficiency of the system.

[0062] In summary, the embodiments of the present application provide an intelligent checkout machine and an intelligent checkout machine plate recognition method. The panoramic image of the tray placement area is obtained in advance, and the panoramic image is analyzed by a pre-trained convolutional neural network model to obtain the position coordinates of each plate, and the plate image is obtained based on the position coordinates. The plate outline and the dish outline in the plate image are detected by the convolutional neural network model to generate a mask image, wherein the mask image includes the color information of the visible area of ​​the plate, and the visible area is the area not covered by the dish in the plate. The color information of the mask image is identified, and the color feature is filled into the inside of the dish outline to generate the original plate image without the dish. The original plate image is feature extracted by the convolutional neural network model to obtain multiple feature vectors of the target plate. The multiple feature vectors of the target plate are compared with the plate feature vectors in the database, and based on the comparison results, the category of the target plate is determined to be the corresponding plate category in the database. Accurate confirmation of the plate category is achieved, and then the name, price and total price of the dish are determined. The problem of low recognition accuracy in the intelligent checkout machine plate recognition in the existing related technology is solved.

[0063] In a third aspect, an embodiment of the present application provides a system for identifying meal plates at an intelligent checkout machine. Figure 5 FIG. 1 is a block diagram of a system for identifying a meal tray of an intelligent checkout machine according to an exemplary embodiment. Figure 5 As shown, the system includes a plate image acquisition module 510, a mask image generation module 520, an original plate image module 530, a target plate feature vector acquisition module 540 and a plate category determination module 550, wherein: The plate image acquisition module 510 is used to acquire a panoramic image of the tray placement area in advance, analyze the panoramic image through a pre-trained convolutional neural network model, acquire the position coordinates of each plate, and acquire the plate image based on the position coordinates; A mask image generation module 520 is used to detect the outline of the dinner plate and the outline of the dish in the dinner plate image by using a convolutional neural network model to generate a mask image, wherein the mask image includes color information of a visible area of ​​the dinner plate, and the visible area is an area of ​​the dinner plate that is not covered by the dish; The original plate image module 530 is used to identify the color information of the mask image, fill the color features into the inside of the dish outline, and generate the original plate image without the dish; A module 540 for obtaining a target plate feature vector is used to extract features of the original plate image through a convolutional neural network model to obtain multiple feature vectors of the target plate; The plate category determination module 550 is used to compare the multiple feature vectors of the target plate with the plate feature vectors in the database, and determine the category of the target plate as the corresponding plate category in the database based on the comparison result.

[0064] In summary, the present application provides a system for intelligent plate recognition of a checkout machine. Through the plate image acquisition module 510, the mask image generation module 520, the original plate image module 530, the target plate feature vector acquisition module 540 and the plate category determination module 550, the plate category is accurately confirmed, and then the dish name, price and total price are determined. The problem of low recognition accuracy of intelligent plate recognition of checkout machines in the existing related technologies is solved.

[0065] It should be noted that the system for identifying meal plates in an intelligent checkout machine provided in this embodiment is used to implement the above-mentioned implementation mode, and the descriptions that have been made will not be repeated. As used above, the terms "module", "unit", "subunit", etc. can be a combination of software and / or hardware that implements a predetermined function. Although the device described in the above embodiment is preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0066] In a third aspect, an embodiment of the present application provides an electronic device, Figure 6 FIG. 1 is a block diagram of an electronic device according to an exemplary embodiment. Figure 6 As shown, the electronic device may include a processor 61 and a memory 62 storing computer program instructions.

[0067] Specifically, the processor 61 may include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits of the embodiments of the present application.

[0068] Among them, the memory 62 may include a large-capacity memory for data or instructions. By way of example and not limitation, the memory 62 may include a hard disk drive (HDD), a floppy disk drive, a solid-state drive (SSD), a flash memory, an optical disk, a magneto-optical disk, a magnetic tape, or a universal serial bus (USB) drive, or a combination of two or more of these. Where appropriate, the memory 62 may include a removable or non-removable (or fixed) medium. Where appropriate, the memory 62 may be inside or outside the data processing device. In a specific embodiment, the memory 62 is a non-volatile memory. In a specific embodiment, the memory 62 includes a read-only memory (ROM) and a random access memory (RAM). Where appropriate, the ROM may be a mask-programmed ROM, a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), an electrically alterable ROM (EAROM) or a flash memory (FLASH), or a combination of two or more of these. Under appropriate circumstances, the RAM can be a static random access memory (SRAM) or a dynamic random access memory (DRAM), wherein the DRAM can be a fast page mode dynamic random access memory (FPMDRAM), an extended data output dynamic random access memory (EDODRAM), a synchronous dynamic random access memory (SDRAM), etc.

[0069] The memory 62 may be used to store or cache various data files required for processing and / or communication, as well as possible computer program instructions executed by the processor 61 .

[0070] The processor 61 reads and executes the computer program instructions stored in the memory 62 to implement any one of the smart checkout machines and the smart checkout machine plate recognition methods in the above embodiments.

[0071] In one embodiment, the device for a smart checkout machine plate recognition method may further include a communication interface 63 and a bus 60. Figure 6 As shown, the processor 61, the memory 62, and the communication interface 63 are connected via a bus 60 and communicate with each other.

[0072] The communication interface 63 is used to implement communication between the modules, devices, units and / or equipment in the embodiment of the present application. The communication port 63 can also implement data communication with other components such as: external devices, image / data acquisition equipment, databases, external storage, and image / data processing workstations.

[0073] The bus 60 includes hardware, software or both, and couples the components of the device of the intelligent checkout machine plate recognition method to each other. The bus 60 includes but is not limited to at least one of the following: data bus, address bus, control bus, expansion bus, and local bus. By way of example and not limitation, bus 60 may include an Accelerated Graphics Port (AGP) or other graphics bus, an Extended Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), a HyperTransport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an InfiniBand interconnect, a Low Pin Count (LPC) bus, a memory bus, a Micro Channel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local Bus (VLB) bus, or other suitable buses or a combination of two or more of the above. Where appropriate, bus 60 may include one or more buses. Although embodiments of the present application describe and illustrate a particular bus, the present application contemplates any suitable bus or interconnect.

[0074] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having a program stored thereon. When the program is executed by a processor, an intelligent settlement machine and an intelligent settlement machine plate recognition method provided in the first aspect are implemented.

[0075] The readable storage medium may include but is not limited to: a portable disk, a hard disk, a random access memory, a read-only memory, an erasable programmable read-only memory, an optical storage device, a magnetic storage device or any suitable combination of the above.

[0076] In a possible implementation, the present invention can also be implemented in the form of a program product, which includes program code. When the program product is run on a terminal device, the program code is used to enable the terminal device to execute the steps of an intelligent settlement machine and an intelligent settlement machine plate recognition method provided in the first aspect.

[0077] Among them, the program code for executing the present invention can be written in any combination of one or more programming languages, and the program code can be executed completely on the user device, partially on the user device, as an independent software package, partially on the user device and partially on a remote device, or completely on the remote device.

[0078] The technical features of the above embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0079] The above embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A method for identifying a meal tray of an intelligent settlement machine, characterized in that: The method comprises: According to the panoramic image of the tray placement area obtained in advance, the panoramic image is analyzed by a pre-trained convolutional neural network model to obtain the position coordinates of each of the dinner plates, and based on the position coordinates, an image of the dinner plate is obtained; Detecting the outline of the dinner plate and the outline of the dish in the dinner plate image by a convolutional neural network model to generate a mask image, wherein the mask image includes color information of a visible area of ​​the dinner plate, and the visible area is an area of ​​the dinner plate that is not covered by the dish; Identify the color information of the mask image, fill the color features into the inside of the dish outline, and generate the original plate image without the dish; Using the convolutional neural network model, extracting features from the original plate image to obtain multiple feature vectors of the target plate; The plurality of feature vectors of the target plate are compared with the plate feature vectors in the database, and based on the comparison result, the category of the target plate is determined to be the corresponding plate category in the database.

2. The method according to claim 1, characterized in that The panoramic image is analyzed by a pre-trained convolutional neural network model to obtain the position coordinates of each of the dinner plates, and the dinner plate image is obtained based on the position coordinates, including: Performing an enhancement operation on the panoramic image by an image preprocessing algorithm, wherein the enhancement operation includes denoising, contrast adjustment and edge sharpening operations; The pre-trained convolutional neural network model is called to analyze the panoramic image after the enhancement operation. The convolutional neural network model is trained with the data of the dining scene to identify the edge contour of each plate and output the position coordinates P1=(x 11 ,y 11 ,x 12 ,y 12 ), P2=(x 21 ,y 21 ,x 22 ,y 22 ), … P i =(x i1 ,y i1 ,x i2 ,y i2 ); Based on the position coordinates of the dinner plate, a corresponding dinner plate image is extracted from the panoramic image after the enhancement operation.

3. The method according to claim 1, characterized in that The method of detecting the dinner plate contour and the dish contour in the dinner plate image by using a convolutional neural network model to generate a mask image includes: Calling a pre-trained convolutional neural network model to detect the dinner plate image, the convolutional neural network model inputs the dinner plate image and outputs a dinner plate outline and a dish outline; Based on the dinner plate outline and the dish outline, a mask image is generated, in which the area inside the dinner plate outline and not covered by the dish is retained as visible color information, while all other areas are set to black or other preset background colors, wherein all other areas include the area outside the dinner plate and the area covered by the dish.

4. The method according to claim 1, characterized in that: The identifying color information of the mask image, filling the color feature into the inside of the dish outline, and generating the original plate image without the dish, includes: Extracting color information of a visible area from a mask image, and identifying the color information of the mask image through a convolutional neural network model, wherein the color information includes hue, saturation, and brightness; Filling the color information into the outline of the dish by texture synthesis, color transfer or neural network learning; The boundary between the filled area and the surrounding original plate color is smoothed to generate the original plate image without the dishes.

5. The method according to claim 1, characterized in that The step of comparing the plurality of feature vectors of the target plate with the plate feature vectors in the database, and determining the category of the target plate as the corresponding plate category in the database based on the comparison result, comprises: Constructing a database, wherein the constructed database includes a feature vector of each known category of dinner plates; Calculate the cosine similarity between the feature vector of the target plate and the feature vectors of plates of known categories in the database; When the cosine similarity is less than or equal to a preset threshold, it is determined that the category of the target plate is the corresponding plate category in the database.

6. The method according to claim 1, characterized in that Before obtaining the panoramic image of the tray placement area in advance and analyzing the panoramic image by a pre-trained convolutional neural network model to obtain the position coordinates of each of the dinner plates, the method further includes: The camera parameters and angles are adjusted to capture images including the tray and all the plates, thereby obtaining a panoramic image of the tray placement area.

7. An intelligent settlement machine, characterized in that: include: A camera is used to photograph the tray recognition area and obtain a panoramic image of the tray placement area; A user display screen is used to display the price, name and total price of the dish selected by the user; Operation display screen, used by operators to modify prices of dishes that have been misidentified and to set up the smart checkout machine; A plate identification area, used for placing a tray, wherein one tray is provided with a plurality of positions for placing plates, and each plate is provided with a dish; The tray identification area is used to place trays to be settled; An operator, used to execute the intelligent settlement machine plate recognition method as described in any one of claims 1 to 6, and to execute the settlement, setting and output display functions of the intelligent settlement machine.

8. The intelligent settlement machine according to claim 7, characterized in that: The dinner plates include multiple types, wherein the same dish is placed on the same type of dinner plate, and different dishes are distinguished by different types of the dinner plates.

9. An intelligent settlement machine plate recognition system, characterized in that: The system comprises a module for acquiring a plate image, a module for generating a mask image, a module for acquiring an original plate image, a module for acquiring a target plate feature vector and a module for determining a plate category, wherein: A module for acquiring a dinner plate image is used to acquire a panoramic image of the tray placement area in advance, analyze the panoramic image through a pre-trained convolutional neural network model, acquire the position coordinates of each dinner plate, and acquire the dinner plate image based on the position coordinates; The mask image generation module is used to detect the outline of the dinner plate and the outline of the dish in the dinner plate image through a convolutional neural network model to generate a mask image, wherein the mask image includes color information of a visible area of ​​the dinner plate, and the visible area is an area of ​​the dinner plate that is not covered by the dish; The original plate image module is used to identify the color information of the mask image, fill the color features into the inside of the dish outline, and generate the original plate image without the dish; The module for obtaining the target plate feature vector is used to extract features of the original plate image through the convolutional neural network model to obtain multiple feature vectors of the target plate; The module for determining the category of the plate is used to compare the multiple feature vectors of the target plate with the feature vectors of the plates in the database, and based on the comparison result, determine the category of the target plate as the corresponding category of the plate in the database.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by the processor, an intelligent settlement machine plate recognition method according to any one of claims 1 to 6 is implemented.

Images