Dinner Plate Information Display Method, Device, Electronic Device, and Computer Readable Medium

Abstract

Embodiments of the present disclosure disclose a method, apparatus, electronic device, and computer-readable medium for displaying dinner plate information. A specific implementation of the method includes: obtaining dinner plate images corresponding to each dinner plate; obtaining instance segmentation images of each dinner plate; performing the following steps: inputting the instance segmentation image of the dinner plate into a feature extraction network in a pre-trained dinner plate recognition model; inputting the dinner plate feature map into a feature processing network in the dinner plate recognition model; inputting the full dinner plate feature information into a classification processing network in the dinner plate recognition model; determining the value information of each dinner plate corresponding to each instance segmentation image of the dinner plate; generating dinner plate information; and displaying the dinner plate information. This implementation can improve the accuracy of recognizing dinner plate images and reduce waste of computing resources.

Description

Technical Field

[0001] Embodiments of the present disclosure relate to the field of computer technologies, and more particularly, to a method, an apparatus, an electronic device, and a computer-readable medium for displaying dinner plate information. Background Art

[0002] Dinner plate recognition technology is a technology for recognizing a dinner plate based on an image of the dinner plate to obtain category information of the dinner plate. Currently, when recognizing a dinner plate, the commonly adopted method is as follows: constructing a dinner plate image classification model. Then, performing feature extraction processing on a pre-acquired dinner plate image through the constructed dinner plate image classification model. Finally, obtaining the category information of the dinner plate according to the extracted dinner plate features.

[0003] However, when recognizing a dinner plate in the above manner, the following technical problems often exist:

[0004] When using a dinner plate image classification model to perform feature extraction on a dinner plate image, it is easily interfered by the background image in the dinner plate image, which easily leads to errors in the extracted dinner plate features. As a result, the accuracy of the dinner plate image classification model in classifying and recognizing the dinner plate image is relatively low, resulting in the need to re-invoke computing resources to perform feature extraction and classification recognition on the same dinner plate image, causing waste of computing resources. Summary of the Invention

[0005] This section of the present disclosure is used to briefly introduce concepts, which will be described in detail in the subsequent Detailed Description section. This section of the present disclosure is not intended to identify the key features or essential features of the claimed technical solution, nor is it intended to be used to limit the scope of the claimed technical solution.

[0006] Some embodiments of the present disclosure propose a method, an apparatus, an electronic device, and a computer-readable medium for displaying dinner plate information to solve one or more of the technical problems mentioned in the above Background Art section.

[0007] In a first aspect, some embodiments of the present disclosure provide a method for displaying plate information, the method including: obtaining plate images corresponding to respective plates; performing instance segmentation processing on the plate images to obtain respective instance segmentation images of the plates; for each instance segmentation image of the respective instance segmentation images of the plates, performing the following steps: inputting the instance segmentation image of the plate into a feature extraction network in a pre-trained plate recognition model to perform feature extraction processing on the instance segmentation image of the plate to obtain a plate feature map corresponding to the instance segmentation image of the plate; inputting the plate feature map into a feature processing network in the plate recognition model to generate plate full-map feature information corresponding to the plate feature map; inputting the plate full-map feature information into a classification processing network in the plate recognition model to obtain plate category information corresponding to the instance segmentation image of the plate; determining respective plate value information corresponding to the respective instance segmentation images of the plates based on the obtained respective plate category information and a preset group of category value information; generating plate information based on the respective plate category information and the respective plate value information, where the plate information includes plate total value information; and displaying the plate information.

[0008] In a second aspect, some embodiments of the present disclosure provide a device for displaying plate information, the device including: an obtaining unit configured to obtain plate images corresponding to respective plates; a segmentation unit configured to perform instance segmentation processing on the plate images to obtain respective instance segmentation images of the plates; an execution unit configured to, for each instance segmentation image of the respective instance segmentation images of the plates, perform the following steps: inputting the instance segmentation image of the plate into a feature extraction network in a pre-trained plate recognition model to perform feature extraction processing on the instance segmentation image of the plate to obtain a plate feature map corresponding to the instance segmentation image of the plate; inputting the plate feature map into a feature processing network in the plate recognition model to generate plate full-map feature information corresponding to the plate feature map; inputting the plate full-map feature information into a classification processing network in the plate recognition model to obtain plate category information corresponding to the instance segmentation image of the plate; a determining unit configured to determine respective plate value information corresponding to the respective instance segmentation images of the plates based on the obtained respective plate category information and a preset group of category value information; a generating unit configured to generate plate information based on the respective plate category information and the respective plate value information, where the plate information includes plate total value information; and a displaying unit configured to display the plate information.

[0009] In a third aspect, some embodiments of the present disclosure provide an electronic device, including: one or more processors; a storage device storing one or more programs thereon, and when the one or more programs are executed by the one or more processors, the one or more processors implement the method described in any implementation manner of the first aspect above.

[0010] In a fourth aspect, some embodiments of the present disclosure provide a computer-readable medium storing a computer program thereon, wherein when the program is executed by a processor, the method described in any implementation manner of the first aspect above is implemented.

[0011] The above-mentioned various embodiments of the present disclosure have the following beneficial effects: Through the method for displaying plate information of the present disclosure, the accuracy of recognizing plate images can be improved, and waste of computing resources can be reduced. Specifically, the reasons for the low accuracy of recognizing plate images and waste of computing resources are as follows: When using a plate image classification model to extract features from a plate image, it is easily interfered by the background image in the plate image, which easily leads to errors in the extracted plate features. As a result, the accuracy of the plate image classification model in classifying and recognizing the plate image is relatively low, resulting in the need to redeploy computing resources to extract features and classify and recognize the same plate image again, causing waste of computing resources. Based on this, in the method for displaying plate information according to some embodiments of the present disclosure, first, plate images corresponding to each plate are obtained. Thus, the original data to be processed can be obtained. Secondly, instance segmentation processing is performed on the above-mentioned plate images to obtain each plate instance segmentation image. Thus, the interference of other factors on the plate image can be reduced, and the accuracy rate of subsequent classification and recognition of the plate image can be improved. Then, for each plate instance segmentation image in the above-mentioned various plate instance segmentation images, the following steps are performed: First, the above-mentioned plate instance segmentation image is input into the feature extraction network in a pre-trained plate recognition model to perform feature extraction processing on the above-mentioned plate instance segmentation image, and a plate feature map corresponding to the above-mentioned plate instance segmentation image is obtained. Thus, a feature map corresponding to the plate image can be obtained. Secondly, the above-mentioned plate feature map is input into the feature processing network in the above-mentioned plate recognition model to generate plate full-image feature information corresponding to the above-mentioned plate feature map. Thus, the global feature information of the plate image can be obtained. Then, the above-mentioned plate full-image feature information is input into the classification processing network in the above-mentioned plate recognition model to obtain plate category information corresponding to the above-mentioned plate instance segmentation image. Thus, the category corresponding to the plate image can be obtained. Then, based on the obtained various plate category information and a preset group of category value information, the respective plate value information corresponding to the above-mentioned various plate instance segmentation images is determined. Thus, the value information corresponding to each plate can be determined through the category corresponding to the plate image. Then, based on the above-mentioned various plate category information and the above-mentioned various plate value information, plate information is generated, where the above-mentioned plate information includes total plate value information. Thus, the total value information of each plate can be obtained. Finally, the above-mentioned plate information is displayed. Thus, the total value information of the plate can be displayed. Also, because instance segmentation processing can be performed on the plate image before classification and recognition, the interference of the background image in the plate image on the plate image can be reduced, and thus the accuracy rate of classifying and recognizing the plate image can be improved.Also, since the global feature information of the dinner plate image can be integrated and then the dinner plate image can be classified and recognized according to the features of the dinner plate image, the accuracy of classifying and recognizing the dinner plate image can be improved, the number of times of calling computing resources to process the same dinner plate image can be reduced, and thus the waste of computing resources can be reduced. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] In combination with the accompanying drawings and with reference to the following specific embodiments, the above and other features, advantages and aspects of the embodiments of the present disclosure will become more obvious. Throughout the accompanying drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic and the elements and elements are not necessarily drawn to scale.

[0013] Figure 1 is a flowchart of some embodiments of the dinner plate information display method according to the present disclosure;

[0014] Figure 2 is a schematic structural diagram of some embodiments of the dinner plate information display device according to the present disclosure;

[0015] Figure 3 is a schematic structural diagram of an electronic device suitable for implementing some embodiments of the present disclosure;

[0016] Figure 4 is a screenshot of a schematic internal test operation page for displaying dinner plate information according to the dinner plate information display method of the present disclosure. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0017] The embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although some embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. On the contrary, these embodiments are provided to more thoroughly and completely understand the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not used to limit the protection scope of the present disclosure.

[0018] In addition, it should be noted that, for the sake of convenience of description, only the parts related to the relevant invention are shown in the drawings. Without conflict, the embodiments in the present disclosure and the features in the embodiments can be combined with each other.

[0019] It should be noted that the concepts such as "first" and "second" mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence relationship of the functions performed by these devices, modules or units.

[0020] It should be noted that the modifications of "one" and "multiple" mentioned in this disclosure are illustrative rather than restrictive. Those skilled in the art should understand that unless otherwise clearly specified in the context, it should be understood as "one or more".

[0021] The names of the messages or information exchanged between multiple devices in the embodiments of this disclosure are only for illustrative purposes and are not used to limit the scope of these messages or information.

[0022] The present disclosure will be described in detail below with reference to the accompanying drawings and in conjunction with embodiments.

[0023] Figure 1 Flow 100 of some embodiments of the dinner plate information display method according to the present disclosure is shown. The dinner plate information display method includes the following steps:

[0024] Step 101, obtain dinner plate images corresponding to each dinner plate.

[0025] In some embodiments, the execution subject (such as a computing device) of the dinner plate information display method may obtain dinner plate images corresponding to each dinner plate. Among them, the above-mentioned dinner plate images may be images of each dinner plate. For example, the above-mentioned dinner plate images may be images captured by a camera before the execution subject recognizes each dinner plate with meals selected by the user in the restaurant. In practice, the execution subject may obtain dinner plate images from a preset database. Among them, the above-mentioned database may be a database for storing dinner plate images. The execution subject may be a server device that recognizes each dinner plate through the dinner plate image and generates dinner plate information corresponding to each dinner plate when the user performs a payment operation in the restaurant. The above-mentioned server device may be a computing device used by restaurant service personnel (such as a cashier).

[0026] Step 102, perform instance segmentation processing on the dinner plate images to obtain instance segmentation images of each dinner plate.

[0027] In some embodiments, the execution subject may perform instance segmentation processing on the above-mentioned dinner plate images to obtain instance segmentation images of each dinner plate. Among them, the above-mentioned instance segmentation images of the dinner plate may be images that only contain one dinner plate obtained after performing instance segmentation processing on the above-mentioned dinner plate images.

[0028] In some optional implementation manners of some embodiments, the execution subject may perform instance segmentation processing on the above-mentioned dinner plate images through the following steps to obtain instance segmentation images of each dinner plate:

[0029] First step, preprocess the above-mentioned dinner plate image to obtain a preprocessed dinner plate image. Among them, the above-mentioned preprocessed dinner plate image can be the dinner plate image after preprocessing. In practice, first, the above-mentioned execution entity can perform denoising processing on the above-mentioned dinner plate image through Gaussian filtering to obtain a denoised dinner plate image. Then, the resolution of the denoised dinner plate image can be adjusted through resampling technology to obtain a preprocessed dinner plate image.

[0030] Second step, normalize the above-mentioned preprocessed dinner plate image to obtain a normalized dinner plate image. Among them, the above-mentioned normalized dinner plate image can be the image obtained after normalizing the above-mentioned preprocessed dinner plate image. In practice, the above-mentioned execution entity can normalize the above-mentioned preprocessed dinner plate image through the extreme value normalization method to obtain a normalized dinner plate image. Among them, the above-mentioned extreme value normalization method can be deviation normalization.

[0031] Third step, perform feature extraction processing on the above-mentioned normalized dinner plate image to obtain a normalized dinner plate feature map. Among them, the above-mentioned normalized dinner plate feature map can be the feature map corresponding to the above-mentioned normalized dinner plate image. In practice, the above-mentioned execution entity can input the above-mentioned normalized dinner plate image into the feature map extraction network of a pre-trained instance segmentation model to obtain a normalized dinner plate feature map. Among them, the above-mentioned instance segmentation model can be a neural network model that takes the normalized dinner plate image as input and outputs the respective instance segmentation masks corresponding to the normalized dinner plate image. The above-mentioned instance segmentation model can include four layers.

[0032] The first layer can be a feature map extraction network. The above-mentioned feature map extraction network can be a neural network that takes the normalized dinner plate image as input and outputs the normalized dinner plate feature map. For example, the above-mentioned feature map extraction network can be a convolutional neural network.

[0033] The second layer can be a feature map processing layer. The above-mentioned feature map processing layer can include a feature map marking network and a classification layer. Among them, the above-mentioned feature map marking network can be a neural network that takes the normalized dinner plate feature map as input and outputs the respective bounding box information corresponding to the normalized dinner plate feature map. For example, the above-mentioned feature map marking network can be a region proposal network. Each piece of bounding box information in the above-mentioned respective bounding box information can be information used to characterize the bounding box. The above-mentioned bounding box information can include the size information corresponding to the bounding box, the center point coordinates, and the offset information. The above-mentioned size information can be used to characterize the width and height of the bounding box. The above-mentioned center point coordinates can be the coordinates of the center point of the bounding box in the normalized dinner plate feature map. The above-mentioned offset information can be used to characterize the offset of the above-mentioned center point coordinates relative to the center point of the normalized dinner plate feature map. Each piece of bounding box information in the above-mentioned respective bounding box information corresponds to a normalized dinner plate feature image region. The above-mentioned normalized dinner plate feature image region can be the image region in the above-mentioned normalized dinner plate feature image.

[0034] The above classification layer can be a neural network layer that takes as input the respective bounding box information corresponding to the normalized plate feature map output by the above feature map labeling network, and outputs the recognition result corresponding to each bounding box information in the respective bounding box information. For example, the above classification layer can be a convolutional layer. Among them, the above recognition result can be the relevant information of the normalized plate feature image area corresponding to the bounding box information. The above recognition result can include class information and confidence. The above class information can be used to characterize the class of the normalized plate feature image area corresponding to the bounding box information. The above confidence can be used to characterize the accuracy of the above class information. For example, the above recognition result can be "the image area is a plate image, 90%".

[0035] The third layer can be a regression layer. The above regression layer can be a neural network layer that takes as input the respective bounding box information corresponding to the normalized plate feature map output by the above feature map labeling network and the recognition result corresponding to each bounding box information in the respective bounding box information output by the above classification layer, and outputs the respective adjusted bounding box information corresponding to the normalized plate feature map. For example, the above regression layer can be a convolutional layer. Among them, the above adjusted bounding box information can be the information used to characterize the adjusted bounding box. The above adjusted bounding box information can include adjusted size information, adjusted center point coordinates, and adjusted offset information. The above adjusted size information can be used to characterize the width and height of the adjusted bounding box. The above adjusted center point coordinates can be the coordinates of the center point of the adjusted bounding box in the normalized plate feature map. The above offset information can be used to characterize the offset of the above adjusted center point coordinates relative to the center point of the normalized plate feature map. The above adjusted bounding box can be a bounding box obtained by performing operations such as enlarging, shrinking, and translating the bounding box characterized by the above bounding box information. Each of the above adjusted bounding box information corresponds to the class information and confidence included in the recognition result. Each of the above adjusted bounding box information corresponds to a normalized plate feature image area.

[0036] The fourth layer can be a mask generation network. The above-mentioned mask generation network can take the respective adjusted bounding box information corresponding to the normalized dinner plate feature map output by the above-mentioned regression layer as input and output the instance segmentation masks of the image regions of the normalized dinner plate features corresponding to each adjusted bounding box information in each adjusted bounding box information. For example, the above-mentioned mask generation network can be a fully convolutional network. Among them, the above-mentioned instance segmentation mask can be a two-dimensional array corresponding to the above-mentioned normalized dinner plate feature image region. Each element in the above-mentioned two-dimensional array can correspond to a pixel point in the above-mentioned normalized dinner plate feature image region. Each element in the above-mentioned two-dimensional array can be used to represent whether the corresponding pixel point belongs to the pixel point corresponding to the dinner plate image. As an example, when the pixel point belongs to the pixel point corresponding to the dinner plate image, the corresponding element in the above-mentioned two-dimensional array can be "1"; when the pixel point does not belong to the pixel point corresponding to the dinner plate image, the corresponding element in the above-mentioned two-dimensional array can be "0".

[0037] In the fourth step, based on the above-mentioned normalized dinner plate feature map, generate respective bounding box information. In practice, the above-mentioned execution entity can generate respective bounding box information on the above-mentioned normalized dinner plate feature map through the feature map marking network of the feature map processing layer in the above-mentioned instance segmentation model according to the respective preset bounding box size information. Among them, the above-mentioned bounding box size information can be used to represent the size of the bounding box.

[0038] In the fifth step, perform recognition processing on the respective normalized dinner plate feature image regions corresponding to the above-mentioned respective bounding box information to obtain respective recognition results. In practice, the above-mentioned execution entity can perform recognition processing on the respective normalized dinner plate feature image regions corresponding to the above-mentioned respective bounding box information through the classification layer of the feature map processing layer in the above-mentioned instance segmentation model to obtain respective recognition results.

[0039] In the sixth step, based on the above-mentioned respective recognition results, perform adjustment processing on the above-mentioned respective bounding box information to obtain respective adjusted bounding box information. In practice, the above-mentioned execution entity can perform adjustment processing on the above-mentioned respective bounding box information through the regression layer in the above-mentioned instance segmentation model to obtain respective adjusted bounding box information.

[0040] In the seventh step, based on the respective confidence levels corresponding to the above-mentioned respective adjusted bounding box information, determine respective target bounding box information. Among them, the above-mentioned target bounding box information can be the adjusted bounding box information after screening. Each target bounding box information in the above-mentioned respective target bounding box information corresponds to a normalized dinner plate feature image region.

[0041] Step 8: For each of the above target bounding box information, based on the normalized dinner plate feature image region corresponding to the above target bounding box information, generate a corresponding instance segmentation mask. In practice, the above execution entity can generate the instance segmentation mask of the normalized dinner plate feature image region corresponding to the above target bounding box information through the mask generation network in the above instance segmentation model.

[0042] Step 9: Determine each target instance segmentation mask based on the confidences corresponding to the generated instance segmentation masks. Among them, the above target instance segmentation mask can be the instance segmentation mask after screening. In practice, in the first step, the above execution entity can sort the above instance segmentation masks in descending order according to the confidences corresponding to the above instance segmentation masks to obtain an instance segmentation mask sequence. In the second step, the instance segmentation mask with the highest corresponding confidence in the above instance segmentation mask sequence can be added to the instance segmentation mask list. In the third step, each of the above instance segmentation masks except the instance segmentation mask with the highest confidence can be determined as each to-be-processed instance segmentation mask. In the fourth step, the intersection over union (IoU) of the instance segmentation mask with the highest confidence and the above to-be-processed instance segmentation masks can be generated through the non-maximum suppression algorithm as each mask IoU. In the fifth step, each of the above mask IoUs greater than a preset mask IoU threshold can be determined as each target mask IoU. Among them, the above preset mask IoU threshold can be a preset value. Here, the specific setting of the above preset mask IoU threshold is not limited. In the sixth step, the instance segmentation mask with the highest confidence and the above to-be-processed instance segmentation masks corresponding to the above target mask IoUs can be cleared through the non-maximum suppression algorithm to update the instance segmentation mask sequence. In the seventh step, the above first step to the above sixth step are re-executed using the updated instance segmentation mask sequence until the updated instance segmentation mask sequence is empty. In the eighth step, each of the instance segmentation masks in the updated instance segmentation mask list is determined as each target instance segmentation mask.

[0043] Tenth step, based on the above-mentioned respective target instance segmentation masks, generate respective plate instance segmentation images. Among them, the above-mentioned plate instance segmentation images can be plate images obtained after instance segmentation processing. In practice, for each target instance segmentation mask among the above-mentioned respective target instance segmentation masks, first, the above-mentioned execution entity can determine the image area corresponding to the target instance segmentation mask in the above-mentioned plate image as the target image area. Then, for each element among the elements of the above-mentioned target instance segmentation mask, the product of the above-mentioned element and the pixel value corresponding to the above-mentioned element in the above-mentioned target image area can be determined as the pixel value of the plate instance segmentation image. Then, the determined respective pixel values of the plate instance segmentation images can be rendered into a plate instance segmentation image through an image processing function. Among them, the above-mentioned image processing function can be a Pillow library function.

[0044] In some optional implementation manners of some embodiments, the above-mentioned execution entity can determine respective target bounding box information based on respective confidences corresponding to the above-mentioned respective adjusted bounding box information through the following steps:

[0045] First step, based on respective confidences corresponding to the above-mentioned respective adjusted bounding box information, perform a sorting process on the above-mentioned respective adjusted bounding box information to obtain an adjusted bounding box information sequence. In practice, the above-mentioned execution entity can perform a descending order sorting on the above-mentioned respective adjusted bounding box information according to the magnitudes of respective confidences corresponding to the above-mentioned respective adjusted bounding box information to obtain an adjusted bounding box information sequence.

[0046] Second step, based on the adjusted bounding box information sequence, perform the following loop steps:

[0047] First sub-step, determine the adjusted bounding box information that satisfies the preset confidence condition in the adjusted bounding box information sequence as the target adjusted bounding box information. Among them, the above-mentioned preset confidence condition can be that the confidence corresponding to the adjusted bounding box information is the highest.

[0048] Second sub-step, add the above-mentioned target adjusted bounding box information to the target adjusted bounding box information list to update the target adjusted bounding box information list.

[0049] Third sub-step, determine respective adjusted bounding box information that does not satisfy the above-mentioned preset confidence condition in the adjusted bounding box information sequence as respective to-be-processed adjusted bounding box information.

[0050] Fourth sub-step, based on the above-mentioned target adjusted bounding box information and the above-mentioned respective to-be-processed adjusted bounding box information, generate respective intersection over union ratios. In practice, for each to-be-processed adjusted bounding box information among the above-mentioned respective to-be-processed adjusted bounding box information, the above-mentioned execution entity can generate the intersection over union ratio between the to-be-processed adjusted bounding box information and the above-mentioned target adjusted bounding box information through a non-maximum suppression algorithm.

[0051] The fifth sub-step is to determine each intersection over union that meets the preset intersection over union condition among the above-mentioned intersections over union as each target intersection over union. Among them, the above-mentioned preset intersection over union condition can be that the intersection over union is greater than a preset intersection over union threshold. The above-mentioned preset intersection over union threshold can be a preset value. Here, the specific setting of the above-mentioned preset intersection over union threshold is not limited.

[0052] The sixth sub-step is to perform a clearing process on the above-mentioned target adjusted bounding box information and each piece of to-be-processed adjusted bounding box information corresponding to each of the above-mentioned target intersection over unions based on the adjusted bounding box information sequence, so as to update the adjusted bounding box information sequence. In practice, the above-mentioned execution entity can perform a clearing process on the above-mentioned target adjusted bounding box information and each piece of to-be-processed adjusted bounding box information corresponding to each of the above-mentioned target intersection over unions through a non-maximum suppression algorithm, so as to update the adjusted bounding box information sequence.

[0053] The seventh sub-step is to, in response to determining that the updated adjusted bounding box information sequence meets the preset loop condition, use the updated adjusted bounding box information sequence to execute the above-mentioned loop steps again. Among them, the above-mentioned preset loop condition can be that the updated adjusted bounding box information sequence is not empty.

[0054] The eighth sub-step is to, in response to determining that the updated adjusted bounding box information sequence does not meet the above-mentioned preset loop condition, determine each piece of target adjusted bounding box information in the updated target adjusted bounding box information list as each piece of target bounding box information.

[0055] Step 103: For each plate instance segmentation image in each plate instance segmentation image, perform the following steps:

[0056] Step 1031: Input the plate instance segmentation image into the feature extraction network in a pre-trained plate recognition model to perform feature extraction processing on the plate instance segmentation image, so as to obtain a plate feature map corresponding to the plate instance segmentation image.

[0057] In some embodiments, the above-mentioned execution entity can input the above-mentioned plate instance segmentation image into the feature extraction network in a pre-trained plate recognition model to perform feature extraction processing on the above-mentioned plate instance segmentation image, so as to obtain a plate feature map corresponding to the above-mentioned plate instance segmentation image. Among them, the above-mentioned plate feature map can be a feature map corresponding to the above-mentioned plate instance segmentation image. The above-mentioned plate recognition model can be a neural network model that takes a plate instance segmentation image as input and outputs the plate category information corresponding to the plate instance segmentation image. For example, the above-mentioned neural network model can be a convolutional neural network. The above-mentioned plate category information can be information used to characterize the color and shape of the plate. The above-mentioned plate recognition model can include three layers.

[0058] The first layer can be a feature extraction network. The above-mentioned feature extraction network can be a neural network that takes the dish instance segmentation image as input and outputs the corresponding dish feature map of the dish instance segmentation image. For example, the above-mentioned feature extraction network can be a convolutional neural network. Among them, the above-mentioned dish feature map can be the feature map corresponding to the above-mentioned dish instance segmentation image.

[0059] The second layer can be a feature processing network. The above-mentioned feature processing network can be a neural network that takes the dish feature map corresponding to the dish instance segmentation image as input and outputs the dish full-image feature information corresponding to the dish instance segmentation image. For example, the above-mentioned feature processing network can be a graph convolutional neural network. Among them, the above-mentioned dish full-image feature information can be a feature vector that synthesizes all features in the above-mentioned dish image.

[0060] The third layer can be a classification processing network. The above-mentioned classification processing network can include a feature vector processing layer and an output layer. The feature vector processing layer can be a fully connected layer that takes the dish full-image feature information corresponding to the dish instance segmentation image as input and outputs the globally updated feature vector corresponding to the dish instance segmentation image. Among them, the above-mentioned globally updated feature vector can be the dish full-image feature information processed by the fully connected layer.

[0061] The above-mentioned output layer can include a probability distribution function and an output function. The above-mentioned probability distribution function can be a probability distribution function that takes the globally updated feature vector output by the above-mentioned feature vector processing layer as input and outputs the dish probability distribution data corresponding to the dish instance segmentation image. Among them, the above-mentioned dish probability distribution data can be the probability distribution of the dish category information corresponding to the dish. As an example, the above-mentioned dish probability distribution data can be "dish category information: square - white, dish probability data: 90%; dish category information: oval - white, dish probability data: 10%". The above-mentioned dish probability distribution data can include each dish probability data. The above-mentioned dish probability data can be the probability that the dish belongs to the dish category information. Each dish probability data in the above-mentioned each dish probability data corresponds to the dish category information. The above-mentioned probability distribution function can be a normalized exponential function. The above-mentioned output function can be an argmax function that takes the above-mentioned dish probability distribution data as input and outputs the dish category information with the largest corresponding dish probability data in the above-mentioned dish probability distribution data.

[0062] In some optional implementation manners of some embodiments, the above-mentioned dish recognition model can be obtained by the above-mentioned execution subject through the following method:

[0063] First step, obtain a sample set. Among them, each sample in the above-mentioned sample set includes a dish instance segmentation image and the corresponding sample dish category information of the above-mentioned dish instance segmentation image. The above-mentioned sample dish category information can be used to represent the true category of the above-mentioned dish instance segmentation image.

[0064] In the second step, based on the sample set, perform the following training steps:

[0065] In the first sub-step, input the dish instance segmentation images of at least one sample in the sample set into the initial neural network respectively, and obtain the dish category information corresponding to each of the above at least one sample. Among them, the structure of the initial neural network can refer to the above-mentioned dish recognition model, which will not be elaborated here. The above dish category information can be the category corresponding to the above dish instance segmentation image.

[0066] In the second sub-step, compare the dish category information corresponding to each of the above at least one sample with the corresponding sample dish category information. In practice, first, the above execution entity can determine the dish probability distribution data corresponding to each of the above at least one sample as the target dish probability distribution data. Then, the above execution entity can generate the probability distribution of the sample dish category information corresponding to each of the above at least one sample through the softmax function. Finally, the above execution entity can generate the loss value between the target dish probability distribution data corresponding to each of the above at least one sample and the probability distribution of the corresponding sample dish category information through the cross-entropy loss function.

[0067] In the third sub-step, determine whether the initial neural network reaches the preset optimization goal according to the comparison result. As an example, when the loss value between the dish category information corresponding to a sample and the corresponding sample dish category information is less than the preset loss threshold, the above sample is determined as a positive sample. Among them, the above positive sample can be used to represent the sample that meets the loss value condition. The above loss value condition can be that the loss value between the dish category information corresponding to a sample and the corresponding sample dish category information is less than the preset loss threshold. The above optimization goal can be that the proportion of positive samples among all samples is greater than the preset accuracy threshold. The above loss threshold can be a preset value. Here, the specific setting of the above loss threshold is not limited. The above accuracy threshold can be a preset value. Here, the specific setting of the above accuracy threshold is not limited.

[0068] In the fourth sub-step, in response to determining that the initial neural network reaches the above optimization goal, determine the initial neural network as the dish recognition model.

[0069] The fifth sub-step, in response to determining that the initial neural network has not achieved the above optimization goal, adjust the network parameters of the initial neural network, and use the unused samples to form a sample set. Use the adjusted initial neural network as the initial neural network and execute the above training steps again. In practice, the above execution subject can use the BackPropagation Algorithm (BP algorithm) and the gradient descent method (such as the mini-batch gradient descent algorithm) to adjust the network parameters of the above initial neural network.

[0070] Step 1032, input the plate feature map into the feature processing network in the plate recognition model to generate the plate full-map feature information corresponding to the plate feature map.

[0071] In some embodiments, the above execution subject can input the above plate feature map into the feature processing network in the above plate recognition model to generate the plate full-map feature information corresponding to the above plate feature map. Among them, the above plate full-map feature information can be a feature vector containing all the information in the above plate instance segmentation image.

[0072] In the process of adopting technical solutions to solve the above technical problems, the following problems often arise:

[0073] When the shapes or colors of plates of different categories are similar, the accuracy of the plate image classification model in recognizing plate images is relatively low. This results in relatively low reliability of the recognized plate category information, leading to the need to redeploy computing resources to recognize the same plate multiple times, causing waste of computing resources.

[0074] Facing the above technical problems, the following solutions are decided to be adopted:

[0075] In some alternative implementation manners of some embodiments, the above execution subject can input the above plate feature map into the feature processing network in the above plate recognition model through the following steps to generate the plate full-map feature information corresponding to the above plate feature map:

[0076] Step 1: Based on the above-mentioned dinner plate feature map, construct a dinner plate feature connection map corresponding to the above-mentioned dinner plate feature map. Among them, the above-mentioned dinner plate feature connection map can be an unweighted undirected graph used to represent the connection relationship between each pixel point in the above-mentioned dinner plate feature map. Each pixel point in the above-mentioned dinner plate feature map corresponds to a feature vector. Each node in the above-mentioned dinner plate feature connection map corresponds to node feature information. The above-mentioned node feature information can be the feature vector corresponding to the node. In practice, first, the above-mentioned execution entity can determine each pixel point in the above-mentioned dinner plate feature map as each node. Then, for each node among the above-mentioned nodes, a dinner plate feature connection map can be obtained by connecting the above-mentioned node with each neighbor node corresponding to the above-mentioned node. In practice, for each pixel point among each pixel point in the above-mentioned dinner plate feature map, first, the above-mentioned execution entity can determine the node corresponding to the above-mentioned pixel point in the above-mentioned dinner plate feature connection map as the target node. Then, the feature vector corresponding to the above-mentioned pixel point can be determined as the node feature information corresponding to the above-mentioned target node.

[0077] Step 2: For each node feature information included in the above-mentioned dinner plate feature connection map, perform the following steps:

[0078] The first sub-step: Based on the above-mentioned node feature information, determine each neighbor node feature information corresponding to the above-mentioned node feature information. Among them, the above-mentioned neighbor node feature information can be the node feature information of the neighbor node corresponding to the above-mentioned node feature information.

[0079] The second sub-step: For each neighbor node feature information among the above-mentioned neighbor node feature information, perform the following steps:

[0080] Sub-step 1: Based on the above-mentioned neighbor node feature information, determine the attention coefficient between the above-mentioned node feature information and the above-mentioned neighbor node feature information. In practice, first, the above-mentioned execution entity can determine the product of the above-mentioned node feature information and a preset weight matrix as the linear node feature information. Then, the product of the above-mentioned neighbor node feature information and the above-mentioned weight matrix can be determined as the linear neighbor node feature information. Then, the linear node feature information and the linear neighbor node feature information can be concatenated into linear feature information through a concatenation function. Then, the transpose of a preset weight vector can be determined as the transposed weight vector. Finally, the product of the above-mentioned linear feature information and the above-mentioned transposed weight vector can be determined as the attention coefficient. Among them, the above-mentioned weight matrix can be the network parameter used to connect each node feature information in the above-mentioned feature processing network. The above-mentioned weight vector can be the network parameter used to control the threshold of the activation function in the above-mentioned feature processing network. The above-mentioned concatenation function can be the concat function.

[0081] Sub-step 2: Determine the weighted neighbor node feature information as the product of the above attention coefficient and the above neighbor node feature information.

[0082] The third sub-step: Determine the sum of the determined weighted neighbor node feature information as the aggregated neighbor node feature information.

[0083] The fourth sub-step: Integrate and process the above node feature information and the above aggregated neighbor node feature information to obtain the spliced node feature information. Among them, the above spliced node feature information can be the feature vector obtained after splicing the above node feature information and the above aggregated neighbor node feature information. In practice, the above execution entity can splice the above node feature information and the above aggregated neighbor node feature information into the spliced node feature information through a splicing function. Among them, the above splicing function can be the concat function.

[0084] The fifth sub-step: Determine the first-level feature information as the product of the above spliced node feature information and the first-level parameter. Among them, the above first-level parameter can be the weight matrix in the above feature processing network.

[0085] The sixth sub-step: Determine the second-level feature information as the sum of the above first-level feature information and the second-level parameter. Among them, the above second-level parameter can be the bias vector in the above feature processing network.

[0086] The seventh sub-step: Generate node feature information based on the above second-level feature information to update the above node feature information. In practice, the above second-level feature information can be input into an activation function to obtain node feature information. Among them, the above activation function can be the rectified linear unit function.

[0087] The third step: In response to determining that each node feature information included in the above plate feature connection graph meets the preset update condition, based on the updated each node feature information, determine the plate full-image feature information corresponding to the above plate feature graph. Among them, the above preset update condition can be that each node feature information in the above plate feature connection graph has been updated. The above plate full-image feature information can be the feature vector that synthesizes all the feature information in the above plate image. In practice, the above execution entity can determine the plate full-image feature information corresponding to the above plate feature graph through the above feature processing network.

[0088] The fourth step: Determine the corresponding memory usage information based on the above plate full-image feature information. Among them, the above memory usage information can be the size of the memory space used by the above plate full-image feature information. In practice, the above execution entity can obtain the memory usage information corresponding to the above plate full-image feature information through the Instrumentation interface.

[0089] Step 5: In response to determining that the above memory usage information is less than the preset allocated memory amount, reduce the physical memory corresponding to the above full-dish map feature information to adjust the storage resource allocation of the above full-dish map feature information. The above preset allocated memory amount can be a memory value preset for storing the above full-dish map feature information. In practice, first, the above execution entity can determine the difference between the above memory usage information and the above preset allocated memory amount as the allocation amount. Then, the memory in the idle state with a storage size of the above allocation amount can be allocated to the above full-dish map feature information through the malloc function to adjust the storage resource allocation of the above full-dish map feature information.

[0090] Step 6: In response to determining that the above memory usage information is greater than the above preset allocated memory amount, increase the physical memory corresponding to the above full-dish map feature information to adjust the storage resource allocation of the above full-dish map feature information. In practice, first, the above execution entity can determine the difference between the above memory usage information and the above preset allocated memory amount as the release amount. Then, the memory in the idle state with a storage size of the above release amount, which is allocated to the above full-dish map feature information, can be allocated to other processes for use through the malloc function to adjust the storage resource allocation of the above full-dish map feature information.

[0091] The above technical solution and its related content, as an inventive point of the embodiments of the present disclosure, solve the problem of "waste of computing resources". The factors that lead to low efficiency of data processing and waste of network resources during marking are often as follows: When the shapes or colors of dinner plates of different categories are similar, the accuracy of the dinner plate image classification model in recognizing dinner plate images is low. This leads to low reliability of the recognized dinner plate category information, resulting in the need to redeploy computing resources to recognize the same dinner plate multiple times, causing waste of computing resources. If the above factors are solved, the waste of computing resources can be reduced. To achieve this effect, the present disclosure first constructs a dinner plate feature connection graph corresponding to the above dinner plate feature map based on the above dinner plate feature map, where each node feature information is included in the above dinner plate feature connection graph. Thus, a feature map of the dinner plate can be obtained. Then, for each node feature information among the various node feature information included in the above dinner plate feature connection graph, the following steps are performed: First, based on the above node feature information, the respective neighbor node feature information corresponding to the above node feature information is determined. Second, for each neighbor node feature information among the above respective neighbor node feature information, the following steps are performed: First, based on the above neighbor node feature information, the attention coefficient between the above node feature information and the above neighbor node feature information is determined. Thus, the attention coefficient between the node feature information and the neighbor node feature information can be obtained. Then, the product of the above attention coefficient and the above neighbor node feature information is determined as the weighted neighbor node feature information. Then, the sum of the determined respective weighted neighbor node feature information is determined as the aggregated neighbor node feature information. Then, the above node feature information and the above aggregated neighbor node feature information are integrally processed to obtain the spliced node feature information. Then, the product of the above spliced node feature information and the first-level parameter is determined as the first-level feature information. Then, the sum of the above first-level feature information and the second-level parameter is determined as the second-level feature information. Then, based on the above second-level feature information, node feature information is generated to update the above node feature information. Thus, through iterative convolution of the respective neighbor node feature information corresponding to the above node feature information, the above node feature information can include the features of the respective neighbor node feature information. Then, in response to determining that each node feature information included in the above dinner plate feature connection graph meets the preset update condition, based on the updated respective node feature information, the dinner plate full map feature information corresponding to the above dinner plate feature map is determined. Thus, a feature vector of the entire dinner plate image can be obtained. Then, based on the above dinner plate full map feature information, the corresponding memory usage information is determined. Thus, the size of the memory space occupied by the dinner plate full map feature information can be obtained. Then, in response to determining that the above memory usage information is less than the preset allocated memory amount, the physical memory corresponding to the above dinner plate full map feature information is reduced to adjust the storage resource allocation of the above dinner plate full map feature information. Thus, the waste of storage resources can be reduced.Finally, in response to determining that the above-mentioned memory usage information is greater than the above-mentioned preset allocated memory amount, increase the physical memory corresponding to the above-mentioned full-dish map feature information to adjust the storage resource allocation of the above-mentioned full-dish map feature information. Thereby, the situation of data loss caused by insufficient storage space can be reduced. Also, when processing the dish image, the feature map of the dish image can be obtained first, and then the features of the dish image can be further processed according to the feature map to obtain the global feature vector of the dish image. After further processing the global feature vector through the fully connected layer, the category corresponding to the dish image is generated according to the feature vector output by the fully connected layer. Therefore, the local features of the dish can be better extracted, and the dishes with similar shapes and colors can be better distinguished through the local features. Furthermore, the accuracy of the generated dish category information can be improved, the number of times of calling computing resources to identify the same dish can be reduced, and thus the waste of computing resources can be reduced.

[0092] Step 1033: Input the full-dish map feature information into the classification processing network in the dish recognition model to obtain the dish category information corresponding to the dish instance segmentation image.

[0093] In some embodiments, the above-mentioned execution subject may input the above-mentioned full-dish map feature information into the classification processing network in the above-mentioned dish recognition model to obtain the dish category information corresponding to the above-mentioned dish instance segmentation image.

[0094] In some optional implementation manners of some embodiments, the above-mentioned execution subject may input the above-mentioned full-dish map feature information into the classification processing network in the above-mentioned dish recognition model through the following steps to obtain the dish category information corresponding to the above-mentioned dish instance segmentation image:

[0095] First step: Generate the dish probability distribution data corresponding to the above-mentioned full-dish map feature information based on the above-mentioned full-dish map feature information. In practice, the above-mentioned execution subject may generate the dish probability distribution data corresponding to the above-mentioned full-dish map feature information through the probability distribution function in the feature vector processing layer and the output layer in the above-mentioned classification processing network.

[0096] Second step: Determine each dish probability data included in the above-mentioned dish probability distribution data as each dish category probability data.

[0097] Third step: Determine the dish category probability data that meets the preset data condition among the above-mentioned each dish category probability data as the target probability data. Among them, the above-mentioned preset data condition may be that the dish category probability data is the largest.

[0098] Fourth step: Determine the dish category information corresponding to the above-mentioned target probability data as the dish category information corresponding to the above-mentioned dish instance segmentation image.

[0099] Step 104: Based on the obtained plate category information and the preset category value information group, determine the plate value information corresponding to each plate instance segmentation image.

[0100] In some embodiments, the above-mentioned execution entity may determine the plate value information corresponding to each of the above-mentioned plate instance segmentation images based on the obtained plate category information and the preset category value information group. Among them, the category value information in the above-mentioned category value information group can be used to represent the value corresponding to plates of each category. Each category value information in the above-mentioned category value information group may include plate category information and the value of the above-mentioned plate category information. The above-mentioned plate value information can be used to represent the value of the plate corresponding to the plate instance segmentation image. In practice, for each plate category information in the above-mentioned plate category information, first, the above-mentioned execution entity may search in the above-mentioned category value information group through a search function to find the plate category information that is the same as the above-mentioned plate category information as the target plate category information. Then, determine the value corresponding to the above-mentioned target plate category information as the target value. Then, determine the above-mentioned target value as the plate value information of the plate instance segmentation image corresponding to the above-mentioned plate category information. For example, the above-mentioned search function may be the SEARCH function. As an example: when the category value information group is "square - white: 3 yuan; round - white: 2 yuan" and the plate category information corresponding to the plate instance segmentation image is "square - white", the plate value information corresponding to the plate instance segmentation image is "3 yuan".

[0101] Step 105: Generate plate information based on the plate category information and the plate value information.

[0102] In some embodiments, the above-mentioned execution entity may generate plate information based on the above-mentioned plate category information and the above-mentioned plate value information. Among them, the above-mentioned plate information may be the relevant information corresponding to the plate. The above-mentioned plate information may include the total plate value information. The above-mentioned total plate value information can be used to represent the sum of the above-mentioned plate value information. In practice, the above-mentioned execution entity may combine the above-mentioned plate category information, the above-mentioned plate value information, and the above-mentioned total plate value information into plate information.

[0103] Step 106: Display the plate information.

[0104] In some embodiments, the above-mentioned execution entity may display the above-mentioned plate information.

[0105] For example, Figure 4 is a schematic screenshot of an internal test operation page for displaying plate information according to the plate information display method of the present disclosure.

[0106] Figure 4"November 4, 2024 17:06:59" in it can be the current time. "Dinner cashiering" can be the name corresponding to the task currently executed by the above-mentioned execution entity. "Clearing the machine" can be a control for clearing data of the above-mentioned execution entity. "Order" can be a control for displaying all orders. "Dinner plate" can be a control for displaying the above-mentioned category value information group. The setting control can be a control for processing such as zooming in and out of the page. The home page control can be a control for exiting the current page and returning to the home page. Figure 4 The "dinner plate recognition information" on the left can be the dinner plate category information, dinner plate value information, and confidence level corresponding to each dinner plate. Among them, the confidence level can be used to represent the accuracy of the above-mentioned dinner plate category information. Take Figure 4 "Ellipse - white 1 yuan [0.90]" in it as an example. Among them, "ellipse - white" is the dinner plate category information. "1 yuan" is the dinner plate value information. "[0.90]" is the confidence level. "A total of 3 copies, totaling 6 yuan" can be information used to represent the quantity of each recognized dinner plate and the total value of each dinner plate. "Server" can be used to display whether the above-mentioned execution entity is connected to the server. "Payment service" can be the name of the task executed by the above-mentioned execution entity. Order information can be the relevant information of all completed orders up to now. The "current order number" in the order information can be the quantity of all completed orders up to now. The "total amount" in the order information can be the sum of the values of all completed orders. Figure 4 On the right, the "total amount" can be the total value of each dinner plate on the left. The "recognized amount" can be the sum of each value information corresponding to each dinner plate when recognizing each dinner plate. The "priced amount" can be the value manually input by the user through the operation keyboard in the "data selection" module. The "preferential amount" can be the value that can be discounted. "Data selection" can be a module for correcting the "recognized amount". The operation keyboard in the "data selection" can be each control for modifying the "recognized amount" and the "preferential amount". "No discount" can be used to display each dinner plate that is not discounted currently. "Circle - white" can be used to display the information corresponding to the dinner plate with a circular shape and white color. "Square - white" can be used to display the information corresponding to the dinner plate with a square shape and white color. The keyboard control can be a control for calling out the keyboard. "Re - recognition" can be used to Figure 4Controls for re-identifying each dinner plate on the left side. "Combined payment" can be a control for combining two or more orders for a single payment. "Automatic payment" can be a control for automatically completing payment through means such as mobile payment. "Cash payment" can be a control for completing payment through a paper-based value (e.g., cash). "Cooperative payment" can be a control for jointly completing payment through the above "Automatic payment" and the above "Cash payment". "Confirm payment" can be a control for confirming payment. "Suspend order" can be a control for saving an order for subsequent continued processing. "Cancel payment" can be a control for canceling payment.

[0107] Optionally, after step 103, the above execution entity can also perform the following steps:

[0108] First step, store the above various dinner plate category information in a preset database. Among them, the above preset database can be a database for storing dinner plate category information.

[0109] Second step, obtain each dinner plate category information within a preset time range as each dinner plate classification information. Among them, the above preset time range can be within a preset time from the current time. The above preset time can be one year. In practice, the above execution entity can obtain each dinner plate category information within the above preset time range from the above preset database.

[0110] Third step, perform classification processing on the above various dinner plate classification information to obtain a set of dinner plate classification information groups. Among them, each dinner plate classification information group in the above set of dinner plate classification information groups corresponds to the same dinner plate classification information. In practice, each dinner plate classification information with the same category name among the above various dinner plate classification information can be determined as a dinner plate classification information group to obtain a set of dinner plate classification information groups.

[0111] Fourth step, for each dinner plate classification information group in the above set of dinner plate classification information groups, perform the following steps:

[0112] First sub-step, generate information on items to be transported corresponding to the above dinner plate classification information group based on the above dinner plate classification information group and preset dinner plate item information. Among them, the above dinner plate item information can be information related to the food placed in dinner plates of various categories. The above dinner plate item information can include the item name and the item weight. The above information on items to be transported can be used to represent the item name and the total item weight corresponding to the items to be transported.

[0113] The second sub-step is to generate the transportation position information of the plate items based on the above plate classification information group and the preset plate placement information. Among them, the above plate placement information can be used to represent the positions where different types of plates should be placed. The above plate placement information may include each plate classification information and the position corresponding to each plate classification information in the above each plate classification information. The above transportation position information of the plate items can be used to represent the position to which the items corresponding to the above item transportation information to be transported should be transported. In practice, for each plate classification information in the above plate classification information group, first, the above execution subject can find the plate classification information identical to the above plate classification information in the above plate placement information as the target plate classification information through a search function. Then, the position corresponding to the above target plate classification information can be determined as the transportation position of the plate items. Finally, the determined transportation positions of each plate item can be combined into the transportation position information of the plate items. For example, the above search function can be the SEARCH function.

[0114] The third sub-step is to control the corresponding item transportation device to transport the items corresponding to the above item transportation information to the target position corresponding to the above transportation position information of the plate items based on the above plate classification information group. Among them, the above item transportation device can be a device for transporting items. For example, the above item transportation device can be an autonomous vehicle. The above target position can be the position represented by the above transportation position information of the plate items.

[0115] The above-mentioned various embodiments of the present disclosure have the following beneficial effects: Through the method for displaying plate information of the present disclosure, the accuracy of identifying plate images can be improved, and waste of computing resources can be reduced. Specifically, the reasons for the low accuracy of identifying plate images and waste of computing resources are as follows: When using a plate image classification model to extract features from a plate image, it is easily interfered by the background image in the plate image, which easily leads to errors in the extracted plate features. As a result, the accuracy of the plate image classification model in classifying and identifying the plate image is relatively low, resulting in the need to redeploy computing resources to extract features and classify and identify the same plate image again, causing waste of computing resources. Based on this, in the method for displaying plate information according to some embodiments of the present disclosure, first, plate images corresponding to each plate are obtained. Thus, the original data to be processed can be obtained. Secondly, instance segmentation processing is performed on the above-mentioned plate images to obtain each plate instance segmentation image. Thus, the interference of other factors on the plate image can be reduced, and the accuracy rate of subsequent classification and identification of the plate image can be improved. Then, for each plate instance segmentation image in the above-mentioned various plate instance segmentation images, the following steps are performed: First, the above-mentioned plate instance segmentation image is input into the feature extraction network in a pre-trained plate recognition model to perform feature extraction processing on the above-mentioned plate instance segmentation image, and a plate feature map corresponding to the above-mentioned plate instance segmentation image is obtained. Thus, a feature map corresponding to the plate image can be obtained. Secondly, the above-mentioned plate feature map is input into the feature processing network in the above-mentioned plate recognition model to generate plate full-image feature information corresponding to the above-mentioned plate feature map. Thus, the global feature information of the plate image can be obtained. Then, the above-mentioned plate full-image feature information is input into the classification processing network in the above-mentioned plate recognition model to obtain plate category information corresponding to the above-mentioned plate instance segmentation image. Thus, the category corresponding to the plate image can be obtained. Then, based on the obtained various plate category information and a preset category value information group, the respective plate value information corresponding to the above-mentioned various plate instance segmentation images is determined. Thus, the value information corresponding to each plate can be determined through the category corresponding to the plate image. Then, based on the above-mentioned various plate category information and the above-mentioned various plate value information, plate information is generated, wherein the above-mentioned plate information includes plate total value information. Thus, the total value information of each plate can be obtained. Finally, the above-mentioned plate information is displayed. Thus, the total value information of the plate can be displayed. Also, because instance segmentation processing can be performed on the plate image before classification and identification, the interference of the background image in the plate image on the plate image can be reduced, and thus the accuracy rate of classifying and identifying the plate image can be improved.Also, since the global feature information of the dinner plate image can be integrated and then the dinner plate image can be classified and recognized according to the features of the dinner plate image, the accuracy of classifying and recognizing the dinner plate image can be improved, the number of times of calling computing resources to process the same dinner plate image can be reduced, and thus the waste of computing resources can be reduced.

[0116] Further referring to Figure 2 , as an implementation of the methods shown in the above figures, the present disclosure provides some embodiments of a dinner plate information display device. These device embodiments correspond to Figure 1 the method embodiments shown, and the device can be specifically applied to various electronic devices.

[0117] As Figure 2 shown, some embodiments of the dinner plate information display device 200 include: an acquisition unit 201, a segmentation unit 202, an execution unit 203, a determination unit 204, a generation unit 205, and a display unit 206. Among them, the acquisition unit 201 is configured to acquire dinner plate images corresponding to each dinner plate; the segmentation unit 202 is configured to perform instance segmentation processing on the above dinner plate images to obtain each dinner plate instance segmentation image; the execution unit 203 is configured to perform the following steps for each dinner plate instance segmentation image in the above each dinner plate instance segmentation image: input the above dinner plate instance segmentation image into the feature extraction network in a pre-trained dinner plate recognition model to perform feature extraction processing on the above dinner plate instance segmentation image to obtain a dinner plate feature map corresponding to the above dinner plate instance segmentation image; input the above dinner plate feature map into the feature processing network in the above dinner plate recognition model to generate dinner plate full map feature information corresponding to the above dinner plate feature map; input the above dinner plate full map feature information into the classification processing network in the above dinner plate recognition model to obtain dinner plate category information corresponding to the above dinner plate instance segmentation image; the determination unit 204 is configured to determine each dinner plate value information corresponding to the above each dinner plate instance segmentation image based on the obtained each dinner plate category information and a preset category value information group; the generation unit 205 is configured to generate dinner plate information based on the above each dinner plate category information and the above each dinner plate value information, where the above dinner plate information includes dinner plate total value information; the display unit 206 is configured to display the above dinner plate information.

[0118] It can be understood that the units described in the device 200 correspond to the respective steps in the method described with reference to Figure 1 . Thus, the operations, features, and beneficial effects described above for the method also apply to the device 200 and the units included therein, and will not be repeated here.

[0119] Next, referring to Figure 3, which shows a schematic structural diagram of an electronic device (such as a computing device) 300 suitable for implementing some embodiments of the present disclosure. Figure 3 The illustrated electronic device is merely an example and should not impose any limitations on the functions and usage scope of the embodiments of the present disclosure.

[0120] As Figure 3 shown, the electronic device 300 may include a processing device (such as a central processing unit, a graphics processing unit, etc.) 301, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 302 or the program loaded from the storage device 308 into the random access memory (RAM) 303. In the RAM 303, various programs and data required for the operation of the electronic device 300 are also stored. The processing device 301, the ROM 302, and the RAM 303 are connected to each other through a bus 304. The input / output (I / O) interface 305 is also connected to the bus 304.

[0121] Generally, the following devices may be connected to the I / O interface 305: an input device 306 including, for example, a touch screen, a touchpad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 307 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 308 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 309. The communication device 309 can allow the electronic device 300 to communicate with other devices wirelessly or wiredly to exchange data. Although Figure 3 the electronic device 300 with various devices is shown, it should be understood that it is not required to implement or have all the shown devices. More or fewer devices may be implemented or had alternatively. Figure 3 Each block shown in

[0122] Specifically, according to some embodiments of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, some embodiments of the present disclosure include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program contains program codes for executing the method shown in the flowchart. In such some embodiments, the computer program can be downloaded and installed from the network through the communication device 309, or installed from the storage device 308, or installed from the ROM 302. When the computer program is executed by the processing device 301, the above functions defined in the method of some embodiments of the present disclosure are executed.

[0123] It should be noted that the computer-readable media described in some embodiments of the present disclosure may be a computer-readable signal medium, a computer-readable storage medium, or any combination of the two. A computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the above. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In some embodiments of the present disclosure, a computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In some embodiments of the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries computer-readable program code. Such a propagated data signal may take many forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on a computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0124] In some embodiments, the client and the server can communicate using any currently known or future-developed network protocol such as HTTP (HyperText Transfer Protocol), and can be interconnected with digital data communication in any form or medium (e.g., a communication network). Examples of communication networks include local area networks ("LANs"), wide area networks ("WANs"), the Internet (e.g., the Internet), and end-to-end networks (e.g., ad hoc end-to-end networks), as well as any currently known or future-developed network.

[0125] The above computer-readable medium may be included in the above electronic device; or it may exist separately and not be assembled into the electronic device. The above computer-readable medium carries one or more programs. When the above one or more programs are executed by the electronic device, the electronic device is caused to: obtain plate images corresponding to each plate; perform instance segmentation processing on the above plate images to obtain instance segmentation images of each plate; for each instance segmentation image of the above instance segmentation images of each plate, perform the following steps: input the above instance segmentation image of the plate into the feature extraction network in the pre-trained plate recognition model to perform feature extraction processing on the above instance segmentation image of the plate to obtain a plate feature map corresponding to the above instance segmentation image of the plate; input the above plate feature map into the feature processing network in the above plate recognition model to generate plate full-image feature information corresponding to the above plate feature map; input the above plate full-image feature information into the classification processing network in the above plate recognition model to obtain plate category information corresponding to the above instance segmentation image of the plate; determine the respective plate value information corresponding to the above instance segmentation images of each plate based on the obtained respective plate category information and a preset group of category value information; generate plate information based on the above respective plate category information and the above respective plate value information, where the above plate information includes total plate value information; and display the above plate information.

[0126] Computer program code for performing the operations of some embodiments of the present disclosure may be written in one or more programming languages or combinations thereof. The above programming languages include object-oriented programming languages such as Java, Smalltalk, C++, and also include conventional procedural programming languages such as the "C" language or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, executed as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (for example, by using an Internet service provider to connect through the Internet).

[0127] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architectures, functions, and operations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagram may represent a module, a segment of a program, or a part of code that contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than that marked in the accompanying drawings. For example, two consecutive blocks shown may actually be executed substantially in parallel, and they may sometimes be executed in the reverse order, depending on the functions involved. It should also be noted that each block in the block diagram and / or flowchart, as well as combinations of blocks in the block diagram and / or flowchart, can be implemented by a dedicated hardware-based system that performs the specified functions or operations, or can be implemented by a combination of dedicated hardware and computer instructions.

[0128] The units described in some embodiments of the present disclosure can be implemented in software or in hardware. The described units can also be provided in a processor. For example, it can be described as: a processor includes an acquisition unit, a segmentation unit, an execution unit, a determination unit, a generation unit, and a display unit. Among them, the names of these units do not constitute a limitation on the unit itself in some cases. For example, the acquisition unit can also be described as "the unit that acquires the plate images corresponding to each plate".

[0129] The functions described above can be at least partially performed by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that can be used include: Field Programmable Gate Array (FPGA), Application Specific Integrated Circuit (ASIC), Application Specific Standard Product (ASSP), System on Chip (SOC), Complex Programmable Logic Device (CPLD), and so on.

[0130] The above description is only some preferred embodiments of the present disclosure and an explanation of the applied technical principles. Those skilled in the art should understand that the scope of the invention involved in the embodiments of the present disclosure is not limited to the technical solutions formed by the specific combination of the above technical features, and should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above inventive concept. For example, the technical solutions formed by mutually replacing the above features with the (but not limited to) technical features having similar functions disclosed in the embodiments of the present disclosure.

Claims

1. A method for displaying plate information, comprising: Get the plate image corresponding to each plate; Performing instance segmentation processing on the dinner plate image to obtain each dinner plate instance segmentation image; For each of the dinner plate instance segmentation images, the following steps are performed: Inputting the meal plate instance segmentation image into a feature extraction network in a pre-trained meal plate recognition model to perform feature extraction processing on the meal plate instance segmentation image to obtain a meal plate feature map corresponding to the meal plate instance segmentation image; Inputting the plate feature image into a feature processing network in the plate recognition model to generate full plate feature information corresponding to the plate feature image, wherein inputting the plate feature image into a feature processing network in the plate recognition model to generate full plate feature information corresponding to the plate feature image comprises: Based on the plate feature graph, construct a plate feature connection graph corresponding to the plate feature graph; For each node feature information in each node feature information included in the dinner plate feature connection graph, the following steps are performed: Based on the node characteristic information, determining each neighbor node characteristic information corresponding to the node characteristic information; For each piece of neighbor node feature information in the respective neighbor node feature information, the following steps are performed: Based on the neighbor node feature information, determining the attention coefficient of the node feature information and the neighbor node feature information; Determine the product of the attention coefficient and the neighbor node feature information as weighted neighbor node feature information; Determining the sum of each determined weighted neighbor node feature information as aggregated neighbor node feature information; Integrate the node feature information and the aggregated neighbor node feature information to obtain concatenated node feature information; Determine the product of the splicing node feature information and the first-level parameter as the first-level feature information; Determining the sum of the first-level feature information and the second-level parameter as the second-level feature information; Based on the second-level feature information, generate node feature information to update the node feature information; In response to determining that each node feature information included in the plate feature connection graph meets a preset update condition, based on the updated each node feature information, determine the plate full graph feature information corresponding to the plate feature graph; Based on the feature information of the full image of the dinner plate, determining corresponding memory usage information; In response to determining that the memory usage information is less than a preset allocated memory amount, reducing the physical memory corresponding to the full plate image feature information to adjust the storage resource allocation of the full plate image feature information; In response to determining that the memory usage information is greater than the preset allocated memory amount, increasing the physical memory corresponding to the full plate image feature information to adjust the storage resource allocation of the full plate image feature information; Inputting the feature information of the whole plate image into the classification processing network in the plate recognition model to obtain the plate category information corresponding to the plate instance segmentation image; Based on the obtained category information of each plate and the preset category value information group, determining the value information of each plate corresponding to each plate instance segmented image; Generate meal plate information based on the meal plate category information and the meal plate value information, wherein the meal plate information includes meal plate total value information; The meal plate information is displayed.

2. The method according to claim 1, wherein: The plate recognition model is trained in the following way: Acquire a sample set, wherein each sample in the sample set includes a meal plate instance segmentation image and sample meal plate category information corresponding to the meal plate instance segmentation image; Based on the sample set, the following training steps are performed: Inputting the plate instance segmentation images of at least one sample in the sample set into the initial neural network respectively to obtain the plate category information corresponding to each sample in the at least one sample, wherein the initial neural network includes a feature extraction network, a feature processing network and a classification processing network; comparing the plate category information corresponding to each sample of the at least one sample with the corresponding sample plate category information; Determine whether the initial neural network reaches the preset optimization goal according to the comparison result; In response to determining that the initial neural network achieves the optimization goal, determining the initial neural network as a plate recognition model; In response to determining that the initial neural network does not achieve the optimization goal, the network parameters of the initial neural network are adjusted, and unused samples are used to form a sample set, and the adjusted initial neural network is used as the initial neural network to perform the training step again.

3. The method according to claim 1, wherein: The step of inputting the full plate image feature information into the classification processing network in the plate recognition model to obtain the plate category information corresponding to the plate instance segmentation image includes: Based on the plate full image feature information, generating plate probability distribution data corresponding to the plate full image feature information, wherein the plate probability distribution data includes individual plate probability data, and each plate probability data in the individual plate probability data corresponds to plate category information; Determine each plate probability data included in the plate probability distribution data as each plate category probability data; Determining the plate category probability data satisfying the preset data condition among the plate category probability data as target probability data; The dinner plate category information corresponding to the target probability data is determined as the dinner plate category information corresponding to the dinner plate instance segmentation image.

4. The method according to claim 1, wherein: The method further comprises: Storing the information of each plate category in a preset database; Acquire the category information of each meal plate within a preset time range as the classification information of each meal plate; Classifying and processing each of the plate classification information to obtain a plate classification information group set, wherein each plate classification information group in the plate classification information group set corresponds to the same plate classification information; For each meal plate classification information group in the meal plate classification information group set, the following steps are performed: Based on the meal tray classification information group and preset meal tray item information, generating information of items to be transported corresponding to the meal tray classification information group; Generate the meal tray item transportation location information based on the meal tray classification information group and the preset meal tray placement information; Based on the meal tray classification information group, the corresponding article transporting equipment is controlled to transport the article corresponding to the to-be-transported article information to the target position corresponding to the meal tray article transporting position information.

5. The method according to claim 1, wherein: The performing instance segmentation processing on the dinner plate image to obtain each dinner plate instance segmentation image comprises: Preprocessing the dinner plate image to obtain a preprocessed dinner plate image; Normalizing the preprocessed plate image to obtain a normalized plate image; Performing feature extraction processing on the normalized dinner plate image to obtain a normalized dinner plate feature map; Based on the normalized plate feature map, generating each bounding box information, wherein each bounding box information in the each bounding box information corresponds to a normalized plate feature image area; Performing recognition processing on each normalized plate feature image region corresponding to each of the bounding box information to obtain each recognition result; Based on the respective recognition results, adjusting the respective bounding box information to obtain respective adjusted bounding box information, wherein each adjusted bounding box information in the respective adjusted bounding box information corresponds to a confidence level; Determine each target bounding box information based on each confidence level corresponding to each adjusted bounding box information, wherein each target bounding box information in the each target bounding box information corresponds to a normalized dinner plate feature image region; For each target bounding box information in the target bounding box information, generating a corresponding instance segmentation mask based on a normalized plate feature image region corresponding to the target bounding box information, wherein the instance segmentation mask corresponds to a confidence level; Determine each target instance segmentation mask based on each confidence level corresponding to each generated instance segmentation mask; Based on the segmentation masks of the target instances, segmentation images of the dinner plates are generated.

6. The method according to claim 5, wherein: The determining each target bounding box information based on each confidence level corresponding to each adjusted bounding box information includes: Based on the confidence levels corresponding to the respective pieces of adjusted bounding box information, the respective pieces of adjusted bounding box information are sorted to obtain an adjusted bounding box information sequence; Based on the sequence of adjusted bounding box information, the following loop steps are performed: Determine the adjusted bounding box information that meets a preset confidence condition in the adjusted bounding box information sequence as the target adjusted bounding box information; Adding the target adjustment bounding box information to a target adjustment bounding box information list to update the target adjustment bounding box information list; determining each piece of adjusted bounding box information that does not satisfy the preset confidence condition in the adjusted bounding box information sequence as each piece of adjusted bounding box information to be processed; Generate each intersection-over-union ratio based on the target adjusted bounding box information and each to-be-processed adjusted bounding box information; Determine each I / O ratio that satisfies a preset I / O ratio condition among the I / O ratios as each target I / O ratio; Based on the adjustment bounding box information sequence, clearing the target adjustment bounding box information and each to-be-processed adjustment bounding box information corresponding to each target intersection-over-union ratio, so as to update the adjustment bounding box information sequence; In response to determining that the updated adjusted bounding box information sequence satisfies a preset loop condition, executing the loop step again using the updated adjusted bounding box information sequence; In response to determining that the updated adjustment bounding box information sequence does not satisfy the preset loop condition, each target adjustment bounding box information in the updated target adjustment bounding box information list is determined as each target bounding box information.

7. A plate information display device, comprising: an acquisition unit, configured to acquire a dinner plate image corresponding to each dinner plate; a segmentation unit configured to perform instance segmentation processing on the dinner plate image to obtain each dinner plate instance segmentation image; The execution unit is configured to perform the following steps for each of the plate instance segmentation images: inputting the plate instance segmentation image into a feature extraction network in a pre-trained plate recognition model to perform feature extraction processing on the plate instance segmentation image to obtain a plate feature map corresponding to the plate instance segmentation image; inputting the plate feature map into a feature processing network in the plate recognition model to generate full plate feature information corresponding to the plate feature map, wherein the inputting the plate feature map into the feature processing network in the plate recognition model to generate full plate feature information corresponding to the plate feature map The method comprises: constructing a plate feature connection graph corresponding to the plate feature graph based on the plate feature graph; performing the following steps for each node feature information in each node feature information included in the plate feature connection graph: determining each neighbor node feature information corresponding to the node feature information based on the node feature information; performing the following steps for each neighbor node feature information in each neighbor node feature information: determining an attention coefficient of the node feature information and the neighbor node feature information based on the neighbor node feature information; determining the product of the attention coefficient and the neighbor node feature information as weighted neighbor node feature information; and The method comprises the steps of: determining the sum of weighted neighbor node feature information of each weighted neighbor node as aggregated neighbor node feature information; integrating the node feature information and the aggregated neighbor node feature information to obtain spliced ​​node feature information; determining the product of the spliced ​​node feature information and the first-level parameter as the first-level feature information; determining the sum of the first-level feature information and the second-level parameter as the second-level feature information; generating node feature information based on the second-level feature information to update the node feature information; in response to determining that each node feature information included in the plate feature connection graph meets a preset update condition, determining the plate feature connection graph based on the updated node feature information. The whole plate image feature information corresponding to the feature image; based on the whole plate image feature information, determining the corresponding memory usage information; in response to determining that the memory usage information is less than the preset allocated memory amount, reducing the physical memory corresponding to the whole plate image feature information to adjust the storage resource allocation of the whole plate image feature information; in response to determining that the memory usage information is greater than the preset allocated memory amount, increasing the physical memory corresponding to the whole plate image feature information to adjust the storage resource allocation of the whole plate image feature information; inputting the whole plate image feature information into the classification processing network in the plate recognition model to obtain the plate category information corresponding to the plate instance segmentation image; a determination unit configured to determine each plate value information corresponding to each plate instance segmented image based on the obtained each plate category information and a preset category value information group; A generating unit configured to generate plate information based on the plate category information and the plate value information, wherein the plate information includes plate total value information; The display unit is configured to display the plate information.

8. An electronic device comprising: one or more processors; a storage device having one or more programs stored thereon; When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 6.

9. A computer readable medium having a computer program stored thereon, wherein: When the program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images